update新结构

report.md

@@ -2,71 +2,46 @@
 # 工业控制系统流量混合扩散生成（HAI 21.03）— 项目报告
 
 ## 1. Project Goal / 项目目标
-Build a **hybrid diffusion-based generator** for industrial control system (ICS) traffic features, targeting **mixed continuous + discrete** feature sequences. The output is **feature-level sequences**, not raw packets. The generator should preserve:
+Build a **hybrid diffusion-based generator** for ICS traffic features, focusing on **mixed continuous + discrete** feature sequences. The output is **feature-level sequences**, not raw packets. The generator should preserve:
-- **Distributional fidelity** (continuous value ranges and discrete frequencies)
+- **Distributional fidelity** (continuous ranges + discrete frequencies)
 - **Temporal consistency** (time correlation and sequence structure)
-- **Protocol/field consistency** (for discrete fields)
+- **Field/logic consistency** for discrete protocol-like columns
 
-构建一个用于工业控制系统（ICS）流量特征的**混合扩散生成模型**，面向**连续+离散混合特征序列**。输出为**特征级序列**而非原始报文。生成结果需要同时保持：
+构建一个用于 ICS 流量特征的**混合扩散生成模型**，处理**连续+离散混合特征序列**。输出为**特征级序列**而非原始报文。生成结果需要保持：
-- **分布一致性**（连续值范围与离散取值频率）
+- **分布一致性**（连续值范围 + 离散频率）
 - **时序一致性**（时间相关性与序列结构）
-- **字段/协议一致性**（离散字段的逻辑一致）
+- **字段/逻辑一致性**（离散字段语义）
-
-This project is aligned with the STOUTER idea of **structure-aware diffusion** for spatiotemporal data, but applied to **ICS feature sequences** rather than cellular traffic.
-
-本项目呼应 STOUTER 的**结构先验+扩散**思想，但应用于**ICS 特征序列**而非蜂窝流量。
 
 ---
 
 ## 2. Data and Scope / 数据与范围
-**Dataset used in the current implementation:** HAI 21.03 (CSV feature traces)
+**Dataset used in current implementation:** HAI 21.03 (CSV feature traces).
 
-**当前实现使用的数据集：** HAI 21.03（CSV 特征序列）
+**当前实现使用数据集：** HAI 21.03（CSV 特征序列）。
 
-**Data location (default in config):**
+**Data path (default in config):**
 - `dataset/hai/hai-21.03/train*.csv.gz`
 
-**数据位置（config 默认）：**
+**特征拆分（固定 schema）：** `example/feature_split.json`
-- `dataset/hai/hai-21.03/train*.csv.gz`
+- Continuous features: sensor/process values
-
+- Discrete features: binary/low-cardinality status/flag fields
-**Feature split (fixed schema):**
+- `time` is excluded from modeling
-- Defined in `example/feature_split.json`
-- **Continuous features:** sensor/process values
-- **Discrete features:** binary/low-cardinality status/flag fields
-- `time` column is excluded from modeling
-
-**特征拆分（固定 schema）：**
-- `example/feature_split.json`
-- **连续特征：** 传感器/过程值
-- **离散特征：** 二值/低基数状态字段
-- `time` 列不参与训练
 
 ---
 
 ## 3. End-to-End Pipeline / 端到端流程
-
+One command pipeline:
-**One command pipeline:**
 ```
 python example/run_all.py --device cuda
 ```
 
-**一键流程：**
+Pipeline stages:
-```
-python example/run_all.py --device cuda
-```
-
-### Pipeline stages / 流程阶段
 1) **Prepare data** (`example/prepare_data.py`)
 2) **Train model** (`example/train.py`)
 3) **Generate samples** (`example/export_samples.py`)
 4) **Evaluate** (`example/evaluate_generated.py`)
-5) **Summarize metrics** (`example/summary_metrics.py`)
 
-1) **数据准备**（统计量与词表）
+一键流程对应：数据准备 → 训练 → 采样导出 → 评估。
-2) **训练模型**
-3) **生成样本并导出**
-4) **评估指标**
-5) **汇总指标**
 
 ---
 
@@ -75,178 +50,82 @@ python example/run_all.py --device cuda
 ### 4.1 Hybrid Diffusion Model (Core) / 混合扩散模型（核心）
 Defined in `example/hybrid_diffusion.py`.
 
-**Key components:**
+**Inputs:**
-- **Continuous branch**: Gaussian diffusion (DDPM style)
+- Continuous projection
-- **Discrete branch**: Mask diffusion for categorical tokens
+- Discrete embeddings
-- **Shared backbone**: GRU + residual MLP + LayerNorm
+- Time embedding (sinusoidal)
-- **Embedding inputs**:
+- Positional embedding (sequence index)
-  - continuous projection
+- Optional condition embedding (`file_id`)
-  - discrete embeddings per column
+
-  - time embedding (sinusoidal)
+**Backbone:**
-  - positional embedding (sequence index)
+- GRU (sequence modeling)
-  - optional condition embedding (`file_id`)
+- Post LayerNorm + residual MLP
 
 **Outputs:**
-- Continuous head: predicts target (`eps`, `x0`, or `v`)
+- Continuous head: predicts target (`eps` or `x0`)
-- Discrete heads: predict logits for each discrete column
+- Discrete heads: logits per discrete column
 
-**核心组成：**
+**连续分支：** Gaussian diffusion
-- **连续分支：** 高斯扩散（DDPM）
+**离散分支：** Mask diffusion
-- **离散分支：** Mask 扩散
-- **共享主干：** GRU + 残差 MLP + LayerNorm
-- **输入嵌入：**
-  - 连续投影
-  - 离散字段嵌入
-  - 时间嵌入（正弦）
-  - 位置嵌入（序列索引）
-  - 条件嵌入（可选，`file_id`）
-
-**输出：**
-- 连续 head：预测 `eps/x0/v`
-- 离散 head：各字段 logits
 
 ---
 
-### 4.2 Feature Graph Mixer (Structure Prior) / 特征图混合器（结构先验）
+### 4.2 Temporal Backbone (GRU) / 共享时序骨干（GRU）
-Implemented in `example/hybrid_diffusion.py` as `FeatureGraphMixer`.
+The GRU is the **shared temporal backbone** that fuses continuous + discrete signals into a unified sequence representation, enabling joint modeling of temporal dynamics and cross-feature dependencies.
 
-Purpose: inject **learnable feature-dependency prior** without dataset-specific hardcoding.
+GRU 是模型的**共享时序核心**，把连续/离散特征统一建模在同一时间结构中。
-
-**Mechanism:**
-- Learns a dense feature relation matrix `A`
-- Applies: `x + x @ A`
-- Symmetric stabilizing constraint: `(A + A^T)/2`
-- Controlled by scale and dropout
-
-**Config:**
-```
-"model_use_feature_graph": true,
-"feature_graph_scale": 0.1,
-"feature_graph_dropout": 0.0
-```
-
-**目的：**在不写死特定数据集关系的情况下，引入**可学习特征依赖先验**。
-
-**机制：**
-- 学习稠密关系矩阵 `A`
-- 特征混合：`x + x @ A`
-- 对称化稳定：`(A + A^T)/2`
-- 通过 scale/dropout 控制强度
 
 ---
 
-### 4.3 Two-Stage Temporal Backbone / 两阶段时序骨干
+## 5. Diffusion Formulations / 扩散形式
-Stage-1 uses a **GRU temporal generator** to model sequence trend in normalized space. Stage-2 diffusion then models the **residual** (x − trend). This decouples temporal consistency from distribution alignment.
-
-第一阶段使用 **GRU 时序生成器**在归一化空间建模序列趋势；第二阶段扩散模型学习**残差**（x − trend），实现时序一致性与分布对齐的解耦。
-
----
-
-## 5. Diffusion Formulations / 扩散建模形式
 
 ### 5.1 Continuous Diffusion / 连续扩散
 Forward process:
 ```
 x_t = sqrt(a_bar_t) * x_0 + sqrt(1 - a_bar_t) * eps
 ```
-
 Targets supported:
-- **eps prediction** (standard DDPM)
+- **eps prediction** (default)
 - **x0 prediction** (direct reconstruction)
-- **v prediction** (v = sqrt(a_bar)*eps − sqrt(1-a_bar)*x0)
 
-Current config default:
+Current config:
 ```
-"cont_target": "v"
+"cont_target": "x0"
 ```
 
-Sampling uses the target to reconstruct `eps` and apply standard DDPM reverse update.
+### 5.2 Discrete Diffusion / 离散扩散
-
+Mask diffusion with cosine schedule:
-**前向扩散：**如上公式。
-
-**支持的目标：**
-- `eps`（噪声预测）
-- `x0`（原样本预测）
-- `v`（v‑prediction）
-
-**当前默认：**`cont_target = v`
-
-**采样：**根据目标反解 `eps` 再执行标准 DDPM 反向步骤。
-
----
-
-### 5.2 Discrete Diffusion (Mask) / 离散扩散（Mask）
-Forward process: replace tokens with `[MASK]` using cosine schedule:
 ```
 p(t) = 0.5 * (1 - cos(pi * t / T))
 ```
-Optional scale: `disc_mask_scale`
+Mask-only cross-entropy is computed on masked positions.
-
-Reverse process: cross-entropy on masked positions only.
-
-**前向：**按 cosine schedule 进行 Mask。
-**反向：**仅在 mask 位置计算交叉熵。
 
 ---
 
-## 6. Loss Design (Current) / 当前损失设计
+## 6. Loss Design / 损失设计
 Total loss:
 ```
-L = λ * L_cont + (1 − λ) * L_disc + w_q * L_quantile
+L = λ * L_cont + (1 − λ) * L_disc
 ```
 
 ### 6.1 Continuous Loss / 连续损失
-Depending on `cont_target`:
+- `eps` target: MSE(eps_pred, eps)
-- eps target: MSE(eps_pred, eps)
+- `x0` target: MSE(x0_pred, x0)
-- x0 target: MSE(x0_pred, x0)
+- Optional inverse-variance weighting: `cont_loss_weighting = "inv_std"`
-- v target: MSE(v_pred, v_target)
-
-Optional inverse-variance weighting:
-```
-cont_loss_weighting = "inv_std"
-```
 
 ### 6.2 Discrete Loss / 离散损失
 Cross-entropy on masked positions only.
 
-### 6.3 Quantile Loss (Distribution Alignment) / 分位数损失（分布对齐）
-Added to improve KS (distribution shape alignment):
-- Compute quantiles on generated vs real x0
-- Loss = Huber or L1 difference on quantiles
-
-Stabilization:
-```
-quantile_loss_warmup_steps
-quantile_loss_clip
-quantile_loss_huber_delta
-```
-
 ---
 
-## 7. Training Strategy / 训练策略
+## 7. Data Processing / 数据处理
-Defined in `example/train.py`.
+Defined in `example/data_utils.py` + `example/prepare_data.py`.
 
-**Key techniques:**
+Key steps:
-- EMA of model weights
+- Streaming mean/std/min/max + int-like detection
-- Gradient clipping
+- Optional **log1p transform** for heavy-tailed continuous columns
-- Shuffle buffer to reduce batch bias
+- Discrete vocab + most frequent token
-- Optional feature graph prior
+- Windowed batching with **shuffle buffer**
-- Quantile loss warmup for stability
-- Optional stage-1 temporal GRU (trend) + residual diffusion
-
-**Config highlights (example/config.json):**
-```
-timesteps: 600
-batch_size: 128
-seq_len: 128
-epochs: 10
-max_batches: 4000
-lambda: 0.7
-cont_target: "v"
-quantile_loss_weight: 0.1
-model_use_feature_graph: true
-use_temporal_stage1: true
-```
 
 ---
 
@@ -255,111 +134,83 @@ Defined in:
 - `example/sample.py`
 - `example/export_samples.py`
 
-**Export steps:**
+Export process:
-- Reverse diffusion with conditional sampling
+- Reverse diffusion sampling
-- Reverse normalize continuous values
+- De-normalize continuous values
 - Clamp to observed min/max
 - Restore discrete tokens from vocab
 - Write to CSV
 
 ---
 
-## 9. Evaluation Metrics / 评估指标
+## 9. Evaluation / 评估指标
-Implemented in `example/evaluate_generated.py`.
+Defined in `example/evaluate_generated.py`.
 
-### Continuous Metrics / 连续指标
+Metrics (with reference):
-- **KS statistic** (distribution similarity per feature)
+- **KS statistic** (continuous distribution)
-- **Quantile errors** (q05/q25/q50/q75/q95)
+- **Quantile diffs** (q05/q25/q50/q75/q95)
 - **Lag‑1 correlation diff** (temporal structure)
-
+- **Discrete JSD** over vocab frequency
-### Discrete Metrics / 离散指标
-- **JSD** over token frequency distribution
 - **Invalid token counts**
 
-### Summary Metrics / 汇总指标
-Auto-logged in:
-- `example/results/metrics_history.csv`
-- via `example/summary_metrics.py`
-
 ---
 
 ## 10. Automation / 自动化
-
+`example/run_all.py` runs all stages with config-driven paths.
-### One‑click pipeline / 一键流程
-```
-python example/run_all.py --device cuda
-```
-
-### Metrics logging / 指标记录
-Each run appends:
-```
-timestamp,avg_ks,avg_jsd,avg_lag1_diff
-```
 
 ---
 
 ## 11. Key Engineering Decisions / 关键工程决策
-
+- Mixed-type diffusion: continuous + discrete split
-### 11.1 Mixed-Type Diffusion / 混合类型扩散
+- Shared temporal backbone (GRU) to align sequence structure
-Continuous + discrete handled separately to respect data types.
+- Positional + time embeddings for stability
-
+- Optional inverse-variance weighting for continuous loss
-### 11.2 Structure Prior / 结构先验
+- Log1p transforms for heavy-tailed signals
-Learnable feature graph added to encode implicit dependencies.
-
-### 11.3 v‑prediction
-Chosen to stabilize training and improve convergence in diffusion.
-
-### 11.4 Distribution Alignment / 分布对齐
-Quantile loss introduced to directly reduce KS.
 
 ---
 
-## 12. Known Issues / Current Limitations / 已知问题与当前局限
+## 12. Code Map (Key Files) / 代码索引
-- **KS remains high** in many experiments, meaning continuous distributions are still misaligned.
+- Core model: `example/hybrid_diffusion.py`
-- **Lag‑1 may degrade** when quantile loss is too strong.
+- Training: `example/train.py`
-- **Loss spikes** observed when quantile loss is unstable (mitigated with warmup + clip + Huber).
+- Data prep: `example/prepare_data.py`
-
+- Data utilities: `example/data_utils.py`
-**当前问题：**
+- Sampling: `example/sample.py`
-- KS 高，说明连续分布仍未对齐
+- Export: `example/export_samples.py`
-- 分位数损失过强时会损害时序相关性
+- Evaluation: `example/evaluate_generated.py`
-- 分位数损失不稳定时会出现 loss 爆炸（已引入 warmup/clip/Huber）
+- Pipeline: `example/run_all.py`
+- Config: `example/config.json`
 
 ---
 
-## 13. Suggested Next Steps (Research Roadmap) / 下一步建议（研究路线）
+## 13. Known Issues / Current Limitations / 已知问题
-1) **SNR-weighted loss** (improve stability across timesteps)
+- KS sometimes remains high → continuous distribution mismatch
-2) **Two-stage training** (distribution first, temporal consistency second)
+- Lag‑1 may fluctuate → distribution vs temporal trade-off
-3) **Upgrade discrete diffusion** (D3PM-style transitions)
+- Continuous loss may dominate → needs careful weighting
-4) **Structured conditioning** (state/phase conditioning)
-5) **Graph-based priors** (explicit feature/plant dependency graphs)
 
 ---
 
-## 14. Code Map (Key Files) / 代码索引（关键文件）
+## 14. Suggested Next Steps / 下一步建议
-
+- Add **SNR-weighted loss** for stable diffusion training
-**Core model**
+- Explore **v‑prediction** for continuous branch
-- `example/hybrid_diffusion.py`
+- Consider **two-stage training** (temporal first, distribution second)
-
+- Strengthen discrete diffusion (e.g., D3PM-style transitions)
-**Training**
-- `example/train.py`
-
-**Sampling & export**
-- `example/sample.py`
-- `example/export_samples.py`
-
-**Pipeline**
-- `example/run_all.py`
-
-**Evaluation**
-- `example/evaluate_generated.py`
-- `example/summary_metrics.py`
-
-**Configs**
-- `example/config.json`
 
 ---
 
 ## 15. Summary / 总结
-This project implements a **hybrid diffusion model for ICS traffic features**, combining continuous Gaussian diffusion with discrete mask diffusion, enhanced with a **learnable feature-graph prior**. The system includes a full pipeline for preparation, training, sampling, exporting, and evaluation. Key research challenges remain in **distribution alignment (KS)** and **joint optimization of distribution fidelity vs temporal consistency**, motivating future improvements such as SNR-weighted loss, staged training, and stronger structural priors.
+This project implements a **hybrid diffusion model** for ICS feature sequences with a GRU backbone, handling continuous and discrete features separately while sharing temporal structure. The pipeline covers data prep, training, sampling, export, and evaluation. The main research challenge remains in balancing **distributional fidelity (KS)** and **temporal consistency (lag‑1)**.
 
-本项目实现了用于 ICS 流量特征的**混合扩散模型**，将连续高斯扩散与离散 Mask 扩散结合，并引入**可学习特征图先验**。系统包含完整的数据准备、训练、采样、导出与评估流程。当前研究挑战集中在**连续分布对齐（KS）**与**分布/时序一致性之间的权衡**，后续可通过 SNR‑weighted loss、分阶段训练与更强结构先验继续改进。
+本项目实现了基于 GRU 的混合扩散模型，连续/离散分支分开建模但共享时序结构，具备完整的训练与评估流程。主要挑战是**分布对齐（KS）与时序一致性（lag‑1）之间的平衡**。
+
+---
+
+## 16. Latest Evaluation Snapshot / 最新评估快照
+Computed averages from the latest `eval.json`:
+- **avg_ks**: 0.5208903596698115
+- **avg_jsd**: 0.010592151023360712
+- **avg_lag1_diff**: 0.8265139723919303
+
+最新评估均值（来自 `eval.json`）：
+- **avg_ks**：0.5208903596698115
+- **avg_jsd**：0.010592151023360712
+- **avg_lag1_diff**：0.8265139723919303