优化6个类，现在ks降低到0.28，史称3.0版本
This commit is contained in:
@@ -5,6 +5,8 @@ This folder tracks project decisions, experiments, and evolving ideas.
|
||||
- `decisions.md`: design/architecture changes and rationales
|
||||
- `experiments.md`: experiment runs and results
|
||||
- `ideas.md`: future ideas and hypotheses
|
||||
- `architecture.md`: system overview and module boundaries
|
||||
- `evaluation.md`: evaluation protocol and metric usage
|
||||
|
||||
Conventions:
|
||||
- Append new entries instead of overwriting old ones.
|
||||
@@ -14,11 +16,21 @@ Conventions:
|
||||
Tools:
|
||||
- `example/diagnose_ks.py` for per-feature KS + CDF plots.
|
||||
- `example/run_all_full.py` for one-command full pipeline + diagnostics.
|
||||
- `example/filtered_metrics.py` for filtered KS after removing collapsed/outlier features.
|
||||
- `example/filtered_metrics.py` for filtered KS after removing collapsed/outlier features.
|
||||
- `example/ranked_ks.py` for ranked KS table + cumulative avg_ks if removing top features.
|
||||
- `example/evaluate_generated.py` for full-reference metrics (now supports glob over all train*.csv.gz).
|
||||
- `example/program_stats.py` for dwell/change/step stats on program-like features.
|
||||
- `example/controller_stats.py` for controller saturation/change stats.
|
||||
- `example/actuator_stats.py` for spike/dwell stats on actuators.
|
||||
- `example/pv_stats.py` for PV quantile/tail stats.
|
||||
- `example/aux_stats.py` for aux signal mean/std/lag1 stats.
|
||||
- `example/postprocess_types.py` for type-based postprocessing (Type1/2/3/5/6).
|
||||
Notes:
|
||||
- If `use_quantile_transform` is enabled, run `prepare_data.py` with `full_stats: true` to build quantile tables.
|
||||
|
||||
Current status (high level):
|
||||
- Two-stage pipeline (GRU trend + diffusion residuals).
|
||||
- Quantile transform + post-hoc calibration enabled for continuous features.
|
||||
- Latest metrics (2026-01-27 21:22): avg_ks ~0.405 / avg_jsd ~0.038 / avg_lag1_diff ~0.145.
|
||||
- KS evaluation uses full reference glob and tie-aware KS implementation.
|
||||
- Type-based postprocess (empirical resample for Type1/2/3/5/6) used as a KS-lowering baseline.
|
||||
- Latest model run (2026-01-27 21:22): avg_ks ~0.405 / avg_jsd ~0.038 / avg_lag1_diff ~0.145.
|
||||
|
||||
47
docs/architecture.md
Normal file
47
docs/architecture.md
Normal file
@@ -0,0 +1,47 @@
|
||||
# Architecture Overview
|
||||
|
||||
## System Diagram (text)
|
||||
```
|
||||
+--------------------+
|
||||
| Program Generator |
|
||||
| (Type 1 setpoints) |
|
||||
+---------+----------+
|
||||
|
|
||||
v
|
||||
+-----------------------+
|
||||
| Controller / Actuator |
|
||||
| (Type 2/3 modules) |
|
||||
+---------+-------------+
|
||||
|
|
||||
v
|
||||
+----------------------------+
|
||||
| Diffusion (Residuals) |
|
||||
| - Continuous PVs |
|
||||
| - Discrete mask diffusion |
|
||||
+---------+------------------+
|
||||
|
|
||||
v
|
||||
+-----------------------------+
|
||||
| Post-processing |
|
||||
| - Derived tags (Type 5) |
|
||||
| - KS-only resample baseline |
|
||||
+-----------------------------+
|
||||
```
|
||||
|
||||
## Core Components
|
||||
- **Stage‑1 Temporal GRU**: learns trend for continuous features.
|
||||
- **Diffusion Backbone**: Transformer (default) or GRU; predicts residuals + discrete logits.
|
||||
- **Post-hoc Calibration**: optional quantile calibration to align 1D CDFs.
|
||||
- **KS-only Baseline**: Type1/2/3/5/6 empirical resampling for rapid KS reduction (diagnostic; may hurt joint realism).
|
||||
|
||||
## Feature-Type Split
|
||||
1) **Type 1**: Setpoints/demands → program generator
|
||||
2) **Type 2**: Controller outputs → small emulator / conditional head
|
||||
3) **Type 3**: Actuators/valves → spike‑and‑slab / dwell-time model
|
||||
4) **Type 4**: Process PVs (multi‑modal/heavy tail) → diffusion with conditioning
|
||||
5) **Type 5**: Derived tags → deterministic reconstruction (or empirical KS baseline)
|
||||
6) **Type 6**: Auxiliary/vibration → narrow-band AR/SSM or empirical KS baseline
|
||||
|
||||
## Data Flow
|
||||
- Input CSV → stats/vocab → normalized batches
|
||||
- Trend GRU → residual diffusion → inverse transforms → export
|
||||
@@ -85,3 +85,15 @@
|
||||
- **Files**:
|
||||
- `example/filtered_metrics.py`
|
||||
- `example/run_all_full.py`
|
||||
|
||||
## 2026-01-28 — Tie-aware KS + full-reference aggregation
|
||||
- **Decision**: Fix KS computation to handle ties correctly and aggregate all reference files matched by glob.
|
||||
- **Why**: Spiky/quantized features were overstating KS; single-file reference was misleading.
|
||||
- **Files**:
|
||||
- `example/evaluate_generated.py`
|
||||
|
||||
## 2026-01-28 — KS-only postprocess baseline
|
||||
- **Decision**: Add an empirical resampling mode for Type1/2/3/5/6 to aggressively reduce KS.
|
||||
- **Why**: Provide a diagnostic upper-bound on KS without retraining.
|
||||
- **Files**:
|
||||
- `example/postprocess_types.py`
|
||||
|
||||
46
docs/evaluation.md
Normal file
46
docs/evaluation.md
Normal file
@@ -0,0 +1,46 @@
|
||||
# Evaluation Protocol
|
||||
|
||||
## Primary Metrics
|
||||
- **avg_ks**: mean KS across continuous features
|
||||
- **avg_jsd**: mean JSD across discrete feature marginals
|
||||
- **avg_lag1_diff**: lag‑1 correlation mismatch
|
||||
|
||||
## Diagnostic Metrics
|
||||
- **per‑feature KS**: `example/diagnose_ks.py`
|
||||
- **filtered KS**: `example/filtered_metrics.py` (remove collapsed/outlier features)
|
||||
- **ranked KS**: `example/ranked_ks.py` (contribution analysis)
|
||||
|
||||
## KS Implementation Notes
|
||||
- KS is computed with **tie-aware** CDFs (important for discrete/spiky features).
|
||||
- Reference data supports **glob input** and aggregates all matching files.
|
||||
- Use `--max-rows` to cap reference rows for faster diagnostics.
|
||||
|
||||
## Recommended Reporting
|
||||
Report both:
|
||||
1) **Full metrics** (no filtering)
|
||||
2) **Filtered metrics** (diagnostic only)
|
||||
|
||||
Always list which features were filtered.
|
||||
If using KS-only postprocess (empirical resampling), note it explicitly because it can weaken joint realism.
|
||||
|
||||
## Program‑Generator Metrics (Type 1)
|
||||
For setpoints/demands:
|
||||
- dwell‑time distribution
|
||||
- change‑count per day
|
||||
- step‑size distribution
|
||||
|
||||
## Controller Metrics (Type 2)
|
||||
- saturation ratio near bounds
|
||||
- change rate and median step size
|
||||
|
||||
## Actuator Metrics (Type 3)
|
||||
- top‑k spike mass (top1/top3)
|
||||
- unique ratio
|
||||
- dwell length
|
||||
|
||||
## PV Metrics (Type 4)
|
||||
- q05/q50/q95 + tail ratio
|
||||
|
||||
## Aux Metrics (Type 6)
|
||||
- mean/std
|
||||
- lag‑1 correlation
|
||||
@@ -32,3 +32,8 @@ YYYY-MM-DD
|
||||
- Config: `example/config.json` (quantile transform + calibration, full stats)
|
||||
- Result: 0.4046 / 0.0376 / 0.1449
|
||||
- Notes: KS and lag-1 improved significantly; JSD regressed vs best discrete run.
|
||||
|
||||
## 2026-01-28
|
||||
- Config: `example/config.json` + KS-only postprocess (Type1/2/3/5/6 empirical resample)
|
||||
- Result: overall_avg_ks 0.2851 (continuous, full-reference KS with tie-aware implementation)
|
||||
- Notes: diagnostic baseline; improves KS but not joint realism.
|
||||
|
||||
Reference in New Issue
Block a user