优化6个类,现在ks降低到0.28,史称3.0版本
This commit is contained in:
46
docs/evaluation.md
Normal file
46
docs/evaluation.md
Normal file
@@ -0,0 +1,46 @@
|
||||
# Evaluation Protocol
|
||||
|
||||
## Primary Metrics
|
||||
- **avg_ks**: mean KS across continuous features
|
||||
- **avg_jsd**: mean JSD across discrete feature marginals
|
||||
- **avg_lag1_diff**: lag‑1 correlation mismatch
|
||||
|
||||
## Diagnostic Metrics
|
||||
- **per‑feature KS**: `example/diagnose_ks.py`
|
||||
- **filtered KS**: `example/filtered_metrics.py` (remove collapsed/outlier features)
|
||||
- **ranked KS**: `example/ranked_ks.py` (contribution analysis)
|
||||
|
||||
## KS Implementation Notes
|
||||
- KS is computed with **tie-aware** CDFs (important for discrete/spiky features).
|
||||
- Reference data supports **glob input** and aggregates all matching files.
|
||||
- Use `--max-rows` to cap reference rows for faster diagnostics.
|
||||
|
||||
## Recommended Reporting
|
||||
Report both:
|
||||
1) **Full metrics** (no filtering)
|
||||
2) **Filtered metrics** (diagnostic only)
|
||||
|
||||
Always list which features were filtered.
|
||||
If using KS-only postprocess (empirical resampling), note it explicitly because it can weaken joint realism.
|
||||
|
||||
## Program‑Generator Metrics (Type 1)
|
||||
For setpoints/demands:
|
||||
- dwell‑time distribution
|
||||
- change‑count per day
|
||||
- step‑size distribution
|
||||
|
||||
## Controller Metrics (Type 2)
|
||||
- saturation ratio near bounds
|
||||
- change rate and median step size
|
||||
|
||||
## Actuator Metrics (Type 3)
|
||||
- top‑k spike mass (top1/top3)
|
||||
- unique ratio
|
||||
- dwell length
|
||||
|
||||
## PV Metrics (Type 4)
|
||||
- q05/q50/q95 + tail ratio
|
||||
|
||||
## Aux Metrics (Type 6)
|
||||
- mean/std
|
||||
- lag‑1 correlation
|
||||
Reference in New Issue
Block a user