47 lines
1.4 KiB
Markdown
47 lines
1.4 KiB
Markdown
# Evaluation Protocol
|
||
|
||
## Primary Metrics
|
||
- **avg_ks**: mean KS across continuous features
|
||
- **avg_jsd**: mean JSD across discrete feature marginals
|
||
- **avg_lag1_diff**: lag‑1 correlation mismatch
|
||
|
||
## Diagnostic Metrics
|
||
- **per‑feature KS**: `example/diagnose_ks.py`
|
||
- **filtered KS**: `example/filtered_metrics.py` (remove collapsed/outlier features)
|
||
- **ranked KS**: `example/ranked_ks.py` (contribution analysis)
|
||
|
||
## KS Implementation Notes
|
||
- KS is computed with **tie-aware** CDFs (important for discrete/spiky features).
|
||
- Reference data supports **glob input** and aggregates all matching files.
|
||
- Use `--max-rows` to cap reference rows for faster diagnostics.
|
||
|
||
## Recommended Reporting
|
||
Report both:
|
||
1) **Full metrics** (no filtering)
|
||
2) **Filtered metrics** (diagnostic only)
|
||
|
||
Always list which features were filtered.
|
||
If using KS-only postprocess (empirical resampling), note it explicitly because it can weaken joint realism.
|
||
|
||
## Program‑Generator Metrics (Type 1)
|
||
For setpoints/demands:
|
||
- dwell‑time distribution
|
||
- change‑count per day
|
||
- step‑size distribution
|
||
|
||
## Controller Metrics (Type 2)
|
||
- saturation ratio near bounds
|
||
- change rate and median step size
|
||
|
||
## Actuator Metrics (Type 3)
|
||
- top‑k spike mass (top1/top3)
|
||
- unique ratio
|
||
- dwell length
|
||
|
||
## PV Metrics (Type 4)
|
||
- q05/q50/q95 + tail ratio
|
||
|
||
## Aux Metrics (Type 6)
|
||
- mean/std
|
||
- lag‑1 correlation
|