# Documentation Index

This folder tracks project decisions, experiments, and evolving ideas.

- `decisions.md`: design/architecture changes and rationales
- `experiments.md`: experiment runs and results
- `ideas.md`: future ideas and hypotheses
- `architecture.md`: system overview and module boundaries
- `evaluation.md`: evaluation protocol and metric usage

Conventions:
- Append new entries instead of overwriting old ones.
- Record exact config file and key overrides when possible.
- Keep metrics in the order: avg_ks / avg_jsd / avg_lag1_diff.

Tools:
- `example/diagnose_ks.py` for per-feature KS + CDF plots.
- `example/run_all_full.py` for one-command full pipeline + diagnostics.
- `example/filtered_metrics.py` for filtered KS after removing collapsed/outlier features.
- `example/ranked_ks.py` for ranked KS table + cumulative avg_ks if removing top features.
- `example/evaluate_generated.py` for full-reference metrics (now supports glob over all train*.csv.gz).
- `example/program_stats.py` for dwell/change/step stats on program-like features.
- `example/controller_stats.py` for controller saturation/change stats.
- `example/actuator_stats.py` for spike/dwell stats on actuators.
- `example/pv_stats.py` for PV quantile/tail stats.
- `example/aux_stats.py` for aux signal mean/std/lag1 stats.
- `example/postprocess_types.py` for type-based postprocessing (Type1/2/3/5/6).
Notes:
- If `use_quantile_transform` is enabled, run `prepare_data.py` with `full_stats: true` to build quantile tables.

Current status (high level):
- Two-stage pipeline (GRU trend + diffusion residuals).
- Quantile transform + post-hoc calibration enabled for continuous features.
- KS evaluation uses full reference glob and tie-aware KS implementation.
- Type-based postprocess (empirical resample for Type1/2/3/5/6) used as a KS-lowering baseline.
- Latest model run (2026-01-27 21:22): avg_ks ~0.405 / avg_jsd ~0.038 / avg_lag1_diff ~0.145.