Skip to content

Commit 8861432

Browse files
committed
Add calibration tolerance experiment app
1 parent 3fe5102 commit 8861432

8 files changed

Lines changed: 879 additions & 40 deletions

File tree

modal_app/README.md

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,8 @@ modal run modal_app/remote_calibration_runner.py --branch <branch> --epochs <n>
2727
| `--push-results` | `False` | Upload weights, blocks, and logs to HuggingFace |
2828
| `--trigger-publish` | `False` | Fire `repository_dispatch` to trigger the Publish workflow |
2929
| `--target-config` | (none) | Target configuration name |
30+
| `--target-policy` | (none) | Target tolerance policy YAML path |
31+
| `--target-tolerance-mode` | (none) | Final tolerance behavior: `enforce` or `evaluate` |
3032
| `--beta` | (none) | L0 relaxation parameter |
3133
| `--lambda-l0` | (none) | L0 penalty weight |
3234
| `--lambda-l2` | (none) | L2 penalty weight |
@@ -78,6 +80,8 @@ Every run produces these local files (whichever the calibration script emits):
7880
- **unified_diagnostics.csv** — Final per-target diagnostics
7981
- **calibration_log.csv** — Per-target metrics across epochs (requires `--log-freq`)
8082
- **unified_run_config.json** — Run configuration and summary stats
83+
- **calibration_target_policy.jsonl** — Resolved per-target tolerance policy
84+
- **calibration_target_policy_summary.json** — Target tolerance summary
8185

8286
## Artifact Upload to HuggingFace
8387

@@ -90,10 +94,36 @@ atomic commit after writing them locally:
9094
| `calibration_log.csv` | `calibration/logs/calibration_log.csv` |
9195
| `unified_diagnostics.csv` | `calibration/logs/unified_diagnostics.csv` |
9296
| `unified_run_config.json` | `calibration/logs/unified_run_config.json` |
97+
| `calibration_target_policy.jsonl` | `calibration/logs/calibration_target_policy.jsonl` |
98+
| `calibration_target_policy_summary.json` | `calibration/logs/calibration_target_policy_summary.json` |
9399

94100
Each upload overwrites the previous files. HF git history provides implicit
95101
versioning — browse past commits to see earlier runs.
96102

103+
## Tolerance Experiment App
104+
105+
`modal_app/calibration_tolerance_experiment.py` runs an isolated full regional
106+
calibration experiment from the current Modal volume inputs. It copies
107+
`policy_data.db` and `source_imputed_stratified_extended_cps.h5` into
108+
`/pipeline/epsilon_calibration_runs/{run_id}`, builds a run-scoped calibration
109+
package, fits weights with `--target-tolerance-mode evaluate`, and writes
110+
diagnostics, resolved target-policy artifacts, and `experiment_summary.json`.
111+
It does not upload Hugging Face artifacts or trigger H5 publication.
112+
113+
Deploy the app when you want to launch repeat experiments from Modal rather
114+
than from a local CLI session:
115+
116+
```bash
117+
modal deploy modal_app/calibration_tolerance_experiment.py
118+
```
119+
120+
```bash
121+
modal run modal_app/calibration_tolerance_experiment.py::main \
122+
--branch codex/epsilon-insensitive-calibration \
123+
--run-id epsilon-smoke \
124+
--epochs 1000
125+
```
126+
97127
## Triggering the Publish Workflow
98128

99129
The `--trigger-publish` flag fires a `repository_dispatch` event

0 commit comments

Comments
 (0)