Problem
Rerunning eval-no-tools-batch against an output dir whose run already completed does not extend it — it replays the stored batch state from round 0 ("round 1: resuming batch batch_…"), stops before the later rounds, and overwrites by_model/<model>.csv with fewer repair rounds applied, silently discarding the completed run's later repairs.
Observed (gpt-5.5 default-effort run, 2026-07-05):
- First invocation: rounds 0–5, final CSV 1,984 rows with 23 missing predictions.
- Second invocation of the identical command on the same dir (intending to continue repairs with
POLICYBENCH_MAX_REPAIR_ROUNDS=8): re-collected rounds 0–2 from the stored OpenAI batch ids, printed "scheduling repair round" after round 2, then wrote the CSV and exited — leaving 51 missing predictions (rounds 3–5's successful repairs discarded).
Whether the raised POLICYBENCH_MAX_REPAIR_ROUNDS (#97) was honored by the resume path is also unclear from the logs — the process stopped scheduling after round 2 despite 18 units still violating.
Expected
Either (a) a completed run dir is a no-op ("already complete: 1,984 rows, 23 missing — pass --force to redo"), or (b) resume continues from the LAST persisted round and only schedules new repair rounds for currently-missing units, never producing a CSV with more missing cells than the one it replaces.
Workaround used
Archived the dir and ran fresh in a new output dir. Evidence preserved at results/local/gpt55_default_run_20260705_attempt1/ (both CSVs' round provenance in batches/).
Problem
Rerunning
eval-no-tools-batchagainst an output dir whose run already completed does not extend it — it replays the stored batch state from round 0 ("round 1: resuming batch batch_…"), stops before the later rounds, and overwritesby_model/<model>.csvwith fewer repair rounds applied, silently discarding the completed run's later repairs.Observed (gpt-5.5 default-effort run, 2026-07-05):
POLICYBENCH_MAX_REPAIR_ROUNDS=8): re-collected rounds 0–2 from the stored OpenAI batch ids, printed "scheduling repair round" after round 2, then wrote the CSV and exited — leaving 51 missing predictions (rounds 3–5's successful repairs discarded).Whether the raised
POLICYBENCH_MAX_REPAIR_ROUNDS(#97) was honored by the resume path is also unclear from the logs — the process stopped scheduling after round 2 despite 18 units still violating.Expected
Either (a) a completed run dir is a no-op ("already complete: 1,984 rows, 23 missing — pass --force to redo"), or (b) resume continues from the LAST persisted round and only schedules new repair rounds for currently-missing units, never producing a CSV with more missing cells than the one it replaces.
Workaround used
Archived the dir and ran fresh in a new output dir. Evidence preserved at
results/local/gpt55_default_run_20260705_attempt1/(both CSVs' round provenance inbatches/).