Skip to content

Commit ab8717b

Browse files
varunursekarclaude
andcommitted
Tell the runbook to copy the session out before killing a run
Learned by losing one. officeqa run #4 was killed while the verifier was blocked on a drain, so it never reached export-session -- and stopping the compose stack removes the containers, taking /state/admin/session with them. Every commit that optimizer made, including the candidate it had deliberately submitted after a 5.5h search, was unrecoverable. The verifier/ directory it left behind holds a single empty file. The fix is one docker cp before the kill, which yields candidates/repository.git and lets scripts/rescore_candidate.py score the submitted candidate on the held-out set afterwards. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 460dbcc commit ab8717b

1 file changed

Lines changed: 21 additions & 0 deletions

File tree

  • harness-engineering-bench/skills/run-benchmark

harness-engineering-bench/skills/run-benchmark/SKILL.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -162,6 +162,27 @@ If you see a clear failure signal, **stop the run and diagnose from disk** rathe
162162
than letting it burn budget. Detached/daemon-owned sandboxes can keep running
163163
after the parent dies, so check for and clean up orphans when you kill a run.
164164

165+
### Copy the session out BEFORE you kill anything
166+
167+
Every commit the optimizer made — including the one it submitted — lives *only*
168+
inside the sidecar container until the verifier's `export-session` step writes
169+
`session.tar.gz`. **Kill a run before that step and the candidates are gone**,
170+
because stopping the stack removes the containers. A run killed mid-verifier
171+
leaves an empty `verifier/` directory and nothing to recover; the entire search
172+
is lost even though it completed successfully.
173+
174+
So the first move when killing a run is always:
175+
176+
```bash
177+
docker cp <sidecar-container>:/state/admin/session ./salvaged-session
178+
```
179+
180+
That gives you `candidates/repository.git` — a real git repo, so
181+
`git --git-dir=.../repository.git log --all` lists every candidate and
182+
`git archive <sha>` extracts one — plus `database.json`, `budgets.json` and the
183+
evaluation job records. With it, the submitted candidate can be re-scored on the
184+
held-out set afterwards. Only once you have it should you stop the containers.
185+
165186
## What "done / green" looks like
166187

167188
- **`finalize.json`** (admin volume) / **`harbor-finalization.json`** (session):

0 commit comments

Comments
 (0)