|
| 1 | +--- |
| 2 | +title: WIP checkpoints |
| 3 | +description: Recover in-progress eval runs from git-backed results repositories. |
| 4 | +sidebar: |
| 5 | + order: 7 |
| 6 | +--- |
| 7 | + |
| 8 | +WIP checkpoints are best-effort snapshots of an eval run while it is still executing. They are designed for long-running evals in CI, pods, or remote agents where losing the process would otherwise lose the completed test rows that were already written locally. |
| 9 | + |
| 10 | +They are **not** a second results mode. They reuse the existing run workspace format and the configured git-backed results repository. |
| 11 | + |
| 12 | +## When checkpoints run |
| 13 | + |
| 14 | +WIP checkpoints are active only when AgentV can resolve a results repo configuration with auto-push enabled: |
| 15 | + |
| 16 | +- In a registered project: `projects[].results.sync.auto_push: true` in `$AGENTV_HOME/config.yaml`. |
| 17 | +- In the top-level fallback config: `results.auto_push: true`. |
| 18 | + |
| 19 | +If no results repo is configured, or auto-push is disabled, `agentv eval` still writes the local run workspace but does not create WIP branches. |
| 20 | + |
| 21 | +## What gets written |
| 22 | + |
| 23 | +| Location | Path or ref | What it contains | |
| 24 | +| --- | --- | --- | |
| 25 | +| Local project | `.agentv/results/runs/<experiment>/<run-id>/benchmark.json` | A run-start stub with `metadata.planned_test_count` and the eval file path when known. This lets Dashboard recognize incomplete local runs as resumable. | |
| 26 | +| Local project | `.agentv/results/runs/<experiment>/<run-id>/index.jsonl` | Result rows appended as test cases finish. Rows use the normal snake_case result JSONL format. | |
| 27 | +| Results repo remote | `agentv/inflight/<hostname>/<run-dir-basename>` | A forced-updated branch containing the checkpointed run under `.agentv/results/runs/<same-relative-run-path>/`. | |
| 28 | +| Results repo storage branch | Configured `results.branch`, or the repo default branch | The final published run after `agentv eval` completes and the normal auto-export succeeds. | |
| 29 | + |
| 30 | +The WIP branch name is derived from the current host and the run directory basename. Non-branch-safe characters are replaced with `-`; the host component is capped at 40 characters and the run component at 60 characters. |
| 31 | + |
| 32 | +## Lifecycle |
| 33 | + |
| 34 | +1. **Run start** — AgentV creates the local run directory and writes the initial `benchmark.json` stub. If auto-push is enabled, it creates a temporary git worktree for a branch named `agentv/inflight/<hostname>/<run-dir-basename>`, based on the configured results storage branch when `results.branch` is set. |
| 35 | +2. **While running** — about every 30 seconds, AgentV copies the current run directory into the WIP worktree, amends a single checkpoint commit, and force-pushes the WIP branch. If nothing changed, it skips the push. |
| 36 | +3. **Successful completion** — AgentV publishes the completed run to the normal results branch. After that publish is confirmed as `published` or `already_published`, it deletes the remote WIP branch. |
| 37 | +4. **Failure, interrupt, or final export failure** — AgentV stops the checkpoint loop and removes the temporary local worktree, but leaves the remote WIP branch intact for recovery. |
| 38 | + |
| 39 | +Checkpoint failures are warnings only. They never fail the eval run. |
| 40 | + |
| 41 | +## Recover from a WIP branch |
| 42 | + |
| 43 | +Use git to retrieve the WIP branch, copy the run workspace back into the eval project, then resume the run with the normal `--resume` flow. |
| 44 | + |
| 45 | +```bash |
| 46 | +# 1. Clone or enter the configured results repo. |
| 47 | +git clone <results-repo-url> /tmp/agentv-results-recovery |
| 48 | +cd /tmp/agentv-results-recovery |
| 49 | + |
| 50 | +# 2. Find WIP branches. |
| 51 | +git fetch origin --prune |
| 52 | +git branch -r --list 'origin/agentv/inflight/*' |
| 53 | + |
| 54 | +# 3. Check out the branch for the interrupted run. |
| 55 | +git switch --detach origin/agentv/inflight/<hostname>/<run-dir-basename> |
| 56 | + |
| 57 | +# 4. Inspect the checkpointed run path. |
| 58 | +find .agentv/results/runs -name benchmark.json |
| 59 | + |
| 60 | +# 5. Copy the run tree into the eval project, preserving paths under runs/. |
| 61 | +PROJECT=/path/to/eval-project |
| 62 | +mkdir -p "$PROJECT/.agentv/results/runs" |
| 63 | +rsync -a .agentv/results/runs/ "$PROJECT/.agentv/results/runs/" |
| 64 | + |
| 65 | +# 6. Resume from the recovered run directory. |
| 66 | +cd "$PROJECT" |
| 67 | +agentv eval <eval-file> --output .agentv/results/runs/<experiment>/<run-id> --resume |
| 68 | +``` |
| 69 | + |
| 70 | +If the recovered `benchmark.json` contains `metadata.eval_file`, use that as `<eval-file>`. If the run lives directly under `.agentv/results/runs/<run-id>/` instead of an experiment directory, pass that path to `--output`. |
| 71 | + |
| 72 | +After the resumed run publishes successfully, AgentV cleans up any WIP branch it creates for the resumed run. Delete the original orphaned branch manually when you no longer need it: |
| 73 | + |
| 74 | +```bash |
| 75 | +git push origin --delete agentv/inflight/<hostname>/<run-dir-basename> |
| 76 | +``` |
| 77 | + |
| 78 | +## Dashboard and `results` surfaces |
| 79 | + |
| 80 | +- **Dashboard local runs:** an interrupted local run can show the one-click **Resume run** and **Rerun failed** actions when `benchmark.json` has `metadata.planned_test_count` greater than the number of result rows, or when any row has `execution_status: execution_error`. |
| 81 | +- **Dashboard remote runs:** normal remote listing reads the configured results storage branch. It does not list `agentv/inflight/...` WIP branches. Recover the checkpoint into the project-local run directory first, or wait for the final publish branch to receive a completed run. |
| 82 | +- **`agentv results` CLI:** the command family manages local run workspaces and reports. It does not have a WIP branch subcommand; use git for remote checkpoint inspection and cleanup. |
| 83 | + |
| 84 | +## Operational caveats |
| 85 | + |
| 86 | +- The first remote checkpoint happens on the periodic interval, so a process that dies immediately after startup may only have the local `benchmark.json` stub. |
| 87 | +- The WIP branch is force-pushed and keeps one snapshot commit. Do not treat it as an audit log. |
| 88 | +- Checkpoint contents can include prompts, outputs, grader evidence, traces, and generated task bundles. Protect the results repo like any other eval artifact store. |
| 89 | +- Authentication and branch permissions are the same as normal results auto-push. If git or GitHub authentication is missing, AgentV warns and keeps evaluating locally. |
| 90 | +- If `results.branch` is configured, create that remote storage branch before running evals. WIP worktrees are based on it. |
| 91 | +- Failed or interrupted runs intentionally leave WIP branches behind. Periodically delete old `agentv/inflight/...` branches once recovered or obsolete. |
| 92 | + |
| 93 | +See also: [Resume an Interrupted Run](/docs/evaluation/running-evals/#resume-an-interrupted-run), [Results](/docs/tools/results/), and [Dashboard Remote Results](/docs/tools/dashboard/#remote-results). |
0 commit comments