Commit e869972
committed
perf(bench): sort instances by (repo, base_commit) for worktree reuse
`ensure_repo` keeps a per-worker worktree path keyed on repo_name and
skips `git worktree add` when the same repo lands twice in a row.
The previous order (manifest insertion) interleaved instances from
different repos, defeating the reuse: every instance triggered a fresh
worktree add, costing ~0.5-2s per call on large repos (django, keras,
transformers).
Sorting by (repo, base_commit) groups instances that share a worktree
so the worker only pays the worktree-add cost once per (repo, commit)
pair. SWE-bench Verified has ~12 instances per repo on average, so the
saving is on the order of `(n_unique_repos x worktree_add_cost)` per
cell -- several minutes per cell on the headline grid, an hour-plus
across the 162-cell matrix.
Pure win: deterministic order, no behavior change in the underlying
eval, no correctness risk. Per-budget checkpoint resume works
identically because instance_id is the resume key, not list index.1 parent ad9e543 commit e869972
1 file changed
Lines changed: 7 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
89 | 89 | | |
90 | 90 | | |
91 | 91 | | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
92 | 99 | | |
93 | 100 | | |
94 | 101 | | |
| |||
0 commit comments