Skip to content

Commit e869972

Browse files
committed
perf(bench): sort instances by (repo, base_commit) for worktree reuse
`ensure_repo` keeps a per-worker worktree path keyed on repo_name and skips `git worktree add` when the same repo lands twice in a row. The previous order (manifest insertion) interleaved instances from different repos, defeating the reuse: every instance triggered a fresh worktree add, costing ~0.5-2s per call on large repos (django, keras, transformers). Sorting by (repo, base_commit) groups instances that share a worktree so the worker only pays the worktree-add cost once per (repo, commit) pair. SWE-bench Verified has ~12 instances per repo on average, so the saving is on the order of `(n_unique_repos x worktree_add_cost)` per cell -- several minutes per cell on the headline grid, an hour-plus across the 162-cell matrix. Pure win: deterministic order, no behavior change in the underlying eval, no correctness risk. Per-budget checkpoint resume works identically because instance_id is the resume key, not list index.
1 parent ad9e543 commit e869972

1 file changed

Lines changed: 7 additions & 0 deletions

File tree

benchmarks/run_final_eval.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,13 @@ def _process_manifest(
8989
name = manifest_path.stem.removeprefix("test_")
9090
ids = read_manifest(manifest_path)
9191
instances = [i for i in filter_instances_by_manifest(adapters, ids) if i.source_benchmark == name]
92+
# Sort by (repo, base_commit) so consecutive worker tasks reuse the same
93+
# git worktree — `ensure_repo` keeps a per-worker worktree path keyed on
94+
# repo_name and skips the worktree-add when the same repo lands twice in
95+
# a row. SWE-bench-style benchmarks have ~12 instances per repo on
96+
# average; this saves on the order of (n_unique_repos x worktree_add_cost)
97+
# per cell, which is several minutes for large repos like django/keras.
98+
instances.sort(key=lambda i: (i.repo, i.base_commit))
9299
if args.limit:
93100
instances = instances[: args.limit]
94101
depth_label = f" L={depth}" if depth is not None else ""

0 commit comments

Comments
 (0)