feat: add a subsampled swe-bench-pro variant (33/66/66) for optimizer runs

[shehabyasser-scale]

Making swe-bench-pro affordable for an optimizer loop, and keep using deepseek.

This is additive. The canonical baseline/build.yaml and its 146/292/293 split are byte-unchanged. This adds a second config beside them, so both can be run.

baseline/build.yaml         146/292/293   gpt-4o        untouched
baseline/build.sample.yaml   33/66/66     deepseek      new
partitions/sample/*.json                                new

Why a variant

Each case builds a real repository and runs its test suite. The full-split held-out baseline alone cost ~7.8h of wall clock for three rounds, and an optimizer evaluates candidates many times over. 33/66/66 matches gaia and browsecomp-plus, the suite's tightest split.

The sample is nested

--sample computes the canonical split first, unchanged, then narrows each partition within itself, so partitions/sample/test.json is a strict subset of partitions/test.json.

This is the load-bearing detail, and I got it wrong first. A flat re-split at the smaller size moves instances between partitions: my initial version produced a "held-out" test set with zero overlap with the real one, silently drawn from what had been dev and validation. Nesting prevents that, and buys a second thing: the baseline needed no re-measuring.

	n	reward	±sd
canonical test partition	293	0.2940	0.0082
sampled test partition	66	0.2923	0.0332

baseline_reward: 0.2923 is the same K=3 trials recomputed over the 66 sampled cases. The recomputation reproduces 0.2940 ±0.0082 exactly on the full set first, which is what validates it. The point estimate barely moves; the spread widens fourfold because n falls 293 → 66, which matters when reading a candidate's margin over that floor.

Caveat, recorded in the file: those trials targeted qwen-3.6-27b, while the variant targets deepseek-v4-flash. So 0.2923 is provisional until a K=3 deepseek pass over the 66 replaces it. That costs ~1.8h at max_concurrency: 24 (calibrated from the real 879-trial run: 131.3h serial, 24 effective slots, ideal 5.47h vs actual 7.8h = 1.43x overhead).

What the variant carries that the canonical config lacks

All places where swe-bench-pro is the lone holdout in the suite: deepseek-v4-flash (prefixed-only, since agents strip just openai/ before the gateway matches exactly), max_concurrency: 24, 4x case budgets (132/264), case_timeout == task_agent_timeout at 3000 so the multiplier is 1.0, score_baseline: false with the baseline pinned, and n_attempts: 3 / mean on the test target.

Two pre-existing inconsistencies, deliberately left alone

Both are in the canonical build.yaml. I did not change them, since this PR is meant to be additive, but they are worth separate one-line fixes:

• case_timeout_seconds: 1800 against task_agent_timeout_seconds: 3000 implies a 0.6 multiplier, but the pinned 0.294 was measured at 1.0: all 879 sampled trial configs record agent_timeout_multiplier=1.0. The config value is wrong, not the measurement.
• model: gpt-4o, but the 0.294 baseline was measured against qwen-3.6-27b.

Also, swe-bench-pro is the only benchmark still using score_baseline: true, so it re-measures the seed on every run rather than pinning it.

Verification

• Canonical partitions, manifest and build.yaml show no diff against pr3
• Canonical --check passes; --sample regenerates byte-identically
• Every sampled partition is a strict subset of its canonical counterpart; all 11 repos appear in each
• Both configs compile side by side: full → 146/292/293, gpt-4o, 1800; sample → 33/66/66, deepseek, 3000, baseline_reward 0.2923, budgets 132/264, n_attempts: 3 / mean on the test backend only
• pytest tests/test_v05_benchmark_configs.py: 13 passed, 6 skipped

Not yet verified: no trial has been executed against the sampled split. An end-to-end optimizer run on it is in flight and I will report the result here. Everything above is config-level correctness.

Note for the optimizer-model axis

The preflight from #59 caught something that affects more than this PR: gpt-5.3-codex has no /responses deployment on the LiteLLM proxy (/chat/completions → 200, /responses → 404 DeploymentNotFound). The codex agent is Responses-native, so any optimizer run using it as the optimizer model fails after setup. gpt-5.6-sol and gpt-4.1 serve both routes; the variant's producer default is now gpt-5.6-sol.

Rebased onto the re-pinned base (2026-07-28)

pr3-harness-bench absorbed pr2-add-vero (merge ed55bc7) and re-pinned all five held-out baselines (44a852e), which collided with this branch in CONFIGURATION.md. Rebased as one commit on ed55bc7; both sides of each hunk are kept:

• Per-benchmark table: upstream's re-pinned numbers (gaia 0.621, officeqa 0.341, swe-atlas-qna 0.068, tau3 0.732, browsecomp-plus 0.462) with this branch's swe-bench-pro column appended to every row. Verified all 19 rows have identical cell counts.
• Prose: upstream's two new paragraphs (why the five went stale; gaia/tau3 noise) are kept ahead of this branch's ◈ footnote, so ◈ still sits next to ◇.

Added a third documented inconsistency. swe-bench-pro is the one benchmark the re-pin did not cover, and its staleness is the strongest case of exactly what upstream's new note describes: the pinned 0.294 was measured on a different agent than the one in the tree. The qwen-era agent used Chat Completions with an explicit local messages list and edited files on 571 of 879 cases; the committed seed is Responses-only, delegates its conversation to previous_response_id, and over 66 sampled cases called write_file/apply_patch/submit zero times for 0.0000. Documented rather than silently repaired, with a pointer to scripts/rescore_candidate.py --seed.

pytest tests/test_v05_benchmark_configs.py: 13 passed, 6 skipped.

Greptile Summary

This PR adds a subsampled (33/66/66) swe-bench-pro variant for optimizer runs alongside the unchanged canonical 146/292/293 config. The approach is additive — baseline/build.yaml and its partition files are byte-identical to pr3-harness-bench.

• New config (build.sample.yaml): targets fireworks_ai/deepseek-v4-flash, bumps max_concurrency from 8 → 24, aligns case_timeout_seconds with task_agent_timeout_seconds at 3000 (fixing the 0.6-multiplier inconsistency in the canonical config), pins score_baseline: false, and carries a baseline_reward: 0.2923 measured from the same K=3 qwen trials — provisional until a deepseek pass replaces it.
• Nested-subsample algorithm (_subsample in partition_swe_bench_pro.py): stratified sampling strictly inside each canonical partition so no instance can migrate between partitions; the largest-remainder loop is mathematically guaranteed to terminate. _tasks_from_manifest lets --sample run without re-exporting 731 task directories.
• Known caveat, explicitly documented: baseline_reward: 0.2923 was measured against qwen-3.6-27b, not the variant's deepseek target; the file notes it's provisional until a ~1.8h K=3 deepseek run replaces it.

Confidence Score: 5/5

Safe to merge — all changes are additive; the canonical config and its partition files are untouched.

The canonical build.yaml and its 146/292/293 partitions are byte-unchanged. The new YAML, partition files, and script additions are self-consistent: partition counts match SAMPLE_COUNTS, the nested-sampling algorithm is correct, and the known baseline-model mismatch is prominently documented. The only defect found is a stray Markdown separator cell in the documentation table.

Files Needing Attention: The extra separator cell in CONFIGURATION.md is cosmetic. No functional files require special attention.

Important Files Changed

Filename	Overview
harness-engineering-bench/CONFIGURATION.md	Adds swe-bench-pro column to the per-benchmark table and a detailed footnote explaining the two-config setup and known inconsistencies. Minor formatting issue: the rate-limit table separator accidentally has an extra cell.
harness-engineering-bench/swe-bench-pro/baseline/build.sample.yaml	New optimizer-variant config (33/66/66, deepseek-v4-flash, max_concurrency 24, case_timeout 3000). Provisional baseline_reward (0.2923, measured on qwen) is prominently documented. All key differences from the canonical config are clearly commented.
harness-engineering-bench/swe-bench-pro/scripts/partition_swe_bench_pro.py	Adds --sample flag, _subsample(), _render_sample(), and _tasks_from_manifest(). The nested-subsample algorithm is correct: quota allocation uses the largest-remainder method within each partition, and the while-loop is guaranteed to terminate because all strata can't simultaneously be at capacity while the total is below target.
harness-engineering-bench/swe-bench-pro/partitions/sample/development.json	33-instance development partition; all 11 repos represented, instances are strict subsets of partitions/development.json.
harness-engineering-bench/swe-bench-pro/partitions/sample/test.json	66-instance held-out test partition; all 11 repos represented, strict subset of partitions/test.json as designed.
harness-engineering-bench/swe-bench-pro/partitions/sample/validation.json	66-instance validation partition; all 11 repos represented, strict subset of partitions/validation.json as designed.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[partition_swe_bench_pro.py] --> B{--sample flag?}
    B -- No --> C[_read_tasks / _tasks_from_manifest]
    C --> D[_allocate: 146/292/293 stratified split]
    D --> E[_render: full manifest + partition files]
    E --> F[partitions/manifest.json]
    F --> G[build.yaml 146/292/293 gpt-4o concurrency 8]
    B -- Yes --> H[_tasks_from_manifest reads existing manifest]
    H --> I[_allocate: same 146/292/293 full split]
    I --> J[_subsample: nested LR sample 33/66/66 within each partition]
    J --> K[_render_sample: partition files only]
    K --> L[partitions/sample/ development/validation/test.json]
    L --> M[build.sample.yaml 33/66/66 deepseek-v4-flash concurrency 24]

Loading

_{Reviews (4): Last reviewed commit: "feat: add a subsampled swe-bench-pro var..." | Re-trigger Greptile}