Making swe-bench-pro affordable for an optimizer loop, and keep using deepseek.

This is additive. The canonical baseline/build.yaml and its 146/292/293 split are byte-unchanged. This adds a second config beside them, so both can be run.

baseline/build.yaml         146/292/293   gpt-4o        untouched
baseline/build.sample.yaml   33/66/66     deepseek      new
partitions/sample/*.json                                new

Why a variant

Each case builds a real repository and runs its test suite. The full-split held-out baseline alone cost ~7.8h of wall clock for three rounds, and an optimizer evaluates candidates many times over. 33/66/66 matches gaia and browsecomp-plus, the suite's tightest split.

The sample is nested

--sample computes the canonical split first, unchanged, then narrows each partition within itself, so partitions/sample/test.json is a strict subset of partitions/test.json.

This is the load-bearing detail, and I got it wrong first. A flat re-split at the smaller size moves instances between partitions: my initial version produced a "held-out" test set with zero overlap with the real one, silently drawn from what had been dev and validation. Nesting prevents that, and buys a second thing: the baseline needed no re-measuring.

baseline_reward: 0.2923 is the same K=3 trials recomputed over the 66 sampled cases. The recomputation reproduces 0.2940 ±0.0082 exactly on the full set first, which is what validates it. The point estimate barely moves; the spread widens fourfold because n falls 293 → 66, which matters when reading a candidate's margin over that floor.

Caveat, recorded in the file: those trials targeted qwen-3.6-27b, while the variant targets deepseek-v4-flash. So 0.2923 is provisional until a K=3 deepseek pass over the 66 replaces it. That costs ~1.8h at max_concurrency: 24 (calibrated from the real 879-trial run: 131.3h serial, 24 effective slots, ideal 5.47h vs actual 7.8h = 1.43x overhead).

What the variant carries that the canonical config lacks

All places where swe-bench-pro is the lone holdout in the suite: deepseek-v4-flash (prefixed-only, since agents strip just openai/ before the gateway matches exactly), max_concurrency: 24, 4x case budgets (132/264), case_timeout == task_agent_timeout at 3000 so the multiplier is 1.0, score_baseline: false with the baseline pinned, and n_attempts: 3 / mean on the test target.

Two pre-existing inconsistencies, deliberately left alone

Both are in the canonical build.yaml. I did not change them, since this PR is meant to be additive, but they are worth separate one-line fixes:

• case_timeout_seconds: 1800 against task_agent_timeout_seconds: 3000 implies a 0.6 multiplier, but the pinned 0.294 was measured at 1.0: all 879 sampled trial configs record agent_timeout_multiplier=1.0. The config value is wrong, not the measurement.
• model: gpt-4o, but the 0.294 baseline was measured against qwen-3.6-27b.

Also, swe-bench-pro is the only benchmark still using score_baseline: true, so it re-measures the seed on every run rather than pinning it.

Verification

• Canonical partitions, manifest and build.yaml show no diff against pr3
• Canonical --check passes; --sample regenerates byte-identically
• Every sampled partition is a strict subset of its canonical counterpart; all 11 repos appear in each
• Both configs compile side by side: full → 146/292/293, gpt-4o, 1800; sample → 33/66/66, deepseek, 3000, baseline_reward 0.2923, budgets 132/264, n_attempts: 3 / mean on the test backend only
• pytest tests/test_v05_benchmark_configs.py: 13 passed, 6 skipped

Not yet verified: no trial has been executed against the sampled split. An end-to-end optimizer run on it is in flight and I will report the result here. Everything above is config-level correctness.

Note for the optimizer-model axis

The preflight from #59 caught something that affects more than this PR: gpt-5.3-codex has no /responses deployment on the LiteLLM proxy (/chat/completions → 200, /responses → 404 DeploymentNotFound). The codex agent is Responses-native, so any optimizer run using it as the optimizer model fails after setup. gpt-5.6-sol and gpt-4.1 serve both routes; the variant's producer default is now gpt-5.6-sol.

Rebased onto the re-pinned base (2026-07-28)

pr3-harness-bench absorbed pr2-add-vero (merge ed55bc7) and re-pinned all five held-out baselines (44a852e), which collided with this branch in CONFIGURATION.md. Rebased as one commit on ed55bc7; both sides of each hunk are kept:

• Per-benchmark table: upstream's re-pinned numbers (gaia 0.621, officeqa 0.341, swe-atlas-qna 0.068, tau3 0.732, browsecomp-plus 0.462) with this branch's swe-bench-pro column appended to every row. Verified all 19 rows have identical cell counts.
• Prose: upstream's two new paragraphs (why the five went stale; gaia/tau3 noise) are kept ahead of this branch's ◈ footnote, so ◈ still sits next to ◇.

Added a third documented inconsistency. swe-bench-pro is the one benchmark the re-pin did not cover, and its staleness is the strongest case of exactly what upstream's new note describes: the pinned 0.294 was measured on a different agent than the one in the tree. The qwen-era agent used Chat Completions with an explicit local messages list and edited files on 571 of 879 cases; the committed seed is Responses-only, delegates its conversation to previous_response_id, and over 66 sampled cases called write_file/apply_patch/submit zero times for 0.0000. Documented rather than silently repaired, with a pointer to scripts/rescore_candidate.py --seed.

pytest tests/test_v05_benchmark_configs.py: 13 passed, 6 skipped.

Greptile Summary

This PR adds a cost-effective 33/66/66 subsampled variant of swe-bench-pro alongside the unchanged canonical 146/292/293 config, making it practical for optimizer loops. The sample is correctly nested — each sampled partition is a strict subset of its canonical counterpart — so no instance can migrate between partitions and the existing per-case trial data can be reused to derive the provisional baseline reward of 0.2923.

• baseline/build.sample.yaml introduces the new optimizer config: fireworks_ai/deepseek-v4-flash, max_concurrency: 24, case_timeout_seconds == task_agent_timeout_seconds = 3000 (1.0 multiplier), score_baseline: false with a pinned baseline, and n_attempts: 3 / mean on the held-out test target only.
• scripts/partition_swe_bench_pro.py gains a --sample flag and a _subsample helper that stratifies within partitions; --tasks-dir is now optional when a manifest is already committed.
• CONFIGURATION.md gains the swe-bench-pro column and a detailed prose section documenting the two-config design and three pre-existing inconsistencies in the canonical config; one stray extra separator cell was introduced in an unrelated table (see inline comment).

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[partition_swe_bench_pro.py] --> B{--tasks-dir given?}
    B -- yes --> C[_read_tasks from disk]
    B -- no --> D[_tasks_from_manifest\nfrom partitions/manifest.json]
    C --> E[_allocate\n731 tasks → 146/292/293]
    D --> E
    E --> F{--sample flag?}
    F -- no --> G[_render\nwrites manifest.json\n+ 3 partition files]
    G --> H[(partitions/\ndevelopment.json\nvalidation.json\ntest.json\nmanifest.json)]
    F -- yes --> I[_subsample\nnarrows inside each partition\n146→33 / 292→66 / 293→66]
    I --> J[_render_sample\nwrites 3 partition files only\nmanifest is shared]
    J --> K[(partitions/sample/\ndevelopment.json\nvalidation.json\ntest.json)]
    H --> L[build.yaml\n146/292/293 gpt-4o]
    K --> M[build.sample.yaml\n33/66/66 deepseek]

_{Reviews (5): Last reviewed commit: "feat: add a subsampled swe-bench-pro var..." | Re-trigger Greptile}