feat: add a subsampled swe-bench-pro variant (33/66/66) for optimizer runs

Additive only. The canonical baseline/build.yaml and its 146/292/293 split are
byte-unchanged; this adds a second config beside them.

Why: the full split is unaffordable for an optimizer loop. Each case builds a
real repository and runs its test suite, so the full-split held-out baseline
alone cost ~7.8h of wall clock for three rounds, and an optimizer evaluates
candidates many times over. 33/66/66 matches gaia and browsecomp-plus, the
suite's tightest split, which is the right reference for its most expensive
benchmark.

The sample is NESTED, which is the load-bearing detail. `--sample` computes the
canonical split first, unchanged, then narrows each partition within itself, so
partitions/sample/test.json is a strict subset of partitions/test.json. Two
things follow. No instance can migrate from test into development: a flat
re-split at the smaller size does move them, and the first version of this
change produced a "held-out" set with zero overlap with the real one, silently
drawn from what had been dev and validation. And the baseline needed no
re-measuring.

  baseline_reward: 0.2923 over the 66 sampled cases, recomputed from the very
  same K=3 trials that produced the canonical 0.2940 +-0.0082. The
  recomputation reproduces 0.2940 +-0.0082 exactly on the full set first, which
  is what validates it. The point estimate barely moves; the spread widens to
  +-0.0332 because n falls 293 -> 66.

  CAVEAT recorded in the file: those trials targeted qwen-3.6-27b while the
  variant targets deepseek-v4-flash, so 0.2923 is provisional until a K=3
  deepseek pass over the 66 replaces it (~1.8h at max_concurrency 24).

Repository stratification and sha256 ordering are unchanged and apply within
each partition, so all 11 upstream repos appear in every sampled partition. The
sample shares the canonical manifest rather than duplicating it: it pins the
same 731 refs and the build loader reads only names from it, so a second copy
would just be a second thing to keep in sync. `--tasks-dir` is now optional,
since the manifest already records every name and stratum.

The variant also carries the settings the canonical config is missing relative
to its siblings: deepseek-v4-flash (prefixed-only, because agents strip just
`openai/` before the gateway matches exactly), max_concurrency 24, 4x case
budgets (132/264), case_timeout equal to task_agent_timeout at 3000 so the
multiplier is 1.0, score_baseline false with the baseline pinned, and
n_attempts 3 / mean on the test target.

CONFIGURATION.md gains the swe-bench-pro column it never had, describing the
canonical config, plus a footnote covering the variant and two pre-existing
inconsistencies in the canonical file that are deliberately left alone here:
case_timeout_seconds 1800 implies a 0.6 multiplier though the pinned 0.294 was
measured at 1.0 (all 879 sampled trial configs say so), and model: gpt-4o though
that baseline was measured against qwen-3.6-27b.

Verified: canonical partitions, manifest and build.yaml show no diff against
pr3; canonical --check passes; --sample regenerates byte-identically; every
sampled partition is a strict subset of its canonical counterpart; both configs
compile side by side (full -> 146/292/293, gpt-4o, 1800; sample -> 33/66/66,
deepseek, 3000, baseline 0.2923, budgets 132/264, n_attempts 3/mean on test
only); 13 passed, 6 skipped in test_v05_benchmark_configs.py.

Author: shehabyasser-scale

harness-engineering-bench/CONFIGURATION.md

@@ -157,25 +157,25 @@ benchmark can be checked against the others at a glance.
 
 ## Per-benchmark values
 
-| | gaia | officeqa | swe-atlas-qna | tau3 | browsecomp-plus |
-|---|---|---|---|---|---|
-| target model | gpt-5.4-mini ◇ | deepseek-v4-flash | gpt-oss-120b | deepseek-v4-flash | deepseek-v4-flash |
-| held-out baseline (K=3) ◆ | 0.574 ±0.010 | 0.360 ±0.042 | 0.097 ±0.011 (agg 0.632) | 0.611 ±0.021 | 0.449 ±0.007 |
-| split dev/val/test | 33/66/66 | 49/98/99 | 25/49/50 | 75/150/150 | 33/66/66 |
-| dev budget (runs / cases) | 100 / 132 | 100 / 196 | 100 / 100 | 100 / 300 | 100 / 132 |
-| val budget (runs / cases) | 100 / 264 | 100 / 392 | 100 / 196 | 100 / 600 | 100 / 264 |
-| gateway max_tokens (evaluation, finalization each) ¶ | 2 B | 3 B | 2 B | 4 B | 2 B |
-| max_concurrency (cases in flight) § | 24 | 24 | 24 | 24 | 24 |
-| timeout_seconds (per eval) ‖ | 7200 | 28800 | 90000 | 79200 | 39600 |
-| case_timeout_seconds = declared † | 600 | 1800 | 10800 | 3600 | 3600 |
-| task_agent_timeout_seconds (declared) | 600 | 1800 | 10800 | 3600 | 3600 |
-| declared `[verifier] timeout_sec` | 300 | 300 | 900 | 300 | 300 |
-| declared `build_timeout_sec` | 300 | 600 | 600 | 600 | 7200 |
-| verifier_timeout_seconds ‖ | 14400 | 54000 | 176400 | 158400 | 75600 |
-| BASH_MAX_TIMEOUT_MS (tool) ¤ | 3600 s | 10800 s | 39600 s | 32400 s | 14400 s |
-| harness_user | harness | harness | null ‡ | null ‡ | null ‡ |
-| task_services_use_upstream | false | false | true (rubric judge) | true (user-sim + grader) | true (answer judge) |
-| task-specific extras | — | `--no-force-build` (prebuilt corpus image) | `keepalive` --ek (ENTRYPOINT images) | `TAU2_*` model pins | pinned 2.2 GB BM25 index |
+| | gaia | officeqa | swe-atlas-qna | tau3 | browsecomp-plus | swe-bench-pro |
+|---|---|---|---|---|---| --- |
+| target model | gpt-5.4-mini ◇ | deepseek-v4-flash | gpt-oss-120b | deepseek-v4-flash | deepseek-v4-flash | gpt-4o ◈ |
+| held-out baseline (K=3) ◆ | 0.574 ±0.010 | 0.360 ±0.042 | 0.097 ±0.011 (agg 0.632) | 0.611 ±0.021 | 0.449 ±0.007 | 0.294 ±0.008 ◈ |
+| split dev/val/test | 33/66/66 | 49/98/99 | 25/49/50 | 75/150/150 | 33/66/66 | 146/292/293 ◈ |
+| dev budget (runs / cases) | 100 / 132 | 100 / 196 | 100 / 100 | 100 / 300 | 100 / 132 | 100 / 146 ◈ |
+| val budget (runs / cases) | 100 / 264 | 100 / 392 | 100 / 196 | 100 / 600 | 100 / 264 | 100 / 292 ◈ |
+| gateway max_tokens (evaluation, finalization each) ¶ | 2 B | 3 B | 2 B | 4 B | 2 B | 100 M |
+| max_concurrency (cases in flight) § | 24 | 24 | 24 | 24 | 24 | 8 ◈ |
+| timeout_seconds (per eval) ‖ | 7200 | 28800 | 90000 | 79200 | 39600 | 28800 |
+| case_timeout_seconds = declared † | 600 | 1800 | 10800 | 3600 | 3600 | 1800 ◈ |
+| task_agent_timeout_seconds (declared) | 600 | 1800 | 10800 | 3600 | 3600 | 3000 |
+| declared `[verifier] timeout_sec` | 300 | 300 | 900 | 300 | 300 | n/a (registry dataset) |
+| declared `build_timeout_sec` | 300 | 600 | 600 | 600 | 7200 | n/a (registry dataset) |
+| verifier_timeout_seconds ‖ | 14400 | 54000 | 176400 | 158400 | 75600 | 28800 |
+| BASH_MAX_TIMEOUT_MS (tool) ¤ | 3600 s | 10800 s | 39600 s | 32400 s | 14400 s | n/a |
+| harness_user | harness | harness | null ‡ | null ‡ | null ‡ | harness |
+| task_services_use_upstream | false | false | true (rubric judge) | true (user-sim + grader) | true (answer judge) | false |
+| task-specific extras | — | `--no-force-build` (prebuilt corpus image) | `keepalive` --ek (ENTRYPOINT images) | `TAU2_*` model pins | pinned 2.2 GB BM25 index | registry dataset; `expose_case_resources: false`; sampled variant ◈ |
 
 ## Conventions
 
@@ -267,6 +267,43 @@ deepseek benchmarks logged zero exceptions over 945 trials, swe-atlas lost
 5/150 to gpt-oss 128k context overflow, and gaia lost 4/198 (infra) after the
 agent's reason/search-only-turn crash was fixed.
 
+◈ swe-bench-pro is the newest benchmark and the only one with **two** configs.
+The column above describes the canonical `baseline/build.yaml` (full 731-instance
+dataset, split 146/292/293). Alongside it sits `baseline/build.sample.yaml`, a
+subsampled variant for optimizer runs; the canonical config is unchanged by it.
+
+*Why a variant.* The full split is unaffordable for an optimizer loop: each case
+builds a real repository and runs its test suite, so the held-out baseline alone
+cost ~7.8h of wall clock for three rounds, and an optimizer evaluates candidates
+many times over. The variant is 33/66/66, matching gaia and browsecomp-plus, on
+`fireworks_ai/deepseek-v4-flash`, `max_concurrency: 24`, 4x case budgets
+(132/264), and `case_timeout_seconds: 3000`.
+
+*The sample is nested.* The full split is computed first and each partition is
+then narrowed within itself (`partition_swe_bench_pro.py --sample`), so
+`partitions/sample/test.json` is a strict subset of `partitions/test.json`. No
+instance can migrate from test into development, and the baseline needed no
+re-measuring: **0.292 ±0.033** over the 66 sampled cases is the same K=3 trials
+that produced the full-partition 0.294 ±0.008, recomputed. The point estimate
+barely moves; the spread widens roughly fourfold because n falls 293 → 66, which
+matters when reading a candidate's margin over that floor.
+
+*Two known inconsistencies in the canonical config*, both pre-existing and left
+alone here rather than silently changed:
+
+- `case_timeout_seconds: 1800` against `task_agent_timeout_seconds: 3000` implies
+  a 0.6 multiplier, but the pinned 0.294 was measured at **1.0**: all 879 sampled
+  trial configs record `agent_timeout_multiplier=1.0`. The config value is wrong,
+  not the measurement. Every sibling has these two equal.
+- `model: gpt-4o`, but the 0.294 baseline was measured against
+  **qwen-3.6-27b**. So the recomputed 0.292 the variant pins is also a qwen
+  number while the variant targets deepseek; treat it as provisional until a K=3
+  deepseek pass over the 66 replaces it (~1.8h at `max_concurrency: 24`).
+
+Also unlike the siblings, swe-bench-pro's canonical config still has
+`score_baseline: true`, so it re-measures the seed every run instead of pinning
+it. The variant sets `false` and pins, as everything else does.
+
 ◇ gaia is the exception to the deepseek-v4-flash default: it is multimodal and
 that model is text-only. Verified against the same litellm endpoint the gateway
 proxies to — gpt-5.4-mini returns 200 for every request shape the gaia agent
@@ -316,7 +353,7 @@ per benchmark from its own case counts; see each `build.yaml` for the arithmetic
 the per-key bucket, so more keys buy proportionally more parallel runs:
 
 | counter | limit | scope |
-|---|---|---|
+|---|---|---| --- |
 | `x-ratelimit-api_key-limit-tokens` | 10 M TPM | **per key** |
 | `x-ratelimit-api_key-limit-requests` | 5 000 RPM | **per key** |
 

harness-engineering-bench/swe-bench-pro/baseline/build.sample.yaml

@@ -0,0 +1,155 @@
+# Subsampled SWE-bench-Pro variant, for optimizer runs.
+#
+# This is a SECOND configuration alongside build.yaml, not a replacement. The
+# canonical build.yaml keeps the full 146/292/293 split and its held-out
+# baseline of 0.2940 +-0.0082; nothing here touches it.
+#
+# Why a variant exists: the full split is unaffordable for an optimizer loop.
+# Each case builds a real repository and runs its (often slow) test suite, so
+# the full-split held-out baseline alone cost ~7.8h of wall clock for three
+# rounds, and an optimizer evaluates candidates many times over. 33/66/66
+# matches gaia and browsecomp-plus, the suite's tightest split.
+#
+# The sampled partitions are NESTED inside the canonical ones: the full split is
+# computed first and each partition is then narrowed within itself, so
+# partitions/sample/test.json is a strict subset of partitions/test.json. No
+# instance can migrate from test into development, and the baseline below did
+# not need re-measuring.
+name: vero/optimize-swe-bench-pro-sample
+description: >-
+  Improve a code-editing SWE-bench-Pro agent while preserving the Harbor agent
+  interface. The target must edit a checked-out repository so the task's hidden
+  test suite passes; the task-source verifier runs the suite for the reward.
+agent_repo: target
+# SWE-bench-Pro ships as the `swebenchpro` dataset in the DEFAULT Harbor registry
+# (731 instances), not as an `<org>/<name>@sha256:<digest>` package like
+# swe-atlas-qna or tau3. Its tasks resolve to git-backed task ids under
+# laude-institute/harbor-datasets at commit c8e8f3fac7097accaacf261d74c3d6f441de45b1,
+# so the version pin is the registry version rather than a content digest.
+task_source: swebenchpro@1.0
+# Shared with the canonical split on purpose: the manifest pins all 731 refs and
+# the build loader reads only task names from it, so a second copy would just be
+# a second thing to keep in sync.
+task_manifest: ../partitions/manifest.json
+agent_import_path: swebench_pro_agent.agent:SweBenchProAgent
+harbor_requirement: harbor[modal]==0.20.0
+
+partition_files:
+  development: ../partitions/sample/development.json
+  validation: ../partitions/sample/validation.json
+  test: ../partitions/sample/test.json
+
+agent_access:
+  - partition: development
+    disclosure: full
+    # false, unlike the sibling benchmarks: VeRO materializes case resources via
+    # PackageDatasetClient, which only accepts `<org>/<name>` package refs. A
+    # registry dataset such as swebenchpro@1.0 cannot be exposed that way.
+    expose_case_resources: false
+    total_runs: 100
+    total_cases: 132  # 4x the partition, the suite-wide budget convention
+  - partition: validation
+    disclosure: aggregate
+    expose_case_resources: false
+    min_aggregate_cases: 5
+    total_runs: 100
+    total_cases: 264  # 4x the partition, the suite-wide budget convention
+
+selection_partition: validation
+targets:
+  - partition: test
+    reward_key: reward
+    # Held-out K=3 mean over the 66 sampled cases. Recomputed from the very same
+    # trials that produced the canonical full-partition 0.2940 +-0.0082, which is
+    # sound only because the sample is a strict subset; the recomputation
+    # reproduces 0.2940 +-0.0082 exactly on the full set first.
+    #
+    # CAVEAT: those trials targeted qwen-3.6-27b, whereas this variant targets
+    # deepseek-v4-flash. Treat 0.2923 as provisional until a K=3 pass on deepseek
+    # over these 66 cases replaces it (~1.8h at max_concurrency 24).
+    #
+    # The +-0.0332 is not a typo: it is 4x the full split's +-0.0082 because n
+    # falls 293 -> 66. Worth remembering when reading a candidate's margin.
+    baseline_reward: 0.2923
+    failure_value: 0.0
+    max_attempts: 1
+    # The held-out eval is noisy: score the selected candidate 3x per case and
+    # average, so the final reward is comparable to the pinned baseline, which was
+    # itself pooled over 3 rounds. Without this the candidate carries ~sqrt(3) more
+    # standard error than the floor it is judged against.
+    # Per-target override - search/validation keep the global n_attempts (1).
+    n_attempts: 3
+    aggregate_attempts: mean
+
+evaluation_set_name: swe-bench-pro-sample
+objective:
+  selector:
+    metric: score
+  direction: maximize
+reward_mode: submit  # agent picks; falls back to auto_best, then current version
+baseline_floor: false  # gates on validation while reward is on test; opt-in only
+score_baseline: false  # pinned as baseline_reward above, like every sibling
+rescore_top_k: 3
+rescore_attempts: 1
+
+model: fireworks_ai/deepseek-v4-flash
+environment_name: ${inner_env:-modal}
+# inner eval sandboxes share a dedicated Modal app instead of the __harbor__ default
+extra_harbor_args: ["--ek", "app_name=harness-engineering-bench", "--ek", "sandbox_idle_timeout_secs=3600"]
+harbor_python_version: "3.12"
+n_attempts: 1
+max_retries: 1
+infrastructure_max_attempts: 3
+infrastructure_retry_delay_seconds: 5
+aggregate_attempts: best
+feedback_transcripts: true
+feedback_max_bytes: 16000
+expose_attempt_detail: false
+# Timeouts are bumped well above the GAIA baseline: SWE-bench-Pro tasks build a
+# real repository and run its (often slow) test suite, so both the per-case and
+# the verifier budgets need substantially more headroom than a short-answer task.
+# Both mirror the task package's own `[agent] timeout_sec = 3000`, so the
+# multiplier is 1.0, matching every sibling benchmark. (The canonical build.yaml
+# still reads 1800 here, implying 0.6, but its published 0.2940 was in fact
+# measured at 1.0: all 879 sampled trial configs record
+# agent_timeout_multiplier=1.0. That file's value is wrong and worth a separate
+# one-line fix; this variant is simply spelled correctly from the start.)
+timeout_seconds: 28800
+case_timeout_seconds: 3000
+task_agent_timeout_seconds: 3000
+max_concurrency: 24  # matching the rest of the suite
+error_rate_threshold: 0.1
+verifier_timeout_seconds: 28800
+secrets:
+  - MODAL_TOKEN_ID
+  - MODAL_TOKEN_SECRET
+  - WANDB_API_KEY
+  - WANDB_BASE_URL  # self-hosted W&B (scaleai.wandb.io)
+
+# candidate harness runs as an unprivileged uid, unable to read held-out state.
+harness_user: harness
+
+wandb:
+  project: vero-swe-bench-pro
+  group: swe-bench-pro-sample  # distinct from the full-split group
+  log_traces: true
+
+inference_gateway:
+  upstream_api_key_env: OPENAI_API_KEY
+  upstream_base_url_env: OPENAI_BASE_URL
+  producer:
+    # NOTE: gpt-5.3-codex has no /responses deployment on the LiteLLM proxy
+    # (/chat/completions is 200, /responses is 404 DeploymentNotFound), and the
+    # codex agent is Responses-native, so launch with
+    # `--model gpt-5.6-sol` (or gpt-4.1) until that deployment exists.
+    allowed_models: ["${optimizer_model:-gpt-5.6-sol}"]
+    max_concurrency: 8
+  evaluation:
+    # Prefixed form only: the agents strip just `openai/` before calling the
+    # gateway, which then matches this allow-list as an exact string.
+    allowed_models: [fireworks_ai/deepseek-v4-flash]
+    max_requests: 15000
+    max_tokens: 100000000
+    max_concurrency: 64
+instruct_multifidelity: true
+instruct_exhaust_budget: true

harness-engineering-bench/swe-bench-pro/partitions/sample/development.json

@@ -0,0 +1,35 @@
+[
+  "instance_ansible__ansible-12734fa21c08a0ce8c84e533abdc560db2eb1955-v7eee2454f617569fd6889f2211f75bc02a35f9f8",
+  "instance_ansible__ansible-1a4644ff15355fd696ac5b9d074a566a80fe7ca3-v30a923fb5c164d6cd18280c02422f75e611e8fb2",
+  "instance_ansible__ansible-34db57a47f875d11c4068567b9ec7ace174ec4cf-v1055803c3a812189a1133297f7f5468579283f86",
+  "instance_ansible__ansible-b8025ac160146319d2b875be3366b60c852dd35d-v0f01c69f1e2528b935359cfe578530722bca2c59",
+  "instance_ansible__ansible-fb144c44144f8bd3542e71f5db62b6d322c7bd85-vba6da65a0f3baefda7a058ebbd0a8dcafb8512f5",
+  "instance_element-hq__element-web-27139ca68eb075a4438c18fca184887002a4ffbc-vnan",
+  "instance_element-hq__element-web-56c7fc1948923b4b3f3507799e725ac16bcf8018-vnan",
+  "instance_flipt-io__flipt-8bd3604dc54b681f1f0f7dd52cbc70b3024184b6",
+  "instance_flipt-io__flipt-967855b429f749c28c112b8cb1b15bc79157f973",
+  "instance_flipt-io__flipt-b2cd6a6dd73ca91b519015fd5924fde8d17f3f06",
+  "instance_flipt-io__flipt-f1bc91a1b999656dbdb2495ccb57bf2105b84920",
+  "instance_future-architect__vuls-0ec945d0510cdebf92cdd8999f94610772689f14",
+  "instance_future-architect__vuls-7e91f5ef7e5712b1a3d7d5066ad6607e9debc21c",
+  "instance_future-architect__vuls-d576b6c6c15e56c47cc3e26f5878867677d4a9ea",
+  "instance_gravitational__teleport-0ecf31de0e98b272a6a2610abe1bbedd379a38a3-vce94f93ad1030e3136852817f2423c1b3ac37bc4",
+  "instance_gravitational__teleport-1a77b7945a022ab86858029d30ac7ad0d5239d00-vee9b09fb20c43af7e520f57e9239bbcf46b7113d",
+  "instance_gravitational__teleport-8302d467d160f869b77184e262adbe2fbc95d9ba-vce94f93ad1030e3136852817f2423c1b3ac37bc4",
+  "instance_internetarchive__openlibrary-630221ab686c64e75a2ce253c893c033e4814b2e-v93c53c13d5f9b383ebb411ee7750b49dcd1a34c6",
+  "instance_internetarchive__openlibrary-bdba0af0f6cbaca8b5fc3be2a3080f38156d9c92-ve8c8d62a2b60610a3c4631f5f23ed866bada9818",
+  "instance_internetarchive__openlibrary-d40ec88713dc95ea791b252f92d2f7b75e107440-v13642507b4fc1f8d234172bf8129942da2c2ca26",
+  "instance_internetarchive__openlibrary-d8162c226a9d576f094dc1830c4c1ffd0be2dd17-v76304ecdb3a5954fcf13feb710e8c40fcf24b73c",
+  "instance_navidrome__navidrome-29bc17acd71596ae92131aca728716baf5af9906",
+  "instance_navidrome__navidrome-8d56ec898e776e7e53e352cb9b25677975787ffc",
+  "instance_nodebb__nodebb-8ca65b0c78c67c1653487c02d1135e1b702185e1-vf2cf3cbd463b7ad942381f1c6d077626485a1e9e",
+  "instance_nodebb__nodebb-97c8569a798075c50e93e585ac741ab55cb7c28b-vf2cf3cbd463b7ad942381f1c6d077626485a1e9e",
+  "instance_protonmail__webclients-6e1873b06df6529a469599aa1d69d3b18f7d9d37",
+  "instance_protonmail__webclients-c5a2089ca2bfe9aa1d85a664b8ad87ef843a1c9c",
+  "instance_protonmail__webclients-fc9d535e9beb3ae30a52a7146398cadfd6e30606",
+  "instance_qutebrowser__qutebrowser-1a9e74bfaf9a9db2a510dc14572d33ded6040a57-v2ef375ac784985212b1805e1d0431dc8f1b3c171",
+  "instance_qutebrowser__qutebrowser-2e961080a85d660148937ee8f0f6b3445a8f2c01-v363c8a7e5ccdf6968fc7ab84a2053ac78036691d",
+  "instance_qutebrowser__qutebrowser-ebfe9b7aa0c4ba9d451f993e08955004aaec4345-v059c6fdc75567943479b23ebca7c07b5e9a7f34c",
+  "instance_qutebrowser__qutebrowser-ff1c025ad3210506fc76e1f604d8c8c27637d88e-v363c8a7e5ccdf6968fc7ab84a2053ac78036691d",
+  "instance_tutao__tutanota-db90ac26ab78addf72a8efaff3c7acc0fbd6d000-vbc0d9ba8f0071fbe982809910959a6ff8884dbbf"
+]