feat: add a subsampled swe-bench-pro variant (33/66/66) for optimizer runs

harness-engineering-bench/CONFIGURATION.md

@@ -157,25 +157,25 @@ benchmark can be checked against the others at a glance.
 
 ## Per-benchmark values
 
-| | gaia | officeqa | swe-atlas-qna | tau3 | browsecomp-plus |
-|---|---|---|---|---|---|
-| target model | gpt-5.4-mini ◇ | deepseek-v4-flash | gpt-oss-120b | deepseek-v4-flash | deepseek-v4-flash |
-| held-out baseline (K=3) ◆ | 0.621 ±0.052 | 0.341 ±0.033 | 0.068 ±0.026 (agg 0.632) | 0.732 ±0.010 | 0.462 ±0.028 |
-| split dev/val/test | 33/66/66 | 49/98/99 | 25/49/50 | 75/150/150 | 33/66/66 |
-| dev budget (runs / cases) | 100 / 132 | 100 / 196 | 100 / 100 | 100 / 300 | 100 / 132 |
-| val budget (runs / cases) | 100 / 264 | 100 / 392 | 100 / 196 | 100 / 600 | 100 / 264 |
-| gateway max_tokens (evaluation, finalization each) ¶ | 2 B | 3 B | 2 B | 4 B | 2 B |
-| max_concurrency (cases in flight) § | 24 | 24 | 24 | 24 | 24 |
-| timeout_seconds (per eval) ‖ | 7200 | 28800 | 90000 | 79200 | 39600 |
-| case_timeout_seconds = declared † | 600 | 1800 | 10800 | 3600 | 3600 |
-| task_agent_timeout_seconds (declared) | 600 | 1800 | 10800 | 3600 | 3600 |
-| declared `[verifier] timeout_sec` | 300 | 300 | 900 | 300 | 300 |
-| declared `build_timeout_sec` | 300 | 600 | 600 | 600 | 7200 |
-| verifier_timeout_seconds ‖ | 14400 | 54000 | 176400 | 158400 | 75600 |
-| BASH_MAX_TIMEOUT_MS (tool) ¤ | 3600 s | 10800 s | 39600 s | 32400 s | 14400 s |
-| harness_user | harness | harness | null ‡ | null ‡ | null ‡ |
-| task_services_use_upstream | false | false | true (rubric judge) | true (user-sim + grader) | true (answer judge) |
-| task-specific extras | — | `--no-force-build` (prebuilt corpus image) | `keepalive` --ek (ENTRYPOINT images) | `TAU2_*` model pins | pinned 2.2 GB BM25 index |
+| | gaia | officeqa | swe-atlas-qna | tau3 | browsecomp-plus | swe-bench-pro |
+|---|---|---|---|---|---| --- |
+| target model | gpt-5.4-mini ◇ | deepseek-v4-flash | gpt-oss-120b | deepseek-v4-flash | deepseek-v4-flash | gpt-4o ◈ |
+| held-out baseline (K=3) ◆ | 0.621 ±0.052 | 0.341 ±0.033 | 0.068 ±0.026 (agg 0.632) | 0.732 ±0.010 | 0.462 ±0.028 | 0.294 ±0.008 ◈ |
+| split dev/val/test | 33/66/66 | 49/98/99 | 25/49/50 | 75/150/150 | 33/66/66 | 146/292/293 ◈ |
+| dev budget (runs / cases) | 100 / 132 | 100 / 196 | 100 / 100 | 100 / 300 | 100 / 132 | 100 / 146 ◈ |
+| val budget (runs / cases) | 100 / 264 | 100 / 392 | 100 / 196 | 100 / 600 | 100 / 264 | 100 / 292 ◈ |
+| gateway max_tokens (evaluation, finalization each) ¶ | 2 B | 3 B | 2 B | 4 B | 2 B | 100 M |
+| max_concurrency (cases in flight) § | 24 | 24 | 24 | 24 | 24 | 8 ◈ |
+| timeout_seconds (per eval) ‖ | 7200 | 28800 | 90000 | 79200 | 39600 | 28800 |
+| case_timeout_seconds = declared † | 600 | 1800 | 10800 | 3600 | 3600 | 1800 ◈ |
+| task_agent_timeout_seconds (declared) | 600 | 1800 | 10800 | 3600 | 3600 | 3000 |
+| declared `[verifier] timeout_sec` | 300 | 300 | 900 | 300 | 300 | n/a (registry dataset) |
+| declared `build_timeout_sec` | 300 | 600 | 600 | 600 | 7200 | n/a (registry dataset) |
+| verifier_timeout_seconds ‖ | 14400 | 54000 | 176400 | 158400 | 75600 | 28800 |
+| BASH_MAX_TIMEOUT_MS (tool) ¤ | 3600 s | 10800 s | 39600 s | 32400 s | 14400 s | n/a |
+| harness_user | harness | harness | null ‡ | null ‡ | null ‡ | harness |
+| task_services_use_upstream | false | false | true (rubric judge) | true (user-sim + grader) | true (answer judge) | false |
+| task-specific extras | — | `--no-force-build` (prebuilt corpus image) | `keepalive` --ek (ENTRYPOINT images) | `TAU2_*` model pins | pinned 2.2 GB BM25 index | registry dataset; `expose_case_resources: false`; sampled variant ◈ |
 
 ## Conventions
 
@@ -271,6 +271,52 @@ agent's reason/search-only-turn crash was fixed.
 
 **gaia and tau3 are too noisy for single-run comparisons.** gaia's own three rounds spanned 0.554-0.682 (sd 0.052), and tau3's optimizer scored one *unchanged* harness at 0.800 and 0.547 on development -- its user-simulator and NL-assertion grader are both LLMs, so their variance rides on every eval. Treat a gaia or tau3 delta under ~0.1 as unresolved. Their splits are not the problem: domain mix matches to the percentage point across all three partitions (airline 13%, banking 26%, retail 30%, telecom 31%), as does telecom persona difficulty.
 
+◈ swe-bench-pro is the newest benchmark and the only one with **two** configs.
+The column above describes the canonical `baseline/build.yaml` (full 731-instance
+dataset, split 146/292/293). Alongside it sits `baseline/build.sample.yaml`, a
+subsampled variant for optimizer runs; the canonical config is unchanged by it.
+
+*Why a variant.* The full split is unaffordable for an optimizer loop: each case
+builds a real repository and runs its test suite, so the held-out baseline alone
+cost ~7.8h of wall clock for three rounds, and an optimizer evaluates candidates
+many times over. The variant is 33/66/66, matching gaia and browsecomp-plus, on
+`fireworks_ai/deepseek-v4-flash`, `max_concurrency: 24`, 4x case budgets
+(132/264), and `case_timeout_seconds: 3000`.
+
+*The sample is nested.* The full split is computed first and each partition is
+then narrowed within itself (`partition_swe_bench_pro.py --sample`), so
+`partitions/sample/test.json` is a strict subset of `partitions/test.json`. No
+instance can migrate from test into development, and the baseline needed no
+re-measuring: **0.292 ±0.033** over the 66 sampled cases is the same K=3 trials
+that produced the full-partition 0.294 ±0.008, recomputed. The point estimate
+barely moves; the spread widens roughly fourfold because n falls 293 → 66, which
+matters when reading a candidate's margin over that floor.
+
+*Three known inconsistencies in the canonical config*, all pre-existing and left
+alone here rather than silently changed:
+
+- `case_timeout_seconds: 1800` against `task_agent_timeout_seconds: 3000` implies
+  a 0.6 multiplier, but the pinned 0.294 was measured at **1.0**: all 879 sampled
+  trial configs record `agent_timeout_multiplier=1.0`. The config value is wrong,
+  not the measurement. Every sibling has these two equal.
+- `model: gpt-4o`, but the 0.294 baseline was measured against
+  **qwen-3.6-27b**. So the recomputed 0.292 the variant pins is also a qwen
+  number while the variant targets deepseek; treat it as provisional until a K=3
+  deepseek pass over the 66 replaces it (~1.8h at `max_concurrency: 24`).
+- **The pinned 0.294 was measured on a different agent than the one in the tree**,
+  which is the same staleness the note above describes, in its strongest form. The
+  qwen-era agent was 630 lines on Chat Completions with an explicit local
+  `messages` list and edited files on 571 of 879 cases (write_file 1108,
+  apply_patch 899, submit 85). The committed seed is Responses-only, delegates its
+  conversation to `previous_response_id`, and edits nothing: over 66 sampled cases
+  it called write_file, apply_patch, and submit **zero** times and scored 0.0000.
+  So this benchmark is the one the re-pin above did not cover, and it needs
+  `scripts/rescore_candidate.py --seed` more than any of the five that did.
+
+Also unlike the siblings, swe-bench-pro's canonical config still has
+`score_baseline: true`, so it re-measures the seed every run instead of pinning
+it. The variant sets `false` and pins, as everything else does.
+
 ◇ gaia is the exception to the deepseek-v4-flash default: it is multimodal and
 that model is text-only. Verified against the same litellm endpoint the gateway
 proxies to — gpt-5.4-mini returns 200 for every request shape the gaia agent
@@ -320,7 +366,7 @@ per benchmark from its own case counts; see each `build.yaml` for the arithmetic
 the per-key bucket, so more keys buy proportionally more parallel runs:
 
 | counter | limit | scope |
-|---|---|---|
+|---|---|---| --- |
 | `x-ratelimit-api_key-limit-tokens` | 10 M TPM | **per key** |
 | `x-ratelimit-api_key-limit-requests` | 5 000 RPM | **per key** |
 

harness-engineering-bench/swe-bench-pro/baseline/build.sample.yaml

@@ -0,0 +1,155 @@
+# Subsampled SWE-bench-Pro variant, for optimizer runs.
+#
+# This is a SECOND configuration alongside build.yaml, not a replacement. The
+# canonical build.yaml keeps the full 146/292/293 split and its held-out
+# baseline of 0.2940 +-0.0082; nothing here touches it.
+#
+# Why a variant exists: the full split is unaffordable for an optimizer loop.
+# Each case builds a real repository and runs its (often slow) test suite, so
+# the full-split held-out baseline alone cost ~7.8h of wall clock for three
+# rounds, and an optimizer evaluates candidates many times over. 33/66/66
+# matches gaia and browsecomp-plus, the suite's tightest split.
+#
+# The sampled partitions are NESTED inside the canonical ones: the full split is
+# computed first and each partition is then narrowed within itself, so
+# partitions/sample/test.json is a strict subset of partitions/test.json. No
+# instance can migrate from test into development, and the baseline below did
+# not need re-measuring.
+name: vero/optimize-swe-bench-pro-sample
+description: >-
+  Improve a code-editing SWE-bench-Pro agent while preserving the Harbor agent
+  interface. The target must edit a checked-out repository so the task's hidden
+  test suite passes; the task-source verifier runs the suite for the reward.
+agent_repo: target
+# SWE-bench-Pro ships as the `swebenchpro` dataset in the DEFAULT Harbor registry
+# (731 instances), not as an `<org>/<name>@sha256:<digest>` package like
+# swe-atlas-qna or tau3. Its tasks resolve to git-backed task ids under
+# laude-institute/harbor-datasets at commit c8e8f3fac7097accaacf261d74c3d6f441de45b1,
+# so the version pin is the registry version rather than a content digest.
+task_source: swebenchpro@1.0
+# Shared with the canonical split on purpose: the manifest pins all 731 refs and
+# the build loader reads only task names from it, so a second copy would just be
+# a second thing to keep in sync.
+task_manifest: ../partitions/manifest.json
+agent_import_path: swebench_pro_agent.agent:SweBenchProAgent
+harbor_requirement: harbor[modal]==0.20.0
+
+partition_files:
+  development: ../partitions/sample/development.json
+  validation: ../partitions/sample/validation.json
+  test: ../partitions/sample/test.json
+
+agent_access:
+  - partition: development
+    disclosure: full
+    # false, unlike the sibling benchmarks: VeRO materializes case resources via
+    # PackageDatasetClient, which only accepts `<org>/<name>` package refs. A
+    # registry dataset such as swebenchpro@1.0 cannot be exposed that way.
+    expose_case_resources: false
+    total_runs: 100
+    total_cases: 132  # 4x the partition, the suite-wide budget convention
+  - partition: validation
+    disclosure: aggregate
+    expose_case_resources: false
+    min_aggregate_cases: 5
+    total_runs: 100
+    total_cases: 264  # 4x the partition, the suite-wide budget convention
+
+selection_partition: validation
+targets:
+  - partition: test
+    reward_key: reward
+    # Held-out K=3 mean over the 66 sampled cases. Recomputed from the very same
+    # trials that produced the canonical full-partition 0.2940 +-0.0082, which is
+    # sound only because the sample is a strict subset; the recomputation
+    # reproduces 0.2940 +-0.0082 exactly on the full set first.
+    #
+    # CAVEAT: those trials targeted qwen-3.6-27b, whereas this variant targets
+    # deepseek-v4-flash. Treat 0.2923 as provisional until a K=3 pass on deepseek
+    # over these 66 cases replaces it (~1.8h at max_concurrency 24).
+    #
+    # The +-0.0332 is not a typo: it is 4x the full split's +-0.0082 because n
+    # falls 293 -> 66. Worth remembering when reading a candidate's margin.
+    baseline_reward: 0.2923
+    failure_value: 0.0
+    max_attempts: 1
+    # The held-out eval is noisy: score the selected candidate 3x per case and
+    # average, so the final reward is comparable to the pinned baseline, which was
+    # itself pooled over 3 rounds. Without this the candidate carries ~sqrt(3) more
+    # standard error than the floor it is judged against.
+    # Per-target override - search/validation keep the global n_attempts (1).
+    n_attempts: 3
+    aggregate_attempts: mean
+
+evaluation_set_name: swe-bench-pro-sample
+objective:
+  selector:
+    metric: score
+  direction: maximize
+reward_mode: submit  # agent picks; falls back to auto_best, then current version
+baseline_floor: false  # gates on validation while reward is on test; opt-in only
+score_baseline: false  # pinned as baseline_reward above, like every sibling
+rescore_top_k: 3
+rescore_attempts: 1
+
+model: fireworks_ai/deepseek-v4-flash
+environment_name: ${inner_env:-modal}
+# inner eval sandboxes share a dedicated Modal app instead of the __harbor__ default
+extra_harbor_args: ["--ek", "app_name=harness-engineering-bench", "--ek", "sandbox_idle_timeout_secs=3600"]
+harbor_python_version: "3.12"
+n_attempts: 1
+max_retries: 1
+infrastructure_max_attempts: 3
+infrastructure_retry_delay_seconds: 5
+aggregate_attempts: best
+feedback_transcripts: true
+feedback_max_bytes: 16000
+expose_attempt_detail: false
+# Timeouts are bumped well above the GAIA baseline: SWE-bench-Pro tasks build a
+# real repository and run its (often slow) test suite, so both the per-case and
+# the verifier budgets need substantially more headroom than a short-answer task.
+# Both mirror the task package's own `[agent] timeout_sec = 3000`, so the
+# multiplier is 1.0, matching every sibling benchmark. (The canonical build.yaml
+# still reads 1800 here, implying 0.6, but its published 0.2940 was in fact
+# measured at 1.0: all 879 sampled trial configs record
+# agent_timeout_multiplier=1.0. That file's value is wrong and worth a separate
+# one-line fix; this variant is simply spelled correctly from the start.)
+timeout_seconds: 28800
+case_timeout_seconds: 3000
+task_agent_timeout_seconds: 3000
+max_concurrency: 24  # matching the rest of the suite
+error_rate_threshold: 0.1
+verifier_timeout_seconds: 28800
+secrets:
+  - MODAL_TOKEN_ID
+  - MODAL_TOKEN_SECRET
+  - WANDB_API_KEY
+  - WANDB_BASE_URL  # self-hosted W&B (scaleai.wandb.io)
+
+# candidate harness runs as an unprivileged uid, unable to read held-out state.
+harness_user: harness
+
+wandb:
+  project: vero-swe-bench-pro
+  group: swe-bench-pro-sample  # distinct from the full-split group
+  log_traces: true
+
+inference_gateway:
+  upstream_api_key_env: OPENAI_API_KEY
+  upstream_base_url_env: OPENAI_BASE_URL
+  producer:
+    # NOTE: gpt-5.3-codex has no /responses deployment on the LiteLLM proxy
+    # (/chat/completions is 200, /responses is 404 DeploymentNotFound), and the
+    # codex agent is Responses-native, so launch with
+    # `--model gpt-5.6-sol` (or gpt-4.1) until that deployment exists.
+    allowed_models: ["${optimizer_model:-gpt-5.6-sol}"]
+    max_concurrency: 8
+  evaluation:
+    # Prefixed form only: the agents strip just `openai/` before calling the
+    # gateway, which then matches this allow-list as an exact string.
+    allowed_models: [fireworks_ai/deepseek-v4-flash]
+    max_requests: 15000
+    max_tokens: 100000000
+    max_concurrency: 64
+instruct_multifidelity: true
+instruct_exhaust_budget: true

harness-engineering-bench/swe-bench-pro/partitions/sample/development.json

@@ -0,0 +1,35 @@
+[
+  "instance_ansible__ansible-12734fa21c08a0ce8c84e533abdc560db2eb1955-v7eee2454f617569fd6889f2211f75bc02a35f9f8",
+  "instance_ansible__ansible-1a4644ff15355fd696ac5b9d074a566a80fe7ca3-v30a923fb5c164d6cd18280c02422f75e611e8fb2",
+  "instance_ansible__ansible-34db57a47f875d11c4068567b9ec7ace174ec4cf-v1055803c3a812189a1133297f7f5468579283f86",
+  "instance_ansible__ansible-b8025ac160146319d2b875be3366b60c852dd35d-v0f01c69f1e2528b935359cfe578530722bca2c59",
+  "instance_ansible__ansible-fb144c44144f8bd3542e71f5db62b6d322c7bd85-vba6da65a0f3baefda7a058ebbd0a8dcafb8512f5",
+  "instance_element-hq__element-web-27139ca68eb075a4438c18fca184887002a4ffbc-vnan",
+  "instance_element-hq__element-web-56c7fc1948923b4b3f3507799e725ac16bcf8018-vnan",
+  "instance_flipt-io__flipt-8bd3604dc54b681f1f0f7dd52cbc70b3024184b6",
+  "instance_flipt-io__flipt-967855b429f749c28c112b8cb1b15bc79157f973",
+  "instance_flipt-io__flipt-b2cd6a6dd73ca91b519015fd5924fde8d17f3f06",
+  "instance_flipt-io__flipt-f1bc91a1b999656dbdb2495ccb57bf2105b84920",
+  "instance_future-architect__vuls-0ec945d0510cdebf92cdd8999f94610772689f14",
+  "instance_future-architect__vuls-7e91f5ef7e5712b1a3d7d5066ad6607e9debc21c",
+  "instance_future-architect__vuls-d576b6c6c15e56c47cc3e26f5878867677d4a9ea",
+  "instance_gravitational__teleport-0ecf31de0e98b272a6a2610abe1bbedd379a38a3-vce94f93ad1030e3136852817f2423c1b3ac37bc4",
+  "instance_gravitational__teleport-1a77b7945a022ab86858029d30ac7ad0d5239d00-vee9b09fb20c43af7e520f57e9239bbcf46b7113d",
+  "instance_gravitational__teleport-8302d467d160f869b77184e262adbe2fbc95d9ba-vce94f93ad1030e3136852817f2423c1b3ac37bc4",
+  "instance_internetarchive__openlibrary-630221ab686c64e75a2ce253c893c033e4814b2e-v93c53c13d5f9b383ebb411ee7750b49dcd1a34c6",
+  "instance_internetarchive__openlibrary-bdba0af0f6cbaca8b5fc3be2a3080f38156d9c92-ve8c8d62a2b60610a3c4631f5f23ed866bada9818",
+  "instance_internetarchive__openlibrary-d40ec88713dc95ea791b252f92d2f7b75e107440-v13642507b4fc1f8d234172bf8129942da2c2ca26",
+  "instance_internetarchive__openlibrary-d8162c226a9d576f094dc1830c4c1ffd0be2dd17-v76304ecdb3a5954fcf13feb710e8c40fcf24b73c",
+  "instance_navidrome__navidrome-29bc17acd71596ae92131aca728716baf5af9906",
+  "instance_navidrome__navidrome-8d56ec898e776e7e53e352cb9b25677975787ffc",
+  "instance_nodebb__nodebb-8ca65b0c78c67c1653487c02d1135e1b702185e1-vf2cf3cbd463b7ad942381f1c6d077626485a1e9e",
+  "instance_nodebb__nodebb-97c8569a798075c50e93e585ac741ab55cb7c28b-vf2cf3cbd463b7ad942381f1c6d077626485a1e9e",
+  "instance_protonmail__webclients-6e1873b06df6529a469599aa1d69d3b18f7d9d37",
+  "instance_protonmail__webclients-c5a2089ca2bfe9aa1d85a664b8ad87ef843a1c9c",
+  "instance_protonmail__webclients-fc9d535e9beb3ae30a52a7146398cadfd6e30606",
+  "instance_qutebrowser__qutebrowser-1a9e74bfaf9a9db2a510dc14572d33ded6040a57-v2ef375ac784985212b1805e1d0431dc8f1b3c171",
+  "instance_qutebrowser__qutebrowser-2e961080a85d660148937ee8f0f6b3445a8f2c01-v363c8a7e5ccdf6968fc7ab84a2053ac78036691d",
+  "instance_qutebrowser__qutebrowser-ebfe9b7aa0c4ba9d451f993e08955004aaec4345-v059c6fdc75567943479b23ebca7c07b5e9a7f34c",
+  "instance_qutebrowser__qutebrowser-ff1c025ad3210506fc76e1f604d8c8c27637d88e-v363c8a7e5ccdf6968fc7ab84a2053ac78036691d",
+  "instance_tutao__tutanota-db90ac26ab78addf72a8efaff3c7acc0fbd6d000-vbc0d9ba8f0071fbe982809910959a6ff8884dbbf"
+]