feat: subsample swe-bench-pro to 33/66/66 and put it on deepseek

A full 146/292/293 split is unaffordable for an optimizer loop. Each case builds
a real repository and runs its test suite, so the held-out baseline alone cost
~7.8h of wall clock for three rounds, and the optimizer evaluates candidates
many times over. 33/66/66 matches browsecomp-plus and gaia, the suite's tightest
split, which is the right reference for its most expensive benchmark.

The sample is NESTED, which is the load-bearing detail. The full split is
computed first, unchanged, and each partition is then sampled within itself, so
the sampled test set is a strict subset of the full test set. Two things follow:
no instance can migrate from test into development (a flat re-split moves them,
which would quietly destroy the held-out guarantee), and the baseline did not
need re-measuring.

  baseline_reward: 0.2923, pinned, with score_baseline: false like every
  sibling. That is the same K=3 trials that produced the published
  full-partition 0.2940 +-0.0082, recomputed over the 66 sampled cases; the
  recomputation reproduces 0.2940 +-0.0082 exactly on the full set first, which
  is what validates it. The point estimate barely moves, but the spread widens
  to +-0.0332 because n falls 293 -> 66. Worth remembering when reading a
  candidate's margin over this floor.

Repository stratification and the sha256 ordering are unchanged and apply within
each partition, so all 11 upstream repos are still represented in every one. The
566 unselected instances stay in the manifest with `partition: null`, keeping
their refs pinned so a later widening is a counts-only change, not a re-pin.
`--tasks-dir` is now optional: the manifest already records every instance's
name and stratum, so changing counts no longer means re-exporting 731 task dirs.

Config corrections that ride along, all aligning with the rest of the suite:

  - case_timeout_seconds 1800 -> 3000, equal to task_agent_timeout_seconds, so
    the multiplier is 1.0 like every sibling. The old value implied 0.6, but the
    published baseline was in fact measured at 1.0: all 879 sampled trial
    configs record agent_timeout_multiplier=1.0. The config was wrong, not the
    measurement.
  - max_concurrency 8 -> 24.
  - total_cases 1x -> 4x the partition (132/264), the convention every other
    benchmark already follows.
  - target model gpt-4o -> fireworks_ai/deepseek-v4-flash, prefixed-only because
    the agents strip just `openai/` before the gateway matches exactly.

CONFIGURATION.md gains the swe-bench-pro column it never had, and a footnote
explaining the subsample and the widened error bar.

Verified: partitions regenerate byte-identically and --check passes; every
partition is a strict subset of its full counterpart; all 11 repos appear in
each; the build compiles and the compiled task carries 132/264 budgets,
baseline_reward 0.2923, deepseek on all three backends and finalization,
case==agent timeout 3000, max_concurrency 24, and n_attempts 3 / mean on the
test backend only; 13 passed, 6 skipped in test_v05_benchmark_configs.py.

Author: shehabyasser-scale

harness-engineering-bench/CONFIGURATION.md

@@ -157,25 +157,25 @@ benchmark can be checked against the others at a glance.
 
 ## Per-benchmark values
 
-| | gaia | officeqa | swe-atlas-qna | tau3 | browsecomp-plus |
-|---|---|---|---|---|---|
-| target model | gpt-5.4-mini ◇ | deepseek-v4-flash | gpt-oss-120b | deepseek-v4-flash | deepseek-v4-flash |
-| held-out baseline (K=3) ◆ | 0.574 ±0.010 | 0.360 ±0.042 | 0.097 ±0.011 (agg 0.632) | 0.611 ±0.021 | 0.449 ±0.007 |
-| split dev/val/test | 33/66/66 | 49/98/99 | 25/49/50 | 75/150/150 | 33/66/66 |
-| dev budget (runs / cases) | 100 / 132 | 100 / 196 | 100 / 100 | 100 / 300 | 100 / 132 |
-| val budget (runs / cases) | 100 / 264 | 100 / 392 | 100 / 196 | 100 / 600 | 100 / 264 |
-| gateway max_tokens (evaluation, finalization each) ¶ | 2 B | 3 B | 2 B | 4 B | 2 B |
-| max_concurrency (cases in flight) § | 24 | 24 | 24 | 24 | 24 |
-| timeout_seconds (per eval) ‖ | 7200 | 28800 | 90000 | 79200 | 39600 |
-| case_timeout_seconds = declared † | 600 | 1800 | 10800 | 3600 | 3600 |
-| task_agent_timeout_seconds (declared) | 600 | 1800 | 10800 | 3600 | 3600 |
-| declared `[verifier] timeout_sec` | 300 | 300 | 900 | 300 | 300 |
-| declared `build_timeout_sec` | 300 | 600 | 600 | 600 | 7200 |
-| verifier_timeout_seconds ‖ | 14400 | 54000 | 176400 | 158400 | 75600 |
-| BASH_MAX_TIMEOUT_MS (tool) ¤ | 3600 s | 10800 s | 39600 s | 32400 s | 14400 s |
-| harness_user | harness | harness | null ‡ | null ‡ | null ‡ |
-| task_services_use_upstream | false | false | true (rubric judge) | true (user-sim + grader) | true (answer judge) |
-| task-specific extras | — | `--no-force-build` (prebuilt corpus image) | `keepalive` --ek (ENTRYPOINT images) | `TAU2_*` model pins | pinned 2.2 GB BM25 index |
+| | gaia | officeqa | swe-atlas-qna | tau3 | browsecomp-plus | swe-bench-pro |
+|---|---|---|---|---|---| --- |
+| target model | gpt-5.4-mini ◇ | deepseek-v4-flash | gpt-oss-120b | deepseek-v4-flash | deepseek-v4-flash | deepseek-v4-flash |
+| held-out baseline (K=3) ◆ | 0.574 ±0.010 | 0.360 ±0.042 | 0.097 ±0.011 (agg 0.632) | 0.611 ±0.021 | 0.449 ±0.007 | 0.292 ±0.033 ◈ |
+| split dev/val/test | 33/66/66 | 49/98/99 | 25/49/50 | 75/150/150 | 33/66/66 | 33/66/66 ◈ |
+| dev budget (runs / cases) | 100 / 132 | 100 / 196 | 100 / 100 | 100 / 300 | 100 / 132 | 100 / 132 |
+| val budget (runs / cases) | 100 / 264 | 100 / 392 | 100 / 196 | 100 / 600 | 100 / 264 | 100 / 264 |
+| gateway max_tokens (evaluation, finalization each) ¶ | 2 B | 3 B | 2 B | 4 B | 2 B | 100 M |
+| max_concurrency (cases in flight) § | 24 | 24 | 24 | 24 | 24 | 24 |
+| timeout_seconds (per eval) ‖ | 7200 | 28800 | 90000 | 79200 | 39600 | 28800 |
+| case_timeout_seconds = declared † | 600 | 1800 | 10800 | 3600 | 3600 | 3000 |
+| task_agent_timeout_seconds (declared) | 600 | 1800 | 10800 | 3600 | 3600 | 3000 |
+| declared `[verifier] timeout_sec` | 300 | 300 | 900 | 300 | 300 | n/a (registry dataset) |
+| declared `build_timeout_sec` | 300 | 600 | 600 | 600 | 7200 | n/a (registry dataset) |
+| verifier_timeout_seconds ‖ | 14400 | 54000 | 176400 | 158400 | 75600 | 28800 |
+| BASH_MAX_TIMEOUT_MS (tool) ¤ | 3600 s | 10800 s | 39600 s | 32400 s | 14400 s | n/a |
+| harness_user | harness | harness | null ‡ | null ‡ | null ‡ | harness |
+| task_services_use_upstream | false | false | true (rubric judge) | true (user-sim + grader) | true (answer judge) | false |
+| task-specific extras | — | `--no-force-build` (prebuilt corpus image) | `keepalive` --ek (ENTRYPOINT images) | `TAU2_*` model pins | pinned 2.2 GB BM25 index | registry dataset, `expose_case_resources: false` |
 
 ## Conventions
 
@@ -267,6 +267,23 @@ deepseek benchmarks logged zero exceptions over 945 trials, swe-atlas lost
 5/150 to gpt-oss 128k context overflow, and gaia lost 4/198 (infra) after the
 agent's reason/search-only-turn crash was fixed.
 
+◈ swe-bench-pro is **subsampled**: its dataset holds 731 instances, which split
+20/40/40 into 146/292/293, but it is by far the most expensive benchmark here
+(each case builds a real repository and runs its test suite, and the held-out
+baseline alone took ~7.8h of wall clock for three rounds). It is therefore cut
+to 33/66/66, matching gaia and browsecomp-plus.
+
+The sample is **nested**: the full split is computed first and each partition is
+then sampled within itself, so the sampled test set is a strict subset of the
+full test set. No instance can migrate from test into development, and the
+baseline did not need re-measuring — 0.292 ±0.033 is the same K=3 trials that
+produced the full-partition 0.294 ±0.008, recomputed over the 66 sampled cases.
+The point estimate barely moves; the spread widens roughly fourfold because n
+falls 293 → 66, which is the real cost of sampling and is worth remembering when
+reading a candidate's margin over this floor. The 566 unselected instances stay
+in the manifest with `partition: null`, so their refs remain pinned and widening
+the split later is a counts-only change.
+
 ◇ gaia is the exception to the deepseek-v4-flash default: it is multimodal and
 that model is text-only. Verified against the same litellm endpoint the gateway
 proxies to — gpt-5.4-mini returns 200 for every request shape the gaia agent
@@ -316,7 +333,7 @@ per benchmark from its own case counts; see each `build.yaml` for the arithmetic
 the per-key bucket, so more keys buy proportionally more parallel runs:
 
 | counter | limit | scope |
-|---|---|---|
+|---|---|---| --- |
 | `x-ratelimit-api_key-limit-tokens` | 10 M TPM | **per key** |
 | `x-ratelimit-api_key-limit-requests` | 5 000 RPM | **per key** |
 

harness-engineering-bench/swe-bench-pro/baseline/build.yaml

@@ -27,20 +27,31 @@ agent_access:
     # registry dataset such as swebenchpro@1.0 cannot be exposed that way.
     expose_case_resources: false
     total_runs: 100
-    total_cases: 146
+    total_cases: 132  # 4x the partition, the suite-wide budget convention
   - partition: validation
     disclosure: aggregate
     expose_case_resources: false
     min_aggregate_cases: 5
     total_runs: 100
-    total_cases: 292
+    total_cases: 264  # 4x the partition, the suite-wide budget convention
 
 selection_partition: validation
 targets:
   - partition: test
     reward_key: reward
+    # Held-out K=3 mean on the sampled 66-case test set, recomputed from the same
+    # trials that produced the full-partition 0.2940 (the sample is a strict
+    # subset, so no re-run was needed). See CONFIGURATION.md.
+    baseline_reward: 0.2923
     failure_value: 0.0
     max_attempts: 1
+    # The held-out eval is noisy: score the selected candidate 3x per case and
+    # average, so the final reward is comparable to the pinned baseline, which was
+    # itself pooled over 3 rounds. Without this the candidate carries ~sqrt(3) more
+    # standard error than the floor it is judged against.
+    # Per-target override - search/validation keep the global n_attempts (1).
+    n_attempts: 3
+    aggregate_attempts: mean
 
 evaluation_set_name: swe-bench-pro
 objective:
@@ -49,11 +60,11 @@ objective:
   direction: maximize
 reward_mode: submit  # agent picks; falls back to auto_best, then current version
 baseline_floor: false  # gates on validation while reward is on test; opt-in only
-score_baseline: true
+score_baseline: false  # pinned as baseline_reward above, like every sibling
 rescore_top_k: 3
 rescore_attempts: 1
 
-model: gpt-4o
+model: fireworks_ai/deepseek-v4-flash
 environment_name: ${inner_env:-modal}
 # inner eval sandboxes share a dedicated Modal app instead of the __harbor__ default
 extra_harbor_args: ["--ek", "app_name=harness-engineering-bench", "--ek", "sandbox_idle_timeout_secs=3600"]
@@ -69,13 +80,15 @@ expose_attempt_detail: false
 # Timeouts are bumped well above the GAIA baseline: SWE-bench-Pro tasks build a
 # real repository and run its (often slow) test suite, so both the per-case and
 # the verifier budgets need substantially more headroom than a short-answer task.
-# task_agent_timeout_seconds mirrors the task package's own `[agent] timeout_sec
-# = 3000`; case_timeout_seconds is the tighter budget VeRO enforces on top, so
-# the baseline runner's --agent-timeout-multiplier is 1800/3000 = 0.6.
+# Both mirror the task package's own `[agent] timeout_sec = 3000`, so the
+# multiplier is 1.0, matching every sibling benchmark. It previously read 1800
+# here (a 0.6 multiplier) while the published held-out baseline of 0.2940 was in
+# fact measured at 1.0: all 879 sampled trial configs record
+# agent_timeout_multiplier=1.0. The config, not the measurement, was wrong.
 timeout_seconds: 28800
-case_timeout_seconds: 1800
+case_timeout_seconds: 3000
 task_agent_timeout_seconds: 3000
-max_concurrency: 8
+max_concurrency: 24  # 8 -> 24, matching the rest of the suite
 error_rate_threshold: 0.1
 verifier_timeout_seconds: 28800
 secrets:
@@ -100,7 +113,7 @@ inference_gateway:
     allowed_models: ["${optimizer_model:-gpt-5.3-codex}"]
     max_concurrency: 8
   evaluation:
-    allowed_models: [gpt-4o]
+    allowed_models: [fireworks_ai/deepseek-v4-flash]
     max_requests: 15000
     max_tokens: 100000000
     max_concurrency: 64