Merge the refreshed swe-bench-pro scaffold base into the capability gate

The base branch now applies this same capability gate to the
swe-bench-pro agent as part of fixing five seed-agent defects, so that
file conflicted. Resolved by taking the base's version verbatim: the gate
is still applied there, just by the base commit rather than by this PR,
which drops swe-bench-pro from this PR's diff without changing the
merged result.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: shehabyasser-scale

harness-engineering-bench/swe-bench-pro/README.md

@@ -1,84 +1,103 @@
-# SWE-bench-Pro (scaffold)
+# SWE-bench-Pro
 
 This benchmark optimizes a code-editing agent on SWE-bench-Pro tasks. Each task
-checks a real project out at `/app/repo` and describes a bug fix or feature. The
+checks a real project out at `/app` and describes a bug fix or feature. The
 editable agent (`baseline/target/src/swebench_pro_agent/agent.py`) edits the
 repository in place; the **task-source verifier** applies the resulting repo state
 and runs the task's hidden test suite for the reward. The agent does not
 self-grade and writes no answer file: reward comes entirely from the suite.
 
-## Status: this is a SCAFFOLD, not a runnable benchmark
-
-Everything except the actual task data is in place: a robust Harbor-native agent,
-a `build.yaml`, partition scaffolding, a split script, and this README. It will
-**not run yet** because the real SWE-bench-Pro task dataset must come from a Harbor
-task source that is not committed here.
-
-### DONE (working skeleton)
-
-- `baseline/target/src/swebench_pro_agent/agent.py`: a tool-using code-editing
-  agent on the OpenAI Responses API (`run_shell`, `read_file`, `write_file`,
-  `apply_patch`, `run_tests`, `submit`), `MAX_TURNS = 50`. The
-  `self._client.responses.create(...)` call is wrapped in a retry-with-backoff
-  helper from the start (the GAIA baseline scored 0.0 because an unguarded call
-  crashed on a transient error; that retry was the optimizer's winning fix).
-- `baseline/build.yaml`: the full VeRO/Harbor optimization config (agent import
-  path, inference gateway, partitions wiring, timeouts, secrets, W&B).
-- `baseline/target/pyproject.toml`, smoke test, `secrets.env.example`, `.gitignore`.
-- `scripts/partition_swe_bench_pro.py`: a deterministic, sha256-keyed stratified
-  split with a `--check` mode, modeled on `gaia/scripts/partition_gaia.py`.
-
-### STUBBED (must be completed before it runs)
-
-Two things must be finished:
-
-1. **Pin the real `task_source`.** `baseline/build.yaml` carries a clearly-marked
-   placeholder:
-
-   ```
-   task_source: scale-ai/swe-bench-pro@sha256:REPLACE_WITH_REAL_DIGEST
-   ```
-
-   SWE-bench-Pro is a Scale benchmark. Find the real pinned task package in the
-   Harbor task registry (ask the DEX-harness / Harbor team if it is not obvious).
-   The baseline in this directory provides only the **agent + config**; the task
-   package provides the **repos, tests, gold patches, and grading**. Replace the
-   placeholder digest here and the `DATASET_DIGEST` constant in
-   `scripts/partition_swe_bench_pro.py`.
-
-2. **Regenerate the partitions from the real task list.** The committed
-   `partitions/{development,validation,test}.json` are empty placeholders and
-   `partitions/manifest.json` is a stub (`"_stub": true`). Once the task source is
-   pinned, set `TOTAL_TASKS`/`TARGET_COUNTS` and confirm `_read_tasks` against the
-   real `task.toml` layout in `scripts/partition_swe_bench_pro.py`, then generate
-   and verify the split:
-
-   ```bash
-   uv run --python 3.12 \
-     harness-engineering-bench/swe-bench-pro/scripts/partition_swe_bench_pro.py \
-     --tasks-dir /path/to/exported/swe-bench-pro \
-     --fetch-registry
-
-   uv run --python 3.12 \
-     harness-engineering-bench/swe-bench-pro/scripts/partition_swe_bench_pro.py \
-     --tasks-dir /path/to/exported/swe-bench-pro \
-     --check
-   ```
-
-   Also update the placeholder `total_cases` under `agent_access` in
-   `baseline/build.yaml` to real disclosure counts, and run `uv lock` in
-   `baseline/target/` so the compiled task pins an exact resolution.
+## Task source
+
+SWE-bench-Pro ships as the **`swebenchpro` dataset in the default Harbor
+registry**, version `1.0`, 731 instances across 11 upstream projects. It is a
+*registry* dataset, not an `<org>/<name>@sha256:<digest>` package like
+swe-atlas-qna or tau3, so the pin is a bare `name@version`:
+
+```
+task_source: swebenchpro@1.0
+```
+
+Its tasks resolve to git-backed task ids under
+`laude-institute/harbor-datasets` at commit
+`c8e8f3fac7097accaacf261d74c3d6f441de45b1`, path
+`datasets/swebenchpro/instance_<owner>__<repo>-<sha>`. Each task ships an
+`environment/Dockerfile` that `FROM`s a public prebuilt DockerHub image
+(`jefzda/sweap-images:<instance>`, 0.5-4.8 GB), resets the repo to the base
+commit, and clears `ENTRYPOINT` (so harbor's default keepalive works and no
+`--ek keepalive` override is needed, unlike swe-atlas-qna).
+
+Grading is entirely offline: `tests/test.sh` checks out the gold test files,
+runs the official `run_script.sh`, parses results with the official `parser.py`,
+and writes `1` to `/logs/verifier/reward.txt` only when every test in
+`fail_to_pass` and `pass_to_pass` passed. **No judge model, so no verifier
+credentials are needed** (`--ve` is unnecessary for this benchmark).
+
+Two consequences of being a registry rather than a package dataset:
+
+- `agent_access[development].expose_case_resources` must be `false`. VeRO
+  materializes case resources through `PackageDatasetClient`, which only parses
+  `<org>/<name>` refs and rejects a bare `swebenchpro@1.0`.
+- The recorded per-task `ref` in `partitions/manifest.json` is
+  `<git_commit>:<path>` rather than a `sha256:` content hash.
 
 ## Split design
 
-The committed split is intended to be deterministic and stratified by source
-repository (so each represented codebase appears across all three partitions):
+The committed split is deterministic and stratified by source repository (so each
+represented codebase appears across all three partitions):
+
+- development: 146 (20%, full result disclosure to the optimizer)
+- validation: 292 (40%, aggregate-only; used to select candidates)
+- test: 293 (40%, held out until Harbor grades the completed outer task)
+
+731 does not divide evenly: the exact ratios are 146.2/292.4/292.4, so the one
+leftover case is assigned by largest remainder, and the .4/.4 tie between
+validation and test breaks towards test. That matches how the sibling benchmarks
+resolved the same tie (officeqa 246 -> 49/98/99, swe-atlas-qna 124 -> 25/49/50).
+
+Regenerate and verify the split with:
+
+```bash
+harbor download dataset swebenchpro@1.0 --output-dir /tmp/swebenchpro   # or a
+# sparse checkout of laude-institute/harbor-datasets at the pinned commit
+
+uv run --python 3.12 \
+  harness-engineering-bench/swe-bench-pro/scripts/partition_swe_bench_pro.py \
+  --tasks-dir /tmp/swebenchpro --fetch-registry
 
-- development: 20% (full result disclosure to the optimizer)
-- validation: 40% (aggregate-only; used to select candidates)
-- test: 40% (held out until Harbor grades the completed outer task)
+uv run --python 3.12 \
+  harness-engineering-bench/swe-bench-pro/scripts/partition_swe_bench_pro.py \
+  --tasks-dir /tmp/swebenchpro --check
+```
 
-Exact counts are TODO until the task source is pinned.
+## Target model: which models the seed agent can actually run on
+
+The seed agent drives the **OpenAI Responses API** and chains turns with
+`previous_response_id`. Measured against the shared LiteLLM proxy
+(`heb-litellm-proxy`), that shape constrains the usable target models:
+
+| model | turn 1 | chained turn (`previous_response_id`) |
+|---|---|---|
+| gpt-4o | OK (with the reasoning gate) | OK |
+| gpt-5.4-mini-2026-03-17 | OK | OK |
+| gpt-5.6-sol | OK | OK |
+| deepseek-v4-flash | OK | OK |
+| gpt-oss-120b | OK | OK |
+| **qwen-3.6-27b** | OK | **fails, deterministically** |
+| gpt-4.1 | 400 on `reasoning` | n/a |
+| gpt-5.3-codex | 404 on the proxy's Responses route | n/a |
+
+`qwen-3.6-27b` answers the first turn but every chained call returns
+`litellm.BadRequestError: Fireworks_aiException ... invalid_request_error`
+(3/3 reproductions, plus an in-harbor trial). The same conversation works on
+`/chat/completions`, so the fault is LiteLLM's Responses-to-ChatCompletions
+bridge replaying a shape Fireworks rejects, not the model. Switching the agent
+to a stateless full-history `input` list also works on qwen, but that changes
+the seed harness being measured, so it has not been done here.
+
+`reasoning: {"effort": ...}` is gated on `_is_reasoning_model()` (the same
+capability predicate the other five benchmark agents use): sending it to gpt-4o
+or gpt-4.1 is a hard 400 on the very first turn.
 
 ## Build the outer Harbor task
 
@@ -94,8 +113,35 @@ VERO_SKIP_SECRET_CHECK=1 uv run vero harbor build \
 Omit `VERO_SKIP_SECRET_CHECK=1` for a real build so VeRO verifies that the OpenAI
 and Modal credentials declared in `build.yaml` are present. Set `OPENAI_BASE_URL`
 to your OpenAI-compatible endpoint. The `compiled/` directory is generated and
-intentionally ignored. (This build will fail to resolve tasks until the
-placeholder `task_source` above is replaced with the real digest.)
+intentionally ignored.
+
+## Measure the held-out baseline
+
+A K=3 baseline is three independent single-attempt rounds over the 293-case
+`test` partition, scored by harbor's own
+`stats.evals.*.metrics[0].mean` (an errored trial counts as a zero). Each round
+is a plain `harbor run` against the registry dataset, with
+`--agent-timeout-multiplier 0.6` (`case_timeout_seconds` 1800 over the task's
+declared `[agent] timeout_sec` 3000):
+
+```bash
+harbor run \
+  -a swebench_pro_agent.agent:SweBenchProAgent \
+  -m <target-model> -e modal \
+  -d swebenchpro@1.0 \
+  $(python3 -c "import json;print(' '.join('-i '+t for t in json.load(open('../../partitions/test.json'))))") \
+  --n-attempts 1 --max-retries 3 -n 5 --yes \
+  --ek app_name=harness-engineering-bench --ek sandbox_idle_timeout_secs=3600 \
+  --agent-timeout-multiplier 0.6 \
+  --env-file baseline/secrets.env \
+  -o ../../../runs/baseline-swe-bench-pro/roundN
+```
+
+Run it three times into `round1/`, `round2/`, `round3/`; the pinned
+`baseline_reward` is the mean of the three round means and the `±` is the
+population stdev across them. Concurrency matters: `-n 12` against the shared
+Azure gpt-4o deployment produced 45 `RateLimitError` retries in 15 minutes,
+while `-n 5` produced none.
 
 ## Run the optimization
 

harness-engineering-bench/swe-bench-pro/baseline/build.yaml

@@ -4,11 +4,12 @@ description: >-
   interface. The target must edit a checked-out repository so the task's hidden
   test suite passes; the task-source verifier runs the suite for the reward.
 agent_repo: target
-# TODO: pin the real SWE-bench-Pro Harbor task source (see README). This is a
-# clearly-marked placeholder, NOT a real digest, and will fail to resolve until
-# replaced with the actual `scale-ai/swe-bench-pro@sha256:<digest>` from the
-# Harbor task registry.
-task_source: scale-ai/swe-bench-pro@sha256:REPLACE_WITH_REAL_DIGEST  # TODO: pin the real SWE-bench-Pro Harbor task source (see README)
+# SWE-bench-Pro ships as the `swebenchpro` dataset in the DEFAULT Harbor registry
+# (731 instances), not as an `<org>/<name>@sha256:<digest>` package like
+# swe-atlas-qna or tau3. Its tasks resolve to git-backed task ids under
+# laude-institute/harbor-datasets at commit c8e8f3fac7097accaacf261d74c3d6f441de45b1,
+# so the version pin is the registry version rather than a content digest.
+task_source: swebenchpro@1.0
 task_manifest: ../partitions/manifest.json
 agent_import_path: swebench_pro_agent.agent:SweBenchProAgent
 harbor_requirement: harbor[modal]==0.20.0
@@ -18,18 +19,21 @@ partition_files:
   validation: ../partitions/validation.json
   test: ../partitions/test.json
 
-# TODO: set real disclosure counts once the partitions are regenerated from the
-# pinned task list. total_cases below are placeholders scaled to the stub sizes.
 agent_access:
   - partition: development
     disclosure: full
-    expose_case_resources: true
-    total_cases: 0
+    # false, unlike the sibling benchmarks: VeRO materializes case resources via
+    # PackageDatasetClient, which only accepts `<org>/<name>` package refs. A
+    # registry dataset such as swebenchpro@1.0 cannot be exposed that way.
+    expose_case_resources: false
+    total_runs: 100
+    total_cases: 146
   - partition: validation
     disclosure: aggregate
     expose_case_resources: false
     min_aggregate_cases: 5
-    total_cases: 0
+    total_runs: 100
+    total_cases: 292
 
 selection_partition: validation
 targets:
@@ -65,9 +69,12 @@ expose_attempt_detail: false
 # Timeouts are bumped well above the GAIA baseline: SWE-bench-Pro tasks build a
 # real repository and run its (often slow) test suite, so both the per-case and
 # the verifier budgets need substantially more headroom than a short-answer task.
+# task_agent_timeout_seconds mirrors the task package's own `[agent] timeout_sec
+# = 3000`; case_timeout_seconds is the tighter budget VeRO enforces on top, so
+# the baseline runner's --agent-timeout-multiplier is 1800/3000 = 0.6.
 timeout_seconds: 28800
 case_timeout_seconds: 1800
-task_agent_timeout_seconds: 3600
+task_agent_timeout_seconds: 3000
 max_concurrency: 8
 error_rate_threshold: 0.1
 verifier_timeout_seconds: 28800

harness-engineering-bench/swe-bench-pro/baseline/target/pyproject.toml

@@ -2,9 +2,10 @@
 name = "swebench-pro-agent"
 version = "0.1.0"
 description = "Editable Harbor-native SWE-bench-Pro baseline for VeRO"
-# Cap below 3.14: an uncapped ">=3.11" lets uv resolve to 3.14, whose litellm/pyo3
-# build fails. Keep this bounded until the toolchain supports 3.14.
-requires-python = ">=3.11,<3.14"
+# Floor at 3.12 (harbor 0.20.0 requires it; ">=3.11" made the package's own
+# `uv run pytest` unresolvable). Cap below 3.14: an uncapped range lets uv
+# resolve to 3.14, whose litellm/pyo3 build fails.
+requires-python = ">=3.12,<3.14"
 dependencies = [
     "harbor==0.20.0",
     "openai==2.46.0",