feat: scaffold SWE-bench-Pro baseline for the vero optimizer

WIP scaffold: a working agent + build/config skeleton for a SWE-bench-Pro
optimization target, mirroring the gaia baseline layout. Two things are
stubbed and must be completed before it runs: (1) the task_source digest in
baseline/build.yaml is a clearly-marked placeholder that must be pinned to the
real SWE-bench-Pro Harbor task package, and (2) the partitions/ files are empty
placeholders that must be regenerated from the real task list via
scripts/partition_swe_bench_pro.py. The agent wraps responses.create in a
retry-with-backoff helper from the start (the unguarded call is what scored the
gaia baseline 0.0). See harness-engineering-bench/swe-bench-pro/README.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: shehabyasser-scale

harness-engineering-bench/swe-bench-pro/README.md

@@ -0,0 +1,135 @@
+# SWE-bench-Pro (scaffold)
+
+This benchmark optimizes a code-editing agent on SWE-bench-Pro tasks. Each task
+checks a real project out at `/app/repo` and describes a bug fix or feature. The
+editable agent (`baseline/target/src/swebench_pro_agent/agent.py`) edits the
+repository in place; the **task-source verifier** applies the resulting repo state
+and runs the task's hidden test suite for the reward. The agent does not
+self-grade and writes no answer file: reward comes entirely from the suite.
+
+## Status: this is a SCAFFOLD, not a runnable benchmark
+
+Everything except the actual task data is in place: a robust Harbor-native agent,
+a `build.yaml`, partition scaffolding, a split script, and this README. It will
+**not run yet** because the real SWE-bench-Pro task dataset must come from a Harbor
+task source that is not committed here.
+
+### DONE (working skeleton)
+
+- `baseline/target/src/swebench_pro_agent/agent.py`: a tool-using code-editing
+  agent on the OpenAI Responses API (`run_shell`, `read_file`, `write_file`,
+  `apply_patch`, `run_tests`, `submit`), `MAX_TURNS = 50`. The
+  `self._client.responses.create(...)` call is wrapped in a retry-with-backoff
+  helper from the start (the GAIA baseline scored 0.0 because an unguarded call
+  crashed on a transient error; that retry was the optimizer's winning fix).
+- `baseline/build.yaml`: the full VeRO/Harbor optimization config (agent import
+  path, inference gateway, partitions wiring, timeouts, secrets, W&B).
+- `baseline/target/pyproject.toml`, smoke test, `secrets.env.example`, `.gitignore`.
+- `scripts/partition_swe_bench_pro.py`: a deterministic, sha256-keyed stratified
+  split with a `--check` mode, modeled on `gaia/scripts/partition_gaia.py`.
+
+### STUBBED (must be completed before it runs)
+
+Two things must be finished:
+
+1. **Pin the real `task_source`.** `baseline/build.yaml` carries a clearly-marked
+   placeholder:
+
+   ```
+   task_source: scale-ai/swe-bench-pro@sha256:REPLACE_WITH_REAL_DIGEST
+   ```
+
+   SWE-bench-Pro is a Scale benchmark. Find the real pinned task package in the
+   Harbor task registry (ask the DEX-harness / Harbor team if it is not obvious).
+   The baseline in this directory provides only the **agent + config**; the task
+   package provides the **repos, tests, gold patches, and grading**. Replace the
+   placeholder digest here and the `DATASET_DIGEST` constant in
+   `scripts/partition_swe_bench_pro.py`.
+
+2. **Regenerate the partitions from the real task list.** The committed
+   `partitions/{development,validation,test}.json` are empty placeholders and
+   `partitions/manifest.json` is a stub (`"_stub": true`). Once the task source is
+   pinned, set `TOTAL_TASKS`/`TARGET_COUNTS` and confirm `_read_tasks` against the
+   real `task.toml` layout in `scripts/partition_swe_bench_pro.py`, then generate
+   and verify the split:
+
+   ```bash
+   uv run --python 3.12 \
+     harness-engineering-bench/swe-bench-pro/scripts/partition_swe_bench_pro.py \
+     --tasks-dir /path/to/exported/swe-bench-pro \
+     --fetch-registry
+
+   uv run --python 3.12 \
+     harness-engineering-bench/swe-bench-pro/scripts/partition_swe_bench_pro.py \
+     --tasks-dir /path/to/exported/swe-bench-pro \
+     --check
+   ```
+
+   Also update the placeholder `total_cases` under `agent_access` in
+   `baseline/build.yaml` to real disclosure counts, and run `uv lock` in
+   `baseline/target/` so the compiled task pins an exact resolution.
+
+## Split design
+
+The committed split is intended to be deterministic and stratified by source
+repository (so each represented codebase appears across all three partitions):
+
+- development: 20% (full result disclosure to the optimizer)
+- validation: 40% (aggregate-only; used to select candidates)
+- test: 40% (held out until Harbor grades the completed outer task)
+
+Exact counts are TODO until the task source is pinned.
+
+## Build the outer Harbor task
+
+From the repository root:
+
+```bash
+cd vero
+VERO_SKIP_SECRET_CHECK=1 uv run vero harbor build \
+  --config ../harness-engineering-bench/swe-bench-pro/baseline/build.yaml \
+  --output ../harness-engineering-bench/swe-bench-pro/baseline/compiled
+```
+
+Omit `VERO_SKIP_SECRET_CHECK=1` for a real build so VeRO verifies that the OpenAI
+and Modal credentials declared in `build.yaml` are present. Set `OPENAI_BASE_URL`
+to your OpenAI-compatible endpoint. The `compiled/` directory is generated and
+intentionally ignored. (This build will fail to resolve tasks until the
+placeholder `task_source` above is replaced with the real digest.)
+
+## Run the optimization
+
+`vero harbor run` compiles the `build.yaml` to a temporary task, wires credentials
+(relocating the upstream inference key to the gateway and handing the optimizer a
+scoped producer token), and invokes Harbor. Copy the secrets template first:
+
+```bash
+cp harness-engineering-bench/swe-bench-pro/baseline/secrets.env.example \
+   harness-engineering-bench/swe-bench-pro/baseline/secrets.env   # then edit it
+
+cd vero
+uv run vero harbor run \
+  --config ../harness-engineering-bench/swe-bench-pro/baseline/build.yaml \
+  --env-file ../harness-engineering-bench/swe-bench-pro/baseline/secrets.env \
+  --environment modal \
+  --agent codex \
+  --model gpt-5.3-codex \
+  --yes \
+  -o ../runs/swe-bench-pro-full/jobs
+```
+
+`--env-file` values override the ambient shell and are passed only through the
+subprocess environment, never on the command line. Anything after the known
+options (here `--yes -o ...`) is forwarded verbatim to `harbor run`.
+
+The producer scope's allow-list in `build.yaml` is `gpt-5.3-codex`, so pass
+`--model gpt-5.3-codex` (or an aligned model) to keep the optimizer's model and
+the gateway allow-list in lockstep and avoid a 403 mismatch. The evaluation scope
+is pinned to `gpt-4o` (matching `model: openai/gpt-4o`).
+
+**Modal endpoint requirement:** `--environment modal` runs eval sandboxes on
+Modal, which must be able to reach the model endpoint in `OPENAI_BASE_URL`. Use a
+Modal-reachable endpoint (for example the Azure
+`https://<resource>.openai.azure.com/openai/v1` endpoint), **not** a VPN-internal
+host that only resolves on the Scale network. Use `--environment docker` to run
+the optimizer locally against Modal eval backends instead.

harness-engineering-bench/swe-bench-pro/baseline/.gitignore

@@ -0,0 +1,4 @@
+compiled/
+target/.venv/
+target/.pytest_cache/
+secrets.env

harness-engineering-bench/swe-bench-pro/baseline/build.yaml

@@ -0,0 +1,101 @@
+name: vero/optimize-swe-bench-pro-baseline
+description: >-
+  Improve a code-editing SWE-bench-Pro agent while preserving the Harbor agent
+  interface. The target must edit a checked-out repository so the task's hidden
+  test suite passes; the task-source verifier runs the suite for the reward.
+agent_repo: target
+# TODO: pin the real SWE-bench-Pro Harbor task source (see README). This is a
+# clearly-marked placeholder, NOT a real digest, and will fail to resolve until
+# replaced with the actual `scale-ai/swe-bench-pro@sha256:<digest>` from the
+# Harbor task registry.
+task_source: scale-ai/swe-bench-pro@sha256:REPLACE_WITH_REAL_DIGEST  # TODO: pin the real SWE-bench-Pro Harbor task source (see README)
+task_manifest: ../partitions/manifest.json
+agent_import_path: swebench_pro_agent.agent:SweBenchProAgent
+harbor_requirement: harbor[modal]==0.20.0
+
+partition_files:
+  development: ../partitions/development.json
+  validation: ../partitions/validation.json
+  test: ../partitions/test.json
+
+# TODO: set real disclosure counts once the partitions are regenerated from the
+# pinned task list. total_cases below are placeholders scaled to the stub sizes.
+agent_access:
+  - partition: development
+    disclosure: full
+    expose_case_resources: true
+    total_cases: 0
+  - partition: validation
+    disclosure: aggregate
+    expose_case_resources: false
+    min_aggregate_cases: 5
+    total_cases: 0
+
+selection_partition: validation
+targets:
+  - partition: test
+    reward_key: reward
+    failure_value: 0.0
+    max_attempts: 1
+
+evaluation_set_name: swe-bench-pro
+objective:
+  selector:
+    metric: score
+  direction: maximize
+reward_mode: submit  # agent picks; falls back to auto_best, then current version
+baseline_floor: false  # gates on validation while reward is on test; opt-in only
+score_baseline: true
+rescore_top_k: 3
+rescore_attempts: 1
+
+model: openai/gpt-4o
+environment_name: ${inner_env:-modal}
+# inner eval sandboxes share a dedicated Modal app instead of the __harbor__ default
+extra_harbor_args: ["--ek", "app_name=harness-engineering-bench", "--ek", "sandbox_idle_timeout_secs=3600"]
+harbor_python_version: "3.12"
+n_attempts: 1
+max_retries: 1
+infrastructure_max_attempts: 3
+infrastructure_retry_delay_seconds: 5
+aggregate_attempts: best
+feedback_transcripts: true
+feedback_max_bytes: 16000
+expose_attempt_detail: false
+# Timeouts are bumped well above the GAIA baseline: SWE-bench-Pro tasks build a
+# real repository and run its (often slow) test suite, so both the per-case and
+# the verifier budgets need substantially more headroom than a short-answer task.
+timeout_seconds: 28800
+case_timeout_seconds: 1800
+task_agent_timeout_seconds: 3600
+max_concurrency: 8
+error_rate_threshold: 0.1
+verifier_timeout_seconds: 28800
+secrets:
+  - MODAL_TOKEN_ID
+  - MODAL_TOKEN_SECRET
+  - WANDB_API_KEY
+  - WANDB_BASE_URL  # self-hosted W&B (scaleai.wandb.io)
+
+# candidate harness runs as an unprivileged uid, unable to read held-out state.
+harness_user: harness
+
+wandb:
+  project: vero-swe-bench-pro
+  group: swe-bench-pro
+  log_traces: true
+
+inference_gateway:
+  upstream_api_key_env: OPENAI_API_KEY
+  upstream_base_url_env: OPENAI_BASE_URL
+  producer:
+    # Both prefixed and bare forms: the gateway model match is prefix-sensitive.
+    allowed_models: ["gpt-5.3-codex", "openai/gpt-5.3-codex"]
+    max_concurrency: 8
+  evaluation:
+    allowed_models: ["gpt-4o", "openai/gpt-4o"]
+    max_requests: 15000
+    max_tokens: 100000000
+    max_concurrency: 64
+instruct_multifidelity: true
+instruct_exhaust_budget: true

harness-engineering-bench/swe-bench-pro/baseline/secrets.env.example

@@ -0,0 +1,23 @@
+# Run secrets for `vero harbor run --env-file`.
+# Copy to `secrets.env` (gitignored) and fill in. File values override the
+# ambient shell and never appear on the command line.
+
+# Upstream inference credentials. VeRO reads these as the real provider key/URL
+# (per the compiled gateway's *_source fields), relocates them to the
+# VERO_INFERENCE_UPSTREAM_* target vars, and hands the optimizer a scoped
+# producer token instead — so the raw key never reaches the coding agent.
+#
+# For `--environment modal`, OPENAI_BASE_URL must be reachable from Modal's
+# network (for example the Azure `https://<resource>.openai.azure.com/openai/v1`
+# endpoint), NOT a VPN-internal host. See the README.
+OPENAI_API_KEY=sk-your-upstream-inference-key
+OPENAI_BASE_URL=https://your-openai-compatible-endpoint/v1
+
+# Modal credentials for the eval sidecar backends (declared required by the
+# compiled task.toml). Copy from your ~/.modal.toml profile.
+MODAL_TOKEN_ID=ak-...
+MODAL_TOKEN_SECRET=as-...
+
+# Self-hosted Weights & Biases (scaleai.wandb.io) for trace/metric logging.
+WANDB_API_KEY=your-wandb-key
+WANDB_BASE_URL=https://scaleai.wandb.io

harness-engineering-bench/swe-bench-pro/baseline/target/pyproject.toml

@@ -0,0 +1,30 @@
+[project]
+name = "swebench-pro-agent"
+version = "0.1.0"
+description = "Editable Harbor-native SWE-bench-Pro baseline for VeRO"
+# Cap below 3.14: an uncapped ">=3.11" lets uv resolve to 3.14, whose litellm/pyo3
+# build fails. Keep this bounded until the toolchain supports 3.14.
+requires-python = ">=3.11,<3.14"
+dependencies = [
+    "harbor==0.20.0",
+    "openai==2.46.0",
+]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/swebench_pro_agent"]
+
+[dependency-groups]
+dev = [
+    "pytest>=9.0.2",
+    "pytest-asyncio>=1.3.0",
+]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+# NOTE: no uv.lock is committed yet. Run `uv lock` in this directory when
+# finalizing the baseline so the compiled task pins an exact resolution.

harness-engineering-bench/swe-bench-pro/baseline/target/src/swebench_pro_agent/__init__.py

@@ -0,0 +1,5 @@
+"""Harbor-native SWE-bench-Pro target agent."""
+
+from swebench_pro_agent.agent import SweBenchProAgent
+
+__all__ = ["SweBenchProAgent"]