Merge pull request #46 from scaleapi/pr3-harness-bench

Author: varunursekar

.gitignore

@@ -15,6 +15,3 @@ wheels/
 secrets.env
 *.secrets.env
 !*.example
-
-# Local benchmark scoping notes (paper planning; not tracked)
-harness-engineering-bench/benchmark-scoping.md

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "browsecomp-plus-upstream"]
+	path = harness-engineering-bench/browsecomp-plus/upstream
+	url = https://github.com/texttron/BrowseComp-Plus.git

harness-engineering-bench/CONFIGURATION.md

@@ -0,0 +1,50 @@
+# Harness engineering benchmarks
+
+`harness-engineering-bench` contains end-to-end benchmarks for automatically
+improving the harness of an agent or the code of a component used to build
+agents. Each leaf directory pairs one editable target program with one immutable
+Harbor dataset and compiles them into an outer Harbor optimization task.
+
+The benchmark definitions intentionally keep three boundaries visible:
+
+- `target/` is the program the optimization agent may edit.
+- `partitions/` pins the cases and the development/validation/test split.
+- `build.yaml` is trusted configuration: model, evaluator, access policy,
+  budgets, and final scoring.
+
+In each benchmark, the complete development tasks and attachments are mounted
+read-only for the optimization agent. Development evaluations expose per-case
+results and complete Harbor trial records, including exact failures and
+target-agent logs. Validation remains aggregate-only, and
+test is reachable only by the trusted final verifier.
+
+The paper-era benchmark stack remains available on the `paper/v1` branch and
+the `paper-v1` tag. New benchmarks should use this Harbor-native layout.
+
+## Benchmarks
+
+Promoted benchmarks live at the top level. Task sets still under review live in
+`candidates/`; we work through the list in the paper's `benchmark-scoping.md` and
+promote a task set to the top level once it is ready.
+
+### Promoted
+
+| Benchmark | Editable target | Dataset | Split |
+| --- | --- | --- | --- |
+| [GAIA baseline](gaia/baseline/) | Tool-using Responses API agent | Harbor `gaia/gaia` | 20% / 40% / 40% |
+| [OfficeQA baseline](officeqa/baseline/) | Grounded document-QA agent | Treasury Bulletin corpus | 20% / 40% / 40% |
+| [SWE-Atlas-QnA baseline](swe-atlas-qna/baseline/) | Codebase investigation agent | Harbor `scale-ai/swe-atlas-qna` | 20% / 40% / 40% |
+| [tau3 baseline](tau3/baseline/) | MCP customer-service agent | Harbor `sierra-research/tau3-bench` | 20% / 40% / 40% |
+| [BrowseComp-Plus baseline](browsecomp-plus/baseline/) | Fixed-corpus deep-research agent | Pinned local Harbor tasks | 20% / 40% / 40% |
+
+**`swe-bench-pro/` is at the top level but is not promoted, and its numbers are
+not comparable to the five above.** It predates the normalization pass those five
+went through and still differs on most of it: case budgets are 1x the partition
+size rather than 4x, the held-out target has no `n_attempts: 3` / `mean`
+override so it is scored once, there is no pinned `baseline_reward` (and
+`score_baseline: true` adds a second full held-out pass), the agent clock runs at
+0.6x the declared case timeout, gateway request and token caps are 20-30x tighter
+than the sizing convention, there is no `agent_env` block, and telemetry goes to
+its own W&B project. Treat it as a work in progress: launching it will produce a
+number, but not a measurement of the same quantity. `CONFIGURATION.md` documents
+the conventions it is missing.

harness-engineering-bench/README.md

@@ -0,0 +1,6 @@
+# Generated from the obfuscated upstream dataset. These files contain decrypted
+# benchmark questions and answers and must not be committed.
+tasks/
+baseline/target/.venv/
+**/__pycache__/
+**/*.pyc

harness-engineering-bench/browsecomp-plus/.gitignore

@@ -0,0 +1,53 @@
+# BrowseComp-Plus
+
+This benchmark turns all 830 queries from
+[BrowseComp-Plus](https://github.com/texttron/BrowseComp-Plus) into local Harbor
+tasks. It evaluates a deep-research agent against the benchmark's fixed corpus
+and canonical BM25 index rather than the live web.
+
+## Pinned sources
+
+- Upstream repository submodule: `046949032b0328319cc9a02663a759ec601d9402`
+- Query dataset `Tevatron/browsecomp-plus`:
+  `144cff8e35b5eaef7e526346aa60774a9deb941f`
+- BM25 index `Tevatron/browsecomp-plus-indexes`:
+  `b3f37f70c33829eb09d04784a54277a31871fd63`
+
+The submodule is the Git pointer requested for the integration. The generator
+refuses to run if it is checked out at another commit. Hugging Face revisions
+are full immutable commit ids as well.
+
+## Generate Harbor tasks
+
+Initialize the submodule and run the builder from this directory:
+
+```bash
+git submodule update --init harness-engineering-bench/browsecomp-plus/upstream
+
+cd harness-engineering-bench/browsecomp-plus
+uv run --no-project --python 3.12 --with datasets==4.0.0 -- \
+  python scripts/build_tasks.py
+```
+
+The builder downloads and decrypts the pinned query dataset using the pinned
+upstream implementation, then writes one complete Harbor task per query under
+the ignored `tasks/` directory. It also regenerates the committed deterministic
+166/332/332 development, validation, and test split. Use `--force` to replace
+an existing generated tree or `--check` to verify it byte-for-byte.
+
+The first Harbor image build downloads the pinned BM25 index (about 2.2 GB).
+Every task has an identical environment, so subsequent tasks reuse that image.
+
+## Scoring and trust boundary
+
+Answers use BrowseComp-Plus's required Explanation / Exact Answer / Confidence
+format. The verifier follows the official upstream OpenAI evaluator and its
+default `gpt-4.1` judge. The upstream primary leaderboard instead runs the same
+judge prompt with Qwen3-32B; this Harbor integration deliberately uses the
+repository's supported API evaluator so tasks do not require a local GPU judge.
+
+The judge runs inside the task environment and therefore receives the real
+upstream credential. As with the SWE-Atlas-QnA rubric judge and tau3 task-owned
+LLM services, this disables uid isolation and assumes a non-adversarial
+optimizer. The editable baseline itself exposes no live-web or shell tool and
+uses only the fixed local index.

harness-engineering-bench/browsecomp-plus/README.md

@@ -0,0 +1,3 @@
+compiled/
+target/.venv/
+target/.pytest_cache/

harness-engineering-bench/browsecomp-plus/baseline/.gitignore

@@ -0,0 +1,20 @@
+# BrowseComp-Plus baseline
+
+This editable target is a Responses API deep-research agent with three tools:
+search the pinned BM25 index, open a document, and submit a formatted response.
+The optimization agent may change its prompts, control flow, tool use, or
+dependencies, but not the dataset, index, split, evaluated model, or verifier.
+
+Build the generated tasks first as described in the parent
+[`README.md`](../README.md), then compile from the repository root:
+
+```bash
+cd vero
+VERO_SKIP_SECRET_CHECK=1 uv run vero harbor build \
+  --config ../harness-engineering-bench/browsecomp-plus/baseline/build.yaml \
+  --output ../harness-engineering-bench/browsecomp-plus/baseline/compiled
+```
+
+For a real run, copy `secrets.env.example` to the ignored `secrets.env`, fill it
+in, and use `vero harbor run` in the same way as the other harness-engineering
+benchmarks.

harness-engineering-bench/browsecomp-plus/baseline/README.md

@@ -0,0 +1,148 @@
+name: vero/optimize-browsecomp-plus-baseline
+description: >-
+  Improve a deep-research agent on BrowseComp-Plus using its fixed corpus,
+  canonical BM25 index, and semantic answer evaluator.
+agent_repo: target
+task_source: ../tasks
+task_manifest: ../partitions/manifest.json
+agent_import_path: browsecomp_plus_agent.agent:BrowseCompPlusAgent
+harbor_requirement: harbor[modal]==0.20.0
+
+partition_files:
+  development: ../partitions/development.json
+  validation: ../partitions/validation.json
+  test: ../partitions/test.json
+
+# Four full passes over each agent-visible partition.
+agent_access:
+  - partition: development
+    disclosure: full
+    expose_case_resources: true
+    total_runs: 100
+    total_cases: 132
+  - partition: validation
+    disclosure: aggregate
+    expose_case_resources: false
+    min_aggregate_cases: 5
+    total_runs: 100
+    total_cases: 264
+
+selection_partition: validation
+targets:
+  - partition: test
+    reward_key: reward
+    baseline_reward: 0.4619  # re-pinned 0.424 / 0.470 / 0.492 (sd 0.0283); was 0.4495. See runs/BASELINES.md
+    failure_value: 0.0
+    max_attempts: 1
+    # The held-out eval is noisy: score the selected candidate 3x per case and
+    # average, so the final reward is comparable to the pinned baseline, which was
+    # itself pooled over 3 rounds. Without this the candidate carries ~sqrt(3) more
+    # standard error than the floor it is judged against.
+    # Per-target override - search/validation keep the global n_attempts (1).
+    n_attempts: 3
+    aggregate_attempts: mean
+
+evaluation_set_name: browsecomp-plus
+objective:
+  selector:
+    metric: score
+  direction: maximize
+reward_mode: submit
+baseline_floor: false
+score_baseline: false
+rescore_top_k: 3
+rescore_attempts: 1
+
+model: fireworks_ai/deepseek-v4-flash
+environment_name: ${inner_env:-modal}
+extra_harbor_args: ["--ek", "app_name=harness-engineering-bench", "--ek", "sandbox_idle_timeout_secs=3600"]
+harbor_python_version: "3.12"
+n_attempts: 1
+max_retries: 1
+infrastructure_max_attempts: 3
+infrastructure_retry_delay_seconds: 5
+aggregate_attempts: best
+feedback_transcripts: true
+feedback_max_bytes: 16000
+expose_attempt_detail: false
+# Unreachable: worst case is ceil(198/24) x 3600 = 32400s, every
+# finalize trial (66 held-out x n_attempts=3) hitting its own cap. Assumes
+# max_concurrency=24; recompute if that drops.
+timeout_seconds: 39600
+# Exactly the dataset's declared [agent] timeout_sec, so vero's derived
+# --agent-timeout-multiplier is 1.0 and the target agent gets precisely the
+# clock the benchmark intends. Harbor times agent setup, environment build and
+# verification on separate clocks with separate multipliers, so none of them
+# eat into this budget and no buffer is warranted.
+case_timeout_seconds: 3600
+task_agent_timeout_seconds: 3600
+max_concurrency: 24  # 8 -> 24; see officeqa for the measured headroom argument
+error_rate_threshold: 0.1
+# Unreachable: worst-case finalize (32400) + worst-case rescore_top_k=3
+# validation rescore (32400). A verifier timeout loses the score outright.
+verifier_timeout_seconds: 75600
+secrets:
+  - MODAL_TOKEN_ID
+  - MODAL_TOKEN_SECRET
+  - WANDB_API_KEY
+  - WANDB_BASE_URL
+
+wandb:
+  project: harness-engineering-bench  # one project for the whole suite
+  group: browsecomp-plus  # keeps the benchmark distinguishable in the shared project
+  name: ${wandb_run:-browsecomp-plus}  # per-launch label, e.g. --param wandb_run=browsecomp-plus__claude-sonnet-5
+  tags: [browsecomp-plus]
+  log_traces: true
+
+inference_gateway:
+  upstream_api_key_env: OPENAI_API_KEY
+  upstream_base_url_env: OPENAI_BASE_URL
+  producer:
+    allowed_models: ["${optimizer_model:-openai/gpt-5.4}"]
+    max_concurrency: 8
+  # See officeqa/baseline/build.yaml for the sizing rationale: the case budget is
+  # the spend control, so a token cap only needs to stop a runaway.
+  evaluation:
+    allowed_models: [fireworks_ai/deepseek-v4-flash]
+    max_requests: 200000
+    max_tokens: 2000000000  # 396 agent case-runs (132 dev + 264 validation)
+    max_concurrency: 64
+  # Reserved so a search-phase overspend can never starve held-out scoring.
+  finalization:
+    allowed_models: [fireworks_ai/deepseek-v4-flash]
+    max_requests: 200000
+    max_tokens: 2000000000  # 66 test cases x3 attempts + rescore headroom
+    max_concurrency: 64
+
+instruct_multifidelity: true
+instruct_exhaust_budget: true
+
+# The official upstream OpenAI evaluator runs inside the task container. It
+# uses the raw upstream credential while the editable target agent continues to
+# receive only the evaluation-scoped gateway credential.
+task_services_use_upstream: true
+harness_user: null
+# Optimizer-agent env (forwarded to the harbor claude-code agent as --ae KEY=VALUE).
+# Claude Code's Bash tool caps a single call at BASH_MAX_TIMEOUT_MS (default
+# 600000=10min), well under one inner eval, which pushed the officeqa optimizer
+# into --detach + background-poll + end-turn -- and a headless --print run is
+# never re-woken, so the search died there. Raise the cap so a whole eval fits in
+# one blocking call. The background-task vars are defence in depth only: they gate
+# *automatic* backgrounding and do NOT remove the Bash tool's run_in_background
+# parameter, which the model can still choose. The instruction forbids that.
+agent_env:
+  # Above this benchmark's widest single eval: a full validation pass is
+  # ceil(66/24) x 3600 = 10800s worst case.
+  BASH_MAX_TIMEOUT_MS: "14400000"
+  BASH_DEFAULT_TIMEOUT_MS: "14400000"  # same as max: an un-timed eval must still block
+  ENABLE_BACKGROUND_TASKS: "0"
+  FORCE_AUTO_BACKGROUND_TASKS: "0"
+  # Harnesses installed with `uv tool install` (mini-swe-agent, swe-agent)
+  # default to symlinking their entry point into /usr/local/bin, which the
+  # unprivileged optimizer user cannot write: "Failed to install executable
+  # ... Permission denied". npm/nvm-based harnesses (claude-code, opencode)
+  # are unaffected, so this only bites when the harness changes.
+  UV_TOOL_BIN_DIR: "/home/agent/.local/bin"
+
+task_environment:
+  BROWSECOMP_JUDGE_MODEL: gpt-4.1

harness-engineering-bench/browsecomp-plus/baseline/build.yaml

@@ -0,0 +1,10 @@
+# Optimizer and evaluated-agent inference. OPENAI_BASE_URL may point to an
+# OpenAI-compatible upstream; use https://api.openai.com/v1 for OpenAI.
+OPENAI_API_KEY=sk-your-upstream-inference-key
+OPENAI_BASE_URL=https://api.openai.com/v1
+
+MODAL_TOKEN_ID=your-modal-token-id
+MODAL_TOKEN_SECRET=your-modal-token-secret
+
+WANDB_API_KEY=your-wandb-api-key
+WANDB_BASE_URL=https://api.wandb.ai