Add harness-engineering-bench (benchmarks at top level)

Bring the benchmark harness with each benchmark as a top-level sibling of
gaia/ (ale-bench, officeqa, swe-atlas-qna, tau3), flattening the former
candidates/ grouping.

Author: varunursekar

harness-engineering-bench/README.md

@@ -0,0 +1,43 @@
+# Harness engineering benchmarks
+
+`harness-engineering-bench` contains end-to-end benchmarks for automatically
+improving the harness of an agent or the code of a component used to build
+agents. Each leaf directory pairs one editable target program with one immutable
+Harbor dataset and compiles them into an outer Harbor optimization task.
+
+The benchmark definitions intentionally keep three boundaries visible:
+
+- `target/` is the program the optimization agent may edit.
+- `partitions/` pins the cases and the development/validation/test split.
+- `build.yaml` is trusted configuration: model, evaluator, access policy,
+  budgets, and final scoring.
+
+In each benchmark, the complete development tasks and attachments are mounted
+read-only for the optimization agent. Development evaluations expose per-case
+results and complete Harbor trial records, including exact failures and
+target-agent logs. Validation remains aggregate-only, and
+test is reachable only by the trusted final verifier.
+
+The paper-era benchmark stack remains available on the `paper/v1` branch and
+the `paper-v1` tag. New benchmarks should use this Harbor-native layout.
+
+## Benchmarks
+
+Promoted benchmarks live at the top level. Task sets still under review live in
+`candidates/`; we work through the list in the paper's `benchmark-scoping.md` and
+promote a task set to the top level once it is ready.
+
+### Promoted
+
+| Benchmark | Editable target | Dataset | Split |
+| --- | --- | --- | --- |
+| [GAIA baseline](gaia/baseline/) | Tool-using Responses API agent | Harbor `gaia/gaia` | 20% / 40% / 40% |
+
+### Candidates
+
+| Benchmark | Editable target | Dataset | Split |
+| --- | --- | --- | --- |
+| [SWE-Atlas-QnA baseline](candidates/swe-atlas-qna/baseline/) | Codebase investigation agent | Harbor `scale-ai/swe-atlas-qna` | 20% / 40% / 40% |
+| [tau3 baseline](candidates/tau3/baseline/) | MCP customer-service agent | Harbor `sierra-research/tau3-bench` | 20% / 40% / 40% |
+| [OfficeQA baseline](candidates/officeqa/baseline/) | Grounded document-QA agent | Treasury Bulletin corpus | 20% / 40% / 40% |
+| [ALE-Bench (ahc011)](candidates/ale-bench/) | C++ solver program (component, not an agent) | ALE-Bench AHC problem | public / private seeds |

harness-engineering-bench/ale-bench/.gitignore

@@ -0,0 +1,2 @@
+environment/vero/
+runs/

harness-engineering-bench/ale-bench/environment/Dockerfile

@@ -0,0 +1,19 @@
+# Main service: the optimizer workbench a coding agent works in.
+# Holds the editable target (solution.cpp) plus the VeRO CLI so the agent can
+# call `vero harbor eval` / `vero harbor status` against the trusted sidecar.
+FROM ghcr.io/astral-sh/uv:python3.12-bookworm
+
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends git ca-certificates \
+    && rm -rf /var/lib/apt/lists/*
+
+# VeRO is not on public PyPI; vendor it into the build context first
+# (`cp -r <repo>/vero environment/vero`) then install it.
+COPY vero /opt/vero
+RUN uv pip install --system "/opt/vero[harbor]"
+
+COPY agent-seed /opt/agent-seed
+COPY main/seed.sh /opt/seed.sh
+RUN chmod +x /opt/seed.sh && useradd -m -u 1001 agent
+
+WORKDIR /work/agent

harness-engineering-bench/ale-bench/environment/agent-baseline/solution.cpp

@@ -0,0 +1,8 @@
+#include <bits/stdc++.h>
+int main(){
+    std::ios::sync_with_stdio(false);
+    std::string line;
+    while (std::getline(std::cin, line)) {}
+    std::cout << "\n";
+    return 0;
+}

harness-engineering-bench/ale-bench/environment/agent-seed/solution.cpp

@@ -0,0 +1,8 @@
+#include <bits/stdc++.h>
+int main(){
+    std::ios::sync_with_stdio(false);
+    std::string line;
+    while (std::getline(std::cin, line)) {}
+    std::cout << "\n";
+    return 0;
+}

harness-engineering-bench/ale-bench/environment/docker-compose.yaml

@@ -0,0 +1,60 @@
+# Harbor merges this after its generated main-service configuration.
+services:
+  main:
+    command: ["/opt/seed.sh"]
+    environment:
+      VERO_EVAL_URL: "http://eval-sidecar:8000"
+      OPENAI_API_KEY: "${OPENAI_API_KEY:?OPENAI_API_KEY must be set}"
+      OPENAI_BASE_URL: "${OPENAI_BASE_URL:?OPENAI_BASE_URL must be set}"
+    volumes:
+      - agent_repo:/work/agent
+      - token_state:/state/token:ro
+    depends_on:
+      eval-sidecar:
+        condition: service_healthy
+
+  eval-sidecar:
+    build:
+      context: .
+      dockerfile: sidecar/Dockerfile
+    # Runs its own Docker daemon (DinD) so ALE-Bench's sibling judge/tool
+    # containers share this container's filesystem — required for their bind
+    # mounts to resolve. Needs privileged + host cgroup ns (nested cgroup v2
+    # otherwise fails: "cannot enter cgroupv2 ... it is in threaded mode").
+    privileged: true
+    cgroup: host
+    command:
+      - "vero"
+      - "harbor"
+      - "serve"
+      - "--factory"
+      - "ale_factory:build"
+      - "--config"
+      - "/opt/serve.json"
+      - "--admin-token"
+      - "/state/token/admin.token"
+    environment:
+      # ALE-Bench talks to the in-container dockerd started by the entrypoint.
+      DOCKER_HOST: "unix:///var/run/docker.sock"
+      ALE_BENCH_DATA: "/state/ale-data"
+    volumes:
+      - agent_repo:/work/agent:ro
+      - admin_state:/state/admin
+      - token_state:/state/token
+      - ale_data:/state/ale-data
+      # DinD storage (avoids overlay-on-overlay issues for the inner daemon).
+      - docker_storage:/var/lib/docker
+    healthcheck:
+      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
+      interval: 5s
+      timeout: 10s
+      # The entrypoint starts dockerd + pulls the ~4GB judge image before serving.
+      retries: 40
+      start_period: 180s
+
+volumes:
+  agent_repo:
+  admin_state:
+  token_state:
+  ale_data:
+  docker_storage:

harness-engineering-bench/ale-bench/environment/main/seed.sh

@@ -0,0 +1,12 @@
+#!/bin/sh
+set -eu
+if [ ! -d /work/agent/.git ]; then
+  cp -a /opt/agent-seed/. /work/agent/
+  cd /work/agent
+  git init -q
+  git add -A
+  git -c user.email=seed@vero.test -c user.name=seed commit -qm "baseline"
+fi
+find /work/agent -exec chown agent:agent {} +
+git config --system --add safe.directory /work/agent
+exec sleep infinity

harness-engineering-bench/ale-bench/environment/sidecar/Dockerfile

@@ -0,0 +1,42 @@
+# Trusted evaluation sidecar. It owns scoring, budget, disclosure, and final
+# selection. The inner loop is a plain CommandBackend (harness/score.py), which
+# calls ALE-Bench's deterministic judge. ALE-Bench spawns the C++ judge + its
+# Rust-tool builder as sibling Docker containers with bind mounts, so this
+# container runs its OWN Docker daemon (DinD) — the sibling containers must share
+# this container's filesystem for their mounts to resolve. Requires privileged.
+FROM ghcr.io/astral-sh/uv:python3.12-bookworm
+
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends \
+        git ca-certificates curl \
+        docker.io iptables \
+        libcairo2 libffi8 \
+    && rm -rf /var/lib/apt/lists/*
+
+# VeRO (not on public PyPI): vendor into the build context, then install.
+COPY vero /opt/vero
+RUN uv pip install --system "/opt/vero[harbor]"
+
+# ALE-Bench toolkit (the judge orchestration + scoring).
+RUN uv pip install --system "git+https://github.com/SakanaAI/ALE-Bench.git"
+
+COPY agent-baseline /opt/agent-baseline
+COPY sidecar/ale_factory.py /opt/ale_factory.py
+COPY sidecar/harness /opt/harness
+COPY sidecar/serve.json /opt/serve.json
+COPY sidecar/dind-entrypoint.sh /opt/dind-entrypoint.sh
+RUN chmod +x /opt/dind-entrypoint.sh
+
+RUN cd /opt/agent-baseline \
+    && git init -q \
+    && git add -A \
+    && git -c user.email=baseline@vero.test -c user.name=baseline commit -qm "baseline" \
+    && git config --system --add safe.directory /opt/agent-baseline \
+    && git config --system --add safe.directory /work/agent \
+    && git config --system --add safe.directory /work/agent/.git
+
+ENV PYTHONPATH=/opt
+WORKDIR /opt
+# Start the in-container Docker daemon + bootstrap the judge image, then exec the
+# serve command passed by docker-compose.
+ENTRYPOINT ["/opt/dind-entrypoint.sh"]

harness-engineering-bench/ale-bench/environment/sidecar/ale_factory.py

@@ -0,0 +1,153 @@
+"""Custom Harbor sidecar factory for ALE-Bench (program optimization).
+
+vero drives the OUTER loop: a coding agent edits /work/agent/solution.cpp to
+maximize the ALE-Bench score for a fixed AHC problem. Each candidate is scored by
+a plain CommandBackend (harness/score.py -> ALE-Bench's deterministic judge),
+NOT a nested `harbor run`. Disclosure maps onto ALE-Bench's public/private split:
+development+validation are scored on public seeds, test on private seeds.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+from vero.candidate_repository import GitCandidateRepository
+from vero.evaluation import (
+    BackendRegistry,
+    BudgetLedger,
+    CaseRange,
+    EvaluationBudget,
+    EvaluationDatabase,
+    EvaluationSet,
+    Evaluator,
+    MetricSelector,
+    ObjectiveSpec,
+)
+from vero.evaluation.command import CommandBackend, CommandBackendConfig
+from vero.evaluation.engine import EvaluationEngine
+from vero.harbor.serve import SidecarComponents
+from vero.harbor.sidecar import EvaluationSidecar, SidecarEvaluationPolicy
+from vero.harbor.transport import GitCandidateTransport
+from vero.harbor.verifier import CanonicalVerifier, VerificationSelection, VerificationTarget
+from vero.sandbox import LocalSandbox
+from vero.workspace import GitWorkspace
+
+SET = "ale-bench"
+# ahc011 is a maximization problem; ALE-Bench zeroes invalid/CE/TLE submissions.
+OBJ = ObjectiveSpec(
+    selector=MetricSelector(metric="score"),
+    direction="maximize",
+    failure_value=0.0,
+)
+
+
+def _backend(harness_root: str) -> CommandBackend:
+    return CommandBackend(
+        CommandBackendConfig(
+            harness_root=harness_root,
+            command=[
+                "python", "{harness}/score.py",
+                "--workspace", "{workspace}",
+                "--request", "{request}",
+                "--report", "{report}",
+                "--artifacts", "{artifacts}",
+            ],
+            # The judge runs the submission in Docker; the scorer needs daemon access.
+            passthrough_environment=["PATH", "HOME", "DOCKER_HOST", "HF_TOKEN"],
+        )
+    )
+
+
+async def build(config: dict) -> SidecarComponents:
+    repo_path = config["repo_path"]
+    agent_repo_path = config["agent_repo_path"]
+    session_dir = Path(config["session_dir"])
+    harness_root = config["harness_root"]
+    session_dir.mkdir(parents=True, exist_ok=True)
+
+    sandbox = await LocalSandbox.create(root=Path(repo_path).parent)
+    workspace = await GitWorkspace.from_path(sandbox, repo_path)
+    candidate_repository = await GitCandidateRepository.create(
+        session_dir / "candidates", workspace=workspace
+    )
+    database = EvaluationDatabase.load_reconciled(
+        database_path=session_dir / "database.json",
+        evaluations_dir=session_dir / "evaluations",
+        database_id="ale",
+    )
+    ledger = BudgetLedger(
+        [
+            EvaluationBudget(
+                backend_id="cmd",
+                evaluation_set_key=f"cmd:{SET}:development",
+                total_runs=40,
+            ),
+            EvaluationBudget(
+                backend_id="cmd",
+                evaluation_set_key=f"cmd:{SET}:validation",
+                total_runs=40,
+            ),
+        ],
+        path=session_dir / "budgets.json",
+    )
+    ledger.save()
+
+    engine = EvaluationEngine(
+        evaluator=Evaluator(
+            candidate_repository=candidate_repository,
+            sandbox=workspace.sandbox,
+            session_dir=session_dir,
+            session_id="ale",
+        ),
+        backends=BackendRegistry({"cmd": _backend(harness_root)}),
+        database=database,
+        database_path=session_dir / "database.json",
+        budget_ledger=ledger,
+    )
+    transport = GitCandidateTransport(
+        workspace=workspace,
+        candidate_repository=candidate_repository,
+        agent_repo_path=agent_repo_path,
+    )
+    baseline = await transport.trusted_candidate("HEAD")
+
+    selection = VerificationSelection(
+        mode="auto_best",
+        backend_id="cmd",
+        evaluation_set=EvaluationSet(
+            name=SET, partition="validation", selection=CaseRange(start=0, stop=1)
+        ),
+        objective=OBJ,
+        baseline_candidate=baseline,
+    )
+    sidecar = EvaluationSidecar(
+        engine=engine,
+        candidate_transport=transport,
+        access_policies=[
+            SidecarEvaluationPolicy(
+                backend_id="cmd", evaluation_set_name=SET, partition="development",
+                disclosure="full", expose_case_resources=True, objective=OBJ,
+            ),
+            SidecarEvaluationPolicy(
+                backend_id="cmd", evaluation_set_name=SET, partition="validation",
+                disclosure="aggregate", min_aggregate_cases=1, objective=OBJ,
+            ),
+        ],
+        admin_volume=session_dir / "admin",
+    )
+    await sidecar.initialize_context()
+
+    verifier = CanonicalVerifier(
+        engine=engine,
+        selection=selection,
+        targets=[
+            VerificationTarget(
+                reward_key="score", backend_id="cmd",
+                evaluation_set=EvaluationSet(name=SET, partition="test"),
+                objective=OBJ, max_attempts=1,
+            )
+        ],
+        admin_volume=session_dir / "admin",
+        score_baseline=True,
+    )
+    return SidecarComponents(sidecar=sidecar, verifier=verifier)

harness-engineering-bench/ale-bench/environment/sidecar/dind-entrypoint.sh

@@ -0,0 +1,27 @@
+#!/bin/sh
+# Start an in-container Docker daemon (DinD) so ALE-Bench's judge + Rust-tool
+# containers are SIBLINGS on a daemon whose filesystem IS this container's — the
+# only way their bind mounts resolve. (A host/VM socket mount fails: the sibling
+# containers resolve mount sources against the daemon host, not this container.)
+# Requires the container to run privileged. Then serve (the passed command).
+set -eu
+
+dockerd >/var/log/dockerd.log 2>&1 &
+for i in $(seq 1 60); do
+  docker info >/dev/null 2>&1 && break
+  sleep 1
+done
+if ! docker info >/dev/null 2>&1; then
+  echo "dockerd failed to start" >&2
+  cat /var/log/dockerd.log >&2 || true
+  exit 1
+fi
+
+# Bootstrap the ALE-Bench C++ judge image. ALE-Bench expects the tag
+# ale-bench:cpp20-202301; upstream builds it as `FROM yimjk/ale-bench:cpp20-202301`
+# plus `chown UID:GID /workdir` — a no-op as root — so a pull + retag suffices.
+docker pull yimjk/ale-bench:cpp20-202301
+docker tag yimjk/ale-bench:cpp20-202301 ale-bench:cpp20-202301
+# rust:1.79.0-buster (tool build) and httpd (vis) are pulled on demand by ale_bench.
+
+exec "$@"