Merge pull request #45 from scaleapi/pr2-add-vero

Author: varunursekar

.gitignore

@@ -1,5 +1,6 @@
 # Python-generated files
 __pycache__/
+.pytest_cache/
 *.py[oc]
 build/
 dist/
@@ -9,3 +10,11 @@ wheels/
 # Virtual environments
 .venv
 .env*
+
+# Run secrets loaded via `vero harbor run --env-file` (keep only *.example)
+secrets.env
+*.secrets.env
+!*.example
+
+# Local benchmark scoping notes (paper planning; not tracked)
+harness-engineering-bench/benchmark-scoping.md

README.md

@@ -1,122 +1,112 @@
-# VeRO: Versioning Rewards and Observations
+# VeRO: a harness for agents to optimize programs, text, and agents
+> **Looking for the code from the VeRO paper?** See
+> [Paper reproduction](#paper-reproduction) — reproduce from the `paper-v1`
+> tag, or read the same code in place under [`legacy/`](legacy/).
 
 [![Paper](https://img.shields.io/badge/arXiv-2602.22480-b31b1b.svg)](https://arxiv.org/abs/2602.22480)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/)
 
-VeRO is an evaluation harness for using coding agents to optimize LLM-based agents and workflows. It treats agent code as a versioned artifact — making changes, evaluating results, and hill-climbing toward better performance using git version control.
-
-> **Paper**: [VeRO: An Evaluation Harness for Agents to Optimize Agents](https://arxiv.org/abs/2602.22480)
-
-## Repository Structure
+VeRO gives a coding agent something to edit, an evaluation boundary, and durable
+memory of every candidate it tried. The target is anything you can put under Git
+and score — a **program** (a single function up to a whole codebase), **text**
+(a prompt, spec, or config), or an **agent** (its scaffold, tools, and prompts).
+VeRO was introduced to optimize agents, and the same version / evaluate / select
+loop applies to any of these.
 
 ```
-vero/
-├── vero/               # Core library (scale-vero)
-├── vero-agents/        # Agent implementations (benchmarking targets)
-├── vero-benchmarking/  # Benchmarking scripts and analysis
-└── LICENSE
+  ┌─────────────────────────┐   submit candidate    ┌─────────────────────────┐
+  │  candidate production   ├──────────────────────►│  evaluation service     │
+  │                         │                       │                         │
+  │  coding agent, command, │◄──────────────────────┤  owns cases + scoring   │
+  │  or custom strategy;    │  score + diagnostics  │                         │
+  │  edits its own Git      │                       │  development: may ask   │
+  │  worktree per candidate │                       │  validation:  aggregate │
+  └───────────┬─────────────┘                       │  test:        withheld  │
+              │ commit                              └───────────┬─────────────┘
+              ▼                                                 │ report
+  ┌─────────────────────────┐    next round     ┌───────────────▼─────────────┐
+  │  candidate history:     │◄──────────────────┤  selection: keep the best   │
+  │  every version kept,    │                   │  feasible candidate         │
+  │  each one re-selectable │                   └─────────────────────────────┘
+  └─────────────────────────┘
+
+  Every model call on both sides goes through the inference gateway, which holds
+  the provider key and meters spend in tokens against a per-scope budget.
 ```
 
-### [vero/](vero/)
-
-The core optimization framework. Provides:
-
-- **Policy** — orchestrates the optimization loop (agent + evaluator + git)
-- **Agents** — VeroAgent (OpenAI Agents SDK) and ClaudeCodeAgent (Claude Agent SDK)
-- **Evaluator** — runs task evaluations in isolated subprocess environments
-- **Tools** — MCP-based tools for agents (bash, file I/O, experiment runner, dataset viewer, etc.)
-- **Traces** — session analysis and LLM-based trace interpretation
-
-```bash
-cd vero && uv sync --extra optimize
-```
-
-See [vero/README.md](vero/README.md) for full documentation.
-
-### [vero-agents/](vero-agents/)
+The target and evaluator do not need to be Python. External evaluators and
+candidate producers connect through command protocols; Python benchmarks can
+use the optional, optimizer-independent `scale-vero-tasks` package.
 
-Agent implementations used as optimization targets:
+Targets may live locally or in an isolated sandbox. VeRO keeps optimization
+state and experiment tracking on the host while running Git worktrees, producer
+commands, builds, and evaluation commands in the target sandbox. The core guide
+includes a no-bind-mount `DockerSandbox` example.
 
-| Agent | Description |
-|-------|-------------|
-| **generic-agent** | General-purpose agent for MATH, GPQA, GAIA, GSM8K, etc. |
-| **web_search_agent** | Web search agent for SimpleQA, Facts Search |
-| **KIRA** | Terminal task agent for Terminal Bench 2.0 |
-| **tau-bench** | Customer service tool-use agent |
-| **pharma_summarizer** | Document summarization agent |
+## Repository layout
 
-See [vero-agents/README.md](vero-agents/README.md) for details.
+| Directory | Purpose |
+| --- | --- |
+| [`vero/`](vero/) | The `scale-vero` optimization kernel, runtime, CLI, and coding-agent adapters |
+| [`vero-tasks/`](vero-tasks/) | Narrow Python task types and schema-v1 evaluation runner |
+| [`harness-engineering-bench/`](harness-engineering-bench/) | Harbor-native target programs and end-to-end optimization benchmarks |
+| [`legacy/`](legacy/) | The pre-v0.5 tree, i.e. the original VeRO paper code — reference only, not used by the current system |
 
-### [vero-benchmarking/](vero-benchmarking/)
+Start with the [generic C matrix-multiplication quickstart](vero/examples/c-matmul/),
+try the [26-circle packing benchmark](vero/examples/circle-packing/), or read the
+[core guide](vero/README.md). The C example demonstrates the language-neutral
+command protocol without model credentials. Circle packing is a substantive
+coding-agent benchmark with exact geometry checks and inspectable search
+artifacts.
 
-Scripts and infrastructure for running optimization experiments:
+For a full agent-optimization example, the [GAIA benchmark](harness-engineering-bench/gaia/)
+pairs a tool-using GPT-5.4 mini target with Harbor's canonical GAIA verifier and
+an immutable 20% / 40% / 40% development, validation, and test split.
 
 ```bash
-cd vero-benchmarking && uv sync --all-extras
-
-# Run an optimization experiment
-uv run python scripts/run_benchmark.py --scaffold claude-code-vmf --model sonnet --task math
-
-# Build datasets
-./scripts/build_datasets.sh
+cd vero
+uv sync --all-extras
+uv run vero --help
 ```
 
-See [vero-benchmarking/README.md](vero-benchmarking/README.md) for full documentation.
+## Paper reproduction
 
-## Quick Start
+VeRO was introduced in [*VeRO: A Harness for Agents to Optimize
+Agents*](https://arxiv.org/abs/2602.22480), accepted at ICML 2026. The current
+library generalizes that version/evaluate/select loop from agents to programs.
 
-### Prerequisites
+The paper-era code is available two ways, and they hold the same code:
 
-- Python 3.11+
-- [uv](https://docs.astral.sh/uv/getting-started/installation/)
-- Git
-- Access to an LLM provider (via LiteLLM, OpenAI, Anthropic, etc.)
-
-### Install
+**To reproduce the paper, use the frozen ref.** It is the repository exactly as
+it stood at publication, with the original paths intact:
 
 ```bash
-git clone <repo-url> && cd vero
-
-# Install core library
-cd vero && uv sync --extra optimize
-
-# Install benchmarking tools
-cd ../vero-benchmarking && uv sync --all-extras
+git checkout paper-v1          # tag; the paper/v1 branch points at the same commit
 ```
 
-### Run Your First Optimization
-
-```python
-from agents import Agent as OAIAgent
-from vero.policy import Policy
-from vero.agents.vero import VeroAgent
-
-policy = Policy(
-    project_path="/path/to/my-agent",
-    dataset="/path/to/my-dataset",
-    agent=VeroAgent(
-        oai_agent=OAIAgent(name="VeroAgent", model="anthropic/claude-sonnet-4-5-20250929"),
-    ),
-    task="main",
-    train_budget=10,
-    max_turns=200,
-)
-
-best = await policy.run()
-print(f"Best commit: {best.commit}, score: {best.score}")
-```
+**To read that code alongside the current system, use [`legacy/`](legacy/).** The
+v0.5 redesign relocated the paper-era tree into that directory rather than
+deleting it, so it sits in this branch next to the code that replaced it. Both
+the frozen ref and `legacy/` include the paper-era `vero-agents` and
+`vero-benchmarking` directories; their Harbor-native replacement is
+[`harness-engineering-bench/`](harness-engineering-bench/).
+
+Prefer the frozen ref for reproduction, since it is the state that was actually
+published. Either way, note that both packages are named `scale-vero` and both
+import as `vero` (0.4.7 in `legacy/vero`, 0.5.0 in `vero/`), so **they cannot be
+installed into the same environment** — give the legacy package its own
+virtualenv. Development of the current system continues on `main`.
 
 ## Citation
 
 ```bibtex
 @article{ursekar2026vero,
-  title={VeRO: An Evaluation Harness for Agents to Optimize Agents},
+  title={VeRO: A Harness for Agents to Optimize Agents},
   author={Ursekar, Varun and Shanker, Apaar and Chatrath, Veronica and Xue, Yuan (Emily) and Denton, Sam},
   journal={arXiv preprint arXiv:2602.22480},
   year={2026}
 }
 ```
 
-## License
-
-[MIT](LICENSE)
+VeRO is licensed under the [MIT License](LICENSE).

legacy/README.md

@@ -4,3 +4,14 @@ This directory archives the codebase as it existed before the v0.5 redesign —
 it is the code for the **original VeRO paper**. It is kept for reference and
 reproducibility only; it is not used by the current system, whose code lives at
 the repository root (`vero/`, `harness-engineering-bench/`).
+
+**Reproducing the paper?** Prefer the frozen ref, which is the repository exactly
+as it stood at publication rather than relocated under this directory:
+
+```bash
+git checkout paper-v1          # tag; the paper/v1 branch points at the same commit
+```
+
+**Installing anything from here?** Give it its own virtualenv. `legacy/vero` is
+`scale-vero` 0.4.7 and the current `vero/` is `scale-vero` 0.5.0; both import as
+`vero`, so they cannot share an environment.

runs/.gitignore

@@ -0,0 +1,3 @@
+# Local GAIA/harbor run outputs — inspectable, not committed.
+*
+!.gitignore

vero-tasks/README.md

@@ -0,0 +1,35 @@
+# scale-vero-tasks
+
+`scale-vero-tasks` is the optional Python task protocol used by VeRO benchmark
+targets. It contains no optimizer, workspace, session, dataset-store, or
+experiment-database code.
+
+```python
+from vero_tasks import TaskOutput, TaskResult, create_task
+
+task = create_task("exact_match")
+
+@task.inference()
+async def infer(case, context):
+    return TaskOutput(output=case["question"].upper())
+
+@task.evaluation()
+async def evaluate(case, output, context):
+    return TaskResult.from_task_output(
+        output,
+        score=float(output.output == case["answer"]),
+    )
+```
+
+The runner accepts a VeRO command-evaluation request and an external JSON/JSONL
+case file, imports a task module, and writes a schema-v1 evaluation report. The
+standard VeRO adapter runs this package in a trusted evaluator project while
+overlaying the candidate package as an editable dependency. This keeps Python
+benchmark ergonomics separate from both the target program and VeRO's
+language-neutral evaluation kernel.
+
+The runner applies VeRO's retry policy independently to each non-batch
+inference and evaluation call. Provider rate limits, configured HTTP status
+codes or messages, and timeouts can be retried with bounded exponential
+backoff. Earlier failed attempts are retained in the canonical case error list;
+they are non-terminal when a later attempt succeeds.

vero-tasks/examples/matmul-eval/pyproject.toml

@@ -0,0 +1,16 @@
+[project]
+name = "matmul-eval"
+version = "0.1.0"
+description = "Evaluation task for the matmul kernel — measures correctness and speed"
+requires-python = ">=3.11"
+dependencies = ["scale-vero-tasks>=0.2.0"]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/matmul_eval"]
+
+[tool.uv.sources]
+scale-vero-tasks = { path = "../..", editable = true }

vero-tasks/examples/matmul-eval/src/matmul_eval/__init__.py

@@ -0,0 +1,70 @@
+"""Matrix multiply evaluation task.
+
+Measures correctness and execution speed of a matmul kernel.
+Score = average time per multiply (lower is better).
+Incorrect results get a penalty score of 999999.0.
+"""
+
+import time
+
+from vero_tasks import (
+    TaskContext,
+    TaskOutput,
+    TaskParameters,
+    TaskResult,
+    create_task,
+)
+
+
+class MatmulParameters(TaskParameters):
+    n_repeats: int = 100
+
+
+matmul_task = create_task("matmul", task_parameters_type=MatmulParameters)
+
+
+@matmul_task.inference()
+async def run_inference(task: dict, context: TaskContext) -> TaskOutput:
+    from matmul_kernel import multiply
+
+    a = task["matrix_a"]
+    b = task["matrix_b"]
+
+    parameters = context.parse_task_params(MatmulParameters)
+
+    # Warmup
+    multiply(a, b)
+
+    # Timed runs (like timeit)
+    start = time.perf_counter()
+    for _ in range(parameters.n_repeats):
+        result = multiply(a, b)
+    elapsed_ms = (time.perf_counter() - start) / parameters.n_repeats * 1000
+
+    return TaskOutput(output={"result": result, "time_ms": elapsed_ms})
+
+
+@matmul_task.evaluation()
+async def run_evaluation(
+    task: dict, output: TaskOutput, context: TaskContext
+) -> TaskResult:
+    expected = task["expected"]
+    actual = output.output["result"]
+    time_ms = output.output["time_ms"]
+
+    # Check correctness (within floating point tolerance)
+    correct = all(
+        abs(a_val - e_val) < 1e-6
+        for a_row, e_row in zip(actual, expected)
+        for a_val, e_val in zip(a_row, e_row)
+    )
+
+    # Score = time if correct, penalty if wrong (lower is better)
+    score = time_ms if correct else 999999.0
+
+    return TaskResult.from_task_output(
+        output,
+        score=score,
+        metrics={"time_ms": time_ms, "correct": 1.0 if correct else 0.0},
+        feedback=f"{'Correct' if correct else 'Wrong'}. Time: {time_ms:.2f}ms",
+    )