Skip to content

Commit 4ae9c1d

Browse files
scale-ballenscale-ballen
committed
Initial commit
0 parents  commit 4ae9c1d

381 files changed

Lines changed: 328823 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Python-generated files
2+
__pycache__/
3+
*.py[oc]
4+
build/
5+
dist/
6+
wheels/
7+
*.egg-info
8+
9+
# Virtual environments
10+
.venv
11+
.env*

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026 Scale AI
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
# VeRO: Versioning Rewards and Observations
2+
3+
[![Paper](https://img.shields.io/badge/arXiv-2602.22480-b31b1b.svg)](https://arxiv.org/abs/2602.22480)
4+
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
5+
6+
VeRO is an evaluation harness for using coding agents to optimize LLM-based agents and workflows. It treats agent code as a versioned artifact — making changes, evaluating results, and hill-climbing toward better performance using git version control.
7+
8+
> **Paper**: [VeRO: An Evaluation Harness for Agents to Optimize Agents](https://arxiv.org/abs/2602.22480)
9+
10+
## Repository Structure
11+
12+
```
13+
vero/
14+
├── vero/ # Core library (scale-vero)
15+
├── vero-agents/ # Agent implementations (benchmarking targets)
16+
├── vero-benchmarking/ # Benchmarking scripts and analysis
17+
└── LICENSE
18+
```
19+
20+
### [vero/](vero/)
21+
22+
The core optimization framework. Provides:
23+
24+
- **Policy** — orchestrates the optimization loop (agent + evaluator + git)
25+
- **Agents** — VeroAgent (OpenAI Agents SDK) and ClaudeCodeAgent (Claude Agent SDK)
26+
- **Evaluator** — runs task evaluations in isolated subprocess environments
27+
- **Tools** — MCP-based tools for agents (bash, file I/O, experiment runner, dataset viewer, etc.)
28+
- **Traces** — session analysis and LLM-based trace interpretation
29+
30+
```bash
31+
cd vero && uv sync --extra optimize
32+
```
33+
34+
See [vero/README.md](vero/README.md) for full documentation.
35+
36+
### [vero-agents/](vero-agents/)
37+
38+
Agent implementations used as optimization targets:
39+
40+
| Agent | Description |
41+
|-------|-------------|
42+
| **generic-agent** | General-purpose agent for MATH, GPQA, GAIA, GSM8K, etc. |
43+
| **web_search_agent** | Web search agent for SimpleQA, Facts Search |
44+
| **KIRA** | Terminal task agent for Terminal Bench 2.0 |
45+
| **tau-bench** | Customer service tool-use agent |
46+
| **pharma_summarizer** | Document summarization agent |
47+
48+
See [vero-agents/README.md](vero-agents/README.md) for details.
49+
50+
### [vero-benchmarking/](vero-benchmarking/)
51+
52+
Scripts and infrastructure for running optimization experiments:
53+
54+
```bash
55+
cd vero-benchmarking && uv sync --all-extras
56+
57+
# Run an optimization experiment
58+
uv run python scripts/run_benchmark.py --scaffold claude-code-vmf --model sonnet --task math
59+
60+
# Build datasets
61+
./scripts/build_datasets.sh
62+
```
63+
64+
See [vero-benchmarking/README.md](vero-benchmarking/README.md) for full documentation.
65+
66+
## Quick Start
67+
68+
### Prerequisites
69+
70+
- Python 3.11+
71+
- [uv](https://docs.astral.sh/uv/getting-started/installation/)
72+
- Git
73+
- Access to an LLM provider (via LiteLLM, OpenAI, Anthropic, etc.)
74+
75+
### Install
76+
77+
```bash
78+
git clone <repo-url> && cd vero
79+
80+
# Install core library
81+
cd vero && uv sync --extra optimize
82+
83+
# Install benchmarking tools
84+
cd ../vero-benchmarking && uv sync --all-extras
85+
```
86+
87+
### Run Your First Optimization
88+
89+
```python
90+
from agents import Agent as OAIAgent
91+
from vero.policy import Policy
92+
from vero.agents.vero import VeroAgent
93+
94+
policy = Policy(
95+
project_path="/path/to/my-agent",
96+
dataset="/path/to/my-dataset",
97+
agent=VeroAgent(
98+
oai_agent=OAIAgent(name="VeroAgent", model="anthropic/claude-sonnet-4-5-20250929"),
99+
),
100+
task="main",
101+
train_budget=10,
102+
max_turns=200,
103+
)
104+
105+
best = await policy.run()
106+
print(f"Best commit: {best.commit}, score: {best.score}")
107+
```
108+
109+
## Citation
110+
111+
```bibtex
112+
@article{ursekar2026vero,
113+
title={VeRO: An Evaluation Harness for Agents to Optimize Agents},
114+
author={Ursekar, Varun and Shanker, Apaar and Chatrath, Veronica and Xue, Yuan (Emily) and Denton, Sam},
115+
journal={arXiv preprint arXiv:2602.22480},
116+
year={2026}
117+
}
118+
```
119+
120+
## License
121+
122+
[MIT](LICENSE)

‎vero-agents/.gitignore‎

Lines changed: 180 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
# Logs
2+
logs
3+
*.log
4+
npm-debug.log*
5+
*.pth
6+
7+
# Runtime data
8+
pids
9+
*.pid
10+
*.seed
11+
*.pid.lock
12+
13+
# IntelliJ
14+
**/.idea
15+
*.iml
16+
17+
# VSCode
18+
.vscode
19+
*.code-workspace
20+
21+
# filesystem files
22+
.DS_Store
23+
24+
# Local environment files
25+
*.env
26+
.env.*
27+
*.envrc
28+
frontend/.npmrc
29+
local*.yaml
30+
31+
# filesystem databases
32+
dump.rdb
33+
*.sqlite
34+
*.db
35+
36+
# Temp dirs
37+
tmp
38+
39+
### PYTHON
40+
41+
# Byte-compiled / optimized / DLL files
42+
__pycache__/
43+
*.py[cod]
44+
*$py.class
45+
46+
# C extensions
47+
*.so
48+
49+
# Distribution / packaging
50+
.Python
51+
build/
52+
develop-eggs/
53+
dist/
54+
downloads/
55+
eggs/
56+
.eggs/
57+
lib/
58+
lib64/
59+
parts/
60+
sdist/
61+
var/
62+
wheels/
63+
share/python-wheels/
64+
*.egg-info/
65+
.installed.cfg
66+
*.egg
67+
MANIFEST
68+
69+
# PyInstaller
70+
# Usually these files are written by a python script from a template
71+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
72+
*.manifest
73+
*.spec
74+
75+
# Installer logs
76+
pip-log.txt
77+
pip-delete-this-directory.txt
78+
79+
# Unit test / coverage reports
80+
htmlcov/
81+
.tox/
82+
.nox/
83+
.coverage
84+
.coverage.*
85+
.cache
86+
nosetests.xml
87+
coverage.xml
88+
*.cover
89+
*.py,cover
90+
.hypothesis/
91+
.pytest_cache/
92+
cover/
93+
94+
# Translations
95+
*.mo
96+
*.pot
97+
98+
# Django stuff:
99+
*.log
100+
local_settings.py
101+
db.sqlite3
102+
db.sqlite3-journal
103+
104+
# Flask stuff:
105+
instance/
106+
.webassets-cache
107+
108+
# Scrapy stuff:
109+
.scrapy
110+
111+
# Sphinx documentation
112+
docs/_build/
113+
114+
# PyBuilder
115+
.pybuilder/
116+
target/
117+
118+
# Jupyter Notebook
119+
.ipynb_checkpoints
120+
_temp_extension
121+
junit.xml
122+
[uU]ntitled*
123+
notebook/static/*
124+
!notebook/static/favicons
125+
notebook/labextension
126+
notebook/schemas
127+
docs/source/changelog.md
128+
docs/source/contributing.md
129+
130+
# IPython
131+
profile_default/
132+
ipython_config.py
133+
134+
# pyenv
135+
# For a library or package, you might want to ignore these files since the code is
136+
# intended to run in multiple environments; otherwise, check them in:
137+
# .python-version
138+
139+
# pipenv
140+
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
141+
# However, in case of collaboration, if having platform-specific dependencies or dependencies
142+
# having no cross-platform support, pipenv may install dependencies that don't work, or not
143+
# install all needed dependencies.
144+
#Pipfile.lock
145+
146+
# pdm
147+
.pdm.toml
148+
149+
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
150+
__pypackages__/
151+
152+
# Lock files
153+
uv.lock
154+
155+
# Environments
156+
.env
157+
.venv
158+
env/
159+
venv/
160+
ENV/
161+
env.bak/
162+
venv.bak/
163+
164+
# mypy
165+
.mypy_cache/
166+
.dmypy.json
167+
dmypy.json
168+
169+
# Pyre type checker
170+
.pyre/
171+
172+
# pytype static type analyzer
173+
.pytype/
174+
175+
# Cython debug symbols
176+
cython_debug/
177+
178+
# IDE / local config
179+
.claude/
180+
.python-version

‎vero-agents/README.md‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
# Vero Agents
2+
3+
A collection of Python-based agent implementations optimizable with [VeRO](../vero/). Each agent is a self-contained `uv` package with its own dependencies and evaluation tasks.
4+
5+
## Agents
6+
7+
| Agent | Description | Benchmark Tasks |
8+
|-------|-------------|-----------------|
9+
| **generic-agent** | OpenAI Agents SDK agent for general-purpose benchmarks | MATH, GPQA, GAIA, GSM8K, HotpotQA, HumanEval, DROP, MBPP |
10+
| **web_search_agent** | Web search agent using OpenAI Agents SDK | SimpleQA, Facts Search |
11+
| **KIRA** | Terminal task agent built on Harbor/Terminus | Terminal Bench 2.0 |
12+
| **tau-bench** | Tool-augmented customer service agent | Tau Bench Retail |
13+
| **pharma_summarizer** | Pharmaceutical document summarization agent | Pharma Summarizer |
14+
15+
## Structure
16+
17+
```
18+
vero-agents/
19+
├── agents/
20+
│ ├── generic-agent/ # General-purpose LLM agent
21+
│ ├── web_search_agent/ # Web search + retrieval agent
22+
│ ├── KIRA/ # Terminal Bench agent (Harbor-based)
23+
│ ├── tau-bench/ # Customer service tool-use agent
24+
│ └── pharma_summarizer/ # Document summarization agent
25+
└── pyproject.toml
26+
```
27+
28+
Each agent contains:
29+
- `pyproject.toml` with dependencies and a dev dependency on `scale-vero`
30+
- Source code under `src/<agent_name>/` or `<agent_name>/`
31+
- A `vero_tasks/` module defining evaluation tasks for VeRO
32+
33+
## Adding a New Agent
34+
35+
1. Create a directory under `agents/`
36+
2. Initialize a `uv` package with `pyproject.toml`
37+
3. Add `scale-vero` as a dev dependency:
38+
```toml
39+
[dependency-groups]
40+
dev = ["scale-vero[evaluate]"]
41+
42+
[tool.uv.sources]
43+
scale-vero = { path = "../../../vero", editable = true }
44+
```
45+
4. Create a `vero_tasks/` module with inference and evaluation functions (see `vero init tasks`)
46+
5. Register the task in `vero-benchmarking/src/vero_benchmarking/tasks/`
47+
48+
## Running Evaluations
49+
50+
Agents are evaluated through `vero-benchmarking`. See the [vero-benchmarking README](../vero-benchmarking/README.md) for details.
51+
52+
```bash
53+
# Quick example: optimize generic-agent on MATH
54+
cd ../vero-benchmarking
55+
uv run python scripts/run_benchmark.py --scaffold claude-code-vmf --model sonnet --task math
56+
```

‎vero-agents/agents/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
uv.lock
2+
.env*

0 commit comments

Comments
 (0)