|
1 | 1 | # AGENTS.md |
2 | 2 |
|
3 | | -<!-- Generated for repository development workflows. Do not edit directly. --> |
4 | | - |
5 | | -## Shared Best Practices (All Contexts) |
6 | | - |
7 | | -- Create new environments with `prime env init <MY_ENV_NAME>`. |
8 | | -- Use `prime <command> --plain` to get better formatted command outputs. |
9 | | -- Use the bundled skills first for create, browse, review, eval, optimization, training, and brainstorming workflows. |
10 | | -- Go through the verifiers code as the source of truth. |
11 | | -- Use toml files and validate them first before you run them. |
12 | | -- Always look at the verifiers v1 code first to see whether helper functions or harnesses exist for your given task. |
13 | | -- Never mix v1 and legacy verifiers functions in one environment. Always prefer and use v1.* code. |
14 | | - |
15 | | -## Style Rules |
16 | | - |
17 | | -Use these rules when shaping public v1 APIs, configs, and environment files. |
18 | | - |
19 | | -- Prefer verifiers-native task, trace, server, harness, and runtime interfaces over repeated path/import/discovery plumbing in user packages. |
20 | | -- Expose as few knobs in the configs as possible, but as many as needed. |
21 | | -- Use strict Pydantic models for structured config, tasks, messages, and state. |
22 | | -- A basic taskset should fit in a few dozen idiomatic lines: typed data/task/config classes, `load()`, and decorated scoring on the task. |
23 | | -- Do not override `Taskset.__init__`; implement `load()`. Do not override `Harness.__init__` or `User.__init__`; use `setup()`. |
24 | | -- Refer to the code as the source of truth. |
25 | | -- Avoid mutable module globals. Process-level locks/rate limiters and immutable constants are the narrow exceptions. |
26 | | - |
27 | | -## Repository Development Notes |
28 | | - |
29 | | -Use this guidance when contributing to the `verifiers` repository itself. |
30 | | - |
31 | | -- Ensure that `uv` is installed, see [uv docs](https://docs.astral.sh/uv/getting-started/installation/) for further information. |
32 | | -- Always run `uv run pre-commit install` before making any changes. |
33 | | -- Run the documented contributor checks for touched areas: `uv run ruff check --fix .`, `uv run pytest tests/`, and `uv run pre-commit run --all-files` as needed. (See `docs/v0/development.md`.) |
34 | | -- The documentation (in `docs/`, `skills/`, `configs/`) is intentionally kept minimal. Do not touch them unless your changes break with these assumptions. |
35 | | -- verifiers has two API surfaces under `verifiers/`: `verifiers/v1`, which is referred to as "v1", while the rest is "legacy". Unless specifically requested, always use and assume v1. |
36 | | -- verifiers v1 uses Pydantic objects everywhere, so you should, too, when working in v1. Mimic the style of the existing code. |
37 | | -- Do not add unit tests to your PRs. v1 has a few end to end tests, which are sufficient. Unit tests clog up the repository. You can, however, write temporary scripts to test. |
38 | | -- The repository uses Python 3.11 or older, so you are encouraged to use modern Python functions to cut down on lines. |
39 | | -- Do not add dependencies, optional extras etc. to the main pyproject.toml. |
| 3 | +## Writing code |
| 4 | + |
| 5 | +- **Code is the source of truth**: before writing anything, read the v1 code for existing helpers, harnesses, and interfaces instead of reinventing them. Prefer verifiers-native task, trace, server, harness, and runtime interfaces over repeated path/import/discovery plumbing in user packages. |
| 6 | +- **Minimal config surface**: expose as few knobs as possible, but as many as needed. |
| 7 | +- **Keep tasksets small**: a basic taskset fits in a few dozen idiomatic lines — typed data/task/config classes, `load()`, and decorated scoring on the task. Don't override `Taskset.__init__` (implement `load()`); don't override `Harness.__init__` or `User.__init__` (use `setup()`). |
| 8 | + |
| 9 | +## Running code |
| 10 | + |
| 11 | +- **Always use uv**: run code and commands with `uv run`, never raw `python`. Make sure `uv` is installed ([docs](https://docs.astral.sh/uv/getting-started/installation/)). |
| 12 | +- **Scaffold environments**: create a new taskset/environment with `uv run init <name>` (`uv run init -h` lists options like `-T`/`-U`/`-H`), and run evals with `uv run eval <taskset>`. |
| 13 | +- **Validate TOML first**: validate config `.toml` files before running them. |
| 14 | +- **Don't add dependencies**: never add dependencies or optional extras to the top-level `pyproject.toml`. |
| 15 | + |
| 16 | +## Docs |
| 17 | + |
| 18 | +- **Kept intentionally minimal**: `docs/`, `skills/`, and `configs/` are deliberately sparse. Don't touch them unless your change breaks their assumptions. |
| 19 | +- **Docs reflect `main`, not history**: describe the current state of the codebase only — no removed/legacy fields, migration paths, or "this used to be X" anecdotes. |
| 20 | + |
| 21 | +## Skills |
| 22 | + |
| 23 | +- **Use bundled skills first**: the skills in `skills/` cover the core workflows — `create-environments` (build or migrate a v1 taskset/environment/harness), `evaluate-environments` (configure and run evals), and `brainstorm` (ideation and research planning). Reach for them before doing the work by hand. |
| 24 | + |
| 25 | +## Testing |
| 26 | + |
| 27 | +- **Prefer e2e tests over unit tests**: v1's end-to-end tests are sufficient — extra unit tests clog the repo. Editing existing tests is fine; to check your own work, write a temporary script instead of committing new tests. |
| 28 | +- **Run the contributor checks**: run `uv run pre-commit install` once, then for touched areas `uv run ruff check --fix .`, `uv run pytest tests/`, and `uv run pre-commit run --all-files` (see `docs/v0/development.md`). |
0 commit comments