Stacked on pr3-harness-bench (#46). One new file: .github/workflows/tests.yml.

Why

No test ever ran in CI on this repository. There is no .github directory on any branch, and the only checks on a pull request are Greptile Review and Socket Security, neither of which invokes pytest.

That is not hypothetical. #61 sat open with a red target suite and nothing surfaced it: the change there reaches the OpenAI client through with_options(...), the test fake was a bare SimpleNamespace, and test_agent_submits_response_and_populates_context had been failing on AttributeError: 'types.SimpleNamespace' object has no attribute 'with_options' since the PR was opened. It only came to light because I ran the suite by hand while rebasing.

What it runs

Two jobs, seven suites, roughly 450 vero tests plus 20 across the agents.

fail-fast: false on the matrix, so one broken agent does not mask the state of the other five.

Decisions worth reviewing

Two placeholder credentials in the vero job. The task compiler refuses to emit a task whose declared credentials are absent from the environment (compiler.py: declared task credentials are missing). Without OPENAI_API_KEY and OPENAI_BASE_URL present, five tests in test_v05_harbor_build.py fail, which looks like a code failure and is not one. They are placeholders; nothing dials them, and the base URL points at port 1 so anything that ever tried would fail fast rather than hang.

Setting VERO_SKIP_SECRET_CHECK instead would be the wrong lever: I tried it, and it makes test_compiler_checks_secrets_before_writing_and_rejects_source_overlap fail, because that test asserts the check does fire. Placeholders keep the check live.

uv sync --locked, not uv sync. --locked fails rather than relocking, so a pyproject.toml edit that forgets to update uv.lock becomes a visible error instead of a silent re-resolve. Verified safe: uv lock --check is clean in all seven projects today, so this will not fail spuriously on arrival.

UV_PYTHON: "3.12". vero requires >=3.11,<3.14, the six targets require >=3.12; 3.12 is the only version satisfying all of them without relying on the runner image's default.

uv via the official installer rather than a setup action, so there is no action tag to keep current. No dependency cache: a fresh resolve of harbor + openai is cheap, and adding cache keys before there is evidence they are needed is speculative.

No submodules: recursive. The only submodule is the BrowseComp-Plus upstream corpus, which no test reads; the directory is empty locally and all suites pass.

Known flake risk

vero/tests/test_v05_docker_sandbox.py is guarded by @pytest.mark.skipif(not _docker_available()). It skips where there is no daemon, but ubuntu-latest has Docker, so these will actually execute in CI. I saw test_generic_optimization_without_a_shared_filesystem fail once locally and pass on every other run, so it may be order-dependent. Flagging rather than pre-emptively skipping it: if it proves flaky in CI it should be fixed or marked, not hidden.

Not included

Ruff is configured in vero/pyproject.toml but not run here. Adding a lint gate to a codebase that has never had one is a separate change with its own diff, and mixing it in would make this PR's signal ambiguous.

Greptile Summary

This PR adds the repository's first CI test workflow, running pytest across vero (447 tests) and six baseline agent targets via a matrix strategy. The design choices are well-documented and deliberate: uv sync --locked for reproducible installs, placeholder OpenAI credentials to satisfy the compiler's secret-presence check without enabling real network calls, fail-fast: false so one broken agent doesn't hide the others, and UV_PYTHON: \"3.12\" to satisfy all seven projects' version constraints.

• vero job installs the vero package with locked deps and runs its full test suite with a 25-minute timeout; Docker-backed sandbox tests will execute on ubuntu-latest (flagged as a known flake risk in the PR description).
• targets job fans out over six benchmark directories using a matrix, each with its own locked uv environment and a 15-minute timeout.
• Both jobs install uv via the official astral.sh installer without pinning a specific version, meaning the toolchain version varies between runs.

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    trigger["on: pull_request / push to main"] --> vero_job
    trigger --> targets_job

    subgraph vero_job["Job: vero (timeout 25 min)"]
        v1["actions/checkout@v4"] --> v2["Install uv (latest)"]
        v2 --> v3["uv sync --locked\n(working-directory: vero)"]
        v3 --> v4["uv run pytest -q\n447 passed, 11 skipped"]
    end

    subgraph targets_job["Job: targets (matrix, fail-fast: false, timeout 15 min)"]
        direction TB
        matrix["Matrix: browsecomp-plus, gaia,\nofficeqa, swe-atlas-qna,\nswe-bench-pro, tau3"]
        matrix --> t1["actions/checkout@v4"]
        t1 --> t2["Install uv (latest)"]
        t2 --> t3["uv sync --locked\n(working-directory: harness-engineering-bench/BENCH/baseline/target)"]
        t3 --> t4["uv run pytest -q\n~20 tests across 6 agents"]
    end

_{Reviews (2): Last reviewed commit: "ci: run pytest for vero and the six base..." | Re-trigger Greptile}