ci: run pytest for vero and the six baseline agents

[shehabyasser-scale]

Stacked on pr3-harness-bench (#46). One new file: .github/workflows/tests.yml.

Why

No test ever ran in CI on this repository. There is no .github directory on any branch, and the only checks on a pull request are Greptile Review and Socket Security, neither of which invokes pytest.

That is not hypothetical. #61 sat open with a red target suite and nothing surfaced it: the change there reaches the OpenAI client through with_options(...), the test fake was a bare SimpleNamespace, and test_agent_submits_response_and_populates_context had been failing on AttributeError: 'types.SimpleNamespace' object has no attribute 'with_options' since the PR was opened. It only came to light because I ran the suite by hand while rebasing.

What it runs

Two jobs, seven suites, roughly 450 vero tests plus 20 across the agents.

Job	Covers	Measured locally at 3.12
vero	vero/	447 passed, 11 skipped
targets (matrix)	browsecomp-plus	3 passed
	gaia	2 passed
	officeqa	2 passed
	swe-atlas-qna	2 passed
	swe-bench-pro	3 passed
	tau3	8 passed

fail-fast: false on the matrix, so one broken agent does not mask the state of the other five.

Decisions worth reviewing

Two placeholder credentials in the vero job. The task compiler refuses to emit a task whose declared credentials are absent from the environment (compiler.py: declared task credentials are missing). Without OPENAI_API_KEY and OPENAI_BASE_URL present, five tests in test_v05_harbor_build.py fail, which looks like a code failure and is not one. They are placeholders; nothing dials them, and the base URL points at port 1 so anything that ever tried would fail fast rather than hang.

Setting VERO_SKIP_SECRET_CHECK instead would be the wrong lever: I tried it, and it makes test_compiler_checks_secrets_before_writing_and_rejects_source_overlap fail, because that test asserts the check does fire. Placeholders keep the check live.

uv sync --locked, not uv sync. --locked fails rather than relocking, so a pyproject.toml edit that forgets to update uv.lock becomes a visible error instead of a silent re-resolve. Verified safe: uv lock --check is clean in all seven projects today, so this will not fail spuriously on arrival.

UV_PYTHON: "3.12". vero requires >=3.11,<3.14, the six targets require >=3.12; 3.12 is the only version satisfying all of them without relying on the runner image's default.

uv via the official installer rather than a setup action, so there is no action tag to keep current. No dependency cache: a fresh resolve of harbor + openai is cheap, and adding cache keys before there is evidence they are needed is speculative.

No submodules: recursive. The only submodule is the BrowseComp-Plus upstream corpus, which no test reads; the directory is empty locally and all suites pass.

Known flake risk

vero/tests/test_v05_docker_sandbox.py is guarded by @pytest.mark.skipif(not _docker_available()). It skips where there is no daemon, but ubuntu-latest has Docker, so these will actually execute in CI. I saw test_generic_optimization_without_a_shared_filesystem fail once locally and pass on every other run, so it may be order-dependent. Flagging rather than pre-emptively skipping it: if it proves flaky in CI it should be fixed or marked, not hidden.

Not included

Ruff is configured in vero/pyproject.toml but not run here. Adding a lint gate to a codebase that has never had one is a separate change with its own diff, and mixing it in would make this PR's signal ambiguous.

Greptile Summary

This PR adds the repository's first CI test workflow, running pytest across vero and six baseline agent projects on every pull request and push to main. The workflow is well-structured with thoughtful decisions documented inline (locked installs, placeholder credentials, fail-fast: false on the matrix).

• vero job: installs dependencies via uv sync --locked, runs ~458 tests (447 pass, 11 skip) with placeholder credentials to satisfy the task compiler's secret-presence check.
• targets matrix: fans out across browsecomp-plus, gaia, officeqa, swe-atlas-qna, swe-bench-pro, and tau3; each agent runs its own locked install and ~20 tests with fail-fast: false so a single broken agent doesn't hide the rest.
• Two minor concerns: cancel-in-progress: true is unconditional, so rapid commits to main can cancel the previous run before it reports; and uv is fetched via an unpinned curl | sh — switching to astral-sh/setup-uv with a version pin would make installs reproducible without needing to track an action tag.

Confidence Score: 4/5

Safe to merge; the workflow is additive with no impact on application code.

The change is a single new CI workflow file. The two suggestions (conditional cancel-in-progress and pinned uv install) are non-blocking quality improvements that can be addressed in a follow-up.

Files Needing Attention: .github/workflows/tests.yml — specifically the concurrency block and the Install uv steps in both jobs.

Important Files Changed

Filename	Overview
.github/workflows/tests.yml	Introduces the first CI test workflow: two jobs (vero + 6-agent matrix) running pytest via uv. Two minor concerns: cancel-in-progress applies to main-branch pushes (could cancel runs for intermediate commits), and uv is installed via an unpinned curl

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    trigger["on: pull_request / push to main"] --> vero["Job: vero\n(ubuntu-latest, 25 min timeout)"]
    trigger --> targets["Job: targets\n(ubuntu-latest, 15 min timeout)\nstrategy: fail-fast: false"]

    vero --> v1["actions/checkout@v4"]
    v1 --> v2["curl ... astral.sh/uv/install.sh | sh\n(latest uv, no version pin)"]
    v2 --> v3["uv sync --locked\n(working-directory: vero)"]
    v3 --> v4["uv run pytest -q\n447 passed, 11 skipped"]

    targets --> m1["Matrix: 6 benchmarks\nbrowsecomp-plus / gaia / officeqa\nswe-atlas-qna / swe-bench-pro / tau3"]
    m1 --> t1["actions/checkout@v4"]
    t1 --> t2["curl ... astral.sh/uv/install.sh | sh\n(latest uv, no version pin)"]
    t2 --> t3["uv sync --locked\n(working-directory: harness-engineering-bench/BENCHMARK/baseline/target)"]
    t3 --> t4["uv run pytest -q\n~20 tests across 6 agents"]

    concurrency["concurrency:\ngroup: tests-github.ref\ncancel-in-progress: true\n(applies to both PR and main push)"] -.-> trigger

Loading

Prompt To Fix All With AI

### Issue 1
.github/workflows/tests.yml:15-17
`cancel-in-progress: true` uses `github.ref` as the group key, so on `push` to `main` every new commit cancels the still-running tests for the previous commit. If commit A introduces a regression that is independent of commit B, A's run gets cancelled before it can report the failure — you'd only ever see B's result. Making the flag conditional on the event type keeps the compute-saving behaviour on PRs while letting every main-branch push run to completion.

```suggestion
concurrency:
  group: tests-${{ github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
```

### Issue 2
.github/workflows/tests.yml:41-47
Both jobs install uv via `curl | sh` without pinning a version, so a uv release could silently change behaviour between runs. The `astral-sh/setup-uv` action accepts a `version` pin and verifies a checksum, giving reproducible installs without requiring the action tag to be updated manually. This also avoids the `echo "$HOME/.local/bin" >> "$GITHUB_PATH"` boilerplate. Showing the `vero` job; the same change applies to the `targets` job.

```suggestion
      - name: Install uv
        uses: astral-sh/setup-uv@v6
        with:
          version: "0.7.x"

      - name: Install dependencies
        working-directory: vero
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "ci: run pytest for vero and the six base..." | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

[greptile-apps]

cancel-in-progress: true uses github.ref as the group key, so on push to main every new commit cancels the still-running tests for the previous commit. If commit A introduces a regression that is independent of commit B, A's run gets cancelled before it can report the failure — you'd only ever see B's result. Making the flag conditional on the event type keeps the compute-saving behaviour on PRs while letting every main-branch push run to completion.

Suggested change

	concurrency:
	group: tests-${{ github.ref }}
	cancel-in-progress: true
	concurrency:
	group: tests-${{ github.ref }}
	cancel-in-progress: ${{ github.event_name == 'pull_request' }}

Prompt To Fix With AI

This is a comment left during a code review.
Path: .github/workflows/tests.yml
Line: 15-17

Comment:
`cancel-in-progress: true` uses `github.ref` as the group key, so on `push` to `main` every new commit cancels the still-running tests for the previous commit. If commit A introduces a regression that is independent of commit B, A's run gets cancelled before it can report the failure — you'd only ever see B's result. Making the flag conditional on the event type keeps the compute-saving behaviour on PRs while letting every main-branch push run to completion.

```suggestion
concurrency:
  group: tests-${{ github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.