Skip to content

ci: run pytest for vero and the six baseline agents #63

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
shehabyasser-scale wants to merge 1 commit into
base: pr3-harness-bench
Choose a base branch
from
+89 −0
Open

ci: run pytest for vero and the six baseline agents #63

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 89 additions & 0 deletions .github/workflows/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
name: tests

# Nothing in this repository ran tests before this workflow: the only checks on a
# pull request were Greptile Review and Socket Security, neither of which invokes
# pytest. PR #61 sat open with a red target suite as a result.

on:
pull_request:
push:
branches: [main]

permissions:
contents: read

concurrency:
group: tests-${{ github.ref }}
cancel-in-progress: true
Comment on lines +15 to +17

@greptile-apps greptile-apps Bot Jul 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 cancel-in-progress: true uses github.ref as the group key, so on push to main every new commit cancels the still-running tests for the previous commit. If commit A introduces a regression that is independent of commit B, A's run gets cancelled before it can report the failure — you'd only ever see B's result. Making the flag conditional on the event type keeps the compute-saving behaviour on PRs while letting every main-branch push run to completion.

Suggested change
concurrency:
group: tests-${{ github.ref }}
cancel-in-progress: true
concurrency:
group: tests-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
Prompt To Fix With AI 
This is a comment left during a code review.
Path: .github/workflows/tests.yml
Line: 15-17

Comment:
`cancel-in-progress: true` uses `github.ref` as the group key, so on `push` to `main` every new commit cancels the still-running tests for the previous commit. If commit A introduces a regression that is independent of commit B, A's run gets cancelled before it can report the failure — you'd only ever see B's result. Making the flag conditional on the event type keeps the compute-saving behaviour on PRs while letting every main-branch push run to completion.

```suggestion
concurrency:
  group: tests-${{ github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Cursor Fix in Claude Code Fix in Codex


env:
# Every package here resolves from public PyPI (vero/pyproject.toml pins that
# index explicitly), so no private-registry credentials are needed. Pinned so a
# runner image bump cannot silently change the interpreter: vero requires
# >=3.11,<3.14 and the six targets require >=3.12.
UV_PYTHON: "3.12"

jobs:
vero:
name: vero
runs-on: ubuntu-latest
timeout-minutes: 25
env:
# The task compiler refuses to emit a task whose declared credentials are
# absent ("declared task credentials are missing", compiler.py), so the
# build tests need these present. Placeholders only: nothing dials them,
# and port 1 fails fast instead of hanging if anything ever tries.
OPENAI_API_KEY: ci-not-a-real-key
OPENAI_BASE_URL: http://127.0.0.1:1/v1
steps:
- uses: actions/checkout@v4

- name: Install uv
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
echo "$HOME/.local/bin" >> "$GITHUB_PATH"

- name: Install dependencies
working-directory: vero
# --locked fails rather than relocking, so a pyproject edit that forgets
# to update uv.lock is a visible error instead of a silent resolve.
run: uv sync --locked

- name: Run tests
working-directory: vero
run: uv run pytest -q

targets:
# The six editable baseline agents. Each is its own uv project with its own
# lock, so they get one job apiece rather than a shared environment.
name: ${{ matrix.benchmark }}
runs-on: ubuntu-latest
timeout-minutes: 15
strategy:
# One broken agent should not mask the state of the other five.
fail-fast: false
matrix:
benchmark:
- browsecomp-plus
- gaia
- officeqa
- swe-atlas-qna
- swe-bench-pro
- tau3
steps:
# No `submodules: recursive`: the only submodule is the BrowseComp-Plus
# upstream corpus, which no test reads.
- uses: actions/checkout@v4

- name: Install uv
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
echo "$HOME/.local/bin" >> "$GITHUB_PATH"

- name: Install dependencies
working-directory: harness-engineering-bench/${{ matrix.benchmark }}/baseline/target
run: uv sync --locked

- name: Run tests
working-directory: harness-engineering-bench/${{ matrix.benchmark }}/baseline/target
run: uv run pytest -q
Loading