Skip to content

Commit cf2b4bb

Browse files
Copilotmrjf
andauthored
Add end-to-end install integration test harness
Agent-Logs-Url: https://github.com/githubnext/autoloop/sessions/df881075-c8de-46ce-8314-6945f02f4f05 Co-authored-by: mrjf <180956+mrjf@users.noreply.github.com>
1 parent 2904dae commit cf2b4bb

7 files changed

Lines changed: 791 additions & 0 deletions

File tree

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
name: Install Integration Test
2+
3+
# End-to-end test of install.md against a long-lived target repo.
4+
# Manual-dispatch only -- this exercises real LLM calls and force-pushes a
5+
# remote branch, so it must not run on PRs or schedules.
6+
7+
on:
8+
workflow_dispatch:
9+
inputs:
10+
keep_state_on_failure:
11+
description: "Leave test repo in failure state for inspection"
12+
type: boolean
13+
default: false
14+
install_test_repo:
15+
description: "Target repo for the install (owner/repo)"
16+
type: string
17+
default: "mrjf/autoloop-test"
18+
19+
jobs:
20+
install-integration:
21+
runs-on: ubuntu-latest
22+
timeout-minutes: 30
23+
permissions:
24+
contents: read
25+
steps:
26+
- uses: actions/checkout@v4
27+
28+
- uses: actions/setup-python@v5
29+
with:
30+
python-version: "3.12"
31+
32+
- name: Install gh aw extension
33+
env:
34+
GH_TOKEN: ${{ secrets.INSTALL_TEST_TOKEN }}
35+
run: gh extension install github/gh-aw
36+
37+
- name: Install Copilot CLI
38+
env:
39+
GH_TOKEN: ${{ secrets.INSTALL_TEST_TOKEN }}
40+
# The Copilot CLI is distributed as an npm package. If the install
41+
# path changes upstream, update this single step.
42+
run: npm install -g @github/copilot
43+
44+
- name: Verify gh auth
45+
env:
46+
GH_TOKEN: ${{ secrets.INSTALL_TEST_TOKEN }}
47+
run: gh auth status
48+
49+
- name: Run integration test
50+
env:
51+
GH_TOKEN: ${{ secrets.INSTALL_TEST_TOKEN }}
52+
INSTALL_TEST_REPO: ${{ inputs.install_test_repo }}
53+
KEEP_STATE_ON_FAILURE: ${{ inputs.keep_state_on_failure && '1' || '0' }}
54+
run: ./tests/install-integration/run.sh
Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# install-integration
2+
3+
End-to-end integration test for [`install.md`](../../install.md). Runs the
4+
install flow as a real coding agent (Copilot CLI) against a long-lived
5+
target repo (`mrjf/autoloop-test` by default), then exercises Phase 2 by
6+
running one iteration each of three programs across the program-source ×
7+
strategy matrix.
8+
9+
This test is **manual-dispatch only**. It is not part of CI.
10+
11+
## Local mode
12+
13+
```bash
14+
# from the autoloop repo root:
15+
./tests/install-integration/run.sh
16+
```
17+
18+
Requirements:
19+
20+
- `gh` CLI authenticated as a user with write access to the target repo.
21+
- `copilot` CLI on PATH.
22+
- `python3` and `git` on PATH.
23+
24+
Optional env / flags:
25+
26+
- `INSTALL_TEST_REPO=<owner>/<repo>` -- override the target (default
27+
`mrjf/autoloop-test`).
28+
- `--keep` (or `KEEP_STATE_ON_FAILURE=1`) -- skip teardown on failure so
29+
the failure state can be inspected. Run `teardown.sh <repo> <base-sha>`
30+
manually afterwards.
31+
32+
## Actions mode
33+
34+
Trigger the **Install Integration Test** workflow from the Actions tab. It
35+
runs the same script on a GitHub-hosted runner. Requires the
36+
`INSTALL_TEST_TOKEN` repo secret -- a PAT with `repo` scope on the target
37+
repo (the default `GITHUB_TOKEN` has no access to repos outside the host).
38+
39+
## What it tests
40+
41+
See [the issue that introduced this harness](https://github.com/githubnext/autoloop/issues)
42+
for the full motivation. In short:
43+
44+
- **Phase 1** (file presence + lock idempotency) -- catches regressions in
45+
`install.md` and in `gh aw compile`.
46+
- **Phase 2** (3 programs × 1 iteration each) -- catches regressions in
47+
the scheduler, in strategy discovery, and in the iteration loop. The
48+
three programs cover:
49+
50+
| # | Source | Strategy |
51+
|---|------------|-----------------|
52+
| 1 | file-based | OpenEvolve |
53+
| 2 | issue-based| Test-Driven |
54+
| 3 | file-based | plain (default) |
55+
56+
- **Phase 3** (teardown) -- resets the target repo to the captured base
57+
SHA, closes test issues/PRs, and deletes test branches.
58+
59+
## Files
60+
61+
| File | Purpose |
62+
|-----------------------|--------------------------------------------------|
63+
| `run.sh` | Driver. Orchestrates phases 1-3. |
64+
| `prompt.md` | Prompt fed to Copilot CLI (edit without touching the driver). |
65+
| `verify-phase1.sh` | File-presence + lock-idempotency assertions. |
66+
| `verify-phase2.sh` | Per-program assertions (one call per program). |
67+
| `teardown.sh` | Idempotent cleanup. Safe to re-run. |
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
You are installing autoloop into a freshly-reset GitHub repository.
2+
3+
Your working directory is the root of that repository, cloned locally. The
4+
repository is empty except for the base fixtures in `src/` and `tests/`.
5+
6+
Follow the install instructions at the URL below, EXACTLY AS WRITTEN. Execute
7+
each step using shell commands. Do not skip steps. Do not improvise. Do not
8+
optimize or "improve" the instructions.
9+
10+
Stop after Step 5 (the install PR is opened). Do NOT proceed to Step 6
11+
("Create Your First Program") -- the test harness handles program creation
12+
itself in a deterministic way.
13+
14+
When you finish: print a single line `INSTALL_PR=<url>` with the URL of the
15+
PR you opened in step 5. Then stop.
16+
17+
Install instructions: https://github.com/githubnext/autoloop/blob/main/install.md

0 commit comments

Comments
 (0)