Skip to content

Commit 99aca3b

Browse files
benvinegarbenvinegar
committed
ci: add integration tests on ephemeral DigitalOcean droplets
- bin/ci/droplet.sh: reusable droplet lifecycle (create/destroy/wait-ssh/run) - bin/ci/setup-ubuntu.sh: Ubuntu prereqs + setup.sh + test suite - .github/workflows/integration.yml: matrix-based workflow (Ubuntu now, extensible to Arch etc) - Ephemeral everything: fresh droplet + SSH key per run, destroyed on cleanup - Single secret: DO_API_TOKEN
1 parent 10af489 commit 99aca3b

6 files changed

Lines changed: 380 additions & 54 deletions

File tree

‎.github/workflows/integration.yml‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
name: Integration
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
branches: [main]
8+
workflow_dispatch:
9+
10+
concurrency:
11+
group: integration-${{ github.event.pull_request.number || github.sha }}
12+
cancel-in-progress: true
13+
14+
jobs:
15+
integration:
16+
runs-on: ubuntu-latest
17+
strategy:
18+
fail-fast: false
19+
matrix:
20+
include:
21+
- distro: ubuntu
22+
image: ubuntu-24-04-x64
23+
setup_script: bin/ci/setup-ubuntu.sh
24+
# To add Arch (or other distros), add entries here:
25+
# - distro: arch
26+
# image: arch-linux-x64 # or a custom snapshot ID
27+
# setup_script: bin/ci/setup-arch.sh
28+
29+
name: ${{ matrix.distro }}
30+
timeout-minutes: 10
31+
32+
steps:
33+
- uses: actions/checkout@v4
34+
35+
- name: Generate ephemeral SSH key
36+
run: |
37+
mkdir -p ~/.ssh
38+
ssh-keygen -t ed25519 -f ~/.ssh/ci_key -N "" -q
39+
40+
- name: Create droplet
41+
id: droplet
42+
env:
43+
DO_API_TOKEN: ${{ secrets.DO_API_TOKEN }}
44+
run: |
45+
output=$(bash bin/ci/droplet.sh create \
46+
"ci-${{ matrix.distro }}-${{ github.run_id }}" \
47+
"${{ matrix.image }}" \
48+
~/.ssh/ci_key.pub)
49+
echo "$output" >> "$GITHUB_OUTPUT"
50+
echo "$output"
51+
52+
- name: Wait for SSH
53+
env:
54+
DO_API_TOKEN: ${{ secrets.DO_API_TOKEN }}
55+
run: |
56+
bash bin/ci/droplet.sh wait-ssh \
57+
"${{ steps.droplet.outputs.DROPLET_IP }}" \
58+
~/.ssh/ci_key
59+
60+
- name: Upload source
61+
run: |
62+
tar czf /tmp/hornet-src.tar.gz \
63+
--exclude=node_modules --exclude=.git .
64+
scp -o StrictHostKeyChecking=no -o BatchMode=yes \
65+
-i ~/.ssh/ci_key \
66+
/tmp/hornet-src.tar.gz \
67+
"root@${{ steps.droplet.outputs.DROPLET_IP }}:/tmp/hornet-src.tar.gz"
68+
69+
- name: Setup and test
70+
run: |
71+
bash bin/ci/droplet.sh run \
72+
"${{ steps.droplet.outputs.DROPLET_IP }}" \
73+
~/.ssh/ci_key \
74+
"${{ matrix.setup_script }}"
75+
76+
- name: Cleanup
77+
if: always()
78+
env:
79+
DO_API_TOKEN: ${{ secrets.DO_API_TOKEN }}
80+
run: |
81+
bash bin/ci/droplet.sh destroy \
82+
"${{ steps.droplet.outputs.DROPLET_ID }}" \
83+
"${{ steps.droplet.outputs.SSH_KEY_ID }}"

‎.pi/todos/20e26efc.md‎

Lines changed: 62 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -1,60 +1,92 @@
11
{
22
"id": "20e26efc",
3-
"title": "CI job: run setup + tests on Ubuntu droplet per PR",
3+
"title": "CI job: run setup + tests on fresh Ubuntu droplet per PR",
44
"tags": [
55
"infra",
66
"ci",
77
"ubuntu"
88
],
99
"status": "todo",
10-
"created_at": "2026-02-17T02:30:52.375Z"
10+
"created_at": "2026-02-17T02:30:52.375Z",
11+
"assigned_to_session": "381813d9-c69a-4472-9a00-e232ffb746d1"
1112
}
1213

1314
## Goal
14-
Add a GitHub Actions workflow that SSHes into the DigitalOcean droplet and runs the full hornet setup + test suite on every PR.
15+
Add a GitHub Actions workflow that spins up a fresh DigitalOcean droplet, runs the full hornet setup + test suite, and destroys it — on every PR.
1516

16-
## Depends on
17-
- TODO-cb931656 (manual verification must pass first)
17+
## Verified approach
18+
Tested manually — full destroy/create/setup/test cycle works end-to-end in ~2 minutes.
1819

19-
## Design decisions needed
20-
- **Fresh state per run**: `uninstall.sh` at start of each run? Or snapshot/restore? Or ephemeral droplet via DO API?
21-
- **Secrets**: droplet IP + SSH key stored as GitHub Actions secrets (`DROPLET_IP`, `DROPLET_SSH_KEY`)
22-
- **Concurrency**: only one CI run at a time on the droplet (use GitHub concurrency group)
23-
- **Scope**: full setup + test, or just test.sh (setup is slow, ~2-3 min)?
20+
| Step | Method | Time |
21+
|------|--------|------|
22+
| Fresh env | Create droplet via DO API | ~45s |
23+
| Wait for SSH | Poll until port 22 responds | ~15s |
24+
| Prereqs | `apt-get install git curl tmux iptables docker.io` (wait for unattended-upgrades lock first) | ~20s |
25+
| Source | `tar + scp + extract + git init` | ~5s |
26+
| Setup | `setup.sh hornet_admin` | ~60s |
27+
| Tests | `npm install + bin/test.sh` | ~15s |
28+
| Cleanup | Destroy droplet + delete SSH key from DO (always, even on failure) | instant |
2429

25-
## Proposed workflow
30+
## GitHub Actions secrets needed
31+
- `DO_API_TOKEN` — DigitalOcean API token
32+
33+
That's it. SSH keys are **ephemeral per run**:
34+
1. `ssh-keygen` a throwaway ed25519 keypair on the runner
35+
2. Register pubkey with DO API → get `ssh_key_id`
36+
3. Create droplet with that `ssh_key_id`
37+
4. Use the private key to SSH in
38+
5. Cleanup: destroy droplet AND delete SSH key from DO account
39+
40+
No persistent SSH keys stored anywhere.
41+
42+
## Design
43+
- **Ephemeral everything**: fresh droplet, fresh SSH key, destroyed after. Zero state between runs.
44+
- **Concurrency group**: only one integration run at a time (avoid parallel droplets piling up).
45+
- **Always cleanup**: use `if: always()` so droplet + SSH key are destroyed even if tests fail.
46+
- **Region/size**: `tor1`, `s-2vcpu-4gb` (~$0.003/run at ~2 min).
47+
- **Unattended-upgrades**: fresh Ubuntu runs apt on first boot — CI must wait for the dpkg lock.
48+
49+
## Steps
50+
1. Add `DO_API_TOKEN` secret to GitHub repo via `gh secret set`
51+
2. Write `.github/workflows/integration.yml`
52+
3. Test on a real PR
53+
4. Optionally add status badge to README
54+
55+
## Workflow skeleton
2656
```yaml
2757
name: Integration (Ubuntu)
2858
on:
2959
pull_request:
3060
branches: [main]
3161

3262
concurrency:
33-
group: droplet-integration
63+
group: integration-ubuntu
3464
cancel-in-progress: true
3565

3666
jobs:
3767
integration:
3868
runs-on: ubuntu-latest
3969
steps:
4070
- uses: actions/checkout@v4
41-
- name: Run on droplet
42-
env:
43-
DROPLET_IP: ${{ secrets.DROPLET_IP }}
44-
SSH_KEY: ${{ secrets.DROPLET_SSH_KEY }}
71+
- name: Generate ephemeral SSH key
72+
run: |
73+
ssh-keygen -t ed25519 -f ~/.ssh/ci_key -N "" -q
74+
# Register with DO, save key ID
75+
- name: Create droplet
76+
# POST /v2/droplets with ssh_key_id, poll until active, extract IP
77+
- name: Wait for SSH
78+
# Loop ssh -o ConnectTimeout=5 until success
79+
- name: Install prereqs
80+
# Wait for apt lock, then apt-get install
81+
- name: Upload source
82+
# tar + scp
83+
- name: Run setup
84+
# setup.sh hornet_admin
85+
- name: Run tests
86+
# npm install + bin/test.sh
87+
- name: Cleanup
88+
if: always()
4589
run: |
46-
# SSH into droplet, rsync repo, run uninstall (clean slate),
47-
# run setup.sh, deploy.sh, test.sh, security-audit.sh
90+
# DELETE /v2/droplets/$DROPLET_ID
91+
# DELETE /v2/account/keys/$SSH_KEY_ID
4892
```
49-
50-
## Steps
51-
1. Create SSH key pair for CI, add public key to droplet
52-
2. Add `DROPLET_IP` and `DROPLET_SSH_KEY` as GitHub repo secrets
53-
3. Write the workflow file (`.github/workflows/integration.yml`)
54-
4. Handle cleanup: uninstall.sh at start of run for clean state
55-
5. Fail the PR if any step exits non-zero
56-
6. Consider: also run security-audit.sh (some checks need live system)
57-
58-
## Open questions
59-
- Do we want to spin up/destroy droplets per run (more isolated, costs more) or reuse one?
60-
- Should we test `start.sh` actually booting an agent, or just setup + unit tests?

‎.pi/todos/cb931656.md‎

Lines changed: 19 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -6,32 +6,27 @@
66
"ubuntu",
77
"ci"
88
],
9-
"status": "todo",
10-
"created_at": "2026-02-17T02:30:39.055Z",
11-
"assigned_to_session": "381813d9-c69a-4472-9a00-e232ffb746d1"
9+
"status": "done",
10+
"created_at": "2026-02-17T02:30:39.055Z"
1211
}
1312

14-
## Goal
15-
SSH into the DigitalOcean Ubuntu droplet and manually verify the full hornet setup works end-to-end.
13+
## Result
14+
Verified on DigitalOcean Ubuntu 24.04 droplet (4GB RAM, 2 vCPU).
1615

17-
## Steps
18-
1. SSH into the box as root
19-
2. Install prerequisites: `git`, `curl`, `docker`, `iptables`, `tmux`
20-
3. Clone the hornet repo
21-
4. Run `setup.sh <admin_user>` — creates `hornet_agent` user, installs Node, pi, firewall, etc.
22-
5. Create a minimal `.env` with dummy/test values (enough to pass varlock validation)
23-
6. Run `bin/deploy.sh` — deploy extensions, skills, bridge to runtime
24-
7. Run `bin/test.sh` — all 207 tests must pass
25-
8. Run `bin/security-audit.sh` — verify firewall, perms, proc isolation
26-
9. Boot the agent: `sudo -u hornet_agent ~/runtime/start.sh` — verify it starts without errors
27-
10. Tear down: run `bin/uninstall.sh` to verify clean removal
16+
### Bugs found and fixed
17+
1. **`setup.sh` — shared repo permissions**: ran `git config` as `hornet_agent` on admin's repo → agent can't access admin home. Fix: run as `$ADMIN_USER` instead.
18+
2. **`setup.sh` — harden-permissions path**: called source copy (`$REPO_DIR/bin/harden-permissions.sh`) which agent can't access. Fix: use deployed copy (`$HORNET_HOME/runtime/bin/`).
19+
3. **`setup.sh` — CWD inheritance**: `sudo -u hornet_agent` inherits root's CWD, causing `find`/`git` failures. Fix: `cd /tmp` at top of script.
20+
4. **`harden-permissions.sh` — sessions dir**: `find` on non-existent sessions dir fails under `set -e`. Fix: wrap in `if [ -d ... ]` guard.
21+
5. **`hornet-safe-bash` — `grep -P`**: Perl regex not reliably available on Ubuntu. Fix: all patterns converted to `grep -E`.
2822

29-
## Success criteria
30-
- `setup.sh` completes without errors on Ubuntu
31-
- All tests pass
32-
- Security audit is clean
33-
- Agent boots and varlock validates the env
23+
### Verification results
24+
- ✅ `setup.sh` completes cleanly (user, Node, pi, firewall, hidepid, deploy, harden)
25+
- ✅ All 5 test suites pass (tool-guard, bridge security, extension scanner, safe-bash, log redaction)
26+
- ✅ Varlock validates env schema
27+
- ✅ `start.sh` boots pi agent (fails at API auth with dummy keys — expected)
28+
- ✅ Firewall active with all rules
29+
- ✅ Process isolation working
3430

35-
## Notes
36-
- Need droplet IP, root credentials (store securely, don't commit)
37-
- This is a one-time manual run; the CI todo automates it afterward
31+
### PR
32+
https://github.com/modem-dev/hornet/pull/10

‎AGENTS.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,9 @@ bin/ security & operations scripts
1414
harden-permissions.sh filesystem hardening (runs on boot)
1515
scan-extensions.mjs extension static analysis
1616
redact-logs.sh secret scrubber for session logs
17+
ci/ CI integration scripts
18+
droplet.sh ephemeral DigitalOcean droplet lifecycle (create/destroy/ssh)
19+
setup-ubuntu.sh Ubuntu droplet: prereqs + setup + tests
1720
hooks/
1821
pre-commit blocks agent from modifying security files in git
1922
pi/

0 commit comments

Comments
 (0)