Skip to content

Commit e56f7a9

Browse files
EnderOfWorlds007EnderOfWorlds007
authored
docs(e2e): bootstrap.md + cleanup-cron stub (Phase 8) (#103)
1 parent 635ca56 commit e56f7a9

4 files changed

Lines changed: 320 additions & 1 deletion

File tree

‎.changeset/e2e-phase-8-bootstrap.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"playground-cli": patch
3+
---
4+
5+
Adds `docs/e2e-bootstrap.md` (public maintainer-facing doc covering pre-conditions, idempotent bootstrap commands, and recovery procedures for the E2E suite) and `.github/workflows/e2e-cleanup.yml` (Sunday 04:00 UTC cron stub for sweeping rotating E2E state — actual sweep logic lands with Phase 5e).

‎.github/workflows/e2e-cleanup.yml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
name: E2E Cleanup
2+
3+
on:
4+
schedule:
5+
- cron: "0 4 * * 0" # Sunday 04:00 UTC, after the nightly window
6+
workflow_dispatch:
7+
8+
permissions:
9+
contents: read
10+
issues: write # for filing if cleanup fails
11+
12+
jobs:
13+
sweep:
14+
name: Sweep rotating E2E state
15+
runs-on: ubuntu-latest
16+
timeout-minutes: 10
17+
steps:
18+
- uses: actions/checkout@v4
19+
20+
- name: Sweep rotating modable repos and domains
21+
shell: bash
22+
env:
23+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
24+
run: |
25+
echo "::group::What this workflow sweeps"
26+
echo "Per spec §9c, this cron sweeps rotating per-run E2E state:"
27+
echo " - GH repos matching 'e2e-cli-modable-*' older than 14 days"
28+
echo " - Registry domains matching 'e2e-cli-modable-*' older than 14 days"
29+
echo ""
30+
echo "Phase 5e (modable) hasn't shipped yet, so there's nothing to sweep today."
31+
echo "When Phase 5e lands, this step gets the actual sweep logic:"
32+
echo " gh repo list --topic e2e-test-fixture --limit 100 ..."
33+
echo " bun tools/sweep-modable-domains.ts (would be added then)"
34+
echo "::endgroup::"
35+
36+
# Stub — exit 0. Replace with real sweep when Phase 5e adds rotating state.
37+
echo "no rotating state to sweep — Phase 5e not yet shipped"

‎CLAUDE.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,5 +63,6 @@ These are things that aren't self-evident from reading the code and have bitten
6363
- **Reports directory:** `e2e-reports/junit.xml` + `e2e-reports/dot-runs.log` (gitignored).
6464
- **Tag prefix:** `DOT_TAG=e2e-{ci|local}-{trigger}` so Sentry dashboards filter test traffic. The CLI plumbs `DOT_TAG` into the `cli.tag` root-span attribute via `src/telemetry-config.ts`.
6565
- **CI report job name:** `E2E Report` — aggregates per-leg conclusions, posts a sticky PR comment with marker `<!-- e2e-pr-report -->`, opens an auto-issue on schedule/release fail.
66-
- **Bootstrap:** see `tools/register-e2e-fixtures.ts` for the mod-test fixture; full bootstrap doc TBD in a later phase.
66+
- **Bootstrap:** see `docs/e2e-bootstrap.md` for the maintainer-facing setup + recovery procedures. The tool itself is `tools/register-e2e-fixtures.ts`.
67+
- **Cleanup cron:** `.github/workflows/e2e-cleanup.yml` runs Sunday 04:00 UTC. Stub today; will sweep rotating modable state when Phase 5e ships.
6768
- **Design spec:** `docs-internal/2026-05-02-e2e-test-suite-design.md`.

‎docs/e2e-bootstrap.md‎

Lines changed: 276 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,276 @@
1+
# E2E Bootstrap
2+
3+
Maintainer runbook for setting up and recovering the E2E test suite against Paseo testnet.
4+
Run these steps once per registry-contract lifetime (not per PR).
5+
6+
---
7+
8+
## 1. Pre-conditions
9+
10+
### SIGNER account
11+
12+
The E2E deployer is derived from the canonical dev mnemonic:
13+
14+
```
15+
mnemonic: bottom drive obey lake curtain smoke basket hold race lonely fit walk
16+
derivation: //e2e-deployer
17+
```
18+
19+
Source: `e2e/cli/fixtures/accounts.ts::DEDICATED_E2E_DEPLOYER_MNEMONIC`.
20+
21+
SIGNER must hold enough PAS to cover `registry.publish()` fees.
22+
`globalSetup` auto-tops-up from the CLI funder chain before every test run (`e2e/cli/setup/fund.ts`);
23+
the bootstrap tool (`tools/register-e2e-fixtures.ts`) does the same before registering fixtures.
24+
Minimum held: 500 PAS. Top-up amount: 1000 PAS. Cost per publish: ~0.1 PAS.
25+
26+
### Permanent fixture domains
27+
28+
Two categories. All owned by SIGNER after bootstrap.
29+
30+
**Tool-managed (explicit bootstrap required):**
31+
32+
| Domain | Purpose |
33+
|---|---|
34+
| `dot-cli-mod-fixture.dot` | `dot mod` fixture — has `repository` metadata pointing at Rock-Paper-Scissors |
35+
| `e2e-cli-foundry` | `pr-deploy-foundry` / `nightly-deploy-foundry` cell |
36+
| `e2e-cli-cdm` | `pr-deploy-cdm` cell (currently skipped pending fixture upgrade) |
37+
| `e2e-cli-hardhat` | `nightly-deploy-hardhat` cell |
38+
| `e2e-cli-multi` | `nightly-deploy-multi` cell |
39+
40+
These 5 domains are registered by `tools/register-e2e-fixtures.ts`.
41+
Same-owner re-publish updates metadata in place — idempotent.
42+
43+
**Test-managed (no explicit bootstrap step):**
44+
45+
| Domain | Purpose |
46+
|---|---|
47+
| `e2e-cli-preflight` | Preflight/validation tests (shared by six tests) |
48+
| `e2e-cli-storage` | Storage-phase happy-path test |
49+
| `e2e-cli-redeploy` | Same-owner re-deploy test |
50+
| `e2e-cli-collision` | Cross-owner collision test |
51+
52+
These 4 domains are registered organically on the first CI run that reaches `registry.publish()`.
53+
Subsequent runs same-owner re-publish them. No manual step needed.
54+
After a registry-contract redeploy, the next CI run restores them automatically.
55+
56+
### GitHub secrets
57+
58+
| Secret | Required for | Status |
59+
|---|---|---|
60+
| `MASTER_FUNDER_SEED` | SIGNER top-up in globalSetup | Required — must be set |
61+
| `SENTRY_AUTH_TOKEN` | Telemetry verification (not yet implemented) | Optional |
62+
63+
`GITHUB_TOKEN` (auto-provided) is used by the report job and cleanup cron.
64+
65+
---
66+
67+
## 2. Idempotent bootstrap commands
68+
69+
Run these once per registry-contract lifetime (or to recover — see §3).
70+
71+
```bash
72+
# Step 1: verify SIGNER has sufficient balance
73+
bun tools/probe-registry-resolution.ts e2e-cli-foundry.dot
74+
# If this 404s cleanly → SIGNER is not yet funded or fixtures not registered.
75+
# If this returns a valid CID → SIGNER already owns this domain (re-run is safe).
76+
77+
# Step 2: register (or re-register) all 5 tool-managed fixture domains
78+
bun tools/register-e2e-fixtures.ts
79+
# Output: signer balance, top-up if needed, then publishes each domain.
80+
81+
# Step 3: verify each tool-managed fixture
82+
for d in dot-cli-mod-fixture e2e-cli-foundry e2e-cli-cdm e2e-cli-hardhat e2e-cli-multi; do
83+
bun tools/probe-registry-resolution.ts "$d.dot"
84+
done
85+
# Each line should print a resolved CID, not an error.
86+
```
87+
88+
To register a single fixture (e.g. after adding a new cell):
89+
90+
```bash
91+
bun tools/register-e2e-fixtures.ts --domain e2e-cli-foundry
92+
```
93+
94+
To register with a custom signer (e.g. a staging key):
95+
96+
```bash
97+
bun tools/register-e2e-fixtures.ts --suri "//MyStagingSigner"
98+
```
99+
100+
---
101+
102+
## 3. Recovery procedures
103+
104+
### SIGNER is drained
105+
106+
Nothing to do manually. `globalSetup` (`e2e/cli/setup/fund.ts`) and
107+
`tools/register-e2e-fixtures.ts` both auto-top-up when free balance is below
108+
500 PAS. The top-up source is `MASTER_FUNDER_SEED` (GitHub secret) or the
109+
CLI's built-in funder derivation locally.
110+
111+
If `MASTER_FUNDER_SEED` itself is empty, re-fund the parent funder account on
112+
Paseo and update the secret.
113+
114+
### Registry contract redeployed (all domains wiped)
115+
116+
This happened in PR #78. Recovery:
117+
118+
```bash
119+
# Re-register the 5 tool-managed fixtures in one shot:
120+
bun tools/register-e2e-fixtures.ts
121+
122+
# The 4 test-managed domains (preflight/storage/redeploy/collision)
123+
# are restored automatically on the next CI run that runs those tests.
124+
# No manual step needed for those.
125+
```
126+
127+
### A specific tool-managed fixture is missing or has wrong metadata
128+
129+
```bash
130+
bun tools/register-e2e-fixtures.ts --domain <name>
131+
# e.g.:
132+
bun tools/register-e2e-fixtures.ts --domain dot-cli-mod-fixture.dot
133+
bun tools/register-e2e-fixtures.ts --domain e2e-cli-foundry
134+
```
135+
136+
### `dot-cli-mod-fixture.dot` repository URL changed
137+
138+
Edit `FIXTURES` in `tools/register-e2e-fixtures.ts` (the `repositoryUrl` field)
139+
and re-run:
140+
141+
```bash
142+
bun tools/register-e2e-fixtures.ts --domain dot-cli-mod-fixture.dot
143+
```
144+
145+
Also update `TEST_TEMPLATE_REPO` in `.github/workflows/e2e.yml` and `e2e/cli/helpers/dot.ts`.
146+
147+
### E2E is gating a PR but the check is missing / failed for a non-code reason
148+
149+
**Do not bypass lightly.** Emergency admin-only procedure (auditable via org audit log):
150+
151+
```bash
152+
# 1. Snapshot current required checks
153+
gh api repos/paritytech/playground-cli/branches/main/protection \
154+
--jq '.required_status_checks.contexts' > /tmp/required-checks-before.txt
155+
156+
# 2. Temporarily remove "E2E Report" (keep all other required checks from step 1)
157+
gh api -X PATCH repos/paritytech/playground-cli/branches/main/protection/required_status_checks \
158+
-f 'contexts[]=other-check-1' # fill in from step 1 output
159+
160+
# 3. Merge the PR.
161+
162+
# 4. Restore "E2E Report" immediately.
163+
gh api -X PATCH repos/paritytech/playground-cli/branches/main/protection/required_status_checks \
164+
-f 'contexts[]=other-check-1' \
165+
-f 'contexts[]=E2E Report'
166+
```
167+
168+
Bypass reasons and preferred handling:
169+
170+
| Situation | Action |
171+
|---|---|
172+
| One cell flaked / timed out | Re-run the failed cell from the Actions UI |
173+
| Whole workflow errored | Re-run the entire workflow |
174+
| Real product regression | Do not bypass — fix it |
175+
| Testnet outage (Paseo infra down) | Use admin bypass above + comment linking the testnet incident |
176+
| Renamed check (e.g. "E2E Report" → new name) | Remove stale rule via gh-api; do not merge broken code |
177+
178+
---
179+
180+
## 4. Adding a new fixture domain
181+
182+
1. Add an entry to `FIXTURES` in `tools/register-e2e-fixtures.ts`.
183+
2. Add the corresponding key to `E2E_DOMAINS` in `e2e/cli/fixtures/accounts.ts`.
184+
3. Run `bun tools/register-e2e-fixtures.ts --domain <new-domain>` once.
185+
4. Add the domain to the verify loop in §2 above.
186+
187+
---
188+
189+
## 5. Rotating/permanent state and cleanup
190+
191+
### Permanent fixtures (never swept)
192+
193+
The 9 domains listed in §1 are permanent. Same-owner re-publish by the tool
194+
or by test runs keeps their metadata current without accumulating new registry
195+
entries.
196+
197+
### Per-run rotating state (Phase 5e, not yet shipped)
198+
199+
When Phase 5e (`nightly-deploy-modable`) ships, each run will create:
200+
- A `e2e-cli-modable-<runId>` registry domain
201+
- A GitHub repo tagged `e2e-test-fixture`
202+
203+
These are swept by `.github/workflows/e2e-cleanup.yml` (Sunday 04:00 UTC)
204+
after 14 days. No manual cleanup needed once that phase ships.
205+
206+
---
207+
208+
## 6. Sentry dashboards
209+
210+
Setup is manual, one-time. Both dashboards already exist as of 2026-05-02.
211+
212+
| Dashboard | ID | Filter | Purpose |
213+
|---|---|---|---|
214+
| Playground CLI Health | 2143100 | `!cli.tag:e2e-*` | Production signal — excludes E2E noise |
215+
| Playground CLI E2E Health | 2216096 | `cli.tag:e2e-*` | E2E-only signal |
216+
217+
Dashboard JSON snapshots live in `sentry/dashboards/`. To view traces for a
218+
specific CI run, use the Sentry link in the `E2E Report` PR comment or step
219+
summary — it deep-links to the `cli.tag` value for that run.
220+
221+
No further setup required unless dashboards are reset or the Sentry project
222+
is rotated. If you need to recreate them, see `sentry/` tooling and the
223+
`## Sentry telemetry` section of `CLAUDE.md`.
224+
225+
---
226+
227+
## 7. Branch protection: adding `E2E Report` as a required check
228+
229+
Run once after the repo is freshly created or protection rules are reset:
230+
231+
```bash
232+
# Read existing required checks first
233+
gh api repos/paritytech/playground-cli/branches/main/protection \
234+
--jq '.required_status_checks.contexts'
235+
236+
# Add "E2E Report" to the list (include all existing checks + the new one)
237+
gh api -X PUT repos/paritytech/playground-cli/branches/main/protection \
238+
--input - <<'EOF'
239+
{
240+
"required_status_checks": {
241+
"strict": false,
242+
"contexts": ["E2E Report"]
243+
},
244+
"enforce_admins": false,
245+
"required_pull_request_reviews": null,
246+
"restrictions": null
247+
}
248+
EOF
249+
```
250+
251+
**Renaming the check later** (e.g. from "E2E Report" to something else) is a
252+
three-step dance to avoid deadlocking the rename PR:
253+
254+
1. Add the new name to required checks.
255+
2. Merge the PR that renames the job.
256+
3. Remove the old name from required checks.
257+
258+
Never remove the old name before the rename PR merges — that leaves the branch
259+
unprotected during the window.
260+
261+
---
262+
263+
## 8. Who has admin
264+
265+
```bash
266+
# Org owners (admin on every paritytech repo)
267+
gh api 'orgs/paritytech/members?role=admin' --jq '.[].login'
268+
269+
# Direct-collaborator admins on this repo
270+
gh api repos/paritytech/playground-cli/collaborators \
271+
--jq '[.[] | select(.permissions.admin == true) | .login]'
272+
273+
# Team-based grants (requires org:read scope)
274+
gh api repos/paritytech/playground-cli/teams \
275+
--jq '[.[] | select(.permission == "admin") | .name]'
276+
```

0 commit comments

Comments
 (0)