Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/e2e-phase-8-bootstrap.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"playground-cli": patch
---

Adds `docs/e2e-bootstrap.md` (public maintainer-facing doc covering pre-conditions, idempotent bootstrap commands, and recovery procedures for the E2E suite) and `.github/workflows/e2e-cleanup.yml` (Sunday 04:00 UTC cron stub for sweeping rotating E2E state — actual sweep logic lands with Phase 5e).
37 changes: 37 additions & 0 deletions .github/workflows/e2e-cleanup.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
name: E2E Cleanup

on:
schedule:
- cron: "0 4 * * 0" # Sunday 04:00 UTC, after the nightly window
workflow_dispatch:

permissions:
contents: read
issues: write # for filing if cleanup fails

jobs:
sweep:
name: Sweep rotating E2E state
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4

- name: Sweep rotating modable repos and domains
shell: bash
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
echo "::group::What this workflow sweeps"
echo "Per spec §9c, this cron sweeps rotating per-run E2E state:"
echo " - GH repos matching 'e2e-cli-modable-*' older than 14 days"
echo " - Registry domains matching 'e2e-cli-modable-*' older than 14 days"
echo ""
echo "Phase 5e (modable) hasn't shipped yet, so there's nothing to sweep today."
echo "When Phase 5e lands, this step gets the actual sweep logic:"
echo " gh repo list --topic e2e-test-fixture --limit 100 ..."
echo " bun tools/sweep-modable-domains.ts (would be added then)"
echo "::endgroup::"

# Stub — exit 0. Replace with real sweep when Phase 5e adds rotating state.
echo "no rotating state to sweep — Phase 5e not yet shipped"
3 changes: 2 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,5 +62,6 @@ These are things that aren't self-evident from reading the code and have bitten
- **Reports directory:** `e2e-reports/junit.xml` + `e2e-reports/dot-runs.log` (gitignored).
- **Tag prefix:** `DOT_TAG=e2e-{ci|local}-{trigger}` so Sentry dashboards filter test traffic. The CLI plumbs `DOT_TAG` into the `cli.tag` root-span attribute via `src/telemetry-config.ts`.
- **CI report job name:** `E2E Report` — aggregates per-leg conclusions, posts a sticky PR comment with marker `<!-- e2e-pr-report -->`, opens an auto-issue on schedule/release fail.
- **Bootstrap:** see `tools/register-e2e-fixtures.ts` for the mod-test fixture; full bootstrap doc TBD in a later phase.
- **Bootstrap:** see `docs/e2e-bootstrap.md` for the maintainer-facing setup + recovery procedures. The tool itself is `tools/register-e2e-fixtures.ts`.
- **Cleanup cron:** `.github/workflows/e2e-cleanup.yml` runs Sunday 04:00 UTC. Stub today; will sweep rotating modable state when Phase 5e ships.
- **Design spec:** `docs-internal/2026-05-02-e2e-test-suite-design.md`.
276 changes: 276 additions & 0 deletions docs/e2e-bootstrap.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,276 @@
# E2E Bootstrap

Maintainer runbook for setting up and recovering the E2E test suite against Paseo testnet.
Run these steps once per registry-contract lifetime (not per PR).

---

## 1. Pre-conditions

### SIGNER account

The E2E deployer is derived from the canonical dev mnemonic:

```
mnemonic: bottom drive obey lake curtain smoke basket hold race lonely fit walk
derivation: //e2e-deployer
```

Source: `e2e/cli/fixtures/accounts.ts::DEDICATED_E2E_DEPLOYER_MNEMONIC`.

SIGNER must hold enough PAS to cover `registry.publish()` fees.
`globalSetup` auto-tops-up from the CLI funder chain before every test run (`e2e/cli/setup/fund.ts`);
the bootstrap tool (`tools/register-e2e-fixtures.ts`) does the same before registering fixtures.
Minimum held: 500 PAS. Top-up amount: 1000 PAS. Cost per publish: ~0.1 PAS.

### Permanent fixture domains

Two categories. All owned by SIGNER after bootstrap.

**Tool-managed (explicit bootstrap required):**

| Domain | Purpose |
|---|---|
| `dot-cli-mod-fixture.dot` | `dot mod` fixture — has `repository` metadata pointing at Rock-Paper-Scissors |
| `e2e-cli-foundry` | `pr-deploy-foundry` / `nightly-deploy-foundry` cell |
| `e2e-cli-cdm` | `pr-deploy-cdm` cell (currently skipped pending fixture upgrade) |
| `e2e-cli-hardhat` | `nightly-deploy-hardhat` cell |
| `e2e-cli-multi` | `nightly-deploy-multi` cell |

These 5 domains are registered by `tools/register-e2e-fixtures.ts`.
Same-owner re-publish updates metadata in place — idempotent.

**Test-managed (no explicit bootstrap step):**

| Domain | Purpose |
|---|---|
| `e2e-cli-preflight` | Preflight/validation tests (shared by six tests) |
| `e2e-cli-storage` | Storage-phase happy-path test |
| `e2e-cli-redeploy` | Same-owner re-deploy test |
| `e2e-cli-collision` | Cross-owner collision test |

These 4 domains are registered organically on the first CI run that reaches `registry.publish()`.
Subsequent runs same-owner re-publish them. No manual step needed.
After a registry-contract redeploy, the next CI run restores them automatically.

### GitHub secrets

| Secret | Required for | Status |
|---|---|---|
| `MASTER_FUNDER_SEED` | SIGNER top-up in globalSetup | Required — must be set |
| `SENTRY_AUTH_TOKEN` | Telemetry verification (not yet implemented) | Optional |

`GITHUB_TOKEN` (auto-provided) is used by the report job and cleanup cron.

---

## 2. Idempotent bootstrap commands

Run these once per registry-contract lifetime (or to recover — see §3).

```bash
# Step 1: verify SIGNER has sufficient balance
bun tools/probe-registry-resolution.ts e2e-cli-foundry.dot
# If this 404s cleanly → SIGNER is not yet funded or fixtures not registered.
# If this returns a valid CID → SIGNER already owns this domain (re-run is safe).

# Step 2: register (or re-register) all 5 tool-managed fixture domains
bun tools/register-e2e-fixtures.ts
# Output: signer balance, top-up if needed, then publishes each domain.

# Step 3: verify each tool-managed fixture
for d in dot-cli-mod-fixture e2e-cli-foundry e2e-cli-cdm e2e-cli-hardhat e2e-cli-multi; do
bun tools/probe-registry-resolution.ts "$d.dot"
done
# Each line should print a resolved CID, not an error.
```

To register a single fixture (e.g. after adding a new cell):

```bash
bun tools/register-e2e-fixtures.ts --domain e2e-cli-foundry
```

To register with a custom signer (e.g. a staging key):

```bash
bun tools/register-e2e-fixtures.ts --suri "//MyStagingSigner"
```

---

## 3. Recovery procedures

### SIGNER is drained

Nothing to do manually. `globalSetup` (`e2e/cli/setup/fund.ts`) and
`tools/register-e2e-fixtures.ts` both auto-top-up when free balance is below
500 PAS. The top-up source is `MASTER_FUNDER_SEED` (GitHub secret) or the
CLI's built-in funder derivation locally.

If `MASTER_FUNDER_SEED` itself is empty, re-fund the parent funder account on
Paseo and update the secret.

### Registry contract redeployed (all domains wiped)

This happened in PR #78. Recovery:

```bash
# Re-register the 5 tool-managed fixtures in one shot:
bun tools/register-e2e-fixtures.ts

# The 4 test-managed domains (preflight/storage/redeploy/collision)
# are restored automatically on the next CI run that runs those tests.
# No manual step needed for those.
```

### A specific tool-managed fixture is missing or has wrong metadata

```bash
bun tools/register-e2e-fixtures.ts --domain <name>
# e.g.:
bun tools/register-e2e-fixtures.ts --domain dot-cli-mod-fixture.dot
bun tools/register-e2e-fixtures.ts --domain e2e-cli-foundry
```

### `dot-cli-mod-fixture.dot` repository URL changed

Edit `FIXTURES` in `tools/register-e2e-fixtures.ts` (the `repositoryUrl` field)
and re-run:

```bash
bun tools/register-e2e-fixtures.ts --domain dot-cli-mod-fixture.dot
```

Also update `TEST_TEMPLATE_REPO` in `.github/workflows/e2e.yml` and `e2e/cli/helpers/dot.ts`.

### E2E is gating a PR but the check is missing / failed for a non-code reason

**Do not bypass lightly.** Emergency admin-only procedure (auditable via org audit log):

```bash
# 1. Snapshot current required checks
gh api repos/paritytech/playground-cli/branches/main/protection \
--jq '.required_status_checks.contexts' > /tmp/required-checks-before.txt

# 2. Temporarily remove "E2E Report" (keep all other required checks from step 1)
gh api -X PATCH repos/paritytech/playground-cli/branches/main/protection/required_status_checks \
-f 'contexts[]=other-check-1' # fill in from step 1 output

# 3. Merge the PR.

# 4. Restore "E2E Report" immediately.
gh api -X PATCH repos/paritytech/playground-cli/branches/main/protection/required_status_checks \
-f 'contexts[]=other-check-1' \
-f 'contexts[]=E2E Report'
```

Bypass reasons and preferred handling:

| Situation | Action |
|---|---|
| One cell flaked / timed out | Re-run the failed cell from the Actions UI |
| Whole workflow errored | Re-run the entire workflow |
| Real product regression | Do not bypass — fix it |
| Testnet outage (Paseo infra down) | Use admin bypass above + comment linking the testnet incident |
| Renamed check (e.g. "E2E Report" → new name) | Remove stale rule via gh-api; do not merge broken code |

---

## 4. Adding a new fixture domain

1. Add an entry to `FIXTURES` in `tools/register-e2e-fixtures.ts`.
2. Add the corresponding key to `E2E_DOMAINS` in `e2e/cli/fixtures/accounts.ts`.
3. Run `bun tools/register-e2e-fixtures.ts --domain <new-domain>` once.
4. Add the domain to the verify loop in §2 above.

---

## 5. Rotating/permanent state and cleanup

### Permanent fixtures (never swept)

The 9 domains listed in §1 are permanent. Same-owner re-publish by the tool
or by test runs keeps their metadata current without accumulating new registry
entries.

### Per-run rotating state (Phase 5e, not yet shipped)

When Phase 5e (`nightly-deploy-modable`) ships, each run will create:
- A `e2e-cli-modable-<runId>` registry domain
- A GitHub repo tagged `e2e-test-fixture`

These are swept by `.github/workflows/e2e-cleanup.yml` (Sunday 04:00 UTC)
after 14 days. No manual cleanup needed once that phase ships.

---

## 6. Sentry dashboards

Setup is manual, one-time. Both dashboards already exist as of 2026-05-02.

| Dashboard | ID | Filter | Purpose |
|---|---|---|---|
| Playground CLI Health | 2143100 | `!cli.tag:e2e-*` | Production signal — excludes E2E noise |
| Playground CLI E2E Health | 2216096 | `cli.tag:e2e-*` | E2E-only signal |

Dashboard JSON snapshots live in `sentry/dashboards/`. To view traces for a
specific CI run, use the Sentry link in the `E2E Report` PR comment or step
summary — it deep-links to the `cli.tag` value for that run.

No further setup required unless dashboards are reset or the Sentry project
is rotated. If you need to recreate them, see `sentry/` tooling and the
`## Sentry telemetry` section of `CLAUDE.md`.

---

## 7. Branch protection: adding `E2E Report` as a required check

Run once after the repo is freshly created or protection rules are reset:

```bash
# Read existing required checks first
gh api repos/paritytech/playground-cli/branches/main/protection \
--jq '.required_status_checks.contexts'

# Add "E2E Report" to the list (include all existing checks + the new one)
gh api -X PUT repos/paritytech/playground-cli/branches/main/protection \
--input - <<'EOF'
{
"required_status_checks": {
"strict": false,
"contexts": ["E2E Report"]
},
"enforce_admins": false,
"required_pull_request_reviews": null,
"restrictions": null
}
EOF
```

**Renaming the check later** (e.g. from "E2E Report" to something else) is a
three-step dance to avoid deadlocking the rename PR:

1. Add the new name to required checks.
2. Merge the PR that renames the job.
3. Remove the old name from required checks.

Never remove the old name before the rename PR merges — that leaves the branch
unprotected during the window.

---

## 8. Who has admin

```bash
# Org owners (admin on every paritytech repo)
gh api 'orgs/paritytech/members?role=admin' --jq '.[].login'

# Direct-collaborator admins on this repo
gh api repos/paritytech/playground-cli/collaborators \
--jq '[.[] | select(.permissions.admin == true) | .login]'

# Team-based grants (requires org:read scope)
gh api repos/paritytech/playground-cli/teams \
--jq '[.[] | select(.permission == "admin") | .name]'
```
Loading