Skip to content

Commit e700436

Browse files
test(e2e): WS1-P1 LIVE real-backend E2E safety harness (cohort + ledger + reaper) (#170)
Make non-mocked Playwright E2E safe to run against a staging backend so we can subsequently close UI<->backend integration gaps (the login-outage class). Per docs/sessions/2026-06-04/OBSERVABILITY-AND-INTELLIGENCE-PLAN.md WS1-P1. instanode-web side only (one-tree discipline): - e2e/cohort.ts: test-cohort identity. Every created entity is branded with the greppable `e2e-cohort` marker (email `e2e-cohort+...`, names `e2e-cohort-...`) so it's identifiable for cleanup AND so the follow-up backend skip-cohort guards can no-op quota/churn/billing for it. - e2e/cleanup-ledger.ts: on-disk ledger (record/load/clear) + idempotent reapEntities() (404 == already gone, best-effort, never throws). Persisted so an out-of-process reaper can clean up after a crashed/timed-out run (rule 24). - e2e/reap-cohort.ts: standalone reaper for CI teardown; exits non-zero if any entity can't be reaped (leak surfaced loudly). Run via `npm run reap:live`. - e2e/live-provision-smoke.spec.ts: U4 smoke (anon provision postgres -> backend-assert connection_url + cohort marker -> reap), gated behind E2E_LIVE=1. Skips cleanly in normal CI (no live backend dependency). - playwright.live.config.ts: dedicated config (testMatch live-*.spec.ts). - playwright.config.ts: testIgnore live-*.spec.ts (keeps the mocked per-PR suite cleanly separate; unchanged otherwise). - .github/workflows/e2e-live.yml: manual + nightly job (NOT on pull_request), staging-only (refuses prod), reaper in `if: always()` teardown. - npm scripts test:e2e:live + reap:live; .gitignore for the ledger artifact. Gate green: tsc clean, build OK, vitest 1115 passed / 3 skipped. Mocked Playwright suite unchanged. FOLLOW-UP (separate api/worker PR, required before targeting PROD vs staging): backend skip-cohort guards — quota/abuse scan, churn predictor, and billing-charge paths must no-op for test-cohort teams (key on the `e2e-cohort` marker / a new `is_test_cohort` column), so a LIVE run never emails a "we miss you", burns real quota budget, or attempts a real charge. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent a9c3458 commit e700436

9 files changed

Lines changed: 657 additions & 0 deletions

File tree

.github/workflows/e2e-live.yml

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
# WS1 — real-backend (LIVE) E2E against a STAGING api.
2+
#
3+
# Plan: docs/sessions/2026-06-04/OBSERVABILITY-AND-INTELLIGENCE-PLAN.md, WS1.
4+
#
5+
# WHY this is a SEPARATE, MANUAL/SCHEDULED job (never on pull_request):
6+
# - LIVE specs create REAL backend resources (DBs/caches/deploys) and reap
7+
# them (rule 24). The per-PR gate must NEVER depend on a reachable backend.
8+
# - It targets a STAGING api only. Prod is opt-in and requires the backend
9+
# skip-cohort guards (separate api/worker PR — quota/churn/billing-charge
10+
# must no-op for test-cohort teams) to be live first. Until then this job
11+
# refuses any prod-looking target.
12+
#
13+
# WHAT it runs:
14+
# - playwright.live.config.ts (testMatch live-*.spec.ts) with E2E_LIVE=1.
15+
# - The reaper (npm run reap:live) in an `if: always()` teardown so any
16+
# resource the run leaked (crash/timeout) is still deleted out-of-process.
17+
#
18+
# Triggers:
19+
# - workflow_dispatch (operator runs it on demand, supplies the staging api).
20+
# - schedule (nightly) — only fires if E2E_LIVE_STAGING_API_URL repo var is
21+
# set; otherwise the job no-ops with a clear message (no false red).
22+
#
23+
# Required for a real run:
24+
# - vars.E2E_LIVE_STAGING_API_URL — the staging api base (e.g.
25+
# https://staging-api.instanode.dev). NEVER api.instanode.dev (prod).
26+
27+
name: E2E LIVE (staging, manual/scheduled)
28+
29+
on:
30+
workflow_dispatch:
31+
inputs:
32+
api_url:
33+
description: 'Staging api base URL (NOT prod). Falls back to vars.E2E_LIVE_STAGING_API_URL.'
34+
required: false
35+
default: ''
36+
schedule:
37+
# Nightly 03:17 UTC. No-ops cleanly when the staging api var is unset.
38+
- cron: '17 3 * * *'
39+
40+
concurrency:
41+
# One LIVE run at a time — they create real resources; overlapping runs
42+
# could contend on fingerprint-dedup or interleave ledger writes.
43+
group: e2e-live-${{ github.workflow }}
44+
cancel-in-progress: false
45+
46+
jobs:
47+
e2e-live-staging:
48+
name: LIVE smoke against staging api + reap
49+
runs-on: ubuntu-latest
50+
timeout-minutes: 12
51+
env:
52+
# Route the (possibly attacker-influenced) dispatch input through env: +
53+
# validate before use — workflow-injection hygiene, same pattern as
54+
# auth-contract-e2e.yml. Repo var is the scheduled-run default.
55+
RAW_API_URL: ${{ github.event.inputs.api_url || vars.E2E_LIVE_STAGING_API_URL || '' }}
56+
E2E_LIVE_RUN_ID: ${{ github.run_id }}
57+
steps:
58+
- uses: actions/checkout@v6
59+
60+
- uses: actions/setup-node@v6
61+
with:
62+
node-version: '22'
63+
cache: 'npm'
64+
65+
- name: Validate + resolve staging target
66+
# Refuse prod and any non-https target. The LIVE harness must only ever
67+
# point at a staging api until the backend skip-cohort guards ship.
68+
run: |
69+
set -euo pipefail
70+
api="${RAW_API_URL:-}"
71+
if [ -z "$api" ]; then
72+
echo "::notice::No staging api configured (vars.E2E_LIVE_STAGING_API_URL unset and no input). Skipping LIVE run."
73+
echo "RUN_LIVE=0" >> "$GITHUB_ENV"
74+
exit 0
75+
fi
76+
case "$api" in
77+
https://api.instanode.dev|https://api.instanode.dev/)
78+
echo "::error::LIVE E2E must NOT target prod (api.instanode.dev). Point it at a staging api."
79+
exit 1 ;;
80+
https://*) ;;
81+
*) echo "::error::E2E_API_URL '$api' must be an https staging URL."; exit 1 ;;
82+
esac
83+
api="${api%/}"
84+
echo "E2E_API_URL=$api" >> "$GITHUB_ENV"
85+
echo "RUN_LIVE=1" >> "$GITHUB_ENV"
86+
echo "Resolved LIVE E2E_API_URL=$api"
87+
88+
- name: Install deps
89+
if: env.RUN_LIVE == '1'
90+
run: npm ci
91+
92+
- name: Install Chromium
93+
if: env.RUN_LIVE == '1'
94+
run: npx playwright install --with-deps chromium
95+
96+
- name: Run LIVE E2E
97+
if: env.RUN_LIVE == '1'
98+
env:
99+
E2E_LIVE: '1'
100+
run: npm run test:e2e:live
101+
102+
# Reaper ALWAYS runs (even on test failure/cancel) so a leaked resource
103+
# is deleted out-of-process from the on-disk ledger (rule 24). Fails the
104+
# job if any entity could not be reaped, surfacing the leak loudly.
105+
- name: Reap cohort resources (teardown)
106+
if: always() && env.RUN_LIVE == '1'
107+
run: npm run reap:live
108+
109+
- name: Upload LIVE trace + ledger on failure
110+
if: failure() && env.RUN_LIVE == '1'
111+
uses: actions/upload-artifact@v4
112+
with:
113+
name: e2e-live-trace-${{ github.run_id }}
114+
path: |
115+
test-results/
116+
playwright-report-live/
117+
e2e/.cleanup-ledger.json
118+
if-no-files-found: ignore
119+
retention-days: 14

.gitignore

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,12 @@ dist/
55
!.env.example
66
.DS_Store
77
playwright-report/
8+
playwright-report-live/
89
test-results/
910

11+
# WS1 LIVE E2E cleanup ledger (runtime artifact — the entities to reap)
12+
e2e/.cleanup-ledger.json
13+
1014
# Cloned from InstaNode-dev/content by scripts/fetch-content.mjs before
1115
# every build. Contents are inlined into the bundle by Vite — never
1216
# committed to this repo.

e2e/cleanup-ledger.ts

Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
// WS1-P1 — cleanup ledger + reaper fixture for real-backend (LIVE) E2E.
2+
//
3+
// Plan: docs/sessions/2026-06-04/OBSERVABILITY-AND-INTELLIGENCE-PLAN.md, WS1.
4+
// Enforces rule 24: a test run NEVER leaks a provisioned (and billable) DO
5+
// resource. Every entity a LIVE spec creates is recorded here; an afterEach +
6+
// afterAll reaps them, and a standalone `reap-cohort.ts` re-runs the same
7+
// deletion in CI teardown so cleanup happens even when the test process dies.
8+
//
9+
// Design notes:
10+
// - The ledger is persisted to disk (E2E_LEDGER_PATH, default
11+
// e2e/.cleanup-ledger.json) so the OUT-OF-PROCESS reaper can read it after
12+
// a crash/timeout that skipped the in-process afterAll.
13+
// - Each entry carries enough to delete it WITHOUT the page: the api base
14+
// URL, the auth token (if any), the entity kind, and its id/token.
15+
// - Deletion is best-effort + idempotent: a 404 (already gone) is success;
16+
// other failures are collected and reported but do not throw past the
17+
// reaper, so one stuck entity can't strand the rest.
18+
19+
import { existsSync, mkdirSync, readFileSync, writeFileSync, rmSync } from 'node:fs'
20+
import { dirname } from 'node:path'
21+
22+
import type { APIRequestContext } from '@playwright/test'
23+
24+
export type CohortEntityKind = 'resource' | 'deployment' | 'team' | 'storage-prefix'
25+
26+
export interface CohortEntity {
27+
/** What to delete. */
28+
kind: CohortEntityKind
29+
/** The id/token used to address the DELETE (resource token, deploy id, etc.). */
30+
id: string
31+
/** Absolute api base URL the entity lives on (e.g. https://staging-api...). */
32+
apiUrl: string
33+
/** Bearer token authorizing the DELETE, when the entity is team-scoped. */
34+
token?: string
35+
/** Free-form, for the reaper log + post-mortem (e.g. resource_type, name). */
36+
note?: string
37+
/** When it was recorded (ISO) — lets the reaper age-out backstop sweeps. */
38+
recordedAt: string
39+
}
40+
41+
export interface ReapResult {
42+
attempted: number
43+
deleted: number
44+
alreadyGone: number
45+
failed: Array<{ entity: CohortEntity; status: number; error: string }>
46+
}
47+
48+
const DEFAULT_LEDGER_PATH = 'e2e/.cleanup-ledger.json'
49+
50+
export function ledgerPath(): string {
51+
return process.env.E2E_LEDGER_PATH || DEFAULT_LEDGER_PATH
52+
}
53+
54+
function readLedger(path: string): CohortEntity[] {
55+
if (!existsSync(path)) return []
56+
try {
57+
const raw = readFileSync(path, 'utf8')
58+
const parsed = JSON.parse(raw)
59+
return Array.isArray(parsed) ? (parsed as CohortEntity[]) : []
60+
} catch {
61+
// Corrupt/partial ledger: treat as empty rather than crash the reaper.
62+
return []
63+
}
64+
}
65+
66+
function writeLedger(path: string, entities: CohortEntity[]): void {
67+
const dir = dirname(path)
68+
if (dir && dir !== '.' && !existsSync(dir)) mkdirSync(dir, { recursive: true })
69+
writeFileSync(path, JSON.stringify(entities, null, 2), 'utf8')
70+
}
71+
72+
/**
73+
* Append an entity to the on-disk ledger. Called the instant a LIVE spec
74+
* creates something, BEFORE any assertion that might throw — so a failed
75+
* assertion still leaves a reapable record (rule 24).
76+
*/
77+
export function recordEntity(entity: Omit<CohortEntity, 'recordedAt'>): void {
78+
const path = ledgerPath()
79+
const current = readLedger(path)
80+
current.push({ ...entity, recordedAt: new Date().toISOString() })
81+
writeLedger(path, current)
82+
}
83+
84+
export function loadLedger(): CohortEntity[] {
85+
return readLedger(ledgerPath())
86+
}
87+
88+
/** Wipe the ledger file (called after a clean reap so the next run starts fresh). */
89+
export function clearLedger(): void {
90+
const path = ledgerPath()
91+
if (existsSync(path)) rmSync(path)
92+
}
93+
94+
function deletePath(entity: CohortEntity): string {
95+
switch (entity.kind) {
96+
case 'resource':
97+
return `/api/v1/resources/${entity.id}`
98+
case 'deployment':
99+
return `/api/v1/deployments/${entity.id}`
100+
case 'team':
101+
return `/api/v1/team/${entity.id}`
102+
case 'storage-prefix':
103+
// Storage objects are tenant-prefix-scoped; deleting the owning resource
104+
// (recorded separately as kind:'resource') reaps the bucket prefix. This
105+
// entry exists for the reaper LOG only and is skipped in deletion.
106+
return ''
107+
}
108+
}
109+
110+
/**
111+
* Delete every entity in `entities` against its recorded api base. Idempotent:
112+
* a 404/410 counts as alreadyGone (success). Never throws — failures are
113+
* collected so one stuck entity can't strand the rest. The OUT-OF-PROCESS
114+
* reaper and the in-process afterAll both call this.
115+
*/
116+
export async function reapEntities(
117+
request: APIRequestContext,
118+
entities: CohortEntity[],
119+
): Promise<ReapResult> {
120+
const result: ReapResult = { attempted: 0, deleted: 0, alreadyGone: 0, failed: [] }
121+
for (const entity of entities) {
122+
const path = deletePath(entity)
123+
if (!path) continue // storage-prefix marker: reaped via its owning resource
124+
result.attempted++
125+
const headers: Record<string, string> = {}
126+
if (entity.token) headers.Authorization = `Bearer ${entity.token}`
127+
try {
128+
const resp = await request.fetch(`${entity.apiUrl.replace(/\/$/, '')}${path}`, {
129+
method: 'DELETE',
130+
headers,
131+
failOnStatusCode: false,
132+
})
133+
const status = resp.status()
134+
if (status === 404 || status === 410) {
135+
result.alreadyGone++
136+
} else if (status >= 200 && status < 300) {
137+
result.deleted++
138+
} else {
139+
result.failed.push({
140+
entity,
141+
status,
142+
error: await resp.text().catch(() => '<unreadable>'),
143+
})
144+
}
145+
} catch (e) {
146+
result.failed.push({ entity, status: 0, error: String((e as Error)?.message ?? e) })
147+
}
148+
}
149+
return result
150+
}

e2e/cohort.ts

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
// WS1-P1 — test-cohort identity for real-backend (LIVE) Playwright runs.
2+
//
3+
// Plan: docs/sessions/2026-06-04/OBSERVABILITY-AND-INTELLIGENCE-PLAN.md, WS1.
4+
//
5+
// Every entity a LIVE spec creates (team / account / resource / deploy) is
6+
// tagged with a recognizable, machine-greppable prefix so it is:
7+
// 1. identifiable for cleanup (the reaper sweeps by this prefix as a
8+
// backstop, in addition to the per-run ledger), and
9+
// 2. recognizable by the FOLLOW-UP backend skip-cohort guards (a separate
10+
// api/worker PR) so quota/abuse/churn/billing-charge paths NO-OP for
11+
// test-cohort teams — a LIVE run must never email a "we miss you", burn a
12+
// real quota budget, or attempt a real charge.
13+
//
14+
// This is the instanode-web side ONLY. The backend `is_test_cohort` column +
15+
// the guards that read it are intentionally NOT in this PR (one-tree
16+
// discipline) — see the PR body's follow-up note. Until those guards exist,
17+
// LIVE runs MUST target STAGING, never prod.
18+
//
19+
// The contract the backend guards will key on (kept here as the single source
20+
// of truth for the string the two repos share):
21+
// - cohort emails ⇒ local-part starts with `e2e-cohort+`
22+
// - cohort names ⇒ resource/team names start with `e2e-cohort-`
23+
// - cohort marker ⇒ the literal token COHORT_MARKER appears in the
24+
// email local-part and in every created name, so a
25+
// single `LIKE '%e2e-cohort%'` (or the future
26+
// `is_test_cohort` backfill) catches all of it.
27+
28+
/**
29+
* The literal token that brands every cohort entity. Backend skip-cohort
30+
* guards (follow-up PR) match teams/resources whose email or name contains
31+
* this substring. Keep in sync with the api/worker guard constant when that
32+
* lands. Exported as a const (not an inline string) per the project's
33+
* no-hardcoded-strings rule.
34+
*/
35+
export const COHORT_MARKER = 'e2e-cohort'
36+
37+
/** Email local-part prefix, e.g. `e2e-cohort+<run>-<rand>@instanode.dev`. */
38+
export const COHORT_EMAIL_PREFIX = `${COHORT_MARKER}+`
39+
40+
/** Resource / team name prefix, e.g. `e2e-cohort-smoke-db-<rand>`. */
41+
export const COHORT_NAME_PREFIX = `${COHORT_MARKER}-`
42+
43+
/** Domain used for cohort emails. instanode.dev keeps them on our own domain. */
44+
export const COHORT_EMAIL_DOMAIN = 'instanode.dev'
45+
46+
/**
47+
* A stable id for one Playwright run. Used to namespace the ledger file AND to
48+
* stamp every created entity, so a failed run's leftovers are attributable to
49+
* the run that made them. Honours CI's run id when present.
50+
*/
51+
export function runId(): string {
52+
const ci = process.env.E2E_LIVE_RUN_ID || process.env.GITHUB_RUN_ID
53+
if (ci) return String(ci)
54+
return `${Date.now()}-${Math.random().toString(36).slice(2, 8)}`
55+
}
56+
57+
function rand(): string {
58+
return Math.random().toString(36).slice(2, 10)
59+
}
60+
61+
/**
62+
* A cohort email. Uses the `+`-subaddress form so every cohort send routes to
63+
* one mailbox (`e2e-cohort@instanode.dev`) yet stays unique per call — and so
64+
* the backend guard can match on the `e2e-cohort+` local-part prefix.
65+
*/
66+
export function cohortEmail(label = 'smoke'): string {
67+
return `${COHORT_EMAIL_PREFIX}${label}-${runId()}-${rand()}@${COHORT_EMAIL_DOMAIN}`
68+
}
69+
70+
/** A cohort-branded resource/team name. */
71+
export function cohortName(label = 'res'): string {
72+
return `${COHORT_NAME_PREFIX}${label}-${rand()}`
73+
}
74+
75+
/** Predicate the reaper uses to decide if a name/email belongs to the cohort. */
76+
export function isCohortBranded(value: string | null | undefined): boolean {
77+
return !!value && value.includes(COHORT_MARKER)
78+
}

0 commit comments

Comments
 (0)