Skip to content

Commit 89e6b1b

Browse files
fix(e2e): pin /vector/new to fast anon hot-pool to stop e2e-prod timeout flake (#179)
The e2e-prod LIVE suite flaked: the `provision anon vector (/vector/new)` spec timed out at the 90s per-test limit (run 26999448643) while a same-commit run passed (26999447344). Root cause: vector is an AUTHED dedicated pgvector Postgres on the minted-pro path — slow to provision on prod (15s warm, >90s cold) — the same dedicated-DB slowness for which db & queue were already pinned to forceAnon. Vector was the one remaining slow authed dedicated-DB provision. Fix: set forceAnon on the vector flow so it routes through the fast pgvector hot-pool (same treatment as /db/new and /queue/new), which can never exceed the timeout; the anon resource is TTL-reaped. cache/nosql/storage/webhook stay on the authed-minted path so the authed/minted-account + authed-reap legs are still exercised. Documented the per-service path choice in a table comment. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent b967ac2 commit 89e6b1b

2 files changed

Lines changed: 50 additions & 8 deletions

File tree

e2e/cohort.ts

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -273,14 +273,21 @@ export const ANON_TIER = 'anonymous'
273273
* Resolve the provision identity for a LIVE resource spec. Prefers the minted
274274
* pro account (authed) when E2E_SESSION_JWT is set; else the anon bypass path.
275275
*
276-
* `forceAnon` pins the anonymous bypass path even when a minted session exists.
277-
* Used for the QUEUE flow: the authed (pro) /queue/new path attempts isolated
278-
* per-tenant NATS provisioning, which HANGS on prod until the operator seeds the
279-
* NATS operator/sys NKeys (CLAUDE.md P1 known gap). The anonymous path returns
280-
* fast with auth_mode=legacy_open. The resulting anon resource is reaped by its
281-
* 24h TTL (no authed DELETE exists for anon resources); the spec treats a
282-
* no-bearer resource's reap as best-effort (TTL-backed) rather than asserting a
283-
* 200 it can never get.
276+
* `forceAnon` pins the anonymous fast-hot-pool path even when a minted session
277+
* exists. Used wherever the AUTHED prod provision is too slow / hangs to finish
278+
* inside the LIVE per-test timeout (playwright.live.config.ts, 90s):
279+
* - DB & VECTOR: the authed (pro) /db/new & /vector/new paths provision a
280+
* DEDICATED Postgres / pgvector Postgres for the team — slow on prod (15s
281+
* warm, >90s cold). The anon path uses the fast (pgvector) hot-pool.
282+
* - QUEUE: the authed /queue/new isolated per-tenant NATS path HANGS on prod
283+
* until the operator seeds the NATS operator/sys NKeys (CLAUDE.md P1 known
284+
* gap). The anon path returns fast with auth_mode=legacy_open.
285+
* In every forceAnon case the resulting anon resource is reaped by its 24h TTL
286+
* (no authed DELETE exists for anon resources); the spec treats a no-bearer
287+
* resource's reap as best-effort (TTL-backed) rather than asserting a 200 it can
288+
* never get. The genuinely-fast services (cache/nosql/storage/webhook) stay on
289+
* the authed-minted path so the authed/minted-account + authed-reap legs are
290+
* still exercised.
284291
*/
285292
export function provisionIdentity(
286293
extra: Record<string, string> = {},

e2e/live-anon-provision.spec.ts

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,12 +92,47 @@ interface ProvisionFlow {
9292
// The registry. The matrix's seventh service (db) is covered by
9393
// live-provision-smoke.spec.ts; the six below complete the set. Editing this
9494
// table is the ONLY change needed to add/remove a covered service.
95+
//
96+
// ── Per-service provision-path choice (forceAnon vs authed-minted) ────────────
97+
// Each flow runs as EITHER the minted PRO account (authed, exercises the
98+
// minted-account/authed-reap path) OR the anonymous fast-hot-pool path
99+
// (forceAnon). The deciding factor is whether the AUTHED prod provision is fast
100+
// enough to finish inside the 90s per-test timeout (playwright.live.config.ts):
101+
//
102+
// service | path | why
103+
// ---------|----------------|----------------------------------------------
104+
// db | anon (forceAnon, in smoke spec) | authed = DEDICATED Postgres,
105+
// | | cold-provision > 90s on prod; anon = hot-pool (fast)
106+
// vector | anon (forceAnon)| authed = DEDICATED pgvector Postgres, same
107+
// | | slow dedicated-provision as db (15s warm, >90s
108+
// | | cold) → TIMED OUT the suite (run 26999448643).
109+
// | | anon = fast hot-pool; anon resource TTL-reaped.
110+
// queue | anon (forceAnon)| authed isolated-NATS path HANGS on prod (P1
111+
// | | NKeys gap); anon returns fast (legacy_open).
112+
// cache | authed-minted | Redis provision is fast (no dedicated DB build);
113+
// | | KEEPS authed/minted-account + authed-reap coverage.
114+
// nosql | authed-minted | Mongo provision is fast; KEEPS authed coverage.
115+
// storage | authed-minted | DO Spaces tenant-prefix (no DB build) → fast;
116+
// | | KEEPS authed coverage.
117+
// webhook | authed-minted | Redis-backed receiver, fast; KEEPS authed coverage.
118+
//
119+
// Net: the two genuinely-slow DEDICATED-DB provisions (db, vector) and the
120+
// hanging queue path use the fast anon hot-pool so they can NEVER exceed the
121+
// timeout; cache/nosql/storage/webhook still exercise the authed/minted-account
122+
// (and authed-reap) path. That keeps both paths covered with zero flake.
95123
const PROVISION_FLOWS: ProvisionFlow[] = [
96124
{
97125
label: 'vector',
98126
path: '/vector/new',
99127
connKey: 'connection_url',
100128
connPattern: /^postgres(ql)?:\/\//,
129+
// Pin anon: the authed (pro) /vector/new path provisions a DEDICATED
130+
// pgvector Postgres for the team — slow on prod (15s warm, >90s cold), which
131+
// timed out the suite at the 90s per-test limit (flaky run 26999448643). The
132+
// anon path uses the fast pgvector hot-pool and returns quickly; the anon
133+
// resource is TTL-reaped (no authed DELETE for anon resources). Same
134+
// treatment as /db/new (smoke spec) — both are dedicated-DB provisions.
135+
forceAnon: true,
101136
extra: (b) =>
102137
expect(b.extension, 'vector provision echoes pgvector extension').toBe('pgvector'),
103138
},

0 commit comments

Comments
 (0)