Skip to content

fix(e2e): pin /nosql/new to fast anon hot-pool to stop e2e-prod timeout flake #180

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mastermanas805 merged 1 commit into from
Jun 5, 2026
Merged

fix(e2e): pin /nosql/new to fast anon hot-pool to stop e2e-prod timeout flake #180

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 9 additions & 7 deletions e2e/cohort.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@
// minted-account run — i.e. it carries a mint token (E2E_ACCOUNT_TOKEN, used by
// the CI workflow to mint/reap the account) or an already-minted session JWT
// (E2E_SESSION_JWT). A prod target WITHOUT either is still refused, so a stray /
// mis-configured invocation can never provision-and-leak against prod.

Check warning on line 91 in e2e/cohort.ts

View workflow job for this annotation

GitHub Actions / typos 

"mis" should be "miss" or "mist".

/** The prod api host. A prod target is only allowed for a sanctioned minted run. */
export const PROD_API_HOST = 'api.instanode.dev'
Expand Down Expand Up @@ -275,19 +275,21 @@
*
* `forceAnon` pins the anonymous fast-hot-pool path even when a minted session
* exists. Used wherever the AUTHED prod provision is too slow / hangs to finish
* inside the LIVE per-test timeout (playwright.live.config.ts, 90s):
* - DB & VECTOR: the authed (pro) /db/new & /vector/new paths provision a
* DEDICATED Postgres / pgvector Postgres for the team — slow on prod (15s
* warm, >90s cold). The anon path uses the fast (pgvector) hot-pool.
* inside the LIVE per-test timeout (playwright.live.config.ts):
* - DB, VECTOR & NOSQL: the authed (pro) /db/new, /vector/new & /nosql/new
* paths each provision a DEDICATED backing DB for the team (Postgres /
* pgvector Postgres / MongoDB) — slow on prod (15s warm, >90s cold). The
* anon path uses the fast hot-pool. These are ALL the dedicated-backing-DB
* services; none may hit the slow authed-dedicated-provision path.
* - QUEUE: the authed /queue/new isolated per-tenant NATS path HANGS on prod
* until the operator seeds the NATS operator/sys NKeys (CLAUDE.md P1 known
* gap). The anon path returns fast with auth_mode=legacy_open.
* In every forceAnon case the resulting anon resource is reaped by its 24h TTL
* (no authed DELETE exists for anon resources); the spec treats a no-bearer
* resource's reap as best-effort (TTL-backed) rather than asserting a 200 it can
* never get. The genuinely-fast services (cache/nosql/storage/webhook) stay on
* the authed-minted path so the authed/minted-account + authed-reap legs are
* still exercised.
* never get. The genuinely-fast services (cache/storage/webhook) have NO
* dedicated backing DB and stay on the authed-minted path so the
* authed/minted-account + authed-reap legs are still exercised.
*/
export function provisionIdentity(
extra: Record<string, string> = {},
Expand Down
80 changes: 54 additions & 26 deletions e2e/live-anon-provision.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,11 +80,16 @@ interface ProvisionFlow {
/** Optional extra per-service assertion on the parsed JSON body. */
extra?: (body: Record<string, unknown>) => void
/**
* Pin the anonymous bypass path even when a minted pro session exists. Only
* `queue` sets this: the authed /queue/new path attempts isolated per-tenant
* NATS provisioning which HANGS on prod (operator NKeys unseeded — CLAUDE.md
* P1). The anon path returns fast with auth_mode=legacy_open; the anon
* resource is TTL-reaped (no authed DELETE for anon resources).
* Pin the anonymous bypass path even when a minted pro session exists. Set by
* every service whose authed path is too slow / hangs to finish inside the
* per-test timeout:
* - `vector` & `nosql`: the authed path provisions a DEDICATED backing DB
* (pgvector Postgres / MongoDB) for the team — cold-provision > 90s on
* prod. (`db` does the same in the smoke spec.)
* - `queue`: the authed /queue/new path attempts isolated per-tenant NATS
* provisioning which HANGS on prod (operator NKeys unseeded — CLAUDE.md P1).
* The anon path returns fast (hot-pool / legacy_open); the anon resource is
* TTL-reaped (no authed DELETE for anon resources).
*/
forceAnon?: boolean
}
Expand All @@ -97,29 +102,44 @@ interface ProvisionFlow {
// Each flow runs as EITHER the minted PRO account (authed, exercises the
// minted-account/authed-reap path) OR the anonymous fast-hot-pool path
// (forceAnon). The deciding factor is whether the AUTHED prod provision is fast
// enough to finish inside the 90s per-test timeout (playwright.live.config.ts):
// enough to finish inside the per-test timeout (playwright.live.config.ts):
//
// service | path | why
// ---------|----------------|----------------------------------------------
// db | anon (forceAnon, in smoke spec) | authed = DEDICATED Postgres,
// | | cold-provision > 90s on prod; anon = hot-pool (fast)
// vector | anon (forceAnon)| authed = DEDICATED pgvector Postgres, same
// | | slow dedicated-provision as db (15s warm, >90s
// | | cold) → TIMED OUT the suite (run 26999448643).
// | | anon = fast hot-pool; anon resource TTL-reaped.
// queue | anon (forceAnon)| authed isolated-NATS path HANGS on prod (P1
// | | NKeys gap); anon returns fast (legacy_open).
// cache | authed-minted | Redis provision is fast (no dedicated DB build);
// | | KEEPS authed/minted-account + authed-reap coverage.
// nosql | authed-minted | Mongo provision is fast; KEEPS authed coverage.
// storage | authed-minted | DO Spaces tenant-prefix (no DB build) → fast;
// | | KEEPS authed coverage.
// webhook | authed-minted | Redis-backed receiver, fast; KEEPS authed coverage.
// service | path | backing DB? | why
// ---------|----------------|--------------|-------------------------------
// db | ANON (forceAnon, in smoke spec) | DEDICATED Postgres | authed
// | | | cold-provision > 90s on prod;
// | | | anon = hot-pool (fast).
// vector | ANON (forceAnon)| DEDICATED pgvector Postgres | same slow
// | | | dedicated-provision as db (15s
// | | | warm, >90s cold) → TIMED OUT the
// | | | suite (run 26999448643). anon =
// | | | fast hot-pool; TTL-reaped.
// nosql | ANON (forceAnon)| DEDICATED Mongo | authed = DEDICATED MongoDB;
// | | | cold-provision > 90s on prod
// | | | (run 27000380873 timed out on
// | | | /nosql/new). anon = fast hot-pool;
// | | | TTL-reaped. The LAST dedicated-DB
// | | | service to move off the slow path.
// queue | ANON (forceAnon)| none (NATS) | authed isolated-NATS path HANGS
// | | | on prod (P1 NKeys gap); anon
// | | | returns fast (legacy_open).
// cache | authed-minted | none (Redis) | Redis provision is fast (no
// | | | dedicated DB build); KEEPS authed/
// | | | minted-account + authed-reap.
// storage | authed-minted | none (Spaces)| DO Spaces tenant-prefix (no DB
// | | | build) → fast; KEEPS authed.
// webhook | authed-minted | none (Redis) | Redis-backed receiver, fast;
// | | | KEEPS authed coverage.
//
// Net: the two genuinely-slow DEDICATED-DB provisions (db, vector) and the
// hanging queue path use the fast anon hot-pool so they can NEVER exceed the
// timeout; cache/nosql/storage/webhook still exercise the authed/minted-account
// (and authed-reap) path. That keeps both paths covered with zero flake.
// Net: EVERY service whose authed path provisions a DEDICATED backing DB
// (db, vector, nosql) — plus the hanging queue path — uses the fast anon
// hot-pool so it can NEVER exceed the per-test timeout. The remaining services
// (cache/storage/webhook) have NO dedicated backing DB, provision fast, and
// stay on the authed/minted-account (and authed-reap) path so the
// minted-account + authed-reap legs are still exercised on every run. The auth
// specs, the claim flow, and the minted-account lifecycle continue to exercise
// the on-the-fly minted account — "test accounts on the fly" coverage is
// retained. Both paths covered, zero dedicated-DB-provision flake.
const PROVISION_FLOWS: ProvisionFlow[] = [
{
label: 'vector',
Expand Down Expand Up @@ -147,6 +167,14 @@ const PROVISION_FLOWS: ProvisionFlow[] = [
path: '/nosql/new',
connKey: 'connection_url',
connPattern: /^mongodb(\+srv)?:\/\//,
// Pin anon: the authed (pro) /nosql/new path provisions a DEDICATED MongoDB
// for the team — slow on prod (>90s cold), which timed out the suite at the
// per-test limit (flaky run 27000380873 failed on /nosql/new). The anon path
// uses the fast Mongo hot-pool and returns quickly; the anon resource is
// TTL-reaped (no authed DELETE for anon resources). Same treatment as /db/new
// and /vector/new — all three are dedicated-DB provisions. nosql was the LAST
// dedicated-DB service still on the slow authed path.
forceAnon: true,
},
{
label: 'queue',
Expand Down
8 changes: 5 additions & 3 deletions e2e/live-provision-smoke.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,9 +101,11 @@ test.describe('LIVE smoke — anonymous provision → backend-assert → reap',
// anon path does cleanly: the X-E2E-Test-Token + X-E2E-Source-IP fingerprint
// bypass gets past the recycle gate (fresh fingerprint per call), and the
// anon resource is TTL-reaped (no authed DELETE for anon resources — the
// reap below is best-effort for the no-bearer case). The full authed-provision
// coverage lives in live-anon-provision.spec.ts (vector/cache/nosql as the
// minted pro account). /db/new REQUIRES a name (CLAUDE.md) — sent below.
// reap below is best-effort for the no-bearer case). The authed/minted-account
// (and authed-reap) coverage lives in live-anon-provision.spec.ts via the
// fast, no-dedicated-DB services (cache/storage/webhook); the dedicated-DB
// services (db/vector/nosql) all use the fast anon hot-pool there too.
// /db/new REQUIRES a name (CLAUDE.md) — sent below.
const id = provisionIdentity({}, /* forceAnon */ true)
const resp = await request.fetch(`${API_URL}/db/new`, {
method: 'POST',
Expand Down
7 changes: 6 additions & 1 deletion playwright.live.config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,12 @@ export default defineConfig({
forbidOnly: !!process.env.CI,
retries: process.env.CI ? 1 : 0,
workers: 1,
timeout: 90_000,
// 120s per test (was 90s). Secondary guard only: the PRIMARY fix is pinning
// every dedicated-backing-DB provision (db/vector/nosql) to the fast anon
// hot-pool via forceAnon (see e2e/cohort.ts + live-anon-provision.spec.ts) so
// no test depends on a slow authed-dedicated-provision finishing in time. The
// modest bump gives extra headroom for a cold hot-pool / network slowness.
timeout: 120_000,
expect: { timeout: 20_000 },

reporter: process.env.CI
Expand Down
Loading