Fix cloud dev-stack degradation under sustained MCP load

The cloud e2e stack degrades after a few minutes of sustained suite load
and eventually hangs, which is the CI cascade flake: MCP-heavy shards
(toolkits-mcp et al) run, then unrelated specs die with Playwright
TimeoutErrors because requests hang rather than erroring.

Root cause is postgres connection churn against the dev PGlite server, not
SSE frame buffering or OTel span growth. Two compounding problems:

- The dev-db PGlite socket server (apps/cloud/scripts/dev-db.ts) used the
  library default maxConnections: 1. The cloud worker opens a fresh postgres
  pool per request (the MCP auth seam rebuilds one on every /mcp request to
  respect workerd's per-request I/O rule), so under concurrent load the
  second-and-later connects were answered with "Too many connections" and an
  immediate socket close. postgres.js then reconnected in a tight loop, piling
  up thousands of half-closed sockets, starving real queries and driving
  request latency into the tens of seconds until the stack hung. PGlite runs
  queries serially anyway (its QueryQueueManager uses runExclusive), so
  allowing many connections is safe: they queue instead of storming. Raise the
  cap (overridable via DEV_DB_MAX_CONNECTIONS).

- The postgres pool finalizer (apps/cloud/src/db/db.ts) was fire-and-forget:
  Effect.runFork(sql.end({ timeout: 0 })) returned before the socket was torn
  down, so closing sockets accumulated faster than the server reaped them.
  Await a graceful end() inside the acquireRelease scope so the number of
  live-plus-closing sockets stays bounded to what is actually in flight.
  Extracted as closePostgres() and covered by a unit test.

Measured on a booted cloud stack (3 concurrent MCP clients, 1s think-time):

  baseline (maxConnections=1): ~18 sessions in 135s, avg 15s/session,
    p95 up to 50s, latency climbing, stack stalling, thousands of
    "Too many connections".
  fixed: 347 sessions in 135s, avg ~125ms/session, p95 ~150ms, flat,
    zero "Too many connections".

Author: RhysSullivan

apps/cloud/scripts/dev-db.ts

@@ -82,10 +82,28 @@ const db = await PGlite.create(DB_PATH);
 console.log(`[dev-db] Running migrations from ${MIGRATIONS_FOLDER}`);
 await migrate(drizzle(db), { migrationsFolder: MIGRATIONS_FOLDER });
 
+// `PGLiteSocketServer` defaults to `maxConnections: 1` and answers every extra
+// concurrent connection with "Too many connections" + an immediate socket
+// close. (pglite-socket 0.1.4's published index.d.ts documents "default: 100",
+// but the shipped runtime JS is `maxConnections ?? 1`, verified in the shipped
+// chunk, so the runtime default really is 1.) The cloud worker opens a fresh
+// postgres pool per request (the MCP auth seam rebuilds one on EVERY `/mcp`
+// request, see apps/cloud/src/mcp/auth.ts), so under concurrent load, exactly
+// what the e2e suite generates against one shared dev stack, the
+// second-and-later connects were rejected, and postgres.js reconnected in a
+// tight loop against the closed socket. That reconnect storm piled up
+// thousands of half-closed sockets, starved real queries, drove request
+// latency into the tens of seconds, and eventually hung the stack: the CI e2e
+// "cloud dev stack degrades after minutes of sustained load" cascade flake.
+// PGlite still runs queries serially (its internal QueryQueueManager executes
+// each under `runExclusive`), so allowing many connections is safe: they queue
+// instead of being rejected. Raise the cap so concurrent requests wait their
+// turn instead of storming.
 const server = new PGLiteSocketServer({
   db,
   port: PORT,
   host: "127.0.0.1",
+  maxConnections: Number(process.env.DEV_DB_MAX_CONNECTIONS ?? 1000),
 });
 
 await server.start();

apps/cloud/src/db/db.close.test.ts

@@ -0,0 +1,109 @@
+// Contract: the postgres pool finalizer must AWAIT the connection teardown.
+//
+// The cloud MCP auth seam (`makeMcpOrganizationAuthServices`) builds a fresh
+// postgres pool on EVERY `/mcp` request and closes it in its `acquireRelease`
+// finalizer. That finalizer used to be fire-and-forget (`Effect.runFork(
+// sql.end({ timeout: 0 }))`): it returned before the socket was torn down, so
+// under sustained MCP load closed-but-unreaped sockets piled up against the
+// dev PGlite server (effectively single-connection) faster than it reaped
+// them. New connects queued behind the backlog, request latency climbed into
+// the tens of seconds, and the e2e cloud dev stack hung after a few minutes:
+// the CI cascade flake.
+//
+// These tests characterize the contract the old fire-and-forget close violated
+// (`closePostgres` itself is new alongside them): it must (a) call `sql.end`
+// with a NON-zero drain window (a clean Terminate, not an abandon) and (b)
+// return an Effect that does not complete until `sql.end` has resolved, even
+// when the teardown takes real wall-clock time. All asserted with a fake `sql`
+// whose `end()` completion is observable, so the tests are fast and need no
+// live database.
+
+import { describe, expect, it } from "@effect/vitest";
+import { Effect, Exit } from "effect";
+
+import { POSTGRES_END_TIMEOUT_SECONDS, closePostgres } from "./db";
+
+describe("closePostgres", () => {
+  it.effect("passes a non-zero drain window to sql.end (clean Terminate, not abandon)", () =>
+    Effect.gen(function* () {
+      let received: { timeout?: number } | undefined;
+      const fakeSql = {
+        end: (options?: { timeout?: number }) => {
+          received = options;
+          return Promise.resolve();
+        },
+      };
+
+      yield* closePostgres(fakeSql);
+
+      expect(received?.timeout).toBe(POSTGRES_END_TIMEOUT_SECONDS);
+      expect(POSTGRES_END_TIMEOUT_SECONDS).toBeGreaterThan(0);
+    }),
+  );
+
+  it.effect("does not complete until sql.end resolves (awaits the teardown)", () =>
+    Effect.gen(function* () {
+      // `end` records an ordering marker only after an async tick. If
+      // `closePostgres` awaits it, the "close completed" marker lands AFTER the
+      // "end resolved" marker. The old fire-and-forget close returned before
+      // `end` ran, so "close completed" would land FIRST.
+      const order: string[] = [];
+      const fakeSql = {
+        end: () =>
+          // Defer resolution across a microtask so a non-awaiting close would
+          // observably finish before this runs.
+          Promise.resolve()
+            .then(() => Promise.resolve())
+            .then(() => {
+              order.push("end-resolved");
+            }),
+      };
+
+      yield* closePostgres(fakeSql);
+      order.push("close-completed");
+
+      // Awaiting the teardown means end resolved strictly before close returned.
+      expect(order).toEqual(["end-resolved", "close-completed"]);
+    }),
+  );
+
+  it.effect("awaits a teardown that takes real wall-clock time (bounded by the ceiling)", () =>
+    Effect.gen(function* () {
+      // `end` resolves only after a real timer delay, not just a microtask.
+      // This pins the "we await to completion, the timeout is only a ceiling"
+      // contract: closePostgres must stay suspended across the delay rather
+      // than resolving early.
+      const order: string[] = [];
+      const fakeSql = {
+        end: () =>
+          new Promise<void>((resolve) => {
+            setTimeout(() => {
+              order.push("end-resolved");
+              resolve();
+            }, 75);
+          }),
+      };
+
+      const startedAt = Date.now();
+      yield* closePostgres(fakeSql);
+      order.push("close-completed");
+
+      expect(order).toEqual(["end-resolved", "close-completed"]);
+      // Slack of a few ms: platform timers may fire marginally early.
+      expect(Date.now() - startedAt).toBeGreaterThanOrEqual(70);
+    }),
+  );
+
+  it.effect("swallows sql.end failures (a teardown error must not fail the request scope)", () =>
+    Effect.gen(function* () {
+      const fakeSql = {
+        // A rejected teardown (connection already gone) must not surface as a
+        // scope failure.
+        // oxlint-disable-next-line executor/no-promise-reject -- test fake: model `sql.end` (a raw postgres.js promise) rejecting
+        end: () => new Promise<void>((_resolve, reject) => reject("connection already gone")),
+      };
+      const exit = yield* Effect.exit(closePostgres(fakeSql));
+      expect(Exit.isSuccess(exit)).toBe(true);
+    }),
+  );
+});

apps/cloud/src/db/db.ts

@@ -67,18 +67,53 @@ const makeSql = (): Sql =>
     onnotice: () => undefined,
   });
 
+/**
+ * Graceful drain window (seconds) handed to `postgres`'s `sql.end`. Small but
+ * non-zero: `timeout: 0` closes immediately without waiting for a clean
+ * Terminate, which leaves the socket half-closed for the server to reap on its
+ * own schedule. A short drain lets postgres.js finish the wire teardown.
+ *
+ * This is a `Promise.race` CEILING, not a fixed wait: postgres.js races
+ * `end({ timeout })` against the actual teardown, and an idle connection's
+ * `end()` calls `terminate()` immediately and resolves as soon as the socket
+ * closes (sub-millisecond). Awaiting it in the request scope therefore adds no
+ * meaningful latency on the common path; the 5s only bounds how long a
+ * connection still mid-query can hold the scope open.
+ */
+export const POSTGRES_END_TIMEOUT_SECONDS = 5;
+
+/**
+ * Close a postgres pool and AWAIT its teardown.
+ *
+ * The `DbService` layers ({@link DbService.Live}, {@link makeDbLayer}) run this
+ * as their `acquireRelease` finalizer. The MCP auth seam
+ * (`makeMcpOrganizationAuthServices`) builds a FRESH pool on EVERY `/mcp`
+ * request, so this finalizer runs per request under sustained load.
+ *
+ * It used to be fire-and-forget (`Effect.runFork(sql.end({ timeout: 0 }))`),
+ * which returned before the connection was actually torn down. The abandoned
+ * sockets piled up against the dev PGlite server (effectively single-connection)
+ * faster than it reaped them; new connects then queued behind the backlog, so
+ * request latency climbed into the tens of seconds and the stack eventually
+ * hung — the CI e2e "cloud dev stack degrades after minutes of sustained load"
+ * cascade. Awaiting the close bounds the number of live-plus-closing sockets to
+ * what is actually in flight. It runs inside the request's own Effect scope, so
+ * it respects workerd's per-request I/O rule.
+ */
+export const closePostgres = (sql: Pick<Sql, "end">): Effect.Effect<void> =>
+  Effect.ignore(
+    Effect.tryPromise({
+      try: () => sql.end({ timeout: POSTGRES_END_TIMEOUT_SECONDS }),
+      catch: (cause) => cause,
+    }),
+  );
+
 const makePostgresResource = (): DbResource => {
   const sql = makeSql();
   return {
     sql,
     db: drizzle(sql, { schema: combinedSchema }) as DrizzleDb,
-    close: () =>
-      Effect.ignore(
-        Effect.tryPromise({
-          try: () => sql.end({ timeout: 0 }),
-          catch: (cause) => cause,
-        }),
-      ),
+    close: () => closePostgres(sql),
   };
 };