Summary
The local dev server intermittently crashes on CI right after printing its startup banner, when a PGlite-backed data Block runs its first migration. The @electric-sql/pglite WASM module traps with unreachable inside _pg_initdb (the lazy Postgres init that fires on the first query), surfacing as QueryFailedException: unreachable and taking down the Node process.
The crash is non-deterministic — it hits whichever PGlite-backed cell loses the dice roll on a given run (seen in both a Next.js SQL/KB cell and a React OIDC/DSQL cell).
Impact
- The dev server prints
AWS Blocks local server running on http://localhost:3000 then immediately crashes, so dev_server_started=false, every downstream check scores 0, and the whole cell is lost.
- Intermittent and init-time; a retry usually succeeds. It is an infrastructure/WASM failure, not a code regression.
- Affects any Block that boots
@aws-blocks/bb-distributed-data's DsqlMockEngine (DSQL mock) or a PGlite-backed Database and runs migrations at startup.
Evidence
Verbatim stack (identical WASM module hash wasm://wasm/01edd1ba across runs):
AWS Blocks local server running on http://localhost:3000
wasm://wasm/01edd1ba:1
QueryFailedException: unreachable
at wasm://wasm/01edd1ba:wasm-function[3625]:0x263199
at wasm://wasm/01edd1ba:wasm-function[7801]:0x3e3508
at wasm://wasm/01edd1ba:wasm-function[2533]:0x1c1034
at Object.Module._pg_initdb (.../@electric-sql/pglite/dist/index.js:3:124284)
at pe.qe (.../@electric-sql/pglite/dist/index.js:3:243608)
at async pe._checkReady (.../@electric-sql/pglite/dist/index.js:3:238920)
at async pe.query (.../@electric-sql/pglite/src/base.ts:153:5)
at async DsqlMockEngine.execute (.../@aws-blocks/bb-distributed-data/dist/engines/dsql-mock-engine.js:79:33)
at async runMigrations (.../@aws-blocks/bb-distributed-data/dist/migrations.js:11:5)
at async DsqlMockEngine.withDdl (.../@aws-blocks/bb-distributed-data/dist/engines/dsql-mock-engine.js:61:20)
Node.js v22.23.1
Observed on two separate CI runs (different cells, same signature):
- run
29138896893 — sql-kb-catalog (Next.js) cell
- run
29140413875 — oidc-dsql-notes (React) cell
Crash path. DsqlMockEngine's constructor does new PGlite(dataDir); PGlite defers the real Postgres bootstrap (_pg_initdb) to the first query. runMigrations() issues that first query — CREATE TABLE IF NOT EXISTS _migrations (...) — which drives _checkReady -> qe -> Module._pg_initdb, where the WASM module traps unreachable. The banner prints before this, so it is not a port/startup race — it is a WASM init failure on the first DB operation.
Reproduction
A minimal, dependency-pinned harness boots a fresh PGlite instance per cycle and issues the exact first two statements runMigrations runs (CREATE TABLE IF NOT EXISTS _migrations (...) then SELECT name FROM _migrations ORDER BY name), hammering the _pg_initdb path sequentially and in parallel (the crash is intermittent). No repo checkout required.
mkdir pglite-repro && cd pglite-repro
npm init -y >/dev/null && npm pkg set type=module >/dev/null
npm i @electric-sql/pglite@0.2.17
# save repro.mjs (below), then:
node repro.mjs 40 16 # 40 sequential + 16 parallel boots (args: SEQ PAR)
repro.mjs:
// Minimal reproduction for the @electric-sql/pglite `_pg_initdb` WASM trap
// ("unreachable") that crashes the AWS Blocks dev server right after the
// "AWS Blocks local server running" banner.
//
// It boots a FRESH PGlite instance per cycle (as DsqlMockEngine does in its
// constructor: `new PGlite(dataDir)`) and issues the first query, which is
// exactly what `runMigrations` does first ("CREATE TABLE IF NOT EXISTS
// _migrations ..."). PGlite lazily runs `_pg_initdb` on that first query — the
// trap site. We hammer it N times sequentially and M in parallel, since the
// crash is intermittent (surfaces under CI memory/concurrency pressure).
//
// Run: node repro.mjs [SEQ=30] [PAR=8]
import { PGlite } from '@electric-sql/pglite';
import { mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
const SEQ = Number(process.argv[2] ?? 30);
const PAR = Number(process.argv[3] ?? 8);
// The exact first two statements runMigrations issues (bb-distributed-data).
const MIGRATIONS_DDL = `
CREATE TABLE IF NOT EXISTS _migrations (
id TEXT PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL UNIQUE,
applied_at TIMESTAMP NOT NULL DEFAULT NOW()
)
`;
async function bootAndMigrateOnce(label) {
const dir = mkdtempSync(join(tmpdir(), 'pglite-repro-'));
const db = new PGlite(dir); // DsqlMockEngine ctor does exactly this
try {
await db.query(MIGRATIONS_DDL); // first query → triggers _pg_initdb
await db.query('SELECT name FROM _migrations ORDER BY name');
return { label, ok: true };
} finally {
try { await db.close(); } catch {}
try { rmSync(dir, { recursive: true, force: true }); } catch {}
}
}
const isUnreachable = (e) => /unreachable/i.test(String(e?.message ?? e)) || /unreachable/i.test(String(e?.stack ?? ''));
let failures = 0;
console.log(`[repro] pglite _pg_initdb stress — ${SEQ} sequential + ${PAR} parallel boots`);
for (let i = 0; i < SEQ; i++) {
try {
await bootAndMigrateOnce(`seq#${i}`);
process.stdout.write('.');
} catch (e) {
failures++;
console.error(`\n[repro] SEQ boot #${i} FAILED (unreachable=${isUnreachable(e)}):\n`, e?.stack ?? e);
}
}
process.stdout.write('\n');
const settled = await Promise.allSettled(
Array.from({ length: PAR }, (_, i) => bootAndMigrateOnce(`par#${i}`)),
);
for (const r of settled) {
if (r.status === 'rejected') {
failures++;
console.error(`[repro] PAR boot FAILED (unreachable=${isUnreachable(r.reason)}):\n`, r.reason?.stack ?? r.reason);
}
}
if (failures === 0) {
console.log(`[repro] OK — ${SEQ + PAR} boots, no _pg_initdb trap on this host.`);
process.exit(0);
} else {
console.error(`[repro] REPRODUCED — ${failures}/${SEQ + PAR} boots crashed in _pg_initdb.`);
process.exit(1);
}
Local reproduction status (honest)
I could not reproduce the exact unreachable WASM trap on a dev host — but the init path is provably memory-sensitive there:
| Environment |
Result |
| Node v18.20.2, 30 seq + 8 par |
clean, no trap (38/38 OK) |
| Node v22.22.3, 40 seq + 16 par |
clean, no trap (56/56 OK) |
Node 22, 32-wide parallel + --max-old-space-size=128 |
clean, no trap |
Node 22, ulimit -v (2.2–3.1 GB) + parallel |
fails in the same init path as RangeError: WebAssembly.Memory(): could not allocate memory at pe.qe -> pe._checkReady -> pe.query |
Node 22, cgroup MemoryMax (MemorySwapMax=0) |
sharp cliff: <=350M -> OOM-kill (rc 137); >=450M -> clean — no band that yields unreachable |
Conclusion: the _pg_initdb / _checkReady / qe path is memory-sensitive and fails under pressure, but the specific WASM unreachable trap is CI-environment-specific — it needs Node v22.23.1 on a GitHub ubuntu runner under the memory/overcommit/RSS pressure of the concurrent (10-wide) bench matrix. It is not fabricated here; run the harness in that environment to surface it.
To surface it on CI
Run the harness (or the test below) on an ubuntu-latest GitHub runner with Node 22.23.1, under concurrency mirroring the bench matrix (multiple PGlite boots in flight, constrained memory). Suggested: SEQ=80 PAR=24 ROUNDS=5, several parallel job shards, and/or a container with --memory in the 400–600 MB band.
Automated test
A node --test regression that boots many fresh PGlite instances sequentially and in parallel and asserts none trap unreachable. It fails/flakes when the bug is present and passes when PGlite inits cleanly. Tunable via SEQ / PAR / ROUNDS env vars.
node --test pglite-initdb.test.mjs # knobs: SEQ=40 PAR=12 ROUNDS=3 (defaults)
Local runs (both green — bug did not surface locally, as noted above):
- Node v18.20.2 ->
# pass 2 # fail 0 (~75 s)
- Node v22.22.3 -> clean
pglite-initdb.test.mjs:
// Automated regression test for the @electric-sql/pglite `_pg_initdb` WASM trap
// ("QueryFailedException: unreachable") observed on CI, crashing the AWS Blocks
// dev server during runMigrations' first query.
//
// The test boots many FRESH PGlite instances — sequentially and in parallel —
// each issuing the first migration query (the `_pg_initdb` trigger). It asserts
// that NONE trap with "unreachable". When the bug is present the test FAILS
// (or flakes); when PGlite inits cleanly it passes.
//
// Run: node --test pglite-initdb.test.mjs
// Tune stress via env: SEQ (default 40), PAR (default 12), ROUNDS (default 3).
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { PGlite } from '@electric-sql/pglite';
import { mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
const SEQ = Number(process.env.SEQ ?? 40);
const PAR = Number(process.env.PAR ?? 12);
const ROUNDS = Number(process.env.ROUNDS ?? 3);
const MIGRATIONS_DDL = `
CREATE TABLE IF NOT EXISTS _migrations (
id TEXT PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL UNIQUE,
applied_at TIMESTAMP NOT NULL DEFAULT NOW()
)
`;
const isUnreachable = (e) =>
/unreachable/i.test(String(e?.message ?? e)) || /unreachable/i.test(String(e?.stack ?? ''));
async function bootAndMigrateOnce() {
const dir = mkdtempSync(join(tmpdir(), 'pglite-test-'));
const db = new PGlite(dir); // mirrors DsqlMockEngine constructor
try {
await db.query(MIGRATIONS_DDL); // first query → _pg_initdb
await db.query('SELECT name FROM _migrations ORDER BY name'); // runMigrations' next query
} finally {
try { await db.close(); } catch {}
try { rmSync(dir, { recursive: true, force: true }); } catch {}
}
}
async function runGuarded() {
try {
await bootAndMigrateOnce();
return null;
} catch (e) {
return e;
}
}
test(`PGlite _pg_initdb never traps 'unreachable' across ${SEQ} sequential boots`, async () => {
const crashes = [];
for (let i = 0; i < SEQ; i++) {
const err = await runGuarded();
if (err) crashes.push({ i, unreachable: isUnreachable(err), stack: err?.stack ?? String(err) });
}
assert.deepEqual(
crashes.filter(c => c.unreachable),
[],
`PGlite _pg_initdb trapped 'unreachable':\n${crashes.map(c => `seq#${c.i}: ${c.stack}`).join('\n\n')}`,
);
assert.equal(crashes.length, 0, `Non-unreachable failures:\n${crashes.map(c => c.stack).join('\n\n')}`);
});
test(`PGlite _pg_initdb never traps 'unreachable' across ${ROUNDS}×${PAR} parallel boots`, async () => {
const crashes = [];
for (let r = 0; r < ROUNDS; r++) {
const settled = await Promise.allSettled(Array.from({ length: PAR }, () => bootAndMigrateOnce()));
settled.forEach((s, i) => {
if (s.status === 'rejected') {
crashes.push({ r, i, unreachable: isUnreachable(s.reason), stack: s.reason?.stack ?? String(s.reason) });
}
});
}
assert.deepEqual(
crashes.filter(c => c.unreachable),
[],
`PGlite _pg_initdb trapped 'unreachable':\n${crashes.map(c => `round${c.r}/par#${c.i}: ${c.stack}`).join('\n\n')}`,
);
assert.equal(crashes.length, 0, `Non-unreachable failures:\n${crashes.map(c => c.stack).join('\n\n')}`);
});
Environment
@electric-sql/pglite: 0.2.17 (declared ^0.2.0 in packages/bb-data/package.json and packages/bb-distributed-data/package.json)
- CI Node: v22.23.1 (GitHub
ubuntu runner) — where the crash occurs
- Dev host Node tested: v18.20.2 and v22.22.3 — no trap
- Crash-path packages:
@aws-blocks/bb-distributed-data (DsqlMockEngine, runMigrations) -> @electric-sql/pglite
- WASM module hash across all crashes:
wasm://wasm/01edd1ba
Possible directions
- Bump/pin
@electric-sql/pglite and re-test under Node 22.23.1 (upstream fixed several WASM init/OOM traps in later 0.2.x / 0.3.x).
- Add a bounded retry around the first
_pg_initdb-triggering query in DsqlMockEngine / runMigrations so a transient WASM init trap doesn't kill the dev server.
- Cap PGlite boot concurrency in the bench harness (serialize migrations across cells) to reduce peak memory pressure on the runner.
Summary
The local dev server intermittently crashes on CI right after printing its startup banner, when a PGlite-backed data Block runs its first migration. The
@electric-sql/pgliteWASM module traps withunreachableinside_pg_initdb(the lazy Postgres init that fires on the first query), surfacing asQueryFailedException: unreachableand taking down the Node process.The crash is non-deterministic — it hits whichever PGlite-backed cell loses the dice roll on a given run (seen in both a Next.js SQL/KB cell and a React OIDC/DSQL cell).
Impact
AWS Blocks local server running on http://localhost:3000then immediately crashes, sodev_server_started=false, every downstream check scores0, and the whole cell is lost.@aws-blocks/bb-distributed-data'sDsqlMockEngine(DSQL mock) or a PGlite-backedDatabaseand runs migrations at startup.Evidence
Verbatim stack (identical WASM module hash
wasm://wasm/01edd1baacross runs):Observed on two separate CI runs (different cells, same signature):
29138896893—sql-kb-catalog(Next.js) cell29140413875—oidc-dsql-notes(React) cellCrash path.
DsqlMockEngine's constructor doesnew PGlite(dataDir); PGlite defers the real Postgres bootstrap (_pg_initdb) to the first query.runMigrations()issues that first query —CREATE TABLE IF NOT EXISTS _migrations (...)— which drives_checkReady->qe->Module._pg_initdb, where the WASM module trapsunreachable. The banner prints before this, so it is not a port/startup race — it is a WASM init failure on the first DB operation.Reproduction
A minimal, dependency-pinned harness boots a fresh PGlite instance per cycle and issues the exact first two statements
runMigrationsruns (CREATE TABLE IF NOT EXISTS _migrations (...)thenSELECT name FROM _migrations ORDER BY name), hammering the_pg_initdbpath sequentially and in parallel (the crash is intermittent). No repo checkout required.repro.mjs:Local reproduction status (honest)
I could not reproduce the exact
unreachableWASM trap on a dev host — but the init path is provably memory-sensitive there:--max-old-space-size=128ulimit -v(2.2–3.1 GB) + parallelRangeError: WebAssembly.Memory(): could not allocate memoryatpe.qe -> pe._checkReady -> pe.queryMemoryMax(MemorySwapMax=0)unreachableConclusion: the
_pg_initdb/_checkReady/qepath is memory-sensitive and fails under pressure, but the specific WASMunreachabletrap is CI-environment-specific — it needs Node v22.23.1 on a GitHububunturunner under the memory/overcommit/RSS pressure of the concurrent (10-wide) bench matrix. It is not fabricated here; run the harness in that environment to surface it.To surface it on CI
Run the harness (or the test below) on an
ubuntu-latestGitHub runner with Node22.23.1, under concurrency mirroring the bench matrix (multiple PGlite boots in flight, constrained memory). Suggested:SEQ=80 PAR=24 ROUNDS=5, several parallel job shards, and/or a container with--memoryin the 400–600 MB band.Automated test
A
node --testregression that boots many fresh PGlite instances sequentially and in parallel and asserts none trapunreachable. It fails/flakes when the bug is present and passes when PGlite inits cleanly. Tunable viaSEQ/PAR/ROUNDSenv vars.Local runs (both green — bug did not surface locally, as noted above):
# pass 2 # fail 0(~75 s)pglite-initdb.test.mjs:Environment
@electric-sql/pglite: 0.2.17 (declared^0.2.0inpackages/bb-data/package.jsonandpackages/bb-distributed-data/package.json)ubunturunner) — where the crash occurs@aws-blocks/bb-distributed-data(DsqlMockEngine,runMigrations) ->@electric-sql/pglitewasm://wasm/01edd1baPossible directions
@electric-sql/pgliteand re-test under Node 22.23.1 (upstream fixed several WASM init/OOM traps in later 0.2.x / 0.3.x)._pg_initdb-triggering query inDsqlMockEngine/runMigrationsso a transient WASM init trap doesn't kill the dev server.