Skip to content

PGlite WASM _pg_initdb crash (QueryFailedException: unreachable) intermittently kills the dev server during runMigrations on CI #188

Description

@hfurkanbozkurt

Summary

The local dev server intermittently crashes on CI right after printing its startup banner, when a PGlite-backed data Block runs its first migration. The @electric-sql/pglite WASM module traps with unreachable inside _pg_initdb (the lazy Postgres init that fires on the first query), surfacing as QueryFailedException: unreachable and taking down the Node process.

The crash is non-deterministic — it hits whichever PGlite-backed cell loses the dice roll on a given run (seen in both a Next.js SQL/KB cell and a React OIDC/DSQL cell).

Impact

  • The dev server prints AWS Blocks local server running on http://localhost:3000 then immediately crashes, so dev_server_started=false, every downstream check scores 0, and the whole cell is lost.
  • Intermittent and init-time; a retry usually succeeds. It is an infrastructure/WASM failure, not a code regression.
  • Affects any Block that boots @aws-blocks/bb-distributed-data's DsqlMockEngine (DSQL mock) or a PGlite-backed Database and runs migrations at startup.

Evidence

Verbatim stack (identical WASM module hash wasm://wasm/01edd1ba across runs):

AWS Blocks local server running on http://localhost:3000
wasm://wasm/01edd1ba:1

QueryFailedException: unreachable
    at wasm://wasm/01edd1ba:wasm-function[3625]:0x263199
    at wasm://wasm/01edd1ba:wasm-function[7801]:0x3e3508
    at wasm://wasm/01edd1ba:wasm-function[2533]:0x1c1034
    at Object.Module._pg_initdb (.../@electric-sql/pglite/dist/index.js:3:124284)
    at pe.qe (.../@electric-sql/pglite/dist/index.js:3:243608)
    at async pe._checkReady (.../@electric-sql/pglite/dist/index.js:3:238920)
    at async pe.query (.../@electric-sql/pglite/src/base.ts:153:5)
    at async DsqlMockEngine.execute (.../@aws-blocks/bb-distributed-data/dist/engines/dsql-mock-engine.js:79:33)
    at async runMigrations (.../@aws-blocks/bb-distributed-data/dist/migrations.js:11:5)
    at async DsqlMockEngine.withDdl (.../@aws-blocks/bb-distributed-data/dist/engines/dsql-mock-engine.js:61:20)

Node.js v22.23.1

Observed on two separate CI runs (different cells, same signature):

  • run 29138896893sql-kb-catalog (Next.js) cell
  • run 29140413875oidc-dsql-notes (React) cell

Crash path. DsqlMockEngine's constructor does new PGlite(dataDir); PGlite defers the real Postgres bootstrap (_pg_initdb) to the first query. runMigrations() issues that first query — CREATE TABLE IF NOT EXISTS _migrations (...) — which drives _checkReady -> qe -> Module._pg_initdb, where the WASM module traps unreachable. The banner prints before this, so it is not a port/startup race — it is a WASM init failure on the first DB operation.

Reproduction

A minimal, dependency-pinned harness boots a fresh PGlite instance per cycle and issues the exact first two statements runMigrations runs (CREATE TABLE IF NOT EXISTS _migrations (...) then SELECT name FROM _migrations ORDER BY name), hammering the _pg_initdb path sequentially and in parallel (the crash is intermittent). No repo checkout required.

mkdir pglite-repro && cd pglite-repro
npm init -y >/dev/null && npm pkg set type=module >/dev/null
npm i @electric-sql/pglite@0.2.17
# save repro.mjs (below), then:
node repro.mjs 40 16     # 40 sequential + 16 parallel boots  (args: SEQ PAR)

repro.mjs:

// Minimal reproduction for the @electric-sql/pglite `_pg_initdb` WASM trap
// ("unreachable") that crashes the AWS Blocks dev server right after the
// "AWS Blocks local server running" banner.
//
// It boots a FRESH PGlite instance per cycle (as DsqlMockEngine does in its
// constructor: `new PGlite(dataDir)`) and issues the first query, which is
// exactly what `runMigrations` does first ("CREATE TABLE IF NOT EXISTS
// _migrations ..."). PGlite lazily runs `_pg_initdb` on that first query — the
// trap site. We hammer it N times sequentially and M in parallel, since the
// crash is intermittent (surfaces under CI memory/concurrency pressure).
//
// Run:  node repro.mjs [SEQ=30] [PAR=8]
import { PGlite } from '@electric-sql/pglite';
import { mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';

const SEQ = Number(process.argv[2] ?? 30);
const PAR = Number(process.argv[3] ?? 8);

// The exact first two statements runMigrations issues (bb-distributed-data).
const MIGRATIONS_DDL = `
  CREATE TABLE IF NOT EXISTS _migrations (
    id TEXT PRIMARY KEY DEFAULT gen_random_uuid(),
    name TEXT NOT NULL UNIQUE,
    applied_at TIMESTAMP NOT NULL DEFAULT NOW()
  )
`;

async function bootAndMigrateOnce(label) {
  const dir = mkdtempSync(join(tmpdir(), 'pglite-repro-'));
  const db = new PGlite(dir); // DsqlMockEngine ctor does exactly this
  try {
    await db.query(MIGRATIONS_DDL); // first query → triggers _pg_initdb
    await db.query('SELECT name FROM _migrations ORDER BY name');
    return { label, ok: true };
  } finally {
    try { await db.close(); } catch {}
    try { rmSync(dir, { recursive: true, force: true }); } catch {}
  }
}

const isUnreachable = (e) => /unreachable/i.test(String(e?.message ?? e)) || /unreachable/i.test(String(e?.stack ?? ''));

let failures = 0;
console.log(`[repro] pglite _pg_initdb stress — ${SEQ} sequential + ${PAR} parallel boots`);

for (let i = 0; i < SEQ; i++) {
  try {
    await bootAndMigrateOnce(`seq#${i}`);
    process.stdout.write('.');
  } catch (e) {
    failures++;
    console.error(`\n[repro] SEQ boot #${i} FAILED (unreachable=${isUnreachable(e)}):\n`, e?.stack ?? e);
  }
}
process.stdout.write('\n');

const settled = await Promise.allSettled(
  Array.from({ length: PAR }, (_, i) => bootAndMigrateOnce(`par#${i}`)),
);
for (const r of settled) {
  if (r.status === 'rejected') {
    failures++;
    console.error(`[repro] PAR boot FAILED (unreachable=${isUnreachable(r.reason)}):\n`, r.reason?.stack ?? r.reason);
  }
}

if (failures === 0) {
  console.log(`[repro] OK — ${SEQ + PAR} boots, no _pg_initdb trap on this host.`);
  process.exit(0);
} else {
  console.error(`[repro] REPRODUCED — ${failures}/${SEQ + PAR} boots crashed in _pg_initdb.`);
  process.exit(1);
}

Local reproduction status (honest)

I could not reproduce the exact unreachable WASM trap on a dev host — but the init path is provably memory-sensitive there:

Environment Result
Node v18.20.2, 30 seq + 8 par clean, no trap (38/38 OK)
Node v22.22.3, 40 seq + 16 par clean, no trap (56/56 OK)
Node 22, 32-wide parallel + --max-old-space-size=128 clean, no trap
Node 22, ulimit -v (2.2–3.1 GB) + parallel fails in the same init path as RangeError: WebAssembly.Memory(): could not allocate memory at pe.qe -> pe._checkReady -> pe.query
Node 22, cgroup MemoryMax (MemorySwapMax=0) sharp cliff: <=350M -> OOM-kill (rc 137); >=450M -> clean — no band that yields unreachable

Conclusion: the _pg_initdb / _checkReady / qe path is memory-sensitive and fails under pressure, but the specific WASM unreachable trap is CI-environment-specific — it needs Node v22.23.1 on a GitHub ubuntu runner under the memory/overcommit/RSS pressure of the concurrent (10-wide) bench matrix. It is not fabricated here; run the harness in that environment to surface it.

To surface it on CI

Run the harness (or the test below) on an ubuntu-latest GitHub runner with Node 22.23.1, under concurrency mirroring the bench matrix (multiple PGlite boots in flight, constrained memory). Suggested: SEQ=80 PAR=24 ROUNDS=5, several parallel job shards, and/or a container with --memory in the 400–600 MB band.

Automated test

A node --test regression that boots many fresh PGlite instances sequentially and in parallel and asserts none trap unreachable. It fails/flakes when the bug is present and passes when PGlite inits cleanly. Tunable via SEQ / PAR / ROUNDS env vars.

node --test pglite-initdb.test.mjs      # knobs: SEQ=40 PAR=12 ROUNDS=3 (defaults)

Local runs (both green — bug did not surface locally, as noted above):

  • Node v18.20.2 -> # pass 2 # fail 0 (~75 s)
  • Node v22.22.3 -> clean

pglite-initdb.test.mjs:

// Automated regression test for the @electric-sql/pglite `_pg_initdb` WASM trap
// ("QueryFailedException: unreachable") observed on CI, crashing the AWS Blocks
// dev server during runMigrations' first query.
//
// The test boots many FRESH PGlite instances — sequentially and in parallel —
// each issuing the first migration query (the `_pg_initdb` trigger). It asserts
// that NONE trap with "unreachable". When the bug is present the test FAILS
// (or flakes); when PGlite inits cleanly it passes.
//
// Run:  node --test pglite-initdb.test.mjs
//   Tune stress via env: SEQ (default 40), PAR (default 12), ROUNDS (default 3).
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { PGlite } from '@electric-sql/pglite';
import { mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';

const SEQ = Number(process.env.SEQ ?? 40);
const PAR = Number(process.env.PAR ?? 12);
const ROUNDS = Number(process.env.ROUNDS ?? 3);

const MIGRATIONS_DDL = `
  CREATE TABLE IF NOT EXISTS _migrations (
    id TEXT PRIMARY KEY DEFAULT gen_random_uuid(),
    name TEXT NOT NULL UNIQUE,
    applied_at TIMESTAMP NOT NULL DEFAULT NOW()
  )
`;

const isUnreachable = (e) =>
  /unreachable/i.test(String(e?.message ?? e)) || /unreachable/i.test(String(e?.stack ?? ''));

async function bootAndMigrateOnce() {
  const dir = mkdtempSync(join(tmpdir(), 'pglite-test-'));
  const db = new PGlite(dir); // mirrors DsqlMockEngine constructor
  try {
    await db.query(MIGRATIONS_DDL);                                  // first query → _pg_initdb
    await db.query('SELECT name FROM _migrations ORDER BY name');    // runMigrations' next query
  } finally {
    try { await db.close(); } catch {}
    try { rmSync(dir, { recursive: true, force: true }); } catch {}
  }
}

async function runGuarded() {
  try {
    await bootAndMigrateOnce();
    return null;
  } catch (e) {
    return e;
  }
}

test(`PGlite _pg_initdb never traps 'unreachable' across ${SEQ} sequential boots`, async () => {
  const crashes = [];
  for (let i = 0; i < SEQ; i++) {
    const err = await runGuarded();
    if (err) crashes.push({ i, unreachable: isUnreachable(err), stack: err?.stack ?? String(err) });
  }
  assert.deepEqual(
    crashes.filter(c => c.unreachable),
    [],
    `PGlite _pg_initdb trapped 'unreachable':\n${crashes.map(c => `seq#${c.i}: ${c.stack}`).join('\n\n')}`,
  );
  assert.equal(crashes.length, 0, `Non-unreachable failures:\n${crashes.map(c => c.stack).join('\n\n')}`);
});

test(`PGlite _pg_initdb never traps 'unreachable' across ${ROUNDS}×${PAR} parallel boots`, async () => {
  const crashes = [];
  for (let r = 0; r < ROUNDS; r++) {
    const settled = await Promise.allSettled(Array.from({ length: PAR }, () => bootAndMigrateOnce()));
    settled.forEach((s, i) => {
      if (s.status === 'rejected') {
        crashes.push({ r, i, unreachable: isUnreachable(s.reason), stack: s.reason?.stack ?? String(s.reason) });
      }
    });
  }
  assert.deepEqual(
    crashes.filter(c => c.unreachable),
    [],
    `PGlite _pg_initdb trapped 'unreachable':\n${crashes.map(c => `round${c.r}/par#${c.i}: ${c.stack}`).join('\n\n')}`,
  );
  assert.equal(crashes.length, 0, `Non-unreachable failures:\n${crashes.map(c => c.stack).join('\n\n')}`);
});

Environment

  • @electric-sql/pglite: 0.2.17 (declared ^0.2.0 in packages/bb-data/package.json and packages/bb-distributed-data/package.json)
  • CI Node: v22.23.1 (GitHub ubuntu runner) — where the crash occurs
  • Dev host Node tested: v18.20.2 and v22.22.3 — no trap
  • Crash-path packages: @aws-blocks/bb-distributed-data (DsqlMockEngine, runMigrations) -> @electric-sql/pglite
  • WASM module hash across all crashes: wasm://wasm/01edd1ba

Possible directions

  • Bump/pin @electric-sql/pglite and re-test under Node 22.23.1 (upstream fixed several WASM init/OOM traps in later 0.2.x / 0.3.x).
  • Add a bounded retry around the first _pg_initdb-triggering query in DsqlMockEngine / runMigrations so a transient WASM init trap doesn't kill the dev server.
  • Cap PGlite boot concurrency in the bench harness (serialize migrations across cells) to reduce peak memory pressure on the runner.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpending-maintainer-responseWaiting on a response from the maintainerspending-triageIssue is pending triage by the maintainers

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions