docs(claude-notes): root-cause Firefox q2-preview peer-connection timeout (bd-jit6pdwq)

Research note + repro scripts for the sporadic Firefox-only
'Timeout waiting for peer connection' failure in q2 preview.

Root cause: Firefox serializes WebSocket opening handshakes per
resolved IP address browser-wide (nsWSAdmissionManager, per RFC 6455
S4.1), so any tab holding a hung handshake to 127.0.0.1 — any port —
starves all other localhost WS handshakes for up to 20s per attempt.
The preview SPA's 5s peer budget then expires and the boot fails
permanently. Reproduced with Playwright Firefox + a blackhole TCP
server; Chromium control unaffected; closed-port control unaffected.

Scripts in claude-notes/tmp/ are the evidence chain referenced by the
note and by the bd-jit6pdwq strand comment.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: cscheid

claude-notes/research/2026-06-11-firefox-ws-handshake-serialization.md

@@ -0,0 +1,156 @@
+# Firefox `q2 preview` peer-connection timeout: root cause
+
+**Date:** 2026-06-11
+**Strand:** bd-jit6pdwq (q2 preview: Firefox flaky 'Document automerge:<id> is unavailable' on cold start)
+**Symptom:** sporadic, Firefox-only `Peer connection failed, continuing in
+offline mode: Error: Timeout waiting for peer connection`, followed by a
+permanent `Document … is unavailable` boot error in the preview SPA.
+
+## TL;DR
+
+Firefox serializes WebSocket *opening handshakes* per host, browser-wide:
+only one WebSocket to `127.0.0.1` may be in the CONNECTING state at a
+time, across **all tabs**. If any tab holds a WebSocket handshake that
+hangs (TCP accepted, HTTP 101 never sent — e.g. a wedged/suspended
+`q2 preview` server, or any unresponsive localhost endpoint), every other
+localhost WebSocket handshake in the browser queues behind it for up to
+Firefox's `network.websocket.timeout.open` (20 s default) per attempt.
+The preview SPA budgets only 5 s for the samod `peer` event, so it falls
+into "offline mode", `repo.find()` then marks the (ephemeral, never
+cached) index doc unavailable, and the boot fails **permanently** — even
+though the WebSocket typically connects fine a few seconds later.
+
+Chromium does not serialize handshakes this way and is immune.
+
+## Evidence (all scripts in `claude-notes/tmp/`)
+
+1. **Server exonerated** (`peer-handshake-probe.mjs`): 10/10 Node
+   connections to the live failing server (port 59776) reached `peer` in
+   0.5–8 ms. The Rust hub answers WS upgrades instantly.
+
+2. **Clean Firefox exonerated** (`firefox-repro.mjs`): 15/15 cold loads
+   of the preview SPA in a fresh Playwright Firefox connected in ~260 ms.
+   The bug does not reproduce in an idle browser → environmental trigger.
+
+3. **Reproduction** (`firefox-serialization-test.mjs` +
+   `blackhole-server.mjs`): one tab holding a WebSocket to a localhost
+   port that accepts TCP but never answers the upgrade ("blackhole",
+   simulating a wedged/suspended server). A second tab then loads the
+   preview SPA:
+
+   - **Firefox**: `Peer connection failed, continuing in offline mode`
+     at t=5.29 s — the exact production failure. Retries at +5 s and
+     +10 s also starved.
+   - **Chromium** (control): `Peer connected - online mode` at t=59 ms
+     under identical conditions.
+   - **Firefox + stale tab on a *closed* port** (control): peer at
+     430 ms. Fast TCP-reset failures do *not* starve the queue; a
+     genuinely hung handshake is required.
+
+   The per-host serialization lives in Firefox's `nsWSAdmissionManager`
+   (`netwerk/protocol/websocket/WebSocketChannel.cpp`).
+
+## Source-level confirmation that the key excludes the port
+
+Verified against mozilla-central master (2026-06-11),
+`netwerk/protocol/websocket/WebSocketChannel.cpp`:
+
+- `nsWSAdmissionManager::ConditionallyConnect` defers a handshake when
+  `IndexOf(ws->mAddress, ws->mOriginSuffix) >= 0` — the queue lookup
+  compares **resolved IP address + origin attributes only**; `nsOpenConn`
+  carries no port or path. Code comment: "If there is already another WS
+  channel connecting to this IP address, defer BeginOpen and mark as
+  waiting in queue."
+- `mAddress` is the **DNS-resolved IP** (`GetNextAddrAsString`), so
+  `localhost` and `127.0.0.1` share one key.
+- The **port-aware** structure is the separate `FailDelayManager`
+  (reconnect backoff), keyed by `(address, path, port)` — which is why
+  the closed-port control didn't interfere cross-port but the hung
+  handshake did.
+- This implements RFC 6455 §4.1 literally: "If multiple connections to
+  the same IP address are attempted simultaneously, the client MUST
+  serialize them so that there is no more than one connection at a
+  time." The spec's serialization clause is per-IP, portless. Chromium
+  interprets it loosely (our control: 59 ms with a pending handshake).
+- Caveat: `mOriginSuffix` carries container-tab / private-browsing
+  identity, so handshakes in a Firefox container or private window do
+  **not** share the slot with normal tabs. (Diagnostic implication: the
+  bug "disappearing" in a private window is consistent with this root
+  cause, not evidence against it.)
+
+## Why this user, why sporadic
+
+The workflow accumulates localhost WebSocket clients: every `q2 preview`
+session opens a tab whose automerge-repo `WebSocketClientAdapter` retries
+`ws://127.0.0.1:<port>/ws` every 5 s, **forever**, after the server goes
+away. Q1 (`quarto preview`) live-reload tabs do the same on their ports.
+Any one of these endpoints entering an accept-but-don't-respond state
+(server wedged, process suspended, kernel listen-backlog of a dying
+process, single-threaded server busy) turns its tab into a perpetual
+slot-holder: each retry occupies the browser-wide handshake slot for up
+to 20 s. While that state persists, *every* new preview tab fails its
+5 s peer wait; close the offending tab (or the endpoint recovers) and
+previews work again. Hence "sporadic", and "reproduces every time" on a
+bad day (bd-jit6pdwq).
+
+## Aggravating design factors found along the way
+
+These make a transient handshake delay into a hard, unrecoverable failure:
+
+1. **5 s peer budget == 5 s adapter retry interval.**
+   `PreviewApp.tsx` passes `peerTimeoutMs: 5000`;
+   `WebSocketClientAdapter`'s `retryInterval` default is also 5000 ms.
+   Any first-attempt failure loses the race by construction — recovery
+   at t≥5 s can never beat the deadline at t=5 s.
+
+2. **No recovery after the timeout.** After `waitForPeer` rejects,
+   `connect()` proceeds to `findDoc()`; `networkSubsystem.whenReady()`
+   force-resolves 1 s after adapter creation even when unconnected, so
+   `handle.request()` runs against zero peers → handle resolves
+   UNAVAILABLE → `connect()` throws → SPA sets `boot: 'error'`
+   permanently. The adapter often connects seconds later (we observed
+   this), but nothing retries the boot.
+
+3. **IndexedDB gates the WebSocket `join`.** automerge-repo's
+   `NetworkSubsystem` defers `adapter.connect()` (which creates the
+   WebSocket and sends `join`) until `storageSubsystem.id()` — an
+   IndexedDB open+read — resolves (verified empirically: a 6 s storage
+   delay produces the identical timeout; `slow-storage-probe.mjs`).
+   The preview SPA gets IndexedDB unconditionally from
+   `buildStorageAdapter()`, yet its docs are ephemeral — the cache can
+   never hit, and each preview port pollutes the Firefox profile with a
+   new origin's database. Any IDB slowness (Firefox is notorious)
+   silently eats the peer budget. Not the trigger observed here, but a
+   second independent path to the same failure.
+
+4. **Stale preview tabs retry forever** (no backoff cap, no give-up),
+   which is what keeps a hung endpoint's slot occupied perpetually and
+   adds localhost WS churn generally.
+
+## Fix directions (not implemented in this session)
+
+- **Make boot resilient instead of deadline-bound** (primary): for the
+  preview SPA, offline mode is meaningless — don't hard-fail on the 5 s
+  peer timeout. Wait for the `peer` event (with status UI), or retry
+  `findDoc` when a peer connects after a failed boot. automerge-repo
+  handles can recover from UNAVAILABLE via `progress`/re-request.
+- **De-align the deadlines**: if a finite budget is kept, make it ≫ the
+  adapter retry interval (e.g. 15–30 s), or pass a smaller
+  `retryInterval` to the adapter.
+- **Memory storage for the preview SPA**: thread a storage option
+  through `createSyncClient()` so preview uses `MemoryStorageAdapter`;
+  removes the IDB gate on `join` and stops per-port profile pollution.
+- **Cap stale-tab retries**: exponential backoff and/or a "server gone —
+  reload when ready" terminal state in the SPA, so dead preview tabs
+  stop hammering localhost. (Both kindness to Firefox's handshake queue
+  and battery.)
+- **User-side mitigation meanwhile**: close old preview tabs; if a
+  preview suddenly shows the offline error, suspect some localhost tab
+  whose server is hung (check DevTools Network → WS for a pending
+  handshake) rather than the new preview itself.
+
+## Loose end (resolved)
+
+The user's 59776 preview server went down mid-investigation; the user
+confirmed they closed it themselves. No spontaneous server death
+observed; a sibling instance survived identical probing.

claude-notes/tmp/blackhole-server.mjs

@@ -0,0 +1,10 @@
+// TCP server that accepts connections and never responds — simulates a
+// wedged/suspended q2 preview server holding WS handshakes in CONNECTING.
+import net from 'node:net';
+const port = Number(process.argv[2] ?? 59001);
+const server = net.createServer((socket) => {
+  // accept, read, never write, never close
+  socket.on('data', () => {});
+  socket.on('error', () => {});
+});
+server.listen(port, '127.0.0.1', () => console.log(`blackhole listening on ${port}`));

claude-notes/tmp/firefox-repro.mjs

@@ -0,0 +1,45 @@
+// Reproduce q2-preview peer-connection timeout in Firefox via Playwright.
+// Usage: node firefox-repro.mjs [url] [iterations]
+import { firefox } from 'playwright';
+
+const url = process.argv[2] ?? 'http://127.0.0.1:59899/';
+const iterations = Number(process.argv[3] ?? 15);
+
+const browser = await firefox.launch();
+let failures = 0;
+
+for (let i = 0; i < iterations; i++) {
+  const context = await browser.newContext(); // fresh storage each time
+  const page = await context.newPage();
+  const t0 = Date.now();
+  const events = [];
+  page.on('console', (msg) => {
+    const text = msg.text();
+    if (/peer|offline|Timeout|error|Error|unavailable/i.test(text)) {
+      events.push(`[${Date.now() - t0}ms] console.${msg.type()}: ${text}`);
+    }
+  });
+  page.on('websocket', (ws) => {
+    events.push(`[${Date.now() - t0}ms] websocket opened: ${ws.url()}`);
+    ws.on('close', () => events.push(`[${Date.now() - t0}ms] websocket closed`));
+  });
+  page.on('pageerror', (err) => events.push(`[${Date.now() - t0}ms] pageerror: ${err.message}`));
+
+  try {
+    await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 20000 });
+    // give the SPA time to boot: wasm init + connect (5s peer budget) + render
+    await page.waitForTimeout(9000);
+  } catch (e) {
+    events.push(`goto failed: ${e.message}`);
+  }
+
+  const failed = events.some((e) => /Timeout waiting for peer/.test(e));
+  if (failed) failures++;
+  console.log(`--- run ${i}${failed ? '  *** PEER TIMEOUT ***' : ''}`);
+  for (const e of events) console.log('   ', e);
+  await context.close();
+}
+
+console.log(`\n${failures}/${iterations} runs hit the peer timeout`);
+await browser.close();
+process.exit(0);

claude-notes/tmp/firefox-serialization-test.mjs

@@ -0,0 +1,46 @@
+// Does a hung WebSocket handshake to 127.0.0.1:<portA> delay a fresh
+// WebSocket handshake to 127.0.0.1:<portB> in the same browser?
+// Usage: node firefox-serialization-test.mjs <browser:firefox|chromium> <previewUrl> <blackholePort>
+import { firefox, chromium } from 'playwright';
+
+const browserName = process.argv[2] ?? 'firefox';
+const previewUrl = process.argv[3] ?? 'http://127.0.0.1:59899/';
+const blackholePort = Number(process.argv[4] ?? 59001);
+
+const browser = await (browserName === 'firefox' ? firefox : chromium).launch();
+const context = await browser.newContext();
+
+// Tab A: stale preview tab analog — WS to the blackhole port, retrying.
+const tabA = await context.newPage();
+await tabA.goto(previewUrl); // same-origin page so we can open ws from it
+await tabA.evaluate((port) => {
+  const tryConnect = () => {
+    const ws = new WebSocket(`ws://127.0.0.1:${port}/ws`);
+    ws.onclose = () => setTimeout(tryConnect, 500);
+    ws.onerror = () => {};
+  };
+  tryConnect();
+}, blackholePort);
+console.log('tab A: hung-handshake websocket started');
+
+// Give tab A's handshake a moment to enter CONNECTING.
+await new Promise((r) => setTimeout(r, 1500));
+
+// Tab B: fresh preview load. Measure time to peer.
+const tabB = await context.newPage();
+const t0 = Date.now();
+const events = [];
+tabB.on('console', (msg) => {
+  const text = msg.text();
+  if (/peer|offline|Timeout/i.test(text)) {
+    events.push(`[${Date.now() - t0}ms] ${msg.type()}: ${text}`);
+  }
+});
+tabB.on('websocket', (ws) => events.push(`[${Date.now() - t0}ms] websocket opened: ${ws.url()}`));
+await tabB.goto(previewUrl, { waitUntil: 'domcontentloaded' });
+await tabB.waitForTimeout(12000);
+
+console.log(`\n=== ${browserName} tab B events ===`);
+for (const e of events) console.log('  ', e);
+await browser.close();
+process.exit(0);

claude-notes/tmp/peer-handshake-probe.mjs

@@ -0,0 +1,31 @@
+// Probe: time the samod/automerge-repo peer handshake against a running
+// q2 preview server. Usage: node peer-handshake-probe.mjs [url] [iterations]
+import { Repo } from '@automerge/automerge-repo';
+import { WebSocketClientAdapter } from '@automerge/automerge-repo-network-websocket';
+
+const url = process.argv[2] ?? 'ws://127.0.0.1:59776/ws';
+const iterations = Number(process.argv[3] ?? 10);
+
+async function once(i) {
+  const t0 = performance.now();
+  const adapter = new WebSocketClientAdapter(url);
+  const repo = new Repo({ network: [adapter] });
+  const result = await new Promise((resolve) => {
+    const timeout = setTimeout(() => resolve({ ok: false, ms: performance.now() - t0 }), 10000);
+    repo.networkSubsystem.on('peer', (p) => {
+      clearTimeout(timeout);
+      resolve({ ok: true, ms: performance.now() - t0, peerId: p.peerId });
+    });
+  });
+  adapter.disconnect();
+  console.log(`run ${i}: ${result.ok ? 'peer in ' + result.ms.toFixed(1) + ' ms (' + result.peerId + ')' : 'TIMEOUT after ' + result.ms.toFixed(0) + ' ms'}`);
+  return result;
+}
+
+let failures = 0;
+for (let i = 0; i < iterations; i++) {
+  const r = await once(i);
+  if (!r.ok) failures++;
+}
+console.log(`\n${failures}/${iterations} timed out`);
+process.exit(0);

claude-notes/tmp/slow-storage-probe.mjs

@@ -0,0 +1,43 @@
+// Mechanism proof: does a slow storage adapter delay the websocket
+// join/peer handshake in automerge-repo? Mirrors quarto-sync-client's
+// connect(): new Repo({network, storage}) then waitForPeer(5000).
+import { Repo } from '@automerge/automerge-repo';
+import { WebSocketClientAdapter } from '@automerge/automerge-repo-network-websocket';
+
+const url = process.argv[2] ?? 'ws://127.0.0.1:59776/ws';
+const storageDelayMs = Number(process.argv[3] ?? 6000);
+
+class SlowMemoryStorage {
+  data = new Map();
+  delay() { return new Promise(r => setTimeout(r, storageDelayMs)); }
+  async load(key) { await this.delay(); return this.data.get(key.join('.')); }
+  async save(key, value) { await this.delay(); this.data.set(key.join('.'), value); }
+  async remove(key) { this.data.delete(key.join('.')); }
+  async loadRange(prefix) {
+    await this.delay();
+    const p = prefix.join('.');
+    return [...this.data.entries()]
+      .filter(([k]) => k.startsWith(p))
+      .map(([k, data]) => ({ key: k.split('.'), data }));
+  }
+  async removeRange() {}
+}
+
+function waitForPeer(repo, timeoutMs) {
+  return new Promise((resolve, reject) => {
+    const t = setTimeout(() => reject(new Error('Timeout waiting for peer connection')), timeoutMs);
+    repo.networkSubsystem.on('peer', () => { clearTimeout(t); resolve(); });
+  });
+}
+
+const t0 = performance.now();
+const adapter = new WebSocketClientAdapter(url);
+const repo = new Repo({ network: [adapter], storage: new SlowMemoryStorage() });
+try {
+  await waitForPeer(repo, 5000);
+  console.log(`peer in ${(performance.now() - t0).toFixed(1)} ms — storage delay ${storageDelayMs} ms did NOT block handshake`);
+} catch (e) {
+  console.log(`TIMEOUT at ${(performance.now() - t0).toFixed(1)} ms with storage delay ${storageDelayMs} ms — storage gates the handshake`);
+}
+adapter.disconnect();
+process.exit(0);

claude-notes/tmp/slow-storage-probe2.mjs

@@ -0,0 +1,45 @@
+// Instrumented variant: trace storage id() resolution and adapter.connect timing.
+import { Repo } from '@automerge/automerge-repo';
+import { WebSocketClientAdapter } from '@automerge/automerge-repo-network-websocket';
+
+const url = process.argv[2] ?? 'ws://127.0.0.1:59776/ws';
+const storageDelayMs = Number(process.argv[3] ?? 0);
+const t0 = performance.now();
+const ts = () => (performance.now() - t0).toFixed(1) + ' ms';
+
+class SlowMemoryStorage {
+  data = new Map();
+  delay() { return new Promise(r => setTimeout(r, storageDelayMs)); }
+  async load(key) {
+    await this.delay();
+    console.log(`[${ts()}] storage.load(${key.join('.')})`);
+    return this.data.get(key.join('.'));
+  }
+  async save(key, value) {
+    await this.delay();
+    console.log(`[${ts()}] storage.save(${key.join('.')})`);
+    this.data.set(key.join('.'), value);
+  }
+  async remove() {}
+  async loadRange(prefix) {
+    await this.delay();
+    console.log(`[${ts()}] storage.loadRange(${prefix.join('.')})`);
+    return [];
+  }
+  async removeRange() {}
+}
+
+const adapter = new WebSocketClientAdapter(url);
+const origConnect = adapter.connect.bind(adapter);
+adapter.connect = (...args) => {
+  console.log(`[${ts()}] adapter.connect() called`);
+  return origConnect(...args);
+};
+
+const repo = new Repo({ network: [adapter], storage: new SlowMemoryStorage() });
+const result = await new Promise((resolve) => {
+  const t = setTimeout(() => resolve('timeout'), 5000);
+  repo.networkSubsystem.on('peer', (p) => { clearTimeout(t); resolve('peer:' + p.peerId); });
+});
+console.log(`[${ts()}] result: ${result}`);
+process.exit(0);