test(ci): heartbeat + running-test pointer in backend test diagnostics

Backend tests silently die with code 255 mid-suite ~22% of the time on
develop (most often Windows-with-plugins, Node 24). Each kill lands
300±50 ms after the previous test's clean ✔ teardown line and produces
no failing-test marker, no error, no Mocha summary, and — despite the
unconditional handlers in `diagnostics.ts` — none of the JS-level death
events fire either. Recent example: run 26311025244 (`Windows with
Plugins (24)`); both attempts crashed at completely different "last
test" locations, so the dying test itself isn't to blame.

The existing diagnostics only set lastSeenTest in afterEach, so if the
kill lands during the NEXT test's setup or body — which is exactly the
~300ms gap we observe — the pointer reads as the previous (passing)
test. That hides whether we're between tests or inside one, and which
one.

Two changes:

1. Track currentTest in beforeEach as well as lastFinishedTest in
   afterEach. Every diag line now carries both, so the death point is
   bracketable regardless of which lifecycle phase the kill interrupts.

2. Add a 1Hz heartbeat that writeSyncs the running-test name plus
   `process.memoryUsage()` (rss, heap) and the active-handle and
   active-request counts. The interval is unref'd so it never holds the
   event loop open by itself. Cost is roughly one extra log line per
   second of mocha runtime (~60-120 lines per CI run).

When the next failure fires, the last heartbeat narrows the kill window
to ≤1s, the running pointer names the test on the rails at that moment,
and the handle/memory trace gives a sparkline that exposes sudden
spikes — a leaked socket, an unref'd timer, a runaway map — that
would otherwise be invisible at the runner-log level.

No behavior change on successful runs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: JohnMcLear

src/tests/backend/diagnostics.ts

@@ -10,6 +10,14 @@
 // in a way that bypassed JS handlers — SIGKILL, OOM, or a fatal native
 // error — OR mocha itself called process.exit before the handlers ran.
 //
+// Subsequent runs on develop (e.g. 26311025244, Windows with plugins,
+// Node 24) reproduced the same fingerprint: `[diag +0ms] diagnostics
+// loaded` lands but no other diag line appears before the silent
+// ELIFECYCLE. Death lands 300±50 ms after the previous test's teardown
+// and the lastSeenTest pointer (only updated in afterEach) tells us
+// nothing about whether the next test had started or whether mocha was
+// between tests at the kill moment.
+//
 // This file:
 //   1. Registers handlers UNCONDITIONALLY at mocha startup (common.ts is
 //      only imported by ~27 of 47 specs, so its handlers may register
@@ -18,9 +26,17 @@
 //      complete before the kernel returns from the syscall, so the line
 //      lands in the runner log even if the process is killed
 //      milliseconds later.
-//   3. Tracks the last-seen test via a mocha root afterEach hook so the
-//      death point is identified.
-//   4. Logs exit-related events so we can discriminate:
+//   3. Tracks BOTH the currently-running test (set in beforeEach) and the
+//      last-finished test (set in afterEach), so the death point can be
+//      bracketed even when the kill lands inside the next test's setup or
+//      body, before its own afterEach has a chance to update the pointer.
+//   4. Emits a 1Hz heartbeat carrying the running-test name, RSS / heap
+//      usage, and active handle / request counts. If the process dies
+//      without firing any JS handler, the last heartbeat narrows the
+//      kill window to <=1s and the handle-count trace exposes leaks
+//      (sockets, timers, native bindings) that would otherwise be
+//      invisible at the runner-log level.
+//   5. Logs exit-related events so we can discriminate:
 //        beforeExit + exit  -> clean event-loop drain (Linux CI, local)
 //        only exit          -> process.exit() called — expected when mocha
 //                              is launched with --exit (the Windows CI
@@ -35,7 +51,8 @@
 import {writeSync} from 'node:fs';
 
 const t0 = Date.now();
-let lastSeenTest = '<no test seen yet>';
+let currentTest = '<no test running>';
+let lastFinishedTest = '<no test finished yet>';
 
 const diag = (msg: string): void => {
   const line = `[diag +${Date.now() - t0}ms] ${msg}\n`;
@@ -48,18 +65,33 @@ const diag = (msg: string): void => {
 
 diag('diagnostics loaded');
 
+// Heartbeat. unref()'d so it never holds the event loop open by itself —
+// it only fires if mocha is otherwise alive. The interval cadence (1Hz) is
+// the trade-off between log noise (~60-120 extra lines per run) and how
+// tightly we can bracket the kill timestamp.
+const heartbeat = setInterval(() => {
+  const mem = process.memoryUsage();
+  const handles = (process as any)._getActiveHandles?.().length ?? -1;
+  const requests = (process as any)._getActiveRequests?.().length ?? -1;
+  diag(`hb running="${currentTest}" lastFinished="${lastFinishedTest}" ` +
+    `rss=${Math.round(mem.rss / 1024 / 1024)}M ` +
+    `heap=${Math.round(mem.heapUsed / 1024 / 1024)}M ` +
+    `handles=${handles} requests=${requests}`);
+}, 1000);
+heartbeat.unref();
+
 process.on('unhandledRejection', (reason: any) => {
   diag(`unhandledRejection: ${
     reason && reason.stack ? reason.stack : String(reason)
-  } (lastTest="${lastSeenTest}")`);
+  } (running="${currentTest}", lastFinished="${lastFinishedTest}")`);
   // Re-throw so existing common.ts handlers / mocha behavior is preserved.
   throw reason;
 });
 
 process.on('uncaughtException', (err: any) => {
   diag(`uncaughtException: ${
     err && err.stack ? err.stack : String(err)
-  } (lastTest="${lastSeenTest}")`);
+  } (running="${currentTest}", lastFinished="${lastFinishedTest}")`);
   // Force fail-fast. Specs that don't import common.ts only have THIS handler,
   // and Node won't exit on its own once an uncaughtException listener is
   // registered. Without the explicit exit a fatal error would be swallowed.
@@ -69,18 +101,18 @@ process.on('uncaughtException', (err: any) => {
 
 process.on('beforeExit', (code: number) => {
   diag(`beforeExit code=${code} exitCode=${process.exitCode} ` +
-    `lastTest="${lastSeenTest}"`);
+    `running="${currentTest}" lastFinished="${lastFinishedTest}"`);
 });
 
 process.on('exit', (code: number) => {
-  diag(`exit code=${code} lastTest="${lastSeenTest}"`);
+  diag(`exit code=${code} running="${currentTest}" lastFinished="${lastFinishedTest}"`);
 });
 
 for (const sig of ['SIGINT', 'SIGTERM', 'SIGHUP', 'SIGBREAK'] as const) {
   // SIGHUP / SIGBREAK don't exist on every platform; ignore registration errors.
   try {
     process.on(sig as any, () => {
-      diag(`received ${sig} (lastTest="${lastSeenTest}")`);
+      diag(`received ${sig} (running="${currentTest}", lastFinished="${lastFinishedTest}")`);
       // Let the default behavior (exit) happen.
       process.exit(128);
     });
@@ -89,12 +121,19 @@ for (const sig of ['SIGINT', 'SIGTERM', 'SIGHUP', 'SIGBREAK'] as const) {
   }
 }
 
-// Mocha root hook — only registered if mocha picks up this file via --require.
-// We track the most recently-finished test so the death point is visible.
+// Mocha root hooks — only registered if mocha picks up this file via --require.
+// beforeEach sets the running pointer so a mid-test kill is attributable to a
+// specific test, not just the previous one that successfully finished.
 export const mochaHooks = {
+  beforeEach(this: any) {
+    if (this.currentTest) {
+      currentTest = this.currentTest.fullTitle();
+    }
+  },
   afterEach(this: any) {
     if (this.currentTest) {
-      lastSeenTest = this.currentTest.fullTitle();
+      lastFinishedTest = this.currentTest.fullTitle();
+      currentTest = '<no test running>';
     }
   },
 };