fix: don't time out bb socket startup while the bb process is alive (#24802)

Fixes the `Error: Timeout connecting to bb socket: unknown
(retry=false)` failures reported by an operator since v5.0.0, which
killed proving jobs during epoch top-tree and cost epochs.

## Root cause

The NativeUnixSocket backend gave bb a hard 5s wall-clock budget that
was **shared** between two phases: waiting for bb to create its socket
file, and connecting to it. `connectWithRetry` reused the `startTime`
captured before the file-wait poll loop, so when bb took close to 5s to
create the socket (many bb processes spawning simultaneously during
top-tree checkpoint/merge jobs, or the Node event loop starving the 50ms
polls), the connect phase was entered with its budget already exhausted
and threw without making a single connect attempt — that is what the
`unknown` in the error message means (`lastErr` was never set).

Two aggravating factors:

- 5s is an arbitrary opinion about how fast a loaded machine should
spawn a process. The pre-socket v4 CLI model had no such deadline — you
waited on the child, and the only failures were real process events.
- The resulting plain `Error` reached the proving agent, which only
honours the retry flag on `ProvingError`s, so the failure was reported
`retry=false` and the job failed permanently instead of being
re-enqueued.

## Changes

### `barretenberg/ts` — socket backend restructure (`native_socket.ts`)

- Replaced the constructor + deferred `connectionPromise` wiring with a
`static async new()` factory, matching the shm and wasm backends. An
instance can now only exist once connected, so `call()` no longer awaits
a stashed connection promise (and is no longer `async` — its body has no
awaits).
- Startup is one flat wait-and-connect loop with the correct liveness
condition: **retry for as long as the bb process is alive**. If bb dies,
fail immediately with the real cause (exit code / signal); spawn
failures are caught up front by awaiting the `spawn` event. Both 5s
timers are deleted.
- One generous 60s backstop remains for a bb that is alive but wedged
before `listen()`. It kills the process (routing cleanup through the
exit path) rather than leaving an orphan. It is a broken-process
detector, not a performance expectation: the timed window ends at bb's
`listen()`, which is reached after only exec + linking + minimal init
(the expensive startup work comes after the socket is up), so firing it
requires a machine degraded far beyond ordinary proving load. And if it
ever does fire on a merely-distressed machine, the failure is retryable
(see below), so the cost is a re-enqueue, not an epoch.
- The four copy-pasted reject-pending-callbacks blocks (process
error/exit, socket error/end) are consolidated into `failAllPending()`.
Note one deliberate behavioural improvement: the socket `error`/`end`
handlers now also destroy and null the socket, so subsequent `call()`s
fail fast with `Socket not connected` instead of `write after destroy`.
- New `native_socket.test.ts` covers: prompt startup, bb taking >5s to
create its socket (the incident's failure mode — fails on the old code
by construction, passes now), bb dying before the socket exists, and a
nonexistent binary.

### `yarn-project/bb-prover` — make startup failures retryable

- `BBJsInstance.create` wraps any `Barretenberg.new` failure as
`ProvingError(..., retry: true)`: bb startup failures are environmental
(machine load, wedged process), never a property of the proof inputs, so
the job is always safe to retry.
- The two catch sites in `bb_prover.ts` that re-wrap errors
(`generateProof`, `verifyProof`) previously constructed a fresh
`ProvingError` with the default `retry=false`, silently dropping the
flag. They now propagate the inner error's retryability and attach it as
`cause`. Errors that were non-retryable before remain non-retryable. The
AVM paths don't wrap, so they needed no change.
- New `bb_js_backend.test.ts` asserts a startup failure surfaces as a
retryable `ProvingError`.

## Result for operators

A loaded prover can take as long as it needs to spawn bb; a genuinely
broken bb fails after 60s with a concrete, diagnosable cause instead of
`Timeout ... unknown`; and if that ever happens the broker re-enqueues
the job (up to its retry limit) instead of failing it permanently and
costing the epoch.

## Testing

- `barretenberg/ts`: new jest suite passes (including a 7s-delayed fake
bb that the old implementation fails on).
- `yarn-project`: full `yarn build`, bb-prover lint, and the new unit
test pass. Root `./bootstrap.sh` green.


## Unrelated CI fix included

`docs/examples/ts/aztecjs_runner/run.sh` installed `typescript` and
`tsx` unpinned; the TypeScript 7.0.2 release (which drops `lib/_tsc.js`)
crashes Yarn 4's builtin `compat/typescript` patch at install time,
failing `docs/examples/bootstrap.sh execute` on every branch. Pinned to
`typescript@^5.3.3` / `tsx@^4`, matching the existing pin in
`docs/examples/ts/bootstrap.sh`. This was the only failure in this PR's
first full CI run and needs forward-porting to the `next` line as well.

Author: PhilWindle

barretenberg/ts/src/bb_backends/node/index.ts

@@ -26,7 +26,12 @@ export async function createAsyncBackend(
         throw new Error('Native backend requires bb binary.');
       }
       logger(`Using native Unix socket backend: ${bbPath}`);
-      const socket = new BarretenbergNativeSocketAsyncBackend(bbPath, options.threads, options.logger, options.unref);
+      const socket = await BarretenbergNativeSocketAsyncBackend.new(
+        bbPath,
+        options.threads,
+        options.logger,
+        options.unref,
+      );
       return new Barretenberg(socket, options);
     }
 

barretenberg/ts/src/bb_backends/node/native_socket.test.ts

@@ -0,0 +1,81 @@
+import { jest } from '@jest/globals';
+import * as fs from 'fs';
+import * as os from 'os';
+import * as path from 'path';
+import { BarretenbergNativeSocketAsyncBackend } from './native_socket.js';
+
+jest.setTimeout(30_000);
+
+// Echo server speaking the bb msgpack socket protocol (4-byte LE length prefix), started after
+// an optional delay to simulate bb's startup time on a loaded machine.
+const ECHO_SERVER_JS = `
+const net = require('net');
+const socketPath = process.argv[2];
+const server = net.createServer(sock => {
+  let buf = Buffer.alloc(0);
+  sock.on('data', d => {
+    buf = Buffer.concat([buf, d]);
+    while (buf.length >= 4) {
+      const len = buf.readUInt32LE(0);
+      if (buf.length < 4 + len) break;
+      const payload = buf.subarray(4, 4 + len);
+      const out = Buffer.alloc(4);
+      out.writeUInt32LE(payload.length, 0);
+      sock.write(out);
+      sock.write(payload);
+      buf = buf.subarray(4 + len);
+    }
+  });
+});
+server.listen(socketPath);
+`;
+
+// A fake bb binary: a bash script that optionally sleeps, then runs the echo server on the
+// socket path bb receives via `msgpack run --input <path>` ($4).
+function writeFakeBb(startupDelaySecs: number): string {
+  const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'fake-bb-'));
+  const serverJs = path.join(dir, 'echo_server.cjs');
+  fs.writeFileSync(serverJs, ECHO_SERVER_JS);
+  const file = path.join(dir, 'bb');
+  const sleep = startupDelaySecs > 0 ? `sleep ${startupDelaySecs}\n` : '';
+  fs.writeFileSync(file, `#!/bin/bash\n${sleep}exec node ${serverJs} "$4"\n`, { mode: 0o755 });
+  return file;
+}
+
+function writeFakeBbScript(script: string): string {
+  const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'fake-bb-'));
+  const file = path.join(dir, 'bb');
+  fs.writeFileSync(file, script, { mode: 0o755 });
+  return file;
+}
+
+describe('BarretenbergNativeSocketAsyncBackend', () => {
+  it('connects and echoes when bb starts promptly', async () => {
+    const fakeBb = writeFakeBb(0);
+    const backend = await BarretenbergNativeSocketAsyncBackend.new(fakeBb);
+    const response = await backend.call(new Uint8Array([1, 2, 3, 4]));
+    expect(response).toEqual(new Uint8Array([1, 2, 3, 4]));
+    await backend.destroy();
+  });
+
+  it('connects even when bb takes longer than 5s to create its socket', async () => {
+    const fakeBb = writeFakeBb(7);
+    const backend = await BarretenbergNativeSocketAsyncBackend.new(fakeBb);
+    const response = await backend.call(new Uint8Array([42]));
+    expect(response).toEqual(new Uint8Array([42]));
+    await backend.destroy();
+  });
+
+  it('fails with the exit cause when bb dies before creating its socket', async () => {
+    const fakeBb = writeFakeBbScript(`#!/bin/bash\nexit 17\n`);
+    await expect(BarretenbergNativeSocketAsyncBackend.new(fakeBb)).rejects.toThrow(
+      /exited before socket connection was established \(code=17/,
+    );
+  });
+
+  it('fails with the spawn error when the bb binary does not exist', async () => {
+    await expect(BarretenbergNativeSocketAsyncBackend.new('/nonexistent/bb-binary')).rejects.toThrow(
+      /Native backend process error/,
+    );
+  });
+});