fix(bb-prover): pool long-lived bb verifier processes instead of spawning per-call (#23093)

## Why

Follow-up to #21564 (bb-prover bb.js migration) addressing the IVC
verification perf regression that surfaced in `tx_stats_bench`.

The migration kept the legacy spawn-per-verification model: every
chonk/ultra-honk verification through `BBCircuitVerifier` spawned a
fresh `bb` process and SIGTERMed it after one proof.
`BB_NUM_IVC_VERIFIERS=8` only capped concurrency at the queue layer
(`QueuedIVCVerifier`), not the number of bb processes.

That made the bench spawn ~600 bb processes over its 60s 10 TPS phase
inside an 8-CPU isolate. Two compounding problems:

1. ~50–100 ms of `bb` startup tax on every verification's hot path.
2. The bind→listen race in `NativeUnixSocket`: bb's socket file appears
after `bind()` but before `listen()`. A TS `connect()` landing in that
window gets `ECONNREFUSED`. Vanishingly rare under low load; reliable
flake under contention. Diagnosis at
http://ci.aztec-labs.com/735256f13a268733.

## What

### Make `BB_NUM_IVC_VERIFIERS` mean what its name says (commits
aa99817, 0f4cb77)

Pool of long-lived bb verifier processes instead of fresh-per-call. The
factory class is renamed `BBJsProverFactory` → `BBJsFactory` (it's used
for both proving and verifying) and given a single `getInstance():
Promise<BBJsApi & AsyncDisposable>` method:

- `new BBJsFactory(path)` → no pool. Every `getInstance()` spawns a
fresh bb that is destroyed on dispose. Same as the previous
`withFreshInstance` behaviour — used by `BBNativeRollupProver`, the AVM
proving tester, and ivc-integration helpers, so their semantics are
unchanged.
- `new BBJsFactory(path, { poolSize: N })` → pool of N long-lived bb
processes, lazily spawned on first acquire. Used by `BBCircuitVerifier`
with `poolSize: numConcurrentIVCVerifiers`.

Callers use `await using inst = await factory.getInstance()` for
RAII-style release, matching the codebase's preference for
`AsyncDisposable`. `BBCircuitVerifier.stop` (already wired through to
aztec-node shutdown) tears the pool down.

### Close the bind→listen race in bb.js (commit 8e519b0)

`barretenberg/ts/src/bb_backends/node/native_socket.ts`: retry
`connect()` on `ECONNREFUSED` with exponential backoff (capped at 50 ms)
up to the existing 5 s budget. Other socket errors fail fast as before.
Pool startup still spawns N bb processes in parallel, so the race
surface is reduced from ~600 to N — the retry handles the residual.

### Server-side Chonk proof split (commit 97577cf)

`splitChonkProofToStructured` in TS had three hand-maintained constants
(`MERGE_PROOF_SIZE`, `ECCVM_PROOF_LENGTH`, `JOINT_PROOF_LENGTH`)
duplicating C++ values. When C++ shifted Chonk layout (e.g. databus
relation changes shrinking the oink portion in the previous round of
regressions), these went stale and verification failed deep in the
verifier with an opaque "OinkVerifier: num_public_inputs mismatch with
VK".

Add a new `ChonkVerifyFromFields` bbapi command that takes a flat
`Vec<bb::fr>` and calls `ChonkProof::from_field_elements` server-side,
then runs the verifier. The TS layer now passes flat fields straight
through — no layout knowledge, no hand-maintained constants.

- `bbapi_chonk.{hpp,cpp}`: new struct + `execute()`.
- `bbapi_execute.hpp`: register the variant.
- `bb_js_backend.ts`: `verifyChonkProof` calls the new API;
`splitChonkProofToStructured` and the 3 constants are deleted.

### Disposal robustness (commit 5cde220)

The first cut of `BBJsFactory` had three `.catch(() => {})` clauses that
silently swallowed bb `destroy()` errors, and an `initPool()` that
dropped already-spawned bb children if a sibling creation failed
(`Promise.all` short-circuit). Both would manifest as the Jest "worker
failed to exit gracefully" warning we hit on one test run.

Now: destroy errors propagate (`AggregateError` for the pool path);
`initPool` uses `allSettled` and tears down anything it spawned if any
sibling rejects.

### Playground bundle size (commit 1681d33)

The new `ChonkVerifyFromFields` bbapi variant tipped the playground main
entrypoint over the 1750 KB hard limit. Bumped to 1800 with a bump-log
entry.

## Effect

- `tx_stats_bench`: 600 bb spawns → 8 bb spawns at boot, then 8
long-lived processes serve every verification. The bind→listen race
surface drops 75×, *and* the residual is handled by the connect retry.
Per-call ~50–100 ms `bb` startup cost disappears from the verifier hot
path.
- Brittle TS Chonk constants are gone — Chonk layout changes in C++ can
no longer manifest as opaque verifier errors in TS.
- Disposal failures surface instead of leaking bb children.
- Behaviour for proving paths (`BBNativeRollupProver`, AVM tests,
ivc-integration) is unchanged — they still spawn fresh per call.

ClaudeBox log: https://claudebox.work/s/2d65052b0deaeab2?run=3

---------

Co-authored-by: Charlie <5764343+charlielye@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: 3 people

barretenberg/cpp/src/barretenberg/bbapi/bbapi_chonk.cpp

@@ -189,6 +189,33 @@ ChonkVerify::Response ChonkVerify::execute(const BBApiRequest& /*request*/) &&
     return { .valid = verified };
 }
 
+ChonkVerifyFromFields::Response ChonkVerifyFromFields::execute(const BBApiRequest& /*request*/) &&
+{
+    BB_BENCH_NAME(MSGPACK_SCHEMA_NAME);
+
+    using VerificationKey = Chonk::MegaVerificationKey;
+    validate_vk_size<VerificationKey>(vk);
+
+    auto hiding_kernel_vk = std::make_shared<VerificationKey>(from_buffer<VerificationKey>(vk));
+
+    // Validate total field count: must match num_public_inputs + fixed overhead.
+    const size_t expected_field_count =
+        static_cast<size_t>(hiding_kernel_vk->num_public_inputs) + ChonkProof::PROOF_LENGTH_WITHOUT_PUB_INPUTS;
+    if (proof.size() != expected_field_count) {
+        throw_or_abort("ChonkVerifyFromFields: proof has wrong field count: expected " +
+                       std::to_string(expected_field_count) + ", got " + std::to_string(proof.size()));
+    }
+
+    // Split the flat field array into the structured ChonkProof. Layout knowledge stays here.
+    auto structured = ChonkProof::from_field_elements(proof);
+
+    auto vk_and_hash = std::make_shared<ChonkNativeVerifier::VKAndHash>(hiding_kernel_vk);
+    ChonkNativeVerifier verifier(vk_and_hash);
+    const bool verified = verifier.verify(structured);
+
+    return { .valid = verified };
+}
+
 ChonkBatchVerify::Response ChonkBatchVerify::execute(const BBApiRequest& /*request*/) &&
 {
     BB_BENCH_NAME(MSGPACK_SCHEMA_NAME);

barretenberg/cpp/src/barretenberg/bbapi/bbapi_chonk.hpp

@@ -157,6 +157,36 @@ struct ChonkVerify {
     bool operator==(const ChonkVerify&) const = default;
 };
 
+/**
+ * @struct ChonkVerifyFromFields
+ * @brief Verify a Chonk proof passed as a flat field-element array (with public inputs prepended).
+ *
+ * The split into structured ChonkProof sub-proofs is done server-side via
+ * ChonkProof::from_field_elements, so callers do not need to know the per-component sub-proof
+ * sizes. This is the recommended entry point for TypeScript callers that hold the proof as a
+ * flat Fr[] (e.g. from tx.chonkProof.attachPublicInputs).
+ */
+struct ChonkVerifyFromFields {
+    static constexpr const char MSGPACK_SCHEMA_NAME[] = "ChonkVerifyFromFields";
+
+    struct Response {
+        static constexpr const char MSGPACK_SCHEMA_NAME[] = "ChonkVerifyFromFieldsResponse";
+
+        /** @brief True if the proof is valid */
+        bool valid;
+        SERIALIZATION_FIELDS(valid);
+        bool operator==(const Response&) const = default;
+    };
+
+    /** @brief Flat proof field elements with public inputs prepended */
+    std::vector<bb::fr> proof;
+    /** @brief The verification key */
+    std::vector<uint8_t> vk;
+    Response execute(const BBApiRequest& request = {}) &&;
+    SERIALIZATION_FIELDS(proof, vk);
+    bool operator==(const ChonkVerifyFromFields&) const = default;
+};
+
 /**
  * @struct ChonkComputeVk
  * @brief Compute MegaHonk verification key for a circuit to be accumulated in Chonk

barretenberg/cpp/src/barretenberg/bbapi/bbapi_execute.hpp

@@ -27,6 +27,7 @@ using Command = NamedUnion<AvmProve,
                            ChonkAccumulate,
                            ChonkProve,
                            ChonkVerify,
+                           ChonkVerifyFromFields,
                            ChonkBatchVerify,
                            VkAsFields,
                            MegaVkAsFields,
@@ -90,6 +91,7 @@ using CommandResponse = NamedUnion<ErrorResponse,
                                    ChonkAccumulate::Response,
                                    ChonkProve::Response,
                                    ChonkVerify::Response,
+                                   ChonkVerifyFromFields::Response,
                                    ChonkBatchVerify::Response,
                                    VkAsFields::Response,
                                    MegaVkAsFields::Response,

barretenberg/ts/src/bb_backends/node/native_socket.ts

@@ -173,52 +173,75 @@ export class BarretenbergNativeSocketAsyncBackend implements IMsgpackBackendAsyn
       throw new Error(`Path exists but is not a socket: ${this.socketPath}`);
     }
 
-    // Connect to bb's socket server as a client
-    return new Promise<void>((resolve, reject) => {
-      this.socket = net.connect(this.socketPath);
+    // Connect with retry on ECONNREFUSED. The socket file appears after bb's bind() but
+    // before its listen(); a connect() landing in that window gets ECONNREFUSED. Retry
+    // briefly until bb is listening or we hit the 5s budget.
+    const socket = await this.connectWithRetry(startTime);
+    this.socket = socket;
 
-      // Disable Nagle's algorithm for lower latency
-      this.socket.setNoDelay(true);
-
-      // Set up event handlers
-      this.socket.once('connect', () => {
-        // Socket starts referenced - will be unreferenced when no callbacks pending
+    // Clear connection timeout on successful connection
+    if (this.connectionTimeout) {
+      clearTimeout(this.connectionTimeout);
+      this.connectionTimeout = null;
+    }
 
-        // Clear connection timeout on successful connection
-        if (this.connectionTimeout) {
-          clearTimeout(this.connectionTimeout);
-          this.connectionTimeout = null;
-        }
-        resolve();
-      });
+    // Set up persistent handlers now that we're connected.
+    socket.on('data', (chunk: Buffer) => {
+      this.handleData(chunk);
+    });
 
-      this.socket.once('error', err => {
-        reject(new Error(`Failed to connect to bb socket: ${err.message}`));
-      });
+    socket.on('error', err => {
+      const error = new Error(`Socket error: ${err.message}`);
+      for (const callback of this.pendingCallbacks) {
+        callback.reject(error);
+      }
+      this.pendingCallbacks = [];
+    });
 
-      // Set up data handler after connection is established
-      this.socket.on('data', (chunk: Buffer) => {
-        this.handleData(chunk);
-      });
+    socket.on('end', () => {
+      const error = new Error('Socket connection ended unexpectedly');
+      for (const callback of this.pendingCallbacks) {
+        callback.reject(error);
+      }
+      this.pendingCallbacks = [];
+    });
+  }
 
-      // Handle ongoing errors after initial connection
-      this.socket.on('error', err => {
-        // Reject all pending callbacks
-        const error = new Error(`Socket error: ${err.message}`);
-        for (const callback of this.pendingCallbacks) {
-          callback.reject(error);
+  private async connectWithRetry(startTime: number): Promise<net.Socket> {
+    let attempt = 0;
+    let lastErr: Error | undefined;
+    while (Date.now() - startTime < 5000) {
+      try {
+        return await this.attemptConnect();
+      } catch (err) {
+        lastErr = err as Error;
+        const code = (err as NodeJS.ErrnoException).code;
+        if (code !== 'ECONNREFUSED') {
+          throw new Error(`Failed to connect to bb socket: ${lastErr.message}`);
         }
-        this.pendingCallbacks = [];
-      });
+        // bb has bound the path but not yet called listen(); back off and retry.
+        const delay = Math.min(50, 5 * 2 ** attempt++);
+        await new Promise(resolve => setTimeout(resolve, delay));
+      }
+    }
+    throw new Error(`Timeout connecting to bb socket: ${lastErr?.message ?? 'unknown'}`);
+  }
 
-      this.socket.on('end', () => {
-        // Reject all pending callbacks
-        const error = new Error('Socket connection ended unexpectedly');
-        for (const callback of this.pendingCallbacks) {
-          callback.reject(error);
-        }
-        this.pendingCallbacks = [];
-      });
+  private attemptConnect(): Promise<net.Socket> {
+    return new Promise<net.Socket>((resolve, reject) => {
+      const socket = net.connect(this.socketPath);
+      socket.setNoDelay(true);
+      const onConnect = () => {
+        socket.removeListener('error', onError);
+        resolve(socket);
+      };
+      const onError = (err: Error) => {
+        socket.removeListener('connect', onConnect);
+        socket.destroy();
+        reject(err);
+      };
+      socket.once('connect', onConnect);
+      socket.once('error', onError);
     });
   }
 

playground/vite.config.ts

@@ -136,9 +136,10 @@ export default defineConfig(({ mode }) => {
         // Bump log:
         // - AD: bumped from 1600 => 1680 as we now have a 20kb msgpack lib in bb.js and other logic got us 50kb higher, adding some wiggle room.
         // - MW: bumped from 1700 => 1750 after adding the noble curves pkg to foundation required for blob batching calculations.
+        // - CL: bumped from 1750 => 1800 after adding the ChonkVerifyFromFields bbapi variant (PR #23093).
         {
           pattern: /assets\/index-.*\.js$/,
-          maxSizeKB: 1750,
+          maxSizeKB: 1800,
           description: 'Main entrypoint, hard limit',
         },
         // Bump log:

yarn-project/bb-prover/src/avm_proving_tests/avm_proving_tester.ts

@@ -17,7 +17,7 @@ import { NativeWorldStateService } from '@aztec/world-state';
 
 import path from 'path';
 
-import { BBJsProverFactory } from '../bb/bb_js_backend.js';
+import { BBJsFactory } from '../bb/bb_js_backend.js';
 
 const BB_PATH = path.resolve('../../barretenberg/cpp/build/bin/bb-avm');
 
@@ -32,7 +32,7 @@ const provingConfig: PublicSimulatorConfig = PublicSimulatorConfig.from({
 });
 
 export class AvmProvingTester extends PublicTxSimulationTester {
-  private readonly bbJsFactory = new BBJsProverFactory(BB_PATH);
+  private readonly bbJsFactory = new BBJsFactory(BB_PATH);
 
   constructor(
     private checkCircuitOnly: boolean,
@@ -64,13 +64,15 @@ export class AvmProvingTester extends PublicTxSimulationTester {
     const inputsBuffer = avmCircuitInputs.serializeWithMessagePack();
 
     if (this.checkCircuitOnly) {
-      const { passed, stats } = await this.bbJsFactory.withFreshInstance(i => i.checkAvmCircuit(inputsBuffer));
+      await using instance = await this.bbJsFactory.getInstance();
+      const { passed, stats } = await instance.checkAvmCircuit(inputsBuffer);
       this.recordProverMetrics(stats, txLabel);
       expect(passed).toBe(true);
       return [];
     }
 
-    const { proof, stats } = await this.bbJsFactory.withFreshInstance(i => i.generateAvmProof(inputsBuffer));
+    await using instance = await this.bbJsFactory.getInstance();
+    const { proof, stats } = await instance.generateAvmProof(inputsBuffer);
     this.recordProverMetrics(stats, txLabel);
     return proof;
   }
@@ -81,7 +83,8 @@ export class AvmProvingTester extends PublicTxSimulationTester {
       return;
     }
     const piBuffer = publicInputs.serializeWithMessagePack();
-    const { verified } = await this.bbJsFactory.withFreshInstance(i => i.verifyAvmProof(proof, piBuffer));
+    await using instance = await this.bbJsFactory.getInstance();
+    const { verified } = await instance.verifyAvmProof(proof, piBuffer);
     expect(verified).toBe(true);
   }