Skip to content

feat(ipc): MPSC shared-memory transport with multi-client echo test#23040

Merged
charlielye merged 1 commit into
nextfrom
cl/ipc-1-mpsc-shm
May 12, 2026
Merged

feat(ipc): MPSC shared-memory transport with multi-client echo test#23040
charlielye merged 1 commit into
nextfrom
cl/ipc-1-mpsc-shm

Conversation

@charlielye
Copy link
Copy Markdown
Contributor

@charlielye charlielye commented May 7, 2026

Part 1 — MPSC shared-memory transport (originally cl/ipc-1-mpsc-shm)

Adds a multi-producer, single-consumer shared-memory IPC transport. Internally this is N SPSC rings (one per producer) + a shared doorbell futex — there is no shared producer ring. Each client writes only to its own SPSC ring; producers ring the doorbell to wake the consumer, which then checks which ring has data and reads from it. Each client also has its own SPSC response ring.

This is the same pattern that existed in earlier IPC iterations and is what aztec-wsdb will/could use when serving both the TS layer and a C++ AVM worker pool from the same world state if we need the performance.

Right now though we'll use UDS. This is just if needed in future.


Part 2 — aztec-wsdb binary + IPC supporting code (originally #23035 / cl/ipc-2-wsdb, merged into this PR)

Summary

Adds the standalone aztec-wsdb binary plus all supporting code (C++ client library, TS spawner, IPC adapter) needed to move world state out of the Node.js process. This part is inert: nothing yet uses the new binary. A follow-up PR (cl/ipc-3-avm-wsdb-cutover, #23036) cuts the NAPI AVM and the TS world state over to use it.

What's added

C++:

  • barretenberg/cpp/src/barretenberg/wsdb/aztec-wsdb standalone binary that runs the world state DB as an IPC server. Same WorldState surface as the in-process NAPI module, but exposed over msgpack via UDS or shared memory.
  • barretenberg/cpp/src/barretenberg/wsdb_client/WsdbIpcMerkleDB implements LowLevelMerkleDBInterface over WSDB IPC. The standalone AVM (or NAPI AVM after cutover) will use this in place of an in-process WorldState reference.
  • barretenberg/cpp/src/barretenberg/ipc/mpsc_shm_{client,server}.hpp — multi-producer single-consumer shared-memory transport (see Part 1 above). Lower latency than UDS for the AVM↔WSDB hop.

TypeScript (bb.js):

  • barretenberg/ts/src/aztec-wsdb/WsdbBackend spawns the aztec-wsdb binary and routes msgpack commands via the generated AsyncApi. Implements IMsgpackBackendAsync.
  • barretenberg/ts/src/cbind/cpp_codegen.ts — C++ codegen used by aztec-wsdb's generate.ts to produce wsdb_ipc_client_generated.{cpp,hpp}. Small shared updates to schema_visitor / typescript_codegen / rust_codegen.

yarn-project:

  • yarn-project/world-state/src/native/ipc_world_state_instance.tsIpcWorldState implements NativeWorldStateInstance over WSDB IPC. Not yet wired in.

Why split this way

The full WSDB-out-of-process cutover involves rewiring the NAPI AVM (which currently dereferences an in-process WorldState* pointer) to talk to aztec-wsdb over IPC, plus replacing TS NAPI WorldState with IpcWorldState everywhere it's used. This PR keeps the diff bounded by landing the binary and supporting code first; the cutover lands separately and should be a tiny diff.

Verification

  • aztec-wsdb builds: cd barretenberg/cpp/build && ninja aztec-wsdb
  • wsdb_client static library builds: ninja wsdb_client
  • bb.js builds (esm/cjs/browser): cd barretenberg/ts && yarn build:esm && yarn build:cjs && yarn build:browser
  • @aztec/world-state typechecks clean (the only TS errors in the build output are pre-existing on next in unrelated packages)

Stack

This is part of a stack splitting up #21331. Plan: /mnt/user-data/charlie/.claude/plans/glittery-snuggling-horizon.md.

@charlielye charlielye marked this pull request as ready for review May 7, 2026 14:00
@charlielye charlielye requested a review from ludamad May 7, 2026 16:45
@charlielye charlielye added this pull request to the merge queue May 7, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 7, 2026
@charlielye charlielye added this pull request to the merge queue May 11, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 11, 2026
@AztecBot AztecBot force-pushed the cl/ipc-1-mpsc-shm branch from 1344f0d to 17b93f4 Compare May 12, 2026 09:00
@AztecBot AztecBot enabled auto-merge May 12, 2026 09:00
Adds a multi-producer, single-consumer shared-memory IPC transport. Internally this is **N SPSC rings (one per producer) + a shared doorbell futex** — there is no shared producer ring. Each client writes only to its own SPSC ring; producers ring the doorbell to wake the consumer, which then checks which ring has data and reads from it. Each client also has its own SPSC response ring.

This is the same pattern that existed in earlier IPC iterations and is what \`aztec-wsdb\` will could use when serving both the TS layer and a C++ AVM worker pool from the same world state if we need the performance.

Right now though we'll use UDS. This is just if needed in future.
@AztecBot AztecBot force-pushed the cl/ipc-1-mpsc-shm branch from 17b93f4 to e668757 Compare May 12, 2026 09:02
@AztecBot AztecBot added this pull request to the merge queue May 12, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 12, 2026
@charlielye charlielye added this pull request to the merge queue May 12, 2026
Merged via the queue into next with commit e9c8d80 May 12, 2026
26 checks passed
@charlielye charlielye deleted the cl/ipc-1-mpsc-shm branch May 12, 2026 10:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants