Skip to content

Commit 1344f0d

Browse files
charlielyecharlielye
authored
feat: aztec-wsdb binary and IPC supporting code (inert) (#23035)
## Summary Adds the standalone `aztec-wsdb` binary plus all supporting code (C++ client library, TS spawner, IPC adapter) needed to move world state out of the Node.js process. **This PR is inert: nothing yet uses the new binary.** A follow-up PR (cl/ipc-3-avm-wsdb-cutover) will cut the NAPI AVM and the TS world state over to use it. ## What's added **C++:** - `barretenberg/cpp/src/barretenberg/wsdb/` — \`aztec-wsdb\` standalone binary that runs the world state DB as an IPC server. Same WorldState surface as the in-process NAPI module, but exposed over msgpack via UDS or shared memory. - `barretenberg/cpp/src/barretenberg/wsdb_client/` — \`WsdbIpcMerkleDB\` implements \`LowLevelMerkleDBInterface\` over WSDB IPC. The standalone AVM (or NAPI AVM after cutover) will use this in place of an in-process \`WorldState\` reference. - `barretenberg/cpp/src/barretenberg/ipc/mpsc_shm_{client,server}.hpp` — multi-producer single-consumer shared-memory transport. Lower latency than UDS for the AVM↔WSDB hop. **TypeScript (bb.js):** - `barretenberg/ts/src/aztec-wsdb/` — \`WsdbBackend\` spawns the \`aztec-wsdb\` binary and routes msgpack commands via the generated \`AsyncApi\`. Implements \`IMsgpackBackendAsync\`. - `barretenberg/ts/src/cbind/cpp_codegen.ts` — C++ codegen used by \`aztec-wsdb\`'s \`generate.ts\` to produce \`wsdb_ipc_client_generated.{cpp,hpp}\`. Small shared updates to \`schema_visitor\` / \`typescript_codegen\` / \`rust_codegen\`. **yarn-project:** - `yarn-project/world-state/src/native/ipc_world_state_instance.ts` — \`IpcWorldState\` implements \`NativeWorldStateInstance\` over WSDB IPC. Not yet wired in. ## Why split this way The full WSDB-out-of-process cutover involves rewiring the NAPI AVM (which currently dereferences an in-process \`WorldState*\` pointer) to talk to \`aztec-wsdb\` over IPC, plus replacing TS NAPI WorldState with \`IpcWorldState\` everywhere it's used. This PR keeps the diff bounded by landing the binary and supporting code first; the cutover lands separately and should be a tiny diff. ## Verification - `aztec-wsdb` builds: \`cd barretenberg/cpp/build && ninja aztec-wsdb\` - \`wsdb_client\` static library builds: \`ninja wsdb_client\` - bb.js builds (esm/cjs/browser): \`cd barretenberg/ts && yarn build:esm && yarn build:cjs && yarn build:browser\` - \`@aztec/world-state\` typechecks clean (the only TS errors in the build output are pre-existing on \`next\` in unrelated packages) ## Stack This is part of a stack splitting up #21331. Plan: \`/mnt/user-data/charlie/.claude/plans/glittery-snuggling-horizon.md\`. - PR 2a: this PR — binary + supporting code (inert) - PR 2b (next): NAPI AVM + TS world state cutover (~500 LOC, deletes NAPI WorldState C++ module) - PR 3: standalone \`aztec-avm\` + CDB IPC server, kills NAPI AVM - PR 4-6: pool, cancellation, optional MPSC SHM transport
1 parent 2c88ac1 commit 1344f0d

28 files changed

Lines changed: 3803 additions & 22 deletions

‎barretenberg/cpp/src/CMakeLists.txt‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -127,6 +127,8 @@ if(NOT FUZZING AND NOT WASM AND NOT BB_LITE)
127127
# NOTE: Do not conditionally base this on the AVM flag as it defines a necessary vm2_sim library.
128128
add_subdirectory(barretenberg/vm2)
129129
add_subdirectory(barretenberg/ipc)
130+
add_subdirectory(barretenberg/wsdb)
131+
add_subdirectory(barretenberg/wsdb_client)
130132
add_subdirectory(barretenberg/nodejs_module)
131133
endif()
132134

‎barretenberg/cpp/src/barretenberg/wsdb/CMakeLists.txt‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
if(NOT(FUZZING) AND NOT(WASM))
2+
# IPC client library (used by AVM simulator to talk to aztec-wsdb)
3+
# Generated via: cd barretenberg/ts && npx tsx src/aztec-wsdb/generate.ts
4+
add_library(
5+
wsdb_ipc_client
6+
STATIC
7+
wsdb_ipc_client_generated.cpp
8+
)
9+
target_link_libraries(
10+
wsdb_ipc_client
11+
PUBLIC
12+
barretenberg
13+
ipc
14+
)
15+
16+
# aztec-wsdb binary (standalone world state database server)
17+
add_executable(
18+
aztec-wsdb
19+
main.cpp
20+
cli.cpp
21+
wsdb_execute.cpp
22+
wsdb_ipc_server.cpp
23+
)
24+
target_link_libraries(
25+
aztec-wsdb
26+
PRIVATE
27+
barretenberg
28+
world_state
29+
ipc
30+
env
31+
)
32+
if(ENABLE_STACKTRACES)
33+
target_link_libraries(
34+
aztec-wsdb
35+
PUBLIC
36+
Backward::Interface
37+
)
38+
target_link_options(
39+
aztec-wsdb
40+
PRIVATE
41+
-ldw -lelf
42+
)
43+
endif()
44+
endif()

‎barretenberg/cpp/src/barretenberg/wsdb/cli.cpp‎

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
#include "barretenberg/wsdb/cli.hpp"
2+
#include "barretenberg/common/log.hpp"
3+
#include "barretenberg/common/throw_or_abort.hpp"
4+
#include "barretenberg/serialize/msgpack.hpp"
5+
#include "barretenberg/wsdb/wsdb_execute.hpp"
6+
#include "barretenberg/wsdb/wsdb_ipc_server.hpp"
7+
8+
#include "barretenberg/bb/deps/cli11.hpp"
9+
#include <cstdint>
10+
#include <iostream>
11+
#include <string>
12+
#include <unordered_map>
13+
#include <vector>
14+
15+
namespace bb::wsdb {
16+
17+
using namespace bb::world_state;
18+
using namespace bb::crypto::merkle_tree;
19+
20+
namespace {
21+
22+
struct WsdbApi {
23+
WsdbCommand commands;
24+
WsdbCommandResponse responses;
25+
SERIALIZATION_FIELDS(commands, responses);
26+
};
27+
28+
std::string get_wsdb_schema_as_json()
29+
{
30+
return msgpack_schema_to_string(WsdbApi{});
31+
}
32+
33+
} // namespace
34+
35+
int parse_and_run_wsdb(int argc, char* argv[])
36+
{
37+
CLI::App app{ "aztec-wsdb: Standalone world state database server" };
38+
app.require_subcommand(1);
39+
40+
// -----------------------------------------------------------------------
41+
// Subcommand: msgpack
42+
// -----------------------------------------------------------------------
43+
CLI::App* msgpack_command = app.add_subcommand("msgpack", "Msgpack API interface.");
44+
45+
// msgpack schema
46+
CLI::App* msgpack_schema_command =
47+
msgpack_command->add_subcommand("schema", "Output a msgpack schema encoded as JSON to stdout.");
48+
49+
// msgpack run
50+
CLI::App* msgpack_run_command =
51+
msgpack_command->add_subcommand("run", "Start the world state database IPC server.");
52+
53+
std::string input_path;
54+
msgpack_run_command->add_option(
55+
"-i,--input", input_path, "IPC socket/shm path (.sock for UDS, .shm for shared memory)");
56+
57+
std::string data_dir;
58+
msgpack_run_command->add_option("-d,--data-dir", data_dir, "Data directory for LMDB stores")->required();
59+
60+
// Tree heights (JSON map: treeId -> height)
61+
std::string tree_heights_json;
62+
msgpack_run_command->add_option("--tree-heights", tree_heights_json, "Tree heights as JSON: {0:40,1:32,...}");
63+
64+
// Tree prefill sizes
65+
std::string tree_prefill_json;
66+
msgpack_run_command->add_option(
67+
"--tree-prefill", tree_prefill_json, "Tree prefill sizes as JSON: {0:128,2:128,...}");
68+
69+
// Map sizes (KB)
70+
std::string map_sizes_json;
71+
msgpack_run_command->add_option("--map-sizes", map_sizes_json, "LMDB map sizes in KB as JSON: {0:1024,...}");
72+
73+
uint32_t threads = 16;
74+
msgpack_run_command->add_option("-t,--threads", threads, "Thread pool size (default: 16)")
75+
->check(CLI::PositiveNumber);
76+
77+
uint32_t initial_header_generator_point = 0;
78+
msgpack_run_command->add_option(
79+
"--initial-header-generator-point", initial_header_generator_point, "Header generator point (default: 0)");
80+
81+
// Prefilled public data as JSON array of [slot_hex, value_hex] pairs
82+
std::string prefilled_public_data_json;
83+
msgpack_run_command->add_option(
84+
"--prefilled-public-data", prefilled_public_data_json, "Prefilled public data as JSON array");
85+
86+
uint64_t genesis_timestamp = 0;
87+
msgpack_run_command->add_option("--genesis-timestamp", genesis_timestamp, "Genesis block timestamp (default: 0)");
88+
89+
size_t request_ring_size = 1024 * 1024;
90+
msgpack_run_command
91+
->add_option(
92+
"--request-ring-size", request_ring_size, "Request ring buffer size for shared memory IPC (default: 1MB)")
93+
->check(CLI::PositiveNumber);
94+
95+
size_t response_ring_size = 1024 * 1024;
96+
msgpack_run_command
97+
->add_option("--response-ring-size",
98+
response_ring_size,
99+
"Response ring buffer size for shared memory IPC (default: 1MB)")
100+
->check(CLI::PositiveNumber);
101+
102+
// Parse CLI
103+
try {
104+
app.parse(argc, argv);
105+
} catch (const CLI::ParseError& e) {
106+
return app.exit(e);
107+
}
108+
109+
try {
110+
if (msgpack_schema_command->parsed()) {
111+
std::cout << get_wsdb_schema_as_json() << std::endl;
112+
return 0;
113+
}
114+
115+
if (msgpack_run_command->parsed()) {
116+
return execute_wsdb_server(input_path,
117+
data_dir,
118+
tree_heights_json,
119+
tree_prefill_json,
120+
map_sizes_json,
121+
threads,
122+
initial_header_generator_point,
123+
prefilled_public_data_json,
124+
genesis_timestamp,
125+
request_ring_size,
126+
response_ring_size);
127+
}
128+
} catch (const std::exception& e) {
129+
std::cerr << "Error: " << e.what() << '\n';
130+
return 1;
131+
}
132+
133+
return 0;
134+
}
135+
136+
} // namespace bb::wsdb

‎barretenberg/cpp/src/barretenberg/wsdb/cli.hpp‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
#pragma once
2+
3+
namespace bb::wsdb {
4+
5+
int parse_and_run_wsdb(int argc, char* argv[]);
6+
7+
} // namespace bb::wsdb

‎barretenberg/cpp/src/barretenberg/wsdb/main.cpp‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
#include "barretenberg/wsdb/cli.hpp"
2+
3+
int main(int argc, char* argv[])
4+
{
5+
return bb::wsdb::parse_and_run_wsdb(argc, argv);
6+
}

0 commit comments

Comments
 (0)