Skip to content

Commit 6113a7c

Browse files
charlielyecharlielye
committed
refactor: extract wsdb into native-packages, decoupled from barretenberg
Moves the world-state DB server (+ persistent merkle), the lmdb C++ wrapper (lmdblib), and the kv-store NAPI (kvdb) out of barretenberg into native-packages siblings, leaving barretenberg free of DB code — and in the same step decouples wsdb from barretenberg's C++ headers. The extraction splits barretenberg's merkle_tree headers: barretenberg keeps only the response/types the AVM's LowLevelMerkleDBInterface returns and its hints serialize, while the full async tree engine (content-addressed trees + world_state) and its response/tree_meta/types move into wsdb. wsdb then compiles with zero barretenberg headers — its own FieldElement (32-byte canonical) + a poseidon2 c_bind are the only barretenberg link. Forks the bb-free common utils, moves merkle_tree Signal into wsdb, keeps lmdb keys backward-compatible (little-endian uint256), switches lmdblib to upstream msgpack + FetchContent (bb-free), and wires the lmdblib/wsdb C++ tests into the CI test engine. The standalone aztec-wsdb binary and the kvdb .node build against prebuilt barretenberg; lmdblib/kvdb are fully barretenberg-free.
1 parent 8443e1a commit 6113a7c

200 files changed

Lines changed: 5956 additions & 1874 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Makefile‎

Lines changed: 24 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -47,15 +47,15 @@ endef
4747
# PHONY TARGETS - List every target that has a file/dir of the same name.
4848
#==============================================================================
4949

50-
.PHONY: noir barretenberg noir-projects l1-contracts release-image boxes playground docs aztec-up spartan wsdb bb-avm-sim
50+
.PHONY: noir barretenberg noir-projects l1-contracts release-image boxes playground docs aztec-up spartan lmdblib kvdb wsdb bb-avm-sim
5151

5252
#==============================================================================
5353
# BOOTSTRAP TARGETS
5454
#==============================================================================
5555

5656
# Fast bootstrap.
5757
fast: release-image barretenberg boxes playground docs aztec-up \
58-
bb-tests l1-contracts-tests yarn-project-tests boxes-tests playground-tests aztec-up-tests docs-tests noir-protocol-circuits-tests contract-snapshots-tests release-image-tests spartan claude-tests ipc-codegen-tests
58+
bb-tests l1-contracts-tests yarn-project-tests boxes-tests playground-tests aztec-up-tests docs-tests noir-protocol-circuits-tests contract-snapshots-tests release-image-tests spartan claude-tests ipc-codegen-tests lmdblib-tests wsdb-tests
5959

6060
# Full bootstrap.
6161
full: fast bb-full-tests bb-cpp-full yarn-project-benches
@@ -326,11 +326,29 @@ ipc-runtime-cross-arm64-macos:
326326
ipc-runtime-cross: ipc-runtime ipc-runtime-cross-arm64-linux ipc-runtime-cross-amd64-macos ipc-runtime-cross-arm64-macos
327327

328328
#==============================================================================
329-
# WSDB
329+
# Native packages (lmdblib, kvdb, wsdb)
330330
#==============================================================================
331331

332-
wsdb: ipc-codegen ipc-runtime bb-cpp-native
333-
$(call build,$@,wsdb)
332+
# lmdblib and kvdb are barretenberg-free: they build against their own deps
333+
# (lmdb, msgpack-c, node-addon-api) only, never bb.
334+
.PHONY: lmdblib kvdb lmdblib-tests wsdb-tests
335+
lmdblib:
336+
$(call build,$@,native-packages/lmdblib)
337+
338+
kvdb: lmdblib
339+
$(call build,$@,native-packages/kvdb)
340+
341+
wsdb: ipc-codegen ipc-runtime bb-cpp-native lmdblib
342+
$(call build,$@,native-packages/wsdb)
343+
344+
# Native-package C++ tests (self-contained gtest binaries). kvdb has no C++ tests
345+
# (its NAPI is exercised by yarn-project's kv-store tests). wsdb_tests use no bb
346+
# headers; the bb-header parity/equivalence target is manual (WSDB_BUILD_BB_TESTS).
347+
lmdblib-tests: lmdblib
348+
$(call test,$@,native-packages/lmdblib)
349+
350+
wsdb-tests: wsdb
351+
$(call test,$@,native-packages/wsdb)
334352

335353
#==============================================================================
336354
# .claude tooling
@@ -421,7 +439,7 @@ l1-contracts-tests: l1-contracts-verifier
421439
# Yarn Project - TypeScript monorepo with all TS packages
422440
#==============================================================================
423441

424-
yarn-project: bb-ts noir-projects l1-contracts wsdb bb-avm-sim
442+
yarn-project: bb-ts noir-projects l1-contracts wsdb kvdb bb-avm-sim
425443
$(call build,$@,yarn-project)
426444

427445
yarn-project-tests: yarn-project

‎aztec-up/bootstrap.sh‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,23 @@
11
#!/usr/bin/env bash
22
source $(git rev-parse --show-toplevel)/ci3/source_bootstrap
33

4-
hash=$(hash_str $(cache_content_hash ^aztec-up/) $(../ipc-runtime/bootstrap.sh hash) $(../wsdb/bootstrap.sh hash) $(../barretenberg/ts/bootstrap.sh hash) $(../yarn-project/bootstrap.sh hash))
4+
hash=$(hash_str $(cache_content_hash ^aztec-up/) $(../ipc-runtime/bootstrap.sh hash) $(../native-packages/kvdb/bootstrap.sh hash) $(../native-packages/wsdb/bootstrap.sh hash) $(../barretenberg/ts/bootstrap.sh hash) $(../yarn-project/bootstrap.sh hash))
55

66
# Bare aliases ("nightly", "latest") resolve to this major version.
77
DEFAULT_MAJOR_VERSION=${AZTEC_TOOLCHAIN_DEFAULT_MAJOR_VERSION:-4}
88

99
function wsdb_package_dirs {
10-
for package_dir in "$root"/wsdb/ts/packages/*; do
10+
for package_dir in "$root"/native-packages/wsdb/ts/packages/*; do
1111
[ -d "$package_dir" ] && echo "$package_dir"
1212
done
13-
echo "$root/wsdb/ts"
13+
echo "$root/native-packages/wsdb/ts"
14+
}
15+
16+
function kvdb_package_dirs {
17+
for package_dir in "$root"/native-packages/kvdb/ts/packages/*; do
18+
[ -d "$package_dir" ] && echo "$package_dir"
19+
done
20+
echo "$root/native-packages/kvdb/ts"
1421
}
1522

1623
function barretenberg_ts_package_dirs {
@@ -117,6 +124,7 @@ EOF
117124
echo $root/ipc-runtime/ts
118125
barretenberg_ts_package_dirs
119126
wsdb_package_dirs
127+
kvdb_package_dirs
120128
# l1-artifacts lives under l1-contracts, so it isn't enumerated by yarn-project's get_projects;
121129
# publish it explicitly or @aztec/aztec's portal dependency can't resolve from the local registry.
122130
echo $root/l1-contracts/l1-artifacts

‎barretenberg/cpp/CMakeLists.txt‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ project(
1212
# === Platform Guard Reference ===
1313
# __wasm__ : Compiler-defined for WebAssembly. Guards threading, file I/O, networking.
1414
# _WIN32 : Compiler-defined for Windows. Guards POSIX APIs (mmap, signals, IPC).
15-
# BB_LITE : CMake option. Excludes server-side subsystems (lmdb, world_state, ipc, nodejs_module).
15+
# BB_LITE : CMake option. Excludes server-side subsystems (ipc, vm2_wsdb, cdb, avm).
1616
# Enabled for: iOS, Android, Windows cross-compiles.
1717
# WASM : CMake variable mirroring __wasm__ for CMake-level logic.
1818

@@ -45,7 +45,7 @@ option(ENABLE_PIC "Builds with position independent code" OFF)
4545
option(SYNTAX_ONLY "only check syntax (-fsyntax-only)" OFF)
4646
option(ENABLE_WASM_BENCH "Enable BB_BENCH benchmarking support in WASM builds (dev only, not for releases)" OFF)
4747
option(AVM "enable building of vm2 module and bb-avm" ON)
48-
option(BB_LITE "Exclude server-side subsystems: lmdb, world_state, ipc, nodejs_module" OFF)
48+
option(BB_LITE "Exclude server-side subsystems: ipc, vm2_wsdb, cdb, avm" OFF)
4949

5050
if(CMAKE_SYSTEM_PROCESSOR MATCHES "aarch64" OR CMAKE_SYSTEM_PROCESSOR MATCHES "arm64")
5151
message(STATUS "Compiling for ARM.")

‎barretenberg/cpp/CMakePresets.json‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -801,25 +801,25 @@
801801
"name": "amd64-linux",
802802
"configurePreset": "amd64-linux",
803803
"inheritConfigureEnvironment": true,
804-
"targets": ["bb", "nodejs_module", "bb-external", "aztec-wsdb", "bb-avm-sim"]
804+
"targets": ["bb", "nodejs_module", "bb-external", "bb-avm-sim"]
805805
},
806806
{
807807
"name": "arm64-linux",
808808
"configurePreset": "arm64-linux",
809809
"inheritConfigureEnvironment": true,
810-
"targets": ["bb", "nodejs_module", "bb-external", "aztec-wsdb", "bb-avm-sim"]
810+
"targets": ["bb", "nodejs_module", "bb-external", "bb-avm-sim"]
811811
},
812812
{
813813
"name": "amd64-macos",
814814
"configurePreset": "amd64-macos",
815815
"inheritConfigureEnvironment": true,
816-
"targets": ["bb", "nodejs_module", "bb-external", "aztec-wsdb", "bb-avm-sim"]
816+
"targets": ["bb", "nodejs_module", "bb-external", "bb-avm-sim"]
817817
},
818818
{
819819
"name": "arm64-macos",
820820
"configurePreset": "arm64-macos",
821821
"inheritConfigureEnvironment": true,
822-
"targets": ["bb", "nodejs_module", "bb-external", "aztec-wsdb", "bb-avm-sim"]
822+
"targets": ["bb", "nodejs_module", "bb-external", "bb-avm-sim"]
823823
},
824824
{
825825
"name": "amd64-windows",

‎barretenberg/cpp/scripts/audit/audit_scopes/merkle_tree_audit_scope.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,14 +20,14 @@ Note: Paths relative to `aztec-packages/barretenberg/cpp/src/barretenberg`
2020
### Node Store (Cache + Metadata)
2121
8. `crypto/merkle_tree/node_store/cached_content_addressed_tree_store.hpp`
2222
9. `crypto/merkle_tree/node_store/content_addressed_cache.hpp`
23-
10. `crypto/merkle_tree/node_store/tree_meta.hpp`
23+
10. `crypto/merkle_tree/tree_meta.hpp`
2424

2525
### Append-Only Tree
2626
11. `crypto/merkle_tree/append_only_tree/content_addressed_append_only_tree.hpp`
2727

2828
### Indexed Tree
2929
12. `crypto/merkle_tree/indexed_tree/content_addressed_indexed_tree.hpp`
30-
13. `crypto/merkle_tree/indexed_tree/indexed_leaf.hpp`
30+
13. `crypto/merkle_tree/indexed_leaf.hpp`
3131

3232
## Summary of Module
3333

‎barretenberg/cpp/src/CMakeLists.txt‎

Lines changed: 4 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -116,22 +116,17 @@ add_subdirectory(barretenberg/ultra_honk)
116116
add_subdirectory(barretenberg/vm2_stub)
117117
add_subdirectory(barretenberg/wasi)
118118

119-
if(NOT BB_LITE)
120-
add_subdirectory(barretenberg/lmdblib)
121-
endif()
122-
123119
if(NOT FUZZING AND NOT WASM AND NOT BB_LITE)
124-
# Fuzzing preset cannot be built with world_state as world_state cannot compile with MULTITHREADING=OFF
125-
# Mobile builds exclude these modules that require LMDB or aren't needed on mobile
120+
# Mobile builds exclude these modules that aren't needed on mobile
126121
add_subdirectory(barretenberg/aztec)
127-
add_subdirectory(barretenberg/world_state)
128122
# NOTE: Do not conditionally base this on the AVM flag as it defines a necessary vm2_sim library.
129123
add_subdirectory(barretenberg/vm2)
130124
add_subdirectory(barretenberg/ipc)
131-
add_subdirectory(barretenberg/wsdb)
132125
add_subdirectory(barretenberg/vm2_wsdb)
133126
add_subdirectory(barretenberg/cdb)
134127
add_subdirectory(barretenberg/avm)
128+
# nodejs_module hosts only the msgpack_client NAPI wrappers now (bb.js's SHM
129+
# transport). The lmdb_store kv-store NAPI moved to native-packages/kvdb.
135130
add_subdirectory(barretenberg/nodejs_module)
136131
endif()
137132

@@ -147,7 +142,6 @@ if(FUZZING_AVM)
147142
if(FUZZING)
148143
# Only add these if they weren't added above (when NOT FUZZING AND NOT WASM)
149144
add_subdirectory(barretenberg/aztec)
150-
add_subdirectory(barretenberg/world_state)
151145
add_subdirectory(barretenberg/vm2)
152146
endif()
153147
add_subdirectory(barretenberg/avm_fuzzer)
@@ -216,11 +210,9 @@ set(BARRETENBERG_TARGET_OBJECTS
216210
set(BB_EXTERNAL_TARGET_OBJECTS ${BARRETENBERG_TARGET_OBJECTS})
217211

218212
if(NOT WASM AND NOT FUZZING AND NOT BB_LITE)
219-
# enable merkle trees and lmdb (not for mobile builds)
213+
# enable merkle trees (not for mobile builds)
220214
list(APPEND BARRETENBERG_TARGET_OBJECTS $<TARGET_OBJECTS:crypto_merkle_tree_objects>)
221-
list(APPEND BARRETENBERG_TARGET_OBJECTS $<TARGET_OBJECTS:lmdblib_objects>)
222215
list(APPEND BARRETENBERG_TARGET_OBJECTS $<TARGET_OBJECTS:api_objects>)
223-
list(APPEND BARRETENBERG_TARGET_OBJECTS $<TARGET_OBJECTS:world_state_objects>)
224216
endif()
225217

226218
add_library(

‎barretenberg/cpp/src/barretenberg/avm/CMakeLists.txt‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,6 @@ target_link_libraries(
4949
barretenberg
5050
vm2_sim
5151
wsdb_ipc_merkle_db
52-
wsdb_ipc_client
5352
cdb_ipc_client
5453
ipc_runtime
5554
env

‎barretenberg/cpp/src/barretenberg/avm/avm_execute.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
namespace bb::avm {
1212

1313
using namespace bb::avm2;
14-
using namespace bb::world_state;
14+
using bb::crypto::merkle_tree::WorldStateRevision;
1515

1616
// Cancellation for the single in-flight simulation. bb-avm-sim runs exactly one
1717
// simulation at a time; the SIGUSR1 handler (which may run on any thread) cancels

‎barretenberg/cpp/src/barretenberg/avm/avm_execute.hpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
#include "barretenberg/avm/generated/avm_ipc_server.hpp"
88
#include "barretenberg/cdb/cdb_ipc_client.hpp"
99
#include "barretenberg/vm2/simulation/lib/cancellation_token.hpp"
10-
#include "barretenberg/wsdb/generated/wsdb_ipc_client.hpp"
10+
#include "barretenberg/vm2_wsdb/generated/wsdb_ipc_client.hpp"
1111

1212
#include <atomic>
1313

‎barretenberg/cpp/src/barretenberg/avm/avm_ipc_server.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
#include "barretenberg/avm/avm_execute.hpp"
33
#include "barretenberg/cdb/cdb_ipc_client.hpp"
44
#include "barretenberg/common/log.hpp"
5-
#include "barretenberg/wsdb/generated/wsdb_ipc_client.hpp"
5+
#include "barretenberg/vm2_wsdb/generated/wsdb_ipc_client.hpp"
66
#include "ipc_runtime/ipc_server.hpp"
77
#include "ipc_runtime/serve_helper.hpp"
88
#include "ipc_runtime/signal_handlers.hpp"

0 commit comments

Comments
 (0)