refactor: extract wsdb/lmdblib/kvdb into native-packages, decoupled from barretenberg

[charlielye]

What

Two changes that belong together, in one commit:

1. Extract the world-state DB server + persistent merkle (wsdb), the lmdb C++ wrapper (lmdblib), and the kv-store NAPI (kvdb) out of barretenberg into top-level native-packages/ siblings, leaving barretenberg free of DB code.
2. Decouple the extracted wsdb from barretenberg's C++ headers, so its world-state/merkle/storage core, its test suite, and the aztec-wsdb binary compile with zero barretenberg headers on the include path — barretenberg is only linked (libbarretenberg.a), purely to satisfy a small flat poseidon2 c_bind.

Together this lets wsdb/lmdblib/kvdb and barretenberg eventually live in independently-owned repos / CI without sharing a compiler, stdlib, or build flags — the only things wsdb and barretenberg share are the IPC schema (already the case) and a couple of extern "C" hash calls.

Extraction (barretenberg → native-packages/)

• New native-packages/ siblings: wsdb (world-state DB server + persistent content-addressed merkle), lmdblib (the C++ lmdb wrapper), kvdb (the kv-store nodejs_module.node NAPI addon that owns the LMDBStore wrapper).
• Barretenberg's merkle_tree headers are split: barretenberg keeps only the response/types the AVM's LowLevelMerkleDBInterface returns and its hints serialize; the full async tree engine (content-addressed trees + world_state) and its response/tree_meta/types move into wsdb. world_state, lmdblib, wsdb, and the merkle-tree benchmarks are deleted from barretenberg/cpp.
• Build: the standalone aztec-wsdb binary and the kvdb .node build against prebuilt barretenberg; lmdblib/kvdb are fully barretenberg-free. lmdblib switches to upstream msgpack + FetchContent. The lmdblib/wsdb C++ tests are wired into the CI test engine.

Decoupling wsdb from barretenberg headers

• Hash c_bind (barretenberg/crypto/poseidon2/poseidon2_capi.cpp, in libbarretenberg.a): flat bb_poseidon2_hash + bb_poseidon2_hash_pair_with_separator. Field elements cross as 32 canonical bytes — no bb field type, stdlib, or build-flag agreement required. This is the only computation wsdb can't own (it must match what the AVM proves and what is committed on L1), so it is single-sourced in barretenberg.
• azteclabs::wsdb::FieldElement: a 32-byte field value with no arithmetic (wsdb only stores/orders/compares/serialises field values and feeds them to the hash). A using fr = FieldElement; alias inside namespace azteclabs::wsdb lets the forked merkle headers compile unchanged.
• Forked the shared merkle headers (types, hash_path, merkle_tree_id, tree_meta, indexed_leaf, response, memory_tree, signal) into wsdb under namespace azteclabs::wsdb::merkle_tree, swapping SERIALIZATION_FIELDS → MSGPACK_DEFINE_MAP (msgpack-c directly). The msgpack values are byte-identical to bb::fr, so on-disk lmdb data and the IPC wire are unchanged.
• Owns bb-free common utils: log, assert, thread_pool, bitop, random, a minimal uint256_t (for the reference/test nullifier abs-diff search), aztec_constants, and vendored cli11.

Proof

• aztec-wsdb + wsdb_tests build with BB_SRC removed from the include dirs; 0 bb #includes remain in the decoupled set. (Also verified locally from the combined commit: lmdblib + aztec-wsdb compile and link.)
• 150 decoupled wsdb_tests pass.
• A bb-linked parity target (-DWSDB_BUILD_BB_TESTS=ON) proves FieldElement + the c_bind are byte-identical to bb::fr and bb's Poseidon2HashPolicy (hash, msgpack, ordering, domain separators) — 6 tests pass.
• The decoupled core compiles under libstdc++ (≠ bb's libc++), demonstrating header independence. The shipping build deliberately stays on bb's toolchain (libc++/-march).

Notes / follow-ups

• lmdb key byte order changed from uint256 little-endian to canonical big-endian (FrKeyType = fr, comparator value_cmp<fr>). msgpack values are unchanged. This is fine for a fresh service (re-sync) but is not backward-compatible with databases written before this change.
• The MemoryMerkleDB equivalence test (wsdb/memory_merkle_db.test.cpp) is left in-tree but not wired into the default build; reviving it (it links both wsdb's forked merkle headers and barretenberg's real ones in one TU) is a follow-up. The bb-linked parity target covers the consensus-critical byte-identity guarantee in the meantime.

[charlielye]

!ci

[charlielye]

!ci

[charlielye]

!ci

[charlielye]

!ci

[charlielye]

!ci

[charlielye]

!ci

[charlielye]

!ci

[charlielye]

!ci

[charlielye]

!ci

[charlielye]

!ci

[charlielye]

!ci

[charlielye]

!ci

[charlielye]

!ci

[charlielye]

!ci

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	npm/​@​aztec/​kvdb-darwin-arm64@​0.0.0-use.local	N/A	N/A	N/A	N/A	N/A
	npm/​@​aztec/​kvdb-darwin-x64@​0.0.0-use.local	N/A	N/A	N/A	N/A	N/A
	npm/​@​aztec/​kvdb-linux-arm64@​0.0.0-use.local	N/A	N/A	N/A	N/A	N/A
	npm/​@​aztec/​kvdb-linux-x64@​0.0.0-use.local	N/A	N/A	N/A	N/A	N/A
	npm/​@​aztec/​kvdb@​0.0.0-use.local	N/A	N/A	N/A	N/A	N/A

View full report