chore(port): forward-port remaining v5-next fairies backlog to next#24949
Merged
Conversation
…E_LOGS_PER_TX + headroom, not exactly MAX_PRIVATE_LOGS_PER_TX (#24437) ## Summary Follow-up to #24429. That PR widened `UNFINALIZED_TAGGING_INDEXES_WINDOW_LEN` from `20` to exactly `MAX_PRIVATE_LOGS_PER_TX` (64). This PR argues that's still not the right value for the scenario #24429 was trying to fix, and proposes `MAX_PRIVATE_LOGS_PER_TX + 20`. **Scope note (see "Known limitation" below):** this fixes a real, CI-confirmed problem; multiple ordinary pending txs to the same counterparty stacking up before the first is mined. There is a separate, harder problem captured in [F-783](https://linear.app/aztec-labs/issue/F-783/pxe-unfinalized-tagging-indexes-window-len-cannot-bound-a-squash-heavy). Additional separate breakage follow-up on the sender-side when pending logs exceed the window [F-784](https://linear.app/aztec-labs/issue/F-784/pxe-typed-error-and-resync-recheck-when-pending-tagging-indexes-exceed). ## Investigation `MAX_PRIVATE_LOGS_PER_TX` is a genuine, protocol-enforced bound for ordinary transaction patterns: `noir-protocol-circuits` `constants.nr` defines `MAX_PRIVATE_LOGS_PER_TX` directly as `MAX_NOTE_HASHES_PER_TX`. A fresh secret's very first ordinary tx can legitimately consume that many indexes before anything is finalized. But exactly `MAX_PRIVATE_LOGS_PER_TX` leaves zero headroom for a second unconfirmed tx to the same counterparty once the first one is already near the max. `finalized` only advances once the sender's own node observes a tx as mined (see `syncSenderTaggingIndexes`), so any burst of sends to one counterparty before the first is mined can exhaust the window. This is exactly what happened in #24429's `bench_build_block` failure: `#24387`'s handshake default added extra per-cold-chain overhead, and the bench builds up a backlog of unmined txs by design, so `used` ran past `finalized` (which stayed at `0`) before window `20` could absorb it. ## Confirmed live: reproduced on this branch at bare `20` To settle whether #24429's bump was actually necessary, we reset this branch's constant back to bare `20` and let `bench_build_block` run to completion in CI rather than relying on the reasoning above alone. It failed the same way (see http://ci.aztec-labs.com/4ed074af4b8ab633 from [a4feea1](a4feea1), on the current `merge-train/fairies-v5` tip: ``` ERROR: kv-store:lmdb-v2 pxe-0 Failed to commit transaction: Error: Highest used index 21 is further than window length from the highest finalized index 0. Tagging window length 20 is configured too low. Contact the Aztec team to increase it! ``` So this isn't a synthetic worst case: it's a real failure on real CI infrastructure, at index `21`, i.e. just one past the window. Not even a large burst of txs. ## Red/green evidence Added `sender_tagging_store.test.ts`: "allows an ordinary pending tx to stack on a fresh secret already at the `MAX_PRIVATE_LOGS_PER_TX` floor" — one tx uses the full worst-case 64-log burst from a fresh secret, then a second, ordinary-sized pending tx to the same secret must still be storable before either is mined. - **Red at bare `20`:** the *first* call in that test already throws, with zero other txs involved: ``` Highest used index 63 is further than window length from the highest finalized index 0. Tagging window length 20 is configured too low. ``` - **Green at `MAX_PRIVATE_LOGS_PER_TX + 20`:** same scenario passes. This is the key point for review: the `bench_build_block` failure isn't just "risky under bursty load".`20` is mathematically incapable of tolerating even one legitimate worst-case single ordinary tx, independent of what CI on this branch shows either way. CI passing at `20` only means our current test/bench suite doesn't happen to construct a single tx tagging more than 20 logs to one recipient; it was never testing the general safety of the constant, just this specific workload's exposure to it. The live CI failure above is the same underlying gap, just surfaced through a different, real workload instead of a targeted unit test. We also added tests to pin the window's stacking semantics: a run of window+1 one-log pending txs to one secret fits, while the next tx throws. ## Why `+ 20`, not `* 2` or another additive constant - The floor (`MAX_PRIVATE_LOGS_PER_TX`) covers one worst-case *ordinary* tx. The failure mode above is about *additional, ordinary-sized* pending txs to the same counterparty stacking up, not another maxed-out 64-log tx. - It's the value the team had already judged sufficient for realistic pending-tx headroom before #24429 (it was just misapplied as the *entire* budget instead of a margin on top of the floor). --------- Co-authored-by: Martin Verzilli <martin@aztec-labs.com> (cherry picked from commit f58e212)
Closes #5837 This PR makes it so we no longer derive signing keys from the master privacy key. If there's no user-supplied privacy key seed, we instead derive it _from_ the signing key, which is a safe derivation: PXE never sees the signing key nor any value it is derived from. This has a large knock-on effect on wallets, CLI, tutorials, etc., but there's ultimately no large decisions being made there, it is mostly mechanical changes. In some cases I reworked some internals of wallets to make this distinction clearer. (cherry picked from commit 94af533)
SQLite3MultipleCiphers 2.3.5 (#24293) stopped shipping the bundler-friendly build variant, so the package entry re-exports the plain sqlite3.mjs. That loader always resolves sqlite3.wasm through its Module['locateFile'] hook, which computes new URL(path, import.meta.url) with a dynamic path — invisible to bundlers. Bundled consumers (e.g. Vite apps using @aztec/kv-store/sqlite-opfs) request an unhashed sqlite3.wasm relative to the emitted chunk and 404 at runtime; dev servers are unaffected, so only production builds break. Wrap the init and inject emscriptenLocateFile (the vendored loader's supported escape hatch= resolving the wasm via a static `new URL('../vendor/jswasm/sqlite3.wasm', import.meta.url)` that bundlers detect, emit, and rewrite. Unbundled usage is unaffected: the static URL resolves to the real vendored path. Verified by building a Vite consumer against the patched dest/: the emitted worker chunk resolves the hashed wasm asset through the hook. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> (cherry picked from commit 8c7e7c7)
(cherry picked from commit 23e8e1e)
The fee payer should only be set in the non-revertible phase, but we were not asserting this. (cherry picked from commit da9ac1c)
Upstreaming the utilities build for `aztec-kit` so external projects can benefit from our e2e scaffolding. Essentially the same thing we already use, but pure ts rather than the unpublishable .sh script. Verified with `ci-full-no-test-cache` (cherry picked from commit 7e09008)
…benchmarks (#24275) Fixes [F-704](https://linear.app/aztec-labs/issue/F-704/optimize-constrained-tag-sync) ## Summary Constrained delivery emits a gapless tagging-index stream, so PXE can stop scanning at the first missing constrained tag instead of probing the full unfinalized window on every sync. This PR: - Starts constrained scans with a small probe and **doubles** the probe size while every probed index has a log, capped at the existing window. - Keeps steady-state polling cheap: one tag per constrained secret. - Reduces catch-up round-trips geometrically until the probe reaches the window cap. - Decouples probe advancement from finalized-cursor persistence so unfinalized logs are still fetched without persisting unsafe cursors. - Leaves unconstrained sync and `findHighestIndexes` unchanged. ## Benchmarks Benchmarks compare doubling against fixed-size probes. The takeaway is that doubling preserves the steady-state tag-query floor of `P=1` while avoiding `P=1`'s one-round-trip-per-log catch-up behavior. A sync runs in rounds: each round computes the next batch of tags, sends them to the node, and blocks on the result before deciding the next round. All counts are per sync. - **tag-queries** — total tags looked up on the node. - **round-trips** — sequential client waits. A round's tags are chunked into parallel calls internally, so a wide round is still one round-trip. - **blocking-ms** — measured wall-clock the caller spends blocked on the node with modeled node latency. This is reported only and varies run to run. Scenarios seed a recipient that already synced prior messages, then measure the next single sync. `secrets = N` means N independent sender streams synced together in one batched pass. - **steady-state** — no new logs since the last sync. - **catch-up-K** — K new contiguous logs per secret since the last sync. - **mixed** — 999 idle secrets plus 1 deep straggler (`K=100`) at 1,000 secrets. ### Cost per sync: doubling vs fixed-P alternatives **doubling is the shipped policy.** The fixed-P columns are the selection comparison that motivated it: `P=84` is the current full-window behavior, and `P=1..5` sweep the constant-step alternative. Unconstrained is not shown as its own row: its windowed scan cannot first-miss, so it is invariant to P and reproduces the `P=84` column. **Tag-queries (thousands)** (exact count = value x 1,000; bold = fewest among the fixed-P columns; `doubling (init=1)` is the shipped policy, `doubling (init=2)`/`doubling (init=4)` start the probe at 2/4 instead of 1): | scenario | doubling (init=1) | doubling (init=2) | doubling (init=4) | P=1 | P=2 | P=3 | P=4 | P=5 | P=84 | |---|---|---|---|---|---|---|---|---|---| | steady, 100 | 0.1 | 0.2 | 0.4 | **0.1** | 0.2 | 0.3 | 0.4 | 0.5 | 8.4 | | steady, 1,000 | 1 | 2 | 4 | **1** | 2 | 3 | 4 | 5 | 84 | | catch-up-1, 100 | 0.3 | 0.2 | 0.4 | **0.2** | 0.2 | 0.3 | 0.4 | 0.5 | 8.4 | | catch-up-1, 1,000 | 3 | 2 | 4 | **2** | 2 | 3 | 4 | 5 | 84 | | catch-up-3, 100 | 0.7 | 0.6 | 0.4 | **0.4** | 0.4 | 0.6 | 0.4 | 0.5 | 8.4 | | catch-up-3, 1,000 | 7 | 6 | 4 | **4** | 4 | 6 | 4 | 5 | 84 | | catch-up-84, 100 | 12.7 | 12.6 | 12.4 | **8.5** | 8.6 | 8.7 | 8.8 | 8.5 | 16.8 | | catch-up-84, 1,000 | 127 | 126 | 124 | **85** | 86 | 87 | 88 | 85 | 168 | | catch-up-50, 100 | 6.3 | 6.2 | 6 | **5.1** | 5.2 | 5.1 | 5.2 | 5.5 | 8.4 | | catch-up-50, 1,000 | 63 | 62 | 60 | **51** | 52 | 51 | 52 | 55 | 84 | | catch-up-100, 100 | 12.7 | 12.6 | 12.4 | **10.1** | 10.2 | 10.2 | 10.4 | 10.5 | 16.8 | | catch-up-100, 1,000 | 127 | 126 | 124 | **101** | 102 | 102 | 104 | 105 | 168 | | mixed, 1,000 (999 idle + 1 deep K=100) | 1.126 | 2.124 | 4.12 | **1.1** | 2.1 | 3.099 | 4.1 | 5.1 | 84.084 | **Round-trips** (bold = fewest among the fixed-P columns; `doubling (init=1)` is the shipped policy, `doubling (init=2)`/`doubling (init=4)` start the probe at 2/4 instead of 1): | scenario | doubling (init=1) | doubling (init=2) | doubling (init=4) | P=1 | P=2 | P=3 | P=4 | P=5 | P=84 | |---|---|---|---|---|---|---|---|---|---| | steady, 100 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | | steady, 1,000 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | | catch-up-1, 100 | 2 | 1 | 1 | 2 | **1** | 1 | 1 | 1 | 1 | | catch-up-1, 1,000 | 2 | 1 | 1 | 2 | **1** | 1 | 1 | 1 | 1 | | catch-up-3, 100 | 3 | 2 | 1 | 4 | 2 | 2 | **1** | 1 | 1 | | catch-up-3, 1,000 | 3 | 2 | 1 | 4 | 2 | 2 | **1** | 1 | 1 | | catch-up-84, 100 | 7 | 6 | 5 | 85 | 43 | 29 | 22 | 17 | **2** | | catch-up-84, 1,000 | 7 | 6 | 5 | 85 | 43 | 29 | 22 | 17 | **2** | | catch-up-50, 100 | 6 | 5 | 4 | 51 | 26 | 17 | 13 | 11 | **1** | | catch-up-50, 1,000 | 6 | 5 | 4 | 51 | 26 | 17 | 13 | 11 | **1** | | catch-up-100, 100 | 7 | 6 | 5 | 101 | 51 | 34 | 26 | 21 | **2** | | catch-up-100, 1,000 | 7 | 6 | 5 | 101 | 51 | 34 | 26 | 21 | **2** | | mixed, 1,000 (999 idle + 1 deep K=100) | 7 | 6 | 5 | 101 | 51 | 34 | 26 | 21 | **2** | **Blocking wall-clock (ms)** (reported only, noisy; the three `doubling` columns are from one paired re-run, the fixed-P columns from the earlier comparison sweep, so not every column is from a single run): | scenario | doubling (init=1) | doubling (init=2) | doubling (init=4) | P=1 | P=2 | P=3 | P=4 | P=5 | P=84 | |---|---|---|---|---|---|---|---|---|---| | steady, 100 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 8 | | steady, 1,000 | 6 | 7 | 7 | 7 | 5 | 7 | 7 | 8 | 21 | | catch-up-1, 100 | 14 | 8 | 8 | 13 | 8 | 8 | 8 | 8 | 12 | | catch-up-1, 1,000 | 32 | 28 | 31 | 32 | 27 | 29 | 31 | 28 | 54 | | catch-up-3, 100 | 23 | 18 | 13 | 29 | 18 | 18 | 12 | 12 | 15 | | catch-up-3, 1,000 | 77 | 77 | 71 | 83 | 73 | 79 | 64 | 67 | 95 | | catch-up-84, 100 | 207 | 215 | 211 | 663 | 407 | 335 | 321 | 265 | 205 | | catch-up-84, 1,000 | 1,886 | 1,893 | 1,986 | 2,226 | 1,942 | 1,935 | 1,871 | 1,830 | 2,000 | | catch-up-50, 100 | 132 | 132 | 132 | 397 | 243 | 193 | 171 | 161 | 127 | | catch-up-50, 1,000 | 1,162 | 1,167 | 1,193 | 1,307 | 1,169 | 1,191 | 1,114 | 1,108 | 1,217 | | catch-up-100, 100 | 244 | 244 | 260 | 784 | 480 | 396 | 348 | 321 | 245 | | catch-up-100, 1,000 | 2,242 | 2,265 | 2,361 | 2,632 | 2,348 | 2,341 | 2,212 | 2,236 | 2,366 | | mixed, 1,000 (999 idle + 1 deep K=100) | 43 | 37 | 31 | 590 | 296 | 199 | 150 | 124 | 32 | ### Takeaway `P=1` is the tag-query floor, but it pays one round-trip per new log during catch-up. The full window (`P=84`) minimizes catch-up round-trips, but it charges every idle secret the full-window tag cost on every sync. Doubling is the middle ground: it matches `P=1` at steady state, stays close to `P=1` on tag queries during catch-up, and collapses deep catch-up round-trips geometrically. In the mixed scenario, doubling uses 1,126 tag queries vs 1,100 for `P=1`, but needs 7 round-trips instead of 101; compared with `P=84`, it avoids the 84,084-tag idle tax while staying within 5 round-trips of the full-window catch-up path. ## Testing - Unit tests cover scan shape, doubling behavior, batched-round semantics, and returned-log equivalence. - Benchmark tests report tag queries, round-trips, and modeled blocking time for steady-state, catch-up, mixed, and unconstrained scenarios. (cherry picked from commit 40d3c92)
nchamo
added
ci-draft
nchamo
marked this pull request as ready for review
July 23, 2026 20:37
vezenovm
approved these changes
Jul 23, 2026
nchamo
enabled auto-merge (squash)
July 23, 2026 20:56
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Final slice of the manual v5-next → next backlog sweep, following #24931 (aztec-nr/contracts) and #24932 (pxe/client/txe). Cherry-picks (
-x) the seven Fairies PRs whose content was confirmed absent fromnextby a three-way code audit (content presence, not commit history):Notes on resolutions:
nextwhile the code was not — this PR brings the code in line with the docs. Their duplicate note insertions were dropped in conflict resolution.e2e_constrained_delivery.test.ts(deleted onnext, touched by feat!: stop deriving signing key from privacy keys #24439 on v5) stays deleted; the build confirms no surviving call site needed its change.Verified locally: full
yarn build, pxe tagging suite (79 tests incl. the new headroom and probe tests),key_derivationtests, backwards-compat suites, both oracle-version checks, and contract compilation against the updated aztec-nr. Thephase_checke2e (#24479) and browser bundling (#24529) rely on CI coverage.