Lite wallet#35
Draft
akumaigorodski wants to merge 15 commits into
Draft
Conversation
akumaigorodski
marked this pull request as draft
June 16, 2026 13:50
akumaigorodski
force-pushed
the
master
branch
2 times, most recently
from
June 16, 2026 13:57
95e1d8b to
42637d6
Compare
Store UTXO creation height in LMDB so lite wallets can request delta updates. Add utxos_by_address to serve those wallet queries directly and speed up existing fn get_utxos_by_addresses (now using LMDB range queries). Route UTXO put/delete/spend/unspend paths through helpers that keep the primary outpoint index, address index, and STXO table in sync.
- Add AddressTxidKey keyed as address || txid - Store filled tx, its Merkle proof, block hash/height, and Unix timestamp - Update txdb with new confirmed txs when blocks connect to the active chain - Query txdb by address set and minimum block height - Prune txdb by Unix timestamp every 144 * 7 blocks (~weekly) - Prune txdb records for disconnected blocks during reorgs - Make Merkle proof types serializable General rationale: lite wallet is UTXO-based so strictly speaking it does not need to know tx history, however, it is still nice to show users actual tx history so they can figure out what was going on. However again, storing full tx history is too demanding for a full node so it is capped.
Author
|
Next planned steps:
|
Done: - Add rustreexo dependency. - Add Accumulator wrapper with serde persistence. - Add compact AccumulatorDiff serialization. - Add archive tables for accumulator snapshots and diffs. - Add archive accessors for accumulator get/put/prune. - Add State utreexo accumulator initialization. - Add startup replay from latest snapshot to tip. - Simplify txdb pruning return values. Remaining: - Generate and persist diffs during block connect. - Apply live accumulator updates on connect. - Rewind accumulator during disconnect and reorg. - Store and prune periodic accumulator snapshots. - Prune old accumulator diffs from block processing. - Handle no-snapshot genesis replay edge case. - Consider binding snapshots to block hash.
Make it store [u8; 32] directly since empty utreexo hashes are not allowed there anyway.
…by_address utxo_heights_by_address now just stores the height of a given UTXO. This will save on storage costs once UTXO set gets big.
…or with it and persist both Diff and Accumulator
Add complete disconnect and reorg handling for both sidechain state and the Utreexo accumulator. Apply each reorg as one all-or-nothing operation. First rebuild the accumulator at the common ancestor, then disconnect the old branch and connect the complete replacement branch in one database transaction. If any block fails, none of the replacement branch is committed. Keep the accumulator locked throughout this operation. If the reorg returns an error or panics, rebuild the in-memory accumulator from the last committed tip. Drop the old forest before rebuilding it so that two full forests are never kept in memory at the same time. Make accumulator snapshots work correctly with forks: - Store snapshots by sidechain block hash, not only by height. - Use only snapshots that belong to the requested block's ancestry. - Read the height of each snapshot without deserializing its full forest. - Deserialize only the newest usable snapshot and replay later block diffs. - Include deposits and withdrawal events in accumulator diffs. - Keep Empty, Placeholder, and real accumulator hashes distinct when saved. - Reject attempts to insert a leaf that already exists. - Keep enough snapshots and diffs for an explicit maximum reorg horizon. - Refuse reorgs that go beyond the retained history. Work out the exact range of L1 blocks applied by each L2 block from the L1 hash stored in that block and its parent. Apply those L1 blocks in forward order when connecting, and reverse order when disconnecting. This makes deposits and withdrawal events follow the correct L1 history during an L2-only reorg. Subscribe to raw L1 connect and disconnect events and reconcile the sidechain automatically. On startup, fetch and archive missing L1 history in bounded batches. Only accept BMM confirmations that remain on the current L1 chain. After an L1 reorg, switch to a ready sidechain branch when possible, otherwise rewind to the newest sidechain block still valid on the new L1 chain. Keep withdrawal bundle metadata in a durable archive that is not removed when an L2 branch is disconnected. This allows a withdrawal created on an abandoned branch to be applied later using its exact sidechain outpoints. Add GetWithdrawalBundle as a recovery path for nodes that never saw the branch where a withdrawal bundle was created. Check that peer-provided metadata matches the requested withdrawal transaction ID, its input commitment, and the size limits. Load all UTXO values from local state rather than trusting values supplied by a peer. If withdrawal metadata is missing, leave the complete reorg uncommitted, mark it as blocked, keep requesting the metadata, and retry the reorg when it becomes available. Invalid peer blocks and ordinary rejected branches no longer stop the network task, while database failures and unrecoverable L1 inconsistencies still stop safely. Fix withdrawal rollback ordering so pending, submitted, confirmed, and failed bundles restore their UTXOs correctly. Keep deposit and withdrawal event markers in sync, and reject duplicate deposits instead of overwriting or resurrecting existing UTXOs. Reduce peak memory use during large reorgs and startup: - Keep replacement block hashes in memory, but load one body at a time. - Load one L1 block's event data at a time while applying a branch. - Fetch historical L1 data in bounded batches. - Limit withdrawal inputs before copying and grouping their UTXOs. - Use deterministic ordering when constructing withdrawal bundles. Keep mempool and transaction-index updates in the same transaction as the reorg. Do not restore disconnected transactions that conflict with the current mempool, and remove invalid archived bodies without exposing a partial reorg. Add tests covering fork-specific snapshots, snapshot pruning, accumulator serialization and recovery, reorg limits, failures during atomic replacement, L2-only reorgs with finalized withdrawals and later deposits, raw L1 reorgs, withdrawal metadata lifetime and rollback order, duplicate deposits, and bounded withdrawal bundle construction. This changes accumulator storage, state metadata, and the peer protocol without providing a migration or compatibility layer.
Author
|
At this point a few choices has been made where I would like to hear maintainer opinion on them: basically everything up to commit 4f44ff0. Later commits are devoted mostly to utreexo accumulator handling, not yet finished and can be reviewed later. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Work in progress with an end goal to have a complete lite wallet protocol support outlined in #28