Commit eb2226c
authored
feat: per-wallet filter scan and runtime wallet catch-up (#122)
* feat: per-wallet filter scan and block processing
Filter matching and block processing now operate per wallet, so a wallet added at runtime catches up without disturbing the wallets that are already in sync.
`WalletInterface` is restructured around per-wallet operations:
- `process_block_for_wallets(block, height, wallets)` replaces the global `process_block` and only updates the listed wallets.
- `wallets_behind(height)` returns the wallet ids that still need filter coverage at `height`.
- `monitored_addresses_for(wallet_id)` and `wallet_synced_height(wallet_id)` give per-wallet projections for filter matching.
- `update_wallet_synced_height` and `update_wallet_last_processed_height` advance one wallet at a time and are monotonic.
- `BlockProcessingResult.new_addresses` and `CheckTransactionsResult.new_addresses` carry gap-limit discoveries with wallet attribution.
`FiltersManager.scan_batch` matches each behind wallet's addresses against the batch's filters at heights it hasn't yet covered. The per-block result flows through `BlocksNeeded` to `BlocksManager`, which processes each block only against the wallets whose filters matched it. `FiltersBatch` records the scanned wallet set so commit advances only their `synced_height`.
When a late-added wallet's filter matches a block already in flight, its id is merged into the existing entry. If the block has already been processed, it is re-queued so `BlocksManager` reloads it from local storage and processes it for the late wallet only.
`process_block_for_wallets` refreshes the cached balance even on rescan paths below the wallet's current `last_processed_height`, because UTXOs may change.
* feat: rescan filters when a wallet falls behind committed height
When `wallet.synced_height()` drops below `FiltersManager`'s `progress.committed_height()`, a wallet was added or moved behind the current scan position and needs catch-up coverage. Add a check at the top of `FiltersManager::tick()` that detects this regression, clears in-flight pipeline state, lowers `committed_height` to the new aggregate min, and re-enters `start_download()`. The check runs in `Syncing`, `Synced`, and `WaitForEvents` states so idle additions are caught on the next 100ms tick.
Add `test_wallet_added_at_runtime_catches_up` in `dash-spv/tests/dashd_sync/tests_basic.rs`. After initial sync with `W1`, mine a block funding `W2`'s address and add `W2` at runtime with `birth_height` before that block. Assert the rescan picks up `W2`'s funding transaction and `W1`'s state is unchanged. Then add `W3` with `birth_height` beyond the tip and assert no spurious rescan or regression in either existing wallet.
* refactor: add `wallets_not_yet_at` wrapper on `WalletInterface`
Encapsulate the off-by-one between `wallets_behind` (strict less-than)
and the inclusive height semantics needed by `scan_batch`, so call
sites no longer need to compensate with `saturating_add(1)`.
* test: add `MultiMockWallet` for per-wallet attribution tests
Single-wallet `MockWallet` cannot exercise paths that hinge on multiple
wallets having distinct `synced_height` values or independent address
sets. `MultiMockWallet` holds per-wallet state keyed by `WalletId`.
* fix: correct per-wallet filter scan and catch-up edge cases
Resolves several correctness issues in the per-wallet filter scan and
catch-up paths:
- \`track_block_match\` now distinguishes three states (\`NewlyTracked\`,
\`InFlight\`, \`AlreadyProcessed\`) instead of a boolean. A block that
was already processed in a prior round is no longer silently re-queued
for re-processing, and a block still in flight still re-emits
\`BlocksNeeded\` so the \`BlocksPipeline\` merges late-arriving wallet
ids into its pending wallet set.
- \`scan_batch\` only adds wallets that contributed addresses to the
scan into \`scanned_wallets\`. Empty-address wallets no longer have
their \`synced_height\` advanced for free.
- \`rescan_batch\` no longer drops new wallet ids when the matching
block is in flight; it forwards the wallet ids through a fresh
\`BlocksNeeded\`, which the pipeline merges into its pending set.
- \`tick\` resets \`committed_height\` to the lowest \`synced_height\` of
the stale wallets only, instead of the global wallet \`synced_height\`,
so already-synced wallets are not re-scanned from scratch.
- \`scan_batch\` runs the filters once over the union of behind-wallet
addresses, then attributes each match per-wallet by re-testing the
matched filter against that wallet's scripts. Cost drops from
O(N_wallets * batch_size) to O(batch_size + N_wallets * matches),
and the per-batch borrow of \`active_batches\` is consolidated.
- Field \`filters_matched\` renamed to \`matched_block_hashes\` to
reflect its actual role as a record (not a deduplication gate).
- \`rescan_batch\` now takes \`addresses_by_wallet\` by reference,
saving a clone per later-batch rescan in \`try_commit_batches\`.
- Uses \`wallets_not_yet_at(batch_end)\` to express the inclusive
height intent, hiding the \`+ 1\` off-by-one at the call site.
* test: add wallet-set propagation and queue-merge tests for `BlocksPipeline`
Cover the previously-implicit contract that `queue` and
`add_from_storage` merge wallet sets per block hash, and that
`take_next_ordered_block` returns the merged wallet set.
* fix: address per-wallet filter scan review feedback
Tighten `scan_batch` so wallets at `synced == batch_end` are skipped
(use `wallets_behind(batch_end)` instead of `wallets_not_yet_at`),
ensure behind wallets with zero monitored addresses still advance their
`synced_height` at commit, and surface filter errors and
`AlreadyProcessed` skips through `tracing::warn!` instead of silent
fallthrough.
Add a `min_height` parameter to `check_compact_filters_for_addresses`
so the union pass can skip irrelevant heights without cloning the
filter map. Reuse the existing `batch_filters` borrow for attribution.
Add tests covering:
- `scan_batch` advancing zero-address wallets
- `BlockTrackResult::InFlight` re-emission for late-arriving wallets
- `BlockTrackResult::AlreadyProcessed` skip path
- Union+attribute false-positive elimination
- Rescan from non-zero `synced_height` (not just genesis)
- `BlocksManager::process_buffered_blocks` routing the wallet set
* fix: skip re-enqueue when `BlocksPipeline` already tracks the hash
`BlocksPipeline::queue` unconditionally called `coordinator.enqueue` for every entry, including hashes that were already pending or in flight from a prior call. When a late-arriving wallet match for an already-queued block was emitted as `BlocksNeeded`, the duplicate `enqueue` corrupted the coordinator's pending count and could trigger a duplicate request to the peer. Only enqueue when the hash is not yet tracked, while still merging the late wallet ids into the per-block wallet set.
* fix: record `scanned_wallets` before `scan_batch` early returns
`scan_batch` recorded the per-batch `scanned_wallets` set only after the empty-filters early return, so a batch that hit that path was committed with an empty set and the per-wallet `synced_height` never advanced for that range. The next tick relisted the same wallets via `wallets_behind` and re-scanned the same range. Move the `set_scanned_wallets` call ahead of the empty-filter and empty-address fast paths so every behind wallet's `synced_height` advances when the batch commits.
Also pass each wallet's own `synced_height` as `min_synced` to `check_compact_filters_for_addresses` in `rescan_batch`. Otherwise a new address could spuriously match heights the wallet has already processed and `track_block_match` would route the result into `AlreadyProcessed` and silently drop it.
* test: cover wallet-set exclusion routing and FFI count semantics
Add a `BlocksManager` test that asserts a wallet absent from the pipeline's interested set never receives `process_block_for_wallets`, complementing the existing positive routing test. Add `dash-spv-ffi` callback tests covering the two behavioural changes of this PR: `BlocksNeeded` dispatch reports unique `FilterMatchKey` count (not inflated by the per-block wallet attribution), and `BlockProcessed` dispatch reports the total address count summed across the per-wallet `new_addresses` map.1 parent 53b25c4 commit eb2226c
22 files changed
Lines changed: 2011 additions & 310 deletions
File tree
- dash-spv-ffi/src
- dash-spv
- src/sync
- blocks
- filters
- mempool
- tests/dashd_sync
- key-wallet-ffi/src
- key-wallet-manager
- examples
- src
- test_utils
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
347 | 347 | | |
348 | 348 | | |
349 | 349 | | |
350 | | - | |
| 350 | + | |
351 | 351 | | |
352 | 352 | | |
353 | 353 | | |
| |||
366 | 366 | | |
367 | 367 | | |
368 | 368 | | |
| 369 | + | |
369 | 370 | | |
370 | 371 | | |
371 | 372 | | |
372 | | - | |
| 373 | + | |
373 | 374 | | |
374 | 375 | | |
375 | 376 | | |
| |||
755 | 756 | | |
756 | 757 | | |
757 | 758 | | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
| 778 | + | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
| 784 | + | |
| 785 | + | |
| 786 | + | |
| 787 | + | |
| 788 | + | |
| 789 | + | |
| 790 | + | |
| 791 | + | |
| 792 | + | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
79 | 79 | | |
80 | 80 | | |
81 | 81 | | |
82 | | - | |
| 82 | + | |
83 | 83 | | |
84 | 84 | | |
85 | | - | |
| 85 | + | |
| 86 | + | |
86 | 87 | | |
87 | | - | |
| 88 | + | |
88 | 89 | | |
89 | 90 | | |
90 | 91 | | |
| 92 | + | |
91 | 93 | | |
92 | 94 | | |
93 | 95 | | |
| |||
96 | 98 | | |
97 | 99 | | |
98 | 100 | | |
99 | | - | |
| 101 | + | |
100 | 102 | | |
101 | 103 | | |
102 | 104 | | |
103 | 105 | | |
104 | 106 | | |
105 | 107 | | |
106 | 108 | | |
107 | | - | |
108 | | - | |
| 109 | + | |
| 110 | + | |
109 | 111 | | |
110 | | - | |
| 112 | + | |
111 | 113 | | |
| 114 | + | |
112 | 115 | | |
113 | 116 | | |
114 | 117 | | |
| |||
168 | 171 | | |
169 | 172 | | |
170 | 173 | | |
171 | | - | |
| 174 | + | |
172 | 175 | | |
173 | | - | |
| 176 | + | |
174 | 177 | | |
175 | 178 | | |
176 | 179 | | |
| |||
215 | 218 | | |
216 | 219 | | |
217 | 220 | | |
218 | | - | |
219 | | - | |
| 221 | + | |
| 222 | + | |
220 | 223 | | |
221 | 224 | | |
222 | 225 | | |
| |||
227 | 230 | | |
228 | 231 | | |
229 | 232 | | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
230 | 329 | | |
0 commit comments