Merge #7101: feat!: migrate additional indexes from synchronous to async operation

cc1dd3c fix: harden index migration and document timestampindex prune caveat (UdjinM6)
0695e97 refactor: move BlockDisconnected handling to BaseIndex (UdjinM6)
c41584e refactor: move BlockUntilSyncedToCurrentChain from index queries to RPC callers (UdjinM6)
1ca7374 fix: harden BlockDisconnected sibling check and getaddressbalance height race (UdjinM6)
386acdf fix: add undo data size validation in AddressIndex::Rewind (UdjinM6)
5796588 fix: reverse transaction processing order in AddressIndex::Rewind (UdjinM6)
8f3f62b fix: add vtxundo size validation in SpentIndex::WriteBlock (UdjinM6)
97dcc6d fix: address review suggestions (UdjinM6)
331335e fix: add explicit vtxundo size validation and in-class initialization for m_block_height (UdjinM6)
98c3c1b fix: check index sync status before querying and show sync progress (UdjinM6)
e84b1fa feat: add async indexes to getindexinfo RPC (UdjinM6)
5be629b refactor: apply clang-format (UdjinM6)
fd1de98 refactor: address code review feedback (UdjinM6)
5fd7033 docs: add release notes (UdjinM6)
6e5f7ab fix: add missing include in `src/zmq/zmqpublishnotifier.cpp` (UdjinM6)
fa60ccb feat: migrate old synchronous index data to new async index databases (UdjinM6)
fa7be34 feat: add CompactRange template method to CDBWrapper (UdjinM6)
8ad06be test: update feature_addressindex for async behavior (UdjinM6)
18276a6 feat: migrate AddressIndex from synchronous to async operation (UdjinM6)
3ea9394 test: update feature_spentindex for async behavior (UdjinM6)
464cdf8 feat: migrate SpentIndex from synchronous to async operation (UdjinM6)
33d2fc5 test: update timestampindex functional test for async behavior (UdjinM6)
f3fcb97 feat: migrate TimestampIndex from synchronous to async operation (UdjinM6)

Pull request description:

  ## Issue being fixed or feature implemented
  Migrate three Dash-specific indexes (`AddressIndex`, `SpentIndex`, `TimestampIndex`) from synchronous operation in `CBlockTreeDB` to async operation using `BaseIndex` framework.

  When enabling an index for the first time, the node will build the index in the background while remaining fully operational. Progress can be monitored via the `getindexinfo` RPC.

  Existing nodes with indexes enabled will automatically migrate data from the old location (block index database) to new separate databases on first startup. This migration may take 20-40 minutes or longer depending on hardware specifications and index sizes. The node will log progress during migration.

  ## What was done?
  Architecture Improvements:
  - Indexes now derive from `BaseIndex`, enabling async catch-up during IBD
  - Each index has its own database under `indexes/{addressindex,spentindex,timestampindex}/`
  - No longer blocks validation during index updates
  - Can be enabled/disabled without `-reindex`

  Migration System:
  - Template-based `MigrateIndex<>()` function handles data migration
  - Threshold-based incremental compaction (256 MiB) to reclaim disk space
  - Iterator reset before compaction to allow LevelDB file deletion
  - Best block locator written to ensure index consistency

  ## How Has This Been Tested?
  Run tests. Run a testnet/mainnet node and confirm migration succeeds (backup old datadir cause otherwise you'd have to reindex to go back to develop/master binaries).

  ## Breaking Changes
  - `SpentIndex` and `AddressIndex` are incompatible with pruned nodes as they require access to undo data now
  - `TimestampIndex` can no longer be enabled on a datadir where pruning has already occurred. It still works when started together with `-prune` from a fresh sync or reindex (the index is built forward as blocks arrive), but enabling it on an existing pre-pruned datadir now fails at startup. The old synchronous implementation built forward from the current tip and accepted that case

  ## Checklist:
  - [ ] I have performed a self-review of my own code
  - [ ] I have commented my code, particularly in hard-to-understand areas
  - [ ] I have added or updated relevant unit/integration/functional/e2e tests
  - [ ] I have made corresponding changes to the documentation
  - [ ] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_

Top commit has no ACKs.

Tree-SHA512: b0129ffdec086883c7a8a10fbeec891b6588523e567d1fc083da96eef53b2bf2c72d4249f0d8420174b03bc58652a9fa68ecdb3e55dc9160b0630dda0f282d7e

Author: PastaPastaPasta

doc/release-notes-7101.md

@@ -0,0 +1,13 @@
+## Indexes
+
+### Async Index Migration (#7101)
+
+`TimestampIndex`, `SpentIndex`, and `AddressIndex` have been migrated from synchronous to asynchronous operation, following the same pattern as `TxIndex` and other indexes using `BaseIndex` framework.
+
+When enabling an index for the first time, the node will build the index in the background while remaining fully operational. Progress can be monitored via the `getindexinfo` RPC.
+
+Existing nodes with indexes enabled will automatically migrate data from the old location (block index database) to new separate databases on first startup. This migration may take 20-40 minutes or longer depending on hardware specifications and index sizes. The node will log progress during migration.
+
+Breaking changes:
+- `SpentIndex` and `AddressIndex` are incompatible with pruned nodes as they require access to undo data now
+- `TimestampIndex` can no longer be enabled on a datadir where pruning has already occurred. It still works when started together with `-prune` from a fresh sync or reindex (the index is built forward as blocks arrive), but enabling it on an existing pre-pruned datadir now fails at startup. The old synchronous implementation built forward from the current tip and accepted that case

src/Makefile.am

@@ -160,8 +160,6 @@ BITCOIN_CORE_H = \
   active/dkgsessionhandler.h \
   active/masternode.h \
   addrdb.h \
-  addressindex.h \
-  spentindex.h \
   addrman.h \
   addrman_impl.h \
   attributes.h \
@@ -250,10 +248,17 @@ BITCOIN_CORE_H = \
   httprpc.h \
   httpserver.h \
   i2p.h \
+  index/addressindex.h \
+  index/addressindex_types.h \
+  index/addressindex_util.h \
   index/base.h \
   index/blockfilterindex.h \
   index/coinstatsindex.h \
   index/disktxpos.h \
+  index/spentindex.h \
+  index/spentindex_types.h \
+  index/timestampindex.h \
+  index/timestampindex_types.h \
   index/txindex.h \
   indirectmap.h \
   init.h \
@@ -348,7 +353,6 @@ BITCOIN_CORE_H = \
   rpc/blockchain.h \
   rpc/client.h \
   rpc/evo_util.h \
-  rpc/index_util.h \
   rpc/mempool.h \
   rpc/mining.h \
   rpc/protocol.h \
@@ -383,7 +387,6 @@ BITCOIN_CORE_H = \
   support/events.h \
   support/lockedpool.h \
   sync.h \
-  timestampindex.h \
   threadsafety.h \
   timedata.h \
   torcontrol.h \
@@ -488,7 +491,6 @@ libbitcoin_node_a_SOURCES = \
   active/dkgsessionhandler.cpp \
   active/masternode.cpp \
   addrdb.cpp \
-  addressindex.cpp \
   addrman.cpp \
   banman.cpp \
   batchedlogger.cpp \
@@ -534,9 +536,13 @@ libbitcoin_node_a_SOURCES = \
   httprpc.cpp \
   httpserver.cpp \
   i2p.cpp \
+  index/addressindex.cpp \
+  index/addressindex_util.cpp \
   index/base.cpp \
   index/blockfilterindex.cpp \
   index/coinstatsindex.cpp \
+  index/spentindex.cpp \
+  index/timestampindex.cpp \
   index/txindex.cpp \
   init.cpp \
   instantsend/db.cpp \
@@ -599,7 +605,6 @@ libbitcoin_node_a_SOURCES = \
   rpc/coinjoin.cpp \
   rpc/evo.cpp \
   rpc/fees.cpp \
-  rpc/index_util.cpp \
   rpc/masternode.cpp \
   rpc/mempool.cpp \
   rpc/governance.cpp \

src/addressindex.cpp

@@ -19,8 +19,7 @@
 #include <util/strencodings.h>
 #include <util/system.h>
 
-#include <addressindex.h>
-#include <spentindex.h>
+#include <index/spentindex.h>
 
 #include <evo/assetlocktx.h>
 #include <evo/cbtx.h>

src/core_write.cpp

@@ -392,6 +392,22 @@ class CDBWrapper
     {
         pdb->CompactRange(nullptr, nullptr);
     }
+
+    /**
+     * Compact a specific range of keys in the database.
+     */
+    template<typename K>
+    void CompactRange(const K& key_begin, const K& key_end) const
+    {
+        CDataStream ssKey1(SER_DISK, CLIENT_VERSION), ssKey2(SER_DISK, CLIENT_VERSION);
+        ssKey1.reserve(DBWRAPPER_PREALLOC_KEY_SIZE);
+        ssKey2.reserve(DBWRAPPER_PREALLOC_KEY_SIZE);
+        ssKey1 << key_begin;
+        ssKey2 << key_end;
+        leveldb::Slice slKey1(CharCast(ssKey1.data()), ssKey1.size());
+        leveldb::Slice slKey2(CharCast(ssKey2.data()), ssKey2.size());
+        pdb->CompactRange(&slKey1, &slKey2);
+    }
 };
 
 template<typename CDBTransaction>