[ft] SMT Replenish with > 1000 lastrevealed index

[portlandhodl]

Currently in BDK there is a pretty big problem when loading a wallet where the last revealed index is into the value of 1000+ for even decent hardware because of the need to utilize the CPU to generate SPKs for each index and add them to the graph. The process will only use a single CPU core currently. Large wallets will cause timeouts and/or become a DoS vector.

This will if greater than 1000 last revealed use all available CPU cores to generate the SPKs,

Previous, ~3 minutes for 100000 SPKs

Notice the CPU usage is only a single core at a time.

New, 28 Seconds for 100000 SPKs

All cores are being used!

This speed up is linear to the number of CPU cores.

[notmandatory]

Concept ACK, but need to handle cfg when std feature not enabled.

[evanlinjin]

Thanks for working on this! My main concern is the introduction of an internal Mutex in a library such as BDK. Mutexes should only belong in the application level and is rather problematic in low-level data structures/types. ChatGPT has some nice arguments as to why

We discussed this in the team meetings and the rough consensus for a solution is to cache spks in the changesets. We can structure these changesets in a way so that you can introduce spks for a keychain well in the future.

For example:

pub struct ChangeSet {
    /// Contains for each descriptor_id the last revealed index of derivation
    pub last_revealed: BTreeMap<DescriptorId, u32>,

    /// Cache of script pubkeys.
    ///
    /// We can choose to populate this or leave it empty. The caller can choose to populate this in
    /// any manner they intend; multi-threaded, downloaded asynchronously, etc.
    pub spk_cache: BTreeMap<(DescriptorId, u32), ScriptBuf>,
}

I'm assuming we need an associated cache either in SpkTxOutIndex or KeychainTxOutIndex. The example code above needs some more thought as to where and how we should structure everything. I.e. should we introduce a changeset to SpkTxOutIndex?

---

To understand your requirements better, is there a special way you are loading the wallet? Or is it loaded from a changeset (the typical way)?

[notmandatory]

I suspect there's a way to do this without need of the Mutex by collecting the yielded results of each thread at the end. And still the multi-threaded approach would only be done if the std feature is enabled.

I also support the caching approach, but I don't see why we can't do both with this mult-ithreaded solution going in first since it's a smaller change. Caching will still be useful for both std and no-std users.

[portlandhodl]

I suspect there's a way to do this without need of the Mutex by collecting the yielded results of each thread at the end. And still the multi-threaded approach would only be done if the std feature is enabled.

I also support the caching approach, but I don't see why we can't do both with this mult-ithreaded solution going in first since it's a smaller change. Caching will still be useful for both std and no-std users.

I currently have a version of this but the number of libraries that are modified and touched is high at 2 or 3, between bdk-wallet, bdk, and bdk-sqlx. Getting SMT working seems to be an overall much lighter solution that would also speed up the initial generation of the cache.

[evanlinjin]

@notmandatory Okay I see what you are saying. To avoid having more complex changes, what do you think about creating a new constructor method that takes in the following:

pub fn with_cache(lookahead: u32, cache: impl IntoIterator<Item = KeychainIndexed<K, ScriptBuf>>) -> Self { todo!() }

This way, we don't need to be opinionated about how we construct the cache and can leave it up to the caller. To implement this cleaner, I suggest creating a CachedSpkIterator that wraps a SpkIterator with a cache.

---

However, I do wonder if it really is that much more difficult to cache spks in the changeset as, it seems like that more complete solution since you don't need to re-derive the spks.

[LLFourn]

@portlandhodl how many public keys are there and how long are the derivation paths for the descriptor in your benchmark. It must be quite a big one to actually hit those numbers. On my computer doing 100,000 derivations for a bip86 style derivation takes around 5 seconds!

[portlandhodl]

@portlandhodl how many public keys are there and how long are the derivation paths for the descriptor in your benchmark. It must be quite a big one to actually hit those numbers. On my computer doing 100,000 derivations for a bip86 style derivation takes around 5 seconds!

Currently using bip48 descriptors (depth = 6) along with miniscript with multiple spend conditions.

[portlandhodl]

Want me to close in favor of #1963 ?

[portlandhodl]

Closed as superseded by #1963