perf: speed up dataset-suggestion minimizer search

[ivan-aksamentov]

• Related to: feat(sort): per-dataset minimizer cutoff exponent #1772

The sort dataset-suggestion step hashes each query into minimizers and looks them up against a shared index. Raising a dataset's minimizerIndex.cutoff widens the global params.cutoff (the client hashes each query once, before it knows which dataset matches, so it must hash at the widest cutoff any dataset uses). The wider cutoff admits several times more query k-mers and slows this step for every query and every dataset, not only the one that raised its cutoff. Where that added cost came from was unverified: extra BTreeMap::get lookups, a larger key span in the map, or something else.

This fixes two hot-path inefficiencies in run_minimizer_search, together cutting its time by 18% at the default cutoff and 22% at the widest, with byte-identical sort output at both cutoffs.

What was fixed

• Deduplication. get_ref_search_minimizers deduplicated the surviving query minimizers with Itertools::unique, which fills a hash set that grows with the wider cutoff. It now sorts the small per-query list and drops adjacent duplicates instead [src]. The consumer needs only the distinct set, so iteration order is irrelevant.
• Hit counting. For every matched query minimizer, the old code scanned all references and tested membership with a linear Vec::contains, costing O(references * entries) per hit. Each map entry already lists exactly the references carrying that minimizer, so it now increments those directly, O(entries) per hit [src].

Both are behavior-preserving: the counted hits and the distinct minimizer set are identical, so sort output is unchanged (verified byte-identical at both cutoffs).

How it was measured

Single-threaded sort on one production index (103 references, 44508 entries, k=17), varying only params.cutoff (1<<28 vs 1<<31) so the stored index is identical between runs. Two independent methods agree:

1. A differential benchmark: criterion on run_minimizer_search [src], isolated from CLI startup and IO, splitting each query into the hashing/dedup stage and the map-lookup/hit-count stage. cargo bench --bench bench_minimizer_search reproduces the split and a BTreeMap-vs-HashMap lookup comparison.
2. A sampling profiler: samply/perf on a release build with debug symbols, before and after the fix, for symbol-level confirmation.

Findings

Full run_minimizer_search time (criterion, mixed query set, ms):

Build	cutoff 1<<28	cutoff 1<<31	wider-cutoff penalty
before	446	536	+89 (+20%)
after	365 (-18%)	419 (-22%)	+54 (+15%)

End-to-end CLI wall-clock (includes startup and index parse): default cutoff 0.84 s -> 0.75 s, widest cutoff 1.04 s -> 0.82 s.

The wider-cutoff penalty was not the map lookups. Splitting the pre-fix +89 ms by stage (both cutoffs pre-fix):

Stage	1<<28	1<<31	delta
hashing + dedup	351.8 ms	432.5 ms	+80.7 ms
map lookup + hit count	94.7 ms	103.3 ms	+8.6 ms

90% of the added cost is in get_ref_search_minimizers. The per-k-mer hashing runs for every k-mer regardless of cutoff, so the cutoff-scaling cost is deduplicating the several-times-larger set of surviving minimizers: Itertools::unique stores seen values in a HashSet [doc] keyed with the standard-library default SipHash, and that set grows with the number of surviving minimizers. The profiler confirms it (inclusive share of samples at cutoff 1<<31, run_minimizer_search at ~95%):

Symbol	before	after
Itertools::unique dedup (SipHash HashSet)	14%	0%
...hashbrown resize/rehash	9%	-
...SipHash hashing	6%	-
BTreeMap::get (lookup)	10%	12%
get_hash (per-k-mer, cutoff-independent)	57%	72%

The HashSet spends most of its time in hashbrown reserve/rehash as it grows. BTreeMap::get is a stable 10-12%, secondary and independent of key magnitude (O(log n) in entry count, which barely changes when a single dataset raises its cutoff).

Remaining cost

After the fix, the dominant cost is the intrinsic per-k-mer get_hash (the char-membership "ACGT".contains path), which is cutoff-independent and is the target of the bit-packing in #1649. Because the wider-cutoff penalty was the dedup rather than anything fundamental to hashing more k-mers, wider cutoffs are now cheap, and scoping the client cost per dataset is unnecessary.

Work items

• Deduplicate query minimizers by sorting instead of Itertools::unique
• Count hits by iterating each matched entry's references instead of scanning all references
• Add unit tests for the sorted-unique invariant and multi-reference hit counting
• Add a criterion benchmark attributing search cost across cutoff and lookup structure
• Add a host sampling-profiler script (dev/profile)
• File proposal: perf-minimizer-search-lookup-map.md: profiling findings and hash-map lookup follow-up

Possible improvements

• Replace the search-time BTreeMap lookups with a hash map, measured -44% on the lookup stage (perf-minimizer-search-lookup-map.md)

[github-actions]

Preview: https://nextstrain--nextclade--pr-1773.previews.neherlab.click

(ci)