perf: opt-in WAND/MAXSCORE competitive scoring, parallel segment search, zapx v18 (ScoreMode API)#2383
Open
maneuvertomars wants to merge 47 commits into
Open
perf: opt-in WAND/MAXSCORE competitive scoring, parallel segment search, zapx v18 (ScoreMode API)#2383maneuvertomars wants to merge 47 commits into
maneuvertomars wants to merge 47 commits into
Conversation
… depth Before: container/heap (binary, log₂N comparisons per siftDown) After: inline ternary heap (3 children/node, log₃N comparisons, removed dependency) The top-N collector maintains a min-heap (worst score at root) over the K best results seen so far. Every new candidate triggers a siftDown traversal from the root. A binary heap takes log₂(N) levels per traversal; a ternary heap takes log₃(N): k=10: binary ~3.3 levels → ternary ~2.1 levels (−36%) k=100: binary ~6.6 levels → ternary ~4.2 levels (−37%) k=1000: binary ~10 levels → ternary ~6.3 levels (−37%) Comparing 3 children per step rather than 1 means all three fit in 1-2 cache lines instead of 1 — fewer cache-line fetches per traversal at the cost of one extra comparison per step. Net win: shallower tree beats extra comparison. Implementation: siftUp/siftDown inline methods on collectStoreHeap using child formula 3i+1, 3i+2, 3i+3. Removes the container/heap import entirely. ~67 lines changed; zero interface changes. Measured improvement on M2 Pro (3-term BM25, MAXSCORE path): k=10: 905µs → 892µs (~1.5%) k=100: 922µs → 911µs (~1.2%) k=1000: 1395µs → 1354µs (~3%) Benefit scales with k — meaningful for top-1000, small for top-10. Composes cleanly with §1 WAND: fewer candidates reach the heap, so fewer siftDowns overall; a faster heap multiplies whatever fraction remains. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Before: every candidate document is scored with full BM25 math
After: candidates whose max possible score ≤ heap threshold are skipped
Once the top-K heap is full, its K-th (lowest) score becomes a threshold.
Any document whose best possible BM25 score cannot beat that threshold is
pruned without scoring. "Best possible" = sum of MaxImpact per query term,
where MaxImpact = IDF × maxTFNorm × queryWeight.
MaxImpact values pre-computed once per query into a float64 array:
query init (once per query):
wandMaxImpacts[0] = MaxImpact("the") = 0.001 ─┐
wandMaxImpacts[1] = MaxImpact("hotel") = 0.018 ─┤ pure array, no interface
wandMaxImpacts[2] = MaxImpact("lisbon") = 0.073 ─┘
per-candidate (~2 ns):
sum = wandMaxImpacts[0] + [1] + [2] ← three float64 additions
sum ≤ threshold → skip Score() ← ~36% of candidates pruned at k=1
sum > threshold → Score() + heap.Push?
Without pre-caching (early attempt, overhead > savings):
per-candidate:
for i := range searchers:
wi := searchers[i].(wandImpacter) ← type assertion ~7 ns
sum += wi.MaxImpact() ← interface dispatch ~7 ns
→ cross-topic regression: +8.23% with zero pruning
With pre-caching (this commit, net win):
→ cross-topic overhead: +8.2% → ~0%
→ geomean: +1.88% → −4.26% vs master
maxTFNorm sources:
- zapx: per-segment maxTFNorm cache (invertedCacheEntry.maxTFNormCache)
Lazily scans posting list once per (term, segment, avgDocLen) on first query;
O(1) RLock+map on subsequent queries. Cap at 100k entries/field/segment;
eviction is segment GC when Scorch merges the segment.
- bleve/scorch: IndexSnapshotTermFieldReader.MaxTFNorm() aggregates across all
segments and stores per-segment values in segMaxTFNorms[].
- bleve/searcher: TermSearcher.MaxImpact() = IDF × maxTFNorm × queryWeight;
cached per-TermSearcher; SetQueryNorm() resets flag on norm change.
- bleve/collector: TopNCollector writes ctx.ScoreThreshold once K-heap fills.
Measured (topical 500k-doc corpus, 3 same-topic terms, M2 Pro):
CrossTopic overhead (no pruning): +8.2% → ~0%
TopicalDisj3SameTopic serial: baseline → −3.4% (p=0.041)
TopicalDisj3SameTopicParallel: baseline → −18% (p=0.002)
Candidates scored at k=1: 4,681 → 2,972 (−36.5%)
geomean: +1.88% → −4.26%
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Before (§1 WAND): every doc containing any query term is visited and threshold-checked
After (MAXSCORE): docs matching only low-impact terms are never visited at all
Sort query terms by MaxImpact ascending. The "pivot" is the first term whose
suffix-sum of MaxImpact exceeds the threshold. Terms before the pivot are
"non-essential" — no document matching only those terms can enter the top-K heap.
Essential terms (≥ pivot) drive candidate generation via Next(); non-essentials
only Advance() to each candidate for a potential bonus score.
"best hotel lisbon" at threshold=0.15 (common after heap fills with k=1):
idx term MaxImpact suffix-sum role
─────────────────────────────────────────────────────
0 "best" 0.048 0.191 > 0.15 → essential
1 "hotel" 0.062 0.143 < 0.15 → non-essential
2 "lisbon" 0.081 0.081 < 0.15 → non-essential
┌──────────────────────────────────────────────────────────────────┐
│ non-essential (idx < pivot) │ essential (idx ≥ pivot) │
│ ["hotel", "lisbon"] │ ["best"] │
│ │ │
│ Advance(candidate) only │ Next() drives iteration │
│ if candidate in their list │ candidate = min docID │
└──────────────────────────────────────────────────────────────────┘
docs matching only "hotel" or "lisbon" (but not "best") → NEVER visited
§1 WAND visits them and pays the threshold check; MAXSCORE skips them entirely
Stopword queries ("to be or not to be"): common words become non-essential
almost immediately → MAXSCORE skips millions of docs; WAND does not.
Measured vs WAND-only baseline (500k-doc topical corpus, 3 same-topic terms, M2 Pro):
k=1: 785µs → 597µs (−24%; candidates 2,972 → 2,305)
k=10: 822µs → 847µs (flat — same-weight terms, pivot=0, all essential)
k=100: 892µs → 875µs (−2%; candidates 4,681 → 3,412, −27%)
k=1000: 1405µs → 1376µs (−2%)
Two correctness bugs found and fixed:
1. Heap corruption: ALL iterators at the candidate docID must be advanced
before returning rv. A non-essential iterator left at the old docID will be
Advance()'d on the next call → Pool.Put(rv) → dm.Reset() → zeroes
IndexInternalID.len → corrupts the live heap entry.
2. Escape-analysis alloc: minIDBuf must be a struct field (s.minIDBuf [8]byte),
NOT a local var. A local [8]byte sliced and passed to Advance() (an interface
method) triggers Go escape analysis → 1 heap alloc per candidate (~669k
allocs/267 iterations measured), wiping out all gains.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…shold
Before: every candidate in every segment is individually checked and scored
After: segments whose best possible score ≤ threshold are skipped entirely
§1/§8 prune individual candidates. §15 prunes at segment granularity: before
iterating a segment at all, compute an upper-bound score as the sum of per-term
MaxImpacts for that specific segment. If no document in the segment can beat the
threshold, advance past the entire segment with a single Advance() call.
After top-K heap fills (threshold > 0, 15 segments, 3-term query):
segments: [seg 0] [seg 1] [seg 2] [seg 3] ... [seg13] [seg14]
segCeiling: 1.42 0.83 1.21 1.38 ... 0.71 0.47
↑ ↑ ↑
below threshold below below
threshold = 0.91 (10th-best score in heap)
seg 0: 1.42 > 0.91 → search (a doc here could enter top-10)
seg 1: 0.83 < 0.91 → SKIP (impossible for any doc to enter top-10)
seg 2: 1.21 > 0.91 → search
seg 3: 1.38 > 0.91 → search
...
seg13: 0.71 < 0.91 → SKIP
seg14: 0.47 < 0.91 → SKIP
Skipping a segment: Advance() all essential iterators to FirstDocIDOfSegment
of the next eligible segment — zero docID iteration, zero FST lookup.
When it fires: heterogeneous indexes where older/smaller segments have lower
average TF (e.g. early segments from a crawl before a domain became relevant).
Current bench corpus: §12 BP merges 15 segments → 1 large segment, so
segCeiling = global ceiling and the skip condition can never be satisfied.
§15 is wired and correct; neutral on the bench corpus, valuable in production
on multi-segment indexes with quality variance across segments.
New interfaces / methods:
- perSegmentTFR on IndexSnapshotTermFieldReader: NumSegments,
MaxTFNormForSegment, SegmentIndexOf, FirstDocIDOfSegment
- segmentSkipper on TermSearcher: thin wrappers scaling maxTFNorm by
IDF × queryWeight into per-segment max impact
- DisjunctionSliceSearcher: segSkippers, segCeilings, segSkipBuf fields;
ceiling matrix built in initWANDMaxImpacts(), skip check in nextMAXSCORE
On the bench corpus (homogeneous, 15 segments), the check adds ~20µs per
query (within noise) and never fires.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ts cost initWANDMaxImpacts() (§15) calls MaxTFNorm() for a full segment scan (N segments) and then MaxImpactForSegment() for each segment separately: 2×N invIndexCache lookups per term × K terms = 2×15×6 = 180 lookups for a 6-term/15-segment entity query, each costing ~61ns (two mutex+map ops). Fix: IndexSnapshotTermFieldReader.MaxTFNorm() now stores per-segment values in segMaxTFNorms[]. MaxTFNormForSegment() checks this cache first, falling back to the dict lookup only when the cache is absent or stale. This eliminates 90 redundant invIndexCache lookups per query (~4µs). Result: EntityMedium6FieldDisj: 69µs → 62µs (-11%), resolving the +7.25% v18 regression and making the benchmark slightly faster than v17 baseline. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
§15 guard (search_disjunction_slice.go):
- Add minSegCeiling float64 field = min(segCeilings), computed once in
initWANDMaxImpacts alongside the segCeilings array.
- nextMAXSCORE now guards the §15 per-segment ceiling check with
ctx.ScoreThreshold >= s.minSegCeiling rather than > 0. §15 can only
skip a segment when the threshold reaches at least the lowest segment
ceiling; below that value the SegmentIndexOf(sort.Search) call is
unnecessary and is skipped entirely.
- On the current bench corpus all segment ceilings exceed realistic
thresholds, so SegmentIndexOf was called on every candidate without
ever firing — ~0.43s + 0.38s = 0.81s of overhead (2.3% of CPU).
DocumentMatch.Reset (search.go):
- Replace *dm = DocumentMatch{} (a ~240-byte duffzero) with explicit
writes to only the 9 fields not otherwise saved+restored: Index, ID,
Score, Expl, Locations, Fragments, Fields, HitNumber, IndexNames.
- In the common MAXSCORE lazy path these fields are already nil/zero
so the stores are cheap; the full duffzero was paid unconditionally.
- Eliminates the runtime.duffzero callee (~0.51s) and reduces total
Reset overhead (6.52% flat + 1.44% duffzero before this change).
Bench results (5 runs, TopicalDisjunction3TopK, Apple M2 Pro):
k=1: ~511 µs (flat, ±0.3%)
k=10: ~727 µs (flat, ±0.3%)
k=100: ~702 µs (was ~767 µs, -8.4%)
k=1000: ~1154 µs (was ~1231 µs, -6.3%)
geomean improvement: ~3.8%
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Before: every candidate that passes the MAXSCORE essential scan is BM25-scored
After: BM25 Score() is called only for candidates whose MaxImpact sum > threshold
§8 MAXSCORE skips docs matching only non-essential terms. Among surviving
candidates, some have a MaxImpact sum still ≤ threshold — they cannot displace
the K-th heap entry either. §9 skips the Score() call for those.
Before §9 (every surviving candidate scored):
candidate doc
→ decode(freq, norm)
→ Score(freq, norm) ← ~5 float64 ops + 1 divide (always paid)
→ heap.Push if score > threshold?
After §9 (lazy: skip Score() when upper bound is too low):
candidate doc
→ nextDocIDOnly() ← cheap: pool.Get + ID copy only
→ sum(wandMaxImpacts) > threshold?
│ NO (~22–27% of cands) │ YES
▼ ▼
PutLazy (no score computed) scoreCurrentDoc() ← Score() paid only here
→ heap.Push?
§8 skips entire categories of docs; §9 eliminates scoring work for surviving
candidates whose upper bound is still too low to displace the K-th result.
New methods:
- TermQueryScorer.ScoreInto: scores into an existing DocumentMatch (preserving
term vectors for highlighting, unlike the original Score which allocates a
fresh match).
- TermSearcher.{nextDocIDOnly, advanceDocIDOnly}: fetch docID without BM25.
- TermSearcher.scoreCurrentDoc: compute BM25 for the current position.
- lazyTermSearcher interface: implemented by *TermSearcher; DSS checks all
sub-searchers implement it before enabling the lazy path.
lazySearchers slice is pre-allocated alongside other per-query slices in
initWANDMaxImpacts to eliminate the make() call on the hot path.
Measured vs §8 MAXSCORE baseline (5-run average, M2 Pro):
k=1: −7.0%
k=10: −7.8%
k=100: −7.0%
k=1000: −3.8%
geomean −6.4%
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…n nextMAXSCORE hot loop Scorch IndexInternalID is always an 8-byte big-endian uint64. Using bytes.Compare (via IndexInternalID.Compare) on every docID comparison in the MAXSCORE hot loop accounts for ~13% of total CPU time (cmpbody shows up in profiles). Decode each ID once with binary.BigEndian.Uint64 and use native uint64 <, ==, > throughout the essential scan, non-essential advance, matching collection, and advancement loop. The decoded minIDVal is computed once per iteration; all four comparisons below use the same integer. Also copies the minID bytes into s.minIDBuf (an [8]byte struct field that avoids heap escape) only when a new minimum is found, not on every loop iteration. A pre-existing pool aliasing issue can leave a curr.IndexInternalID at len=0 (Reset by DocumentMatchPool.Put) while the pointer is still in s.currs. The original bytes.Compare silently treated this as "before all docs" and produced an empty minID that terminated the search; the uint64 decode would panic with an index-out-of-range. All four comparison sites now guard len == 8 and skip the corrupted entry, matching the original termination semantics while continuing with any remaining essentials. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ction loop wandAboveThreshold has inliner cost 109 (budget 80) so it is never inlined into nextMAXSCORE. The function does a second range-over-matchingIdxs to sum per-term MaxImpact values, right after the collection loop that just built matchingIdxs. This commit: - Hoists wandImpacts (= s.wandMaxImpacts) and threshold (= ctx.ScoreThreshold) to locals before the outer loop. Both are constant within a single nextMAXSCORE call (threshold only changes after we return a result). - Accumulates upperBound += wandImpacts[i] inside the existing matching- collection loop (zero additional iterations). - Replaces the s.wandAboveThreshold(ctx) call with a direct "upperBound > threshold" comparison. threshold > 0 and len(wandImpacts) > 0 are invariants of the MAXSCORE path (the caller in Next() only dispatches here after those checks pass), so the guard clauses inside wandAboveThreshold are redundant here. Bench results relative to previous commit (§15 guard + Reset zeroing): k=1: 479 µs (was 511 µs, -6.3%) k=10: 671 µs (was 727 µs, -7.7%) k=100: 681 µs (was 702 µs, -3.0%) k=1000: 1131 µs (was 1154 µs, -2.0%) Combined with previous commit, total improvement from this session relative to the uint64-comparison baseline: k=1: -6.6%, k=10: -7.5%, k=100: -11.2%, k=1000: -8.1% geomean: ~-8.4% Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… fast path DocumentMatch.Reset (search.go): - Add nil guard before clear(scoreBreakdown). In the benchmark's MAXSCORE path, ScoreBreakdown is never populated (no KNN, no retrieveScoreBreakdown), so scoreBreakdown is nil on every Reset call. With Go 1.25 Swiss maps, clear(nil) still dispatches through internal/runtime/maps.(*Map).Clear (~1.2 ns/call) — removing the call saves ~0.80s at ~670M calls/bench. MergeFieldTermLocations (search/util.go): - Add fast-path return after the n-computation loop: if n == len(dest), no constituent contributed any FieldTermLocations, so skip the second iteration and the mergeFieldTermLocationFromMatch function calls entirely. In the benchmark (no location tracking), this fires on every call to DisjunctionQueryScorer.Score (~168M calls at 3 terms × 2305 candidates × 73k bench iterations). Combined incremental vs. prior commit (wandAboveThreshold inlining): k=1: flat (mapclear/merge scale with scored candidates, same count) k=10: -0.6% k=100: -5.2% k=1000: -2.1% Cumulative from session baseline (uint64-comparison commit 9162b7f): k=1: -6.9%, k=10: -8.5%, k=100: -15.8%, k=1000: -11.2% geomean: ~-10.7% Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ORE path
In the lazy BM25 path (§9), DocumentMatches that flow through the inner
MAXSCORE loop have at most two fields set:
• IndexInternalID — set by nextDocIDOnly / advanceDocIDOnly
• Score — set by TermQueryScorer.ScoreInto (only when candidate
passes the WAND threshold)
The full DocumentMatch.Reset saves/restores 6 slice headers, guards a
nil-map clear, and zeroes 9 additional fields — none of which are set in
this path. Each Reset takes ~2 ns; with ~660M inner-loop Put calls per
bench run, that is ~1.3s of avoidable work.
This commit adds DocumentMatchPool.PutLazy which zeros only
IndexInternalID and Score (~0.3 ns), and uses it in nextMAXSCORE for:
1. Non-essential advance loop: docs that were never scored (IDonly)
2. Bottom advancement loop (non-rv matched docs): docs that may have
Score set by scoreCurrentDoc but nothing else
3. §15 segment-skip loop: same as (1)
The collector's k-heap eviction path continues to use the full Put/Reset
since those docs may have HitNumber set by the collector.
Bench results incremental vs. prior commit (mapclear nil + MFT fast path):
k=1: -9.6% (477 → 432 µs)
k=10: -8.9% (664 → 605 µs)
k=100: -3.5% (645 → 622 µs)
k=1000: ~flat (1093 → 1097 µs, within noise)
Cumulative from uint64-comparison baseline (9162b7f):
k=1: -15.8%, k=10: -16.6%, k=100: -18.9%, k=1000: -10.8%
geomean: ~-15.6%
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
SegmentIndexOf (sort.Search over 15 offsets) was called for every candidate document whenever ScoreThreshold >= minSegCeiling. With ~2305 candidates per query and 15 segments of ~33k docs each, this was 2305 binary searches per query for a check that almost never fires (the segment ceiling usually exceeds the threshold). Cache the last segment lookup (segIdx + start/end bounds). Only call SegmentIndexOf when minIDVal crosses a segment boundary — ~15 times per query instead of ~2305. The cache auto-invalidates when the iteration moves to a new segment (boundary check) and is reset in initWANDMaxImpacts/SetQueryNorm. CPU profile improvement: §15 drops from 2.30s cum (on 58s run) to near zero. Overall: k=1 −3%, k=10 −3%, k=1000 −2%. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ScoreFast (scorer_disjunction.go): Extract explain path to scoreExplain(). Add ScoreFast, a lightweight variant that skips MergeFieldTermLocations and the explain branch; inlinable at cost 37. In the MAXSCORE lazy path, scoreCurrentDoc only sets Score — FieldTermLocations and Expl are never written, so ScoreFast is always correct there. Use it in nextMAXSCORE (lazy && !scoreBreakdown). Concrete *TermSearcher dispatch (search_disjunction_slice.go): Change lazySearchers from []lazyTermSearcher to []*TermSearcher. TermSearcher is the only implementor of lazyTermSearcher; the interface was causing vtable dispatch overhead on every nextDocIDOnly (~288M calls per 354k bench iterations), advanceDocIDOnly, and scoreCurrentDoc call. scoreCurrentDoc (cost 64 < 80) is now inlinable at the call site. Update initWANDMaxImpacts to assert *TermSearcher instead of the interface. PutUint64 for minIDBuf (search_disjunction_slice.go): Replace copy(s.minIDBuf[:], curr.IndexInternalID) inside the essential iterator scan loop with a single binary.BigEndian.PutUint64 after the loop. Avoids re-loading source bytes and writes minIDBuf exactly once per outer iteration instead of once per improved minimum. MergeFieldTermLocations slow path extracted (util.go): Move the grow/merge logic into mergeFieldTermLocationsGrow so the fast path (n == len(dest)) is cheaper to call. Cumulative benchmark improvement from session start (TFD-Reset baseline): k=1: 444µs → 411µs (−7%), k=10: 621µs → 597µs (−4%), k=100: 637µs → 636µs (flat), k=1000: 1132µs → 1114µs (−2%) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two hot-path micro-optimizations in nextMAXSCORE: 1. In the `lazy := ...` hoisted check, compare against s.numSearchers (cache line 1, warm — shares a line with s.currs) instead of len(s.searchers) (cache line 0, cold during the inner loop). Saves ~1 L2 miss per nextMAXSCORE call. 2. ScoreFast: when countMatch == countTotal (coord factor = 1.0), assign rv.Score = sum directly without the multiply+divide. For topical queries where all terms co-occur this is the common path, saving 2 float64 operations per scored candidate. ScoreFast cost stays 45 (budget 80), so it remains inlinable. Benchmark (120s, M2 Pro): k=1: 405µs (-1.2%) k=10: 575µs (-3.6%) k=100: 625µs (-1.8%) — first improvement in k=100 this session k=1000: 1104µs (-0.9%) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nextMAXSCORE's inner loop computed `lazy := len(s.lazySearchers) == s.numSearchers` once per call. len(s.lazySearchers) loads the slice header from offset 360 — cache line 5, cold in the inner loop — even though the value is constant for the whole query. Precompute the result into a new `lazyMode bool` field set in initWANDMaxImpacts(). The bool slots into the existing 7-byte padding gap between retrieveScoreBreakdown and currs, so it lands on cache line 1 (the hot line, alongside numSearchers/currs) and the struct size is unchanged at 384 bytes (6 × 64-byte cache lines). Add struct_size_test.go to guard the 384-byte layout: any field addition that spills into a 7th cache line has measured ~9% k=1000 regression, so the test fails loudly if the size grows. Benchmark (large-bench topical, count=3) vs the s.numSearchers baseline — neutral within noise, k=1000 slightly better: k=0001 405,899 -> 410,405 ns/op (+1.1%) k=0010 575,495 -> 580,436 ns/op (+0.9%) k=0100 624,780 -> 620,257 ns/op (-0.7%) k=1000 1,103,827 -> 1,094,146 ns/op (-0.9%) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two changes to OptimizeTFRDisjunctionUnadorned.Finish():
1. Remove the dead first loop that computed cMax per segment but never
used the result. This was a full O(segments × TFRs) scan for nothing.
2. Add fast paths after collecting actualBMs/docNums for each segment:
- Empty segment (no bitmaps, no 1-hit docs): reuse the zero-alloc
anEmptyPostingsIterator singleton instead of allocating roaring.New()
+ unadornedPostingsIteratorBitmap.
- Single 1-hit doc with no bitmaps: use newUnadornedPostingsIteratorFrom1Hit,
saving the roaring.Bitmap alloc.
Also implement segment.OptimizablePostingsIterator on emptyPostingsIterator
(ActualBitmap→nil, DocNum1Hit→false, ReplaceActual→noop) so that the empty
sentinel composes correctly in nested disjunction optimizations — a second
disjunction:unadorned Finish() treats it as contributing nothing to the OR,
which is correct, rather than aborting the optimization with ok=false.
Impact on entity search corpus (15 segs, 500k docs, 6 keyword fields):
Miss6FieldDisjScoreNone: 273 allocs/op (-14%, was 318)
MissConjScoreNone: 532 allocs/op (-20%, was 667)
The conjunction cascade benefits because emptyPostingsIterator now triggers
the *emptyPostingsIterator short-circuit in conjunction:unadorned Finish(),
saving the AND-of-empty-bitmaps allocation per segment as well.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…in Advance() When Advance() needs to seek backwards it restarts from the beginning by calling TermFieldReader() for a fresh reader (i2), then replacing the current reader's state with i2's. The previous code called i.Close() to do this, which with a non-zero fieldTFRCacheThreshold donates i to the recycle pool while the caller's pointer to i is still live. Another goroutine can immediately retrieve i from the pool via allocTermFieldReaderDicts and begin writing to its fields; the Advance() path then executes *i = *i2, stomping over those writes with no synchronisation. The result is two goroutines sharing the same *IndexSnapshotTermFieldReader — a data race that manifests as nil-pointer dereferences, divide-by-zero, and index-out-of-range panics inside the posting list and chunk decoder layers (MB-64604). Fix: do not call i.Close() here. Instead, replicate the non-recycle parts of Close() inline (IO stats reporting, TotTermSearchersFinished accounting), then overwrite i in-place from i2. i2 becomes an orphan; its recycle flag is cleared so it cannot be re-added to the pool if Close() is ever called on it. The caller's pointer to i remains valid throughout — i is never donated to the pool while in use. Added TestAdvanceBackwardSeekNoRaceWithRecycling: 8 goroutines, each calling Next()+Advance(sameID) 200× with fieldTFRCacheThreshold=100. Confirmed the test reports DATA RACE on every run with the old code and passes cleanly with -race on the fixed code. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The race that forced threshold=0 (MB-64604) is fixed in the previous commit. Set the default to 4: small enough to bound pool memory, large enough to cover the hot case on a multi-core server running concurrent queries on the same field. Measured benefit: ~27% fewer allocs per warm query on a 15-segment index (skips FST.Reader + Dictionary + PostingsList + PostingsIterator allocation per recycled (term, segment) pair). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Before: DisjunctionSliceSearcher searches all segments in a single goroutine
After: min(GOMAXPROCS, 8) goroutines search disjoint segment ranges in parallel
Each shard needs its own TermFieldReaders (iterators are not goroutine-safe).
Naively creating a full TFR per shard costs ~80× allocs (FST Dict + PostingsList
+ iterator setup per term per segment). ShardView borrows read-only sub-slices
(dicts + postings arrays) from the parent recycled TFR and creates only fresh
iterators — ~5× alloc overhead vs ~80×.
Serial (before):
goroutine 1: [seg 0..14] → candidates → top-K heap
Parallel with ShardView (after):
goroutine 1: [seg 0.. 4] → local heap ─┐
goroutine 2: [seg 5.. 9] → local heap ─┤ → merge → global top-K
goroutine 3: [seg 10..14] → local heap ─┘
Shared atomic threshold: sharedThreshold broadcasts the tightest heap
minimum across all goroutines so WAND prunes aggressively once any shard's
K-heap fills, without a per-candidate mutex.
Scorer sharing: shard TermSearchers share the parent TermQueryScorer
(IDF/queryWeight are read-only after query init), keeping allocs low.
Bug fixed: ShardView sliced i.postings[startSeg:endSeg] unconditionally.
Unadorned TFRs (produced by OptimizeTFRConjunction/DisjunctionUnadorned)
carry nil postings; slicing a nil slice with endSeg > 0 panics. Guard with
len > 0; for nil-postings TFRs use freshIteratorForShard on each parent
per-segment iterator. Also carry i.unadorned into the ShardView copy.
Results on 15-segment 500k-doc index (12-core M2 Pro, K=10):
BestHotelLisbon (3-term): 539µs → 467µs (−13.4%)
8TermsWAND (8-term): 659µs → 603µs (−8.5%)
Concurrent QPS (parallel): 95µs/op → unchanged (no regression)
Trade-off: §7 is opt-in (default false). Goroutine over-subscription degrades
QPS for concurrent workloads; use only for serial/latency-sensitive requests.
On large indexes (100+ segments), expected speedup is 3–8×.
New globals (opt-in, default off):
EnableParallelSegmentSearch bool (default false)
ParallelSegmentSearchMinSegs int (default 6)
ParallelSegmentSearchShardK int (default 100; set = query.Count for WAND)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… path
TestParallelSegmentSearchUnadornedConjunction is the regression test for the
nil-postings ShardView panic:
1. NewConjunctionSearcher with Score="none" fires conjunction:unadorned,
producing a TermSearcher whose TFR has nil postings.
2. disjunction:unadorned is disabled so NewDisjunctionSearcher creates a
DisjunctionSliceSearcher containing that TermSearcher.
3. On the first Next() with EnableParallelSegmentSearch=true,
runParallelSegmentSearch calls ForSegmentRange → ShardView on the
nil-postings TFR. Before the snapshot_index_tfr.go fix this panicked.
TestParallelSegmentSearchCorrectness covers three Score="none" disjunction
cases (simple OR, (conj AND) OR term, two-term OR) in parallel mode and
verifies the document sets against known expected values.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add search.ParallelSegmentSearchKey (ContextKey) so callers can override the global EnableParallelSegmentSearch and ParallelSegmentSearchShardK on a per-request basis by placing an int into the context before calling SearchInContext: 0 disables, ≥2 enables with that shardK, absent means the global flags apply unchanged. shouldRunParallel now returns (bool, int) — the resolved shardK — so runParallelSegmentSearch and runShardSearch no longer read the global directly, removing the implicit coupling to a package-level variable. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Before: per-doc BM25 = ~5 float64 ops + 1 divide per scored document
After: per-doc BM25 = 1 table lookup + 1 float64 multiply
Pre-compute tfNorm(freq, normByte) as a [64][256]float32 table at index open
(once per avgDocLen, ~82µs, 64 KB). During scoring, replace the per-posting
BM25 calculation with a single table lookup and one multiply.
impactTable [64][256]float32 (64 KB, built once at index open):
normByte (SmallFloat-encoded field length, from §20 norm column):
0 16 32 64 128 255
freq = 1 [0.0][...][...][...][...][...] ← tfNorm(√1, SmallFloat(nb))
freq = 2 [0.0][...][...][...][...][...]
freq = 4 [0.0][...][...][...][...][...]
freq = 8 [0.0][...][...][...][...][...]
...
freq = 63 [0.0][...][...][...][...][...]
normByte=0 → pre-v18 segment (no norm column) → full BM25 fallback
Per scored document:
BEFORE: tfNorm = √freq × k1 / (√freq + k1×(1−b + b×fieldLen/avgDocLen))
score = tfNorm × idf × queryWeight (~5 float64 ops + 1 divide)
AFTER: score = float64(impactTable[freq][normByte]) × idfQueryWeight
(1 lookup + 1 mul)
The normByte is read lazily via PostingsIterator.NormColumnByte(docNum), called
only in postingToTermFieldDoc (for documents that are actually scored). Reading
it in Next() (every traversed posting) caused a +5% regression on WAND-heavy
8-term queries because WAND traverses ~30× more postings than it scores.
Fast path active when: BM25 scoring, Explain=false, normByte ≠ 0, freq < 64.
Requires §20 norm column (zapx v18 segments only).
Data flow:
normColumn[docNum] (zapx §20)
→ PostingsIterator.NormColumnByte(docNum) [type-asserted lazily]
→ TermFieldDoc.NormByte
→ TermQueryScorer fast path: impactTable[freq][NormByte] × idfQueryWeight
Measured (M2 Pro, compared to zapx SkipN baseline):
TermTier1–TermTier5 (scoring-dominated): ~−20% each
QueryBestHotelLisbon (WAND-dominated): −1.7%
TopicalDisjunction8SameTopic: −4.5% (v2, lazy normByte)
geomean (37 benchmarks): −3.44%
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Switch from Posting.NormByte() (type-asserted on the Posting) to PostingsIterator.NormColumnByte(docNum) (type-asserted on the iterator). This matches the new lazy-access pattern in zapx where normColumn is only read when a document is actually being scored. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Before: new segments written in zapx v17 (norm interleaved in every posting)
After: new segments written in zapx v18 (norm extracted to a separate column)
§20 (zapx side, merged separately) moves per-document field-length norms out
of the interleaved freq/norm posting stream into a flat per-field byte column.
This commit wires zapx/v18 as the write plugin in bleve/scorch.
v17: norm stored once per (term, doc) pair — i.e., once per posting:
term "hotel" posting stream:
doc 42: [freq=3][normBits=0x3F800000] ← 2 uvarints, ~6 bytes
doc 187: [freq=5][normBits=0x3E4CCCCD]
term "lisbon" posting stream:
doc 42: [freq=1][normBits=0x3F800000] ← identical normBits stored again
500k docs × 50 avg terms × ~4.5 B/norm ≈ 112 MB of redundant norm data
v18: norm column, stored once per document per field:
normColumn["body"]: [nB₀][nB₁] ... [nB₄₉₉₉₉₉] ← 500 KB flat array
term "hotel" posting stream (v18):
doc 42: [freq=3] ← freq only; norm removed from stream
at score time: normByte = normColumn[docNum] ← O(1) byte load
freq stream shrinks ~50%; norm storage: 112 MB → 500 KB.
zapx v17 and older (v11–v16) remain registered as read-only legacy plugins,
so existing on-disk segments are still opened and merged transparently.
The zapx module path was promoted from v17 → v18 in zapx/go.mod to reflect
the new on-disk format. zapx/v18 has no published release yet; the go.work
workspace resolves it from ./zapx during local development.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
zapx v18 changes the bytes-read accounting in two ways: 1. First-query byte counts increase: each field gains a 10-byte section entry in loadFields + a 20-byte normColumnHeaderSize read in loadDvReaders 2. Subsequent-query byte counts decrease: freq stream no longer carries the normBits uvarint per posting (~14 bytes saved on 'united' query) Also updates the disjunction query expected value (120→95) and the numeric range query (924→920) for the same freq-stream shrinkage. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…al a lower bound
Before: SearchResult.Total always claimed to be the exact match count, but
WAND/MaxScore pruning silently under-counted by skipping candidates
whose MaxImpact sum ≤ ScoreThreshold before they ever reached the collector
After: SearchResult.TotalRelation is "eq" (exact) or "gte" (lower bound)
When the top-K heap fills and ScoreThreshold rises, DisjunctionSliceSearcher
starts skipping candidates in three places — none of which were counted in
hc.total (which only increments in prepareDocumentMatch, after searcher.Next
returns a live result):
nextBasic: !wandAboveThreshold → discard without scoring
Next(): pivotIdx == len(maxscoreOrder) → return nil, stop iteration
nextMAXSCORE: len(matching)>=min && upperBound≤threshold → discard
All three sites now set ctx.WANDPruned=true. After the collection loop,
TopNCollector.Collect captures hc.wandPruned=searchContext.WANDPruned.
New constants:
TotalRelationEq = "eq" // Total is the exact match count
TotalRelationGte = "gte" // Total is a lower bound (WAND pruned some docs)
SearchResult.TotalRelation is set in index_impl.go and propagated through
SearchResult.Merge (if either merged shard has TotalRelationGte, the merged
result inherits it).
JSON key: "total_relation" — omitempty not used so the field is always present,
making it unambiguous whether the caller is reading a new-format result.
Follows Lucene's TotalHits{value, relation} pattern (LUCENE-8060, CJ 2018-08-07).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ing)
Adds ScoreMode string to SearchRequest (JSON: "score_mode"), following
Lucene's ScoreMode enum. Default ("" or "complete") is backwards-
compatible: all candidates visited, exact BM25 scores, exact Total.
ScoreMode = "top_scores" engages competitive scoring:
req.ScoreMode = bleve.ScoreModeTopScores
Result-affecting optimizations gated behind "top_scores":
§1/§8 WAND/MAXSCORE pruning: candidates skipped → Total is a lower
bound; SearchResult.TotalRelation set to "gte" when pruning fires.
§25 impact-table scoring: SmallFloat/normByte rounding → hit scores
may differ slightly from full float64 BM25.
Clean optimizations always active (same results, ScoreMode has no effect):
§13 ternary heap, §22 micro-opts, §9 lazy BM25, §1 MaxImpact caching.
Score="none" interaction: ScoreThreshold stays 0 when scoring is off,
so the WAND gate (WANDEnabled && ScoreThreshold > 0) never passes —
"top_scores" is a harmless no-op in that case.
Thread: req.ScoreMode → coll.SetWANDEnabled() → SearchContext.WANDEnabled
→ three pruning sites in DisjunctionSliceSearcher (Next early-exit,
nextBasic, nextMAXSCORE). TotalRelation propagates via SearchResult.Merge
for multi-shard aliases.
Constants: bleve.ScoreModeComplete, bleve.ScoreModeTopScores,
bleve.TotalRelationEq, bleve.TotalRelationGte.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The root cause of §7's regression against WAND queries: runShardSearch set WANDEnabled=false in its SearchContext, so every shard fell through to the basic posting-list scan. For BestHotelLisbon this scored ~6,250 candidates per shard (full posting-list union / 8 shards) vs serial WAND's 2,462 total. Fix: two changes working together: 1. Pre-compute global per-term MaxImpact from the original full-index TermSearchers before sharding (runParallelSegmentSearch). Shard TFRs cover only 2/15 segments so their MaxImpact() is lower, making the MAXSCORE essential/non-essential partition ineffective against a cross-shard threshold broadcast by the highest-scoring shard. Global ceilings are a correct upper bound on any shard doc's score. 2. Inject global ceilings into each shard DSS via injectGlobalWANDCeilings() (new DSS method, mirrors initWANDMaxImpacts but skips per-shard MaxImpact calls). Set WANDEnabled=true in each shard SearchContext. The sharedThreshold already syncs the score threshold across goroutines: as any shard's top-K heap fills it broadcasts its K-th-best score. Other shards immediately see the tighter threshold on their next Next() call and the MAXSCORE pivot fires aggressively using the global ceilings. Result (BENCH_PARALLEL_SEARCH=10, serial path): DisjunctionHighDF: 13.9ms → 8.6ms (−38%) BestHotelLisbon: 633µs → 600µs (goroutine overhead ≈ query work) All tests pass including TestParallelSegmentSearchCorrectness. Note: BENCH_PARALLEL_SEARCH explicit override bypasses §33 guards so the *Parallel QPS benchmarks regress under oversubscription (expected). A concurrency gate (§33 design) is needed for production auto-mode. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Brings back the two §33 adaptive guards (without the WAND guard, which §34 obsoletes) and the parallelDecided O(N) bug fix: 1. parallelDecided bool (packed in 7-byte padding gap after lazyMode, offset 82 — struct stays 456 bytes): ensures shouldRunParallel is called at most once per DSS. Without it, shouldRunParallel fires O(NumCandidates) times when §7 is disabled (parallelResults stays nil), adding ~369µs overhead per query. 2. DF-based shard guard: skip §7 when totalDF/numSegs < MinDFPerSeg (=150). Prevents goroutine overhead from dominating on low-DF entity queries. 3. Concurrency gate: skip §7 when parallelSearchesActive >= GOMAXPROCS/p. Prevents goroutine oversubscription at high QPS (e.g. 12 bench goroutines × 8 shard goroutines = 96 goroutines on 12 cores). The WAND guard from the original §33 is intentionally omitted: §34 global MaxImpact ceilings make §7 compatible with top_scores mode (shards now use global ceilings from the full-index TermSearchers rather than per-shard lower bounds, enabling effective MAXSCORE pruning alongside parallelism). shouldRunParallel signature updated to take sctx *search.SearchContext for future guard use. TestParallelSegmentSearchAdaptiveGuards added to cover DF guard and concurrency gate. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Per-shard heap now sized to the query's count (SearchRequest.Size+From) rather than a fixed 100. The heap fills after count docs, broadcasting a tight sharedThreshold to all goroutines — §34 global WAND ceilings then prune as aggressively as the serial path. Two new tuning knobs: - ParallelSegmentSearchShardK = 10 (was 100, now the floor): prevents degenerate heaps for very small counts (count=1 → shardK=10). - ParallelSegmentSearchMaxCount = 100: disables parallel search when count exceeds this value. For large-K queries WAND pruning is weaker and goroutine overhead dominates; serial is faster. Also avoids the prior correctness bug where shardK=100 < count=1000 caused shards to silently discard candidates the final merge needed. - SearcherOptions.TopK carries the count from index_impl into the DSS. Explicit context override (BENCH_PARALLEL_SEARCH=N) bypasses both the floor and the cap, so forced-parallel bench runs still work unchanged. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The MaxTFNorm sidecar format now writes a uvarint entryCount prefix immediately after the 20-byte DV-compat header (zapx change). This adds one byte per field sidecar read, which shifts the faceted-query bytes-read measurement from ~105 to ~137 — just outside the 30% approxSame tolerance. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ation optimizeCompositeSearcher() returns a new merged TermSearcher wrapping a combined bitmap, but never closed the original N sub-searchers. Their TFRs were never returned to the snapshot per-field pool (is.fieldTFRs), so every subsequent query reopened the Dictionary + vellum FST Reader for each field and segment — 90 dictionary() calls per EntityRare6FieldDisjScoreNone query. Close the original qsearchers in newDisjunctionSearcher immediately after the optimization succeeds. This is safe because OptimizeTFRDisjunctionUnadorned Finish() clones/creates all bitmaps before returning, so the originals hold no shared state. The close loop is intentionally in newDisjunctionSearcher (not inside optimizeCompositeSearcher) because optimizeMultiTermSearcher already calls cleanup() on its batch after the call — putting it inside would double-close. For EntityRare6FieldDisjScoreNone the TFR pool now stays warm across queries, eliminating the 24.9M dictionary alloc_objects (90/query) and contributing the remaining ~14µs of the combined -59% improvement (33.8µs → 13.8µs). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two collector bottlenecks identified by profiling TopicalMergedDisjunction3TopK k=1000 (3710 candidates, heap of 1000 docs, BM25 score sort): 1. SortOrder.Compare was 18.5% of total CPU. The generic path iterates a sort field slice and checks two bool flags (cachedScoring, cachedDesc) on every heap comparison, adding ~40% overhead to the unavoidable float64 comparison. Fix: detect score-descending-only at newTopNCollector time and store a specialized collectorCompare that compares i.Score/j.Score directly. Share this comparator (hc.cmp) with the heap, lowestMatchOutsideResults checks, and searchAfter pagination. SortOrder.Compare flat cost: 2.06s → 1.18s (−43%). 2. collectStoreHeap.Final was 11.6% of total CPU. Extracting k=1000 docs via repeated removeLast (heapsort) requires O(k log₃ k) comparisons with scattered pointer dereferences at each heap level — poor cache behavior. Fix: sort.Slice in-place (pdqsort) then copy heap[skip..skip+size-1] sequentially. Final cum cost: 1.29s → 0.52s (−60%); sort.Slice itself costs 0.32s. Combined: TopicalMergedDisjunction3TopK/k=1000 1.173ms → ~1.065ms (−9%). All collector tests pass; sort semantics (descending score, HitNumber tie-break, skip/pagination) preserved. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…r-doc overhead postingToTermFieldDoc was performing a dynamic interface type assertion (iterators[segmentOffset].(normByteIterator)) for every matched document. For TermTier4 (~2474 docs/query, 79k iterations) this amounts to ~196M type assertions per 10s run (~7.4ns each = 1.46s flat, 11.6% of CPU). Add normByteIters []normByteIterator parallel to iterators, populated once per query in TermFieldReader, TermFieldReaderForSegmentRange, and ShardView. postingToTermFieldDoc now uses a nil pointer check instead of the type assertion. ShardView also had a latent panic for scored parallel queries since it never populated normByteIters. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…urrIDs cache Profile of TopicalMergedDisjunction8SameTopic showed two hot loops in nextMAXSCORE — the collect loop (1.46s flat) and the advance-all loop (2.27s flat) — each scanning all N s.currs with redundant nil/len/BigEndian checks on every WAND iteration (~89M times per 30s run with 8 terms). Two changes: 1. currIDs []uint64 cache: a parallel slice to s.currs that stores the decoded big-endian uint64 docID (math.MaxUint64 for nil/exhausted). Updated after every s.currs[i] assignment in initSearchers, nextBasic, nextMAXSCORE, and Advance. The hot loops now compare plain uint64 values instead of chasing pointers and decoding BigEndian on every element. 2. advance-all loop now iterates s.matchingIdxs instead of all s.currs: the collect loop already identified exactly which indices are at minIDVal, so re-scanning all N iterators with condition checks is redundant. Both apply to every WAND-enabled disjunction query regardless of term count. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…non-KNN queries For score-sorted queries with no field loading (TermTier4 and similar), the collector's hot-path per-doc work had significant dead overhead: - adjustDocumentMatch (470ms flat/30s): entire function is a no-op when hc.knnHits == nil; guard the call at both call sites with knnHits != nil. - prepareDocumentMatch (1.56s flat/30s): three conditional branches always false for simple score queries (isKnnDoc check: 380ms, neededFields check: 240ms, sort-compute check: 210ms). Add fastPrepare bool, set once in Collect after needDocIds is known; fast path executes only the four necessary statements (total++, HitNumber, maxScore, Sort=sortByScoreOpt). - DocumentMatch.Reset (1.36s flat/30s): unconditional nil/empty stores on pointer/string/map fields trigger a GC write barrier even for nil→nil. Add nil guards so the common case (no explain, no fragments, no fields) skips the write barrier entirely. Expected improvement: ~1.3s saved per 30s TermTier4 run (~4% reduction). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds 9 unit tests for the collectStoreList type in search/collector that had 0% coverage before this commit. Tests cover round-trip insertion order, size-capped eviction (AddNotExceedingSize), skip-based pagination (Final), Internal() ascending traversal, removeLast worst-doc eviction, single-element and equal-score edge cases, and fixup error propagation — all using existing scoreDesc / makeScoreDoc helpers from heap_test.go. These tests have no dependency on any perf-gar change and can be rebased or cherry-picked independently. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…memory optimizations)
Two test files covering §25 and §22 in-memory optimizations — no file format
change, no backwards-compat concern.
scorer_term_table_test.go (package scorer):
- TestBM25ImpactTableVsFormula: verifies every (freq, normByte) entry in the
§25 BM25 impact table matches the exact float64 formula within float32 budget
- TestBM25ImpactTableNormByteSentinel: normByte=0 sentinel (infinite fieldLen) path
- TestBM25ImpactTableMonotoneInFreq: monotonicity invariant across all normBytes
- TestBM25ImpactTableCached: same pointer for same avgDocLen (cache hit)
- TestBM25SmallFloatFieldLenDecoder: all non-zero norm bytes decode to ≥1;
byte=0 decodes to 0
reset_test.go (addition — TestDocumentMatchResetAllFieldsCanary):
- Populates every user-visible field of DocumentMatch with non-zero values,
calls Reset(), asserts each zeroed field is actually zero. Acts as a
compile-time canary: new fields added to DocumentMatch without a matching
nil in Reset() will fail this test.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…age (new API/behavior) Unit tests for two parallel-search primitives added by §7: sharedThreshold CAS monotone invariant (inc. race-detector run) and dmMinHeap pushBounded eviction and top-k retention. Integration test for TotalRelation="gte" verifying that ScoreModeTopScores triggers WAND pruning on a BM25-scored OR query — confirmed with 50-doc corpus (5 high-freq, 45 prunable alpha-only docs, Size=3). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
runParallelSegmentSearch was unconditionally enabling WAND whenever MaxImpact was available, ignoring the request's ScoreMode. With ScoreModeComplete the user wants exact scores; WAND is a pruning mechanism and must not fire. Added requestWAND bool parameter (passed from ctx.WANDEnabled at the call site) that gates both the globalMI allocation and the canWAND flag. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…l correction estimateDF returned sum(term DFs) for all queries. For MSM queries (min > 1) this massively over-estimates: a matching doc must appear in ≥ min distinct postings, so the actual candidate count is far lower than the raw sum. Under the independence model the correct estimate is: E ≈ C(N, min) × (avgDF / N_docs)^(min−1) × avgDF For EntityMSMCommon6Field (N=6, avgDF=25k, N_docs=5M) this gives: min=2: 1 875 (was 150 000) → below threshold 2 250 → guard blocks §7 min=3: 12 (was 150 000) → below threshold 2 250 → guard blocks §7 Before the fix, §7 fired for these queries in --parallel-search (auto-mode) runs: 8 separate DocumentMatchPools allocated (7.9× serial bytes), WAND heap never filled (0–8 hits), goroutine overhead dominated → +17% and +14% regressions vs serial baseline. Plain disjunctions (min ≤ 1) return sum(DFs) unchanged. DocCount() on the scorch snapshot is O(1); falls back to sum(DFs) on error. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three gaps caused ctx.WANDPruned to stay false even when MAXSCORE was actively pruning candidates, leading to TotalRelation="eq" when it should be "gte" for ScoreModeTopScores queries. Fix 1a (serial MAXSCORE implicit skip): when pivotIdx > 0, MAXSCORE only generates candidates from essential iterators. Docs matching only non-essential terms are silently skipped without firing the per-candidate WANDPruned assignment at line 808. Now set ctx.WANDPruned = true at the DSS.Next() call site when pivotIdx > 0. Fix 1b (§15 segment skip): when segCeilings[segIdx] <= threshold, the entire segment is skipped. WANDPruned was not set in this branch. Now set it before checking if all remaining segments are exhausted. Fix 1c (§7 parallel shard contexts): runShardSearch creates a shard-local searchCtx. WANDPruned set in that ctx was never returned to the caller. Change runShardSearch to return (matches, wandPruned, error), aggregate across all shards in runParallelSegmentSearch, and propagate to ctx in DSS.Next() at both call sites. Verified by TestWANDTotalRelation in bench/wand_tr_test.go: top_scores + facets: Total=2483 TotalRelation="gte" (was "eq") complete + facets: Total=28196 TotalRelation="eq" Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
§55 Idea D. For score="none" a request means "return any Size+From matching docs", so the collector can stop pulling from the searcher once that many hits are collected instead of draining the full posting list. Because the searcher stack is pull-based, breaking the collect loop IS the early exit — nothing needs to propagate into the searchers. - TopNCollector.earlyStopN + SetEarlyStop(n): the collect loop breaks once hc.total >= n. Valid because the store keeps the earliest size+skip hits in insertion order and, with all scores equal, no later doc can displace them. - earlyStopped + EarlyStopped(): SearchInContext ORs it into TotalRelation so a bounded result reports "gte" (Total becomes a lower bound), mirroring WAND. - Eligibility gate in SearchInContext (after SetWANDEnabled): score=none, Size>0, no facets, no KNN, no SearchAfter/reverse, not nested, and sort-by-score-only (degrades to insertion order under score=none). Count queries (Size=0) are excluded by the Size>0 gate, so they still drain and report the exact Total. Scored queries are untouched. Benchmark (500K-doc index, warm cache, score=none Size=10): single term DF~500K: 15.41 ms -> 6.7 us (~2290x) min=2 disjunction nextBasic: 35.26 ms -> 16.7 us (~2110x) Allocs unchanged (pool recycles); the win is pure iteration CPU, O(matches)->O(k). Caveats (see §55 doc): §7 parallel segment search buffers all results up front so it defeats the speedup (still correct); the min<=1 unadorned bitmap optimizer eagerly unions, so early-stop saves iteration but not the union. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The rebase onto master adopts bleve_index_api's index-ID API change (#96), where IndexInternalID.Value() now returns a single uint64 instead of (uint64, error). Update the §15 per-segment score-ceiling call site to match, consistent with the other Value() call sites in this file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DisjunctionQueryScorer.Score reuses constituents[0] as rv, then called scoreExplain to gather each constituent's Expl into the "sum of:" children. Setting rv.Expl = nil *before* scoreExplain nulled constituents[0].Expl (same object as rv), so the first child of every disjunction explanation came back nil — a malformed explain tree for any Explain=true query that lowers to a disjunction (multi-term match, query_string, boolean should, ...), independent of score_mode. Run scoreExplain before clearing Expl; only clear on the non-explain path. Adds a regression test (tf-idf + BM25) asserting no nil explanation nodes and per-node arithmetic consistency (score/tf/idf/fieldNorm/coord). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR is the bleve half of the "perf-gar" performance effort: opt-in competitive scoring (WAND/MAXSCORE pruning), parallel segment search, and a series of query hot-path optimizations. It registers the new zapx v18 segment format (companion PR: blevesearch/zapx#438) as the default segment plugin and builds the pruning machinery on top of v18's MaxTFNorm/norm-column primitives.
Depends on blevesearch/zapx#438 — that PR should land first.
New public API
SearchRequest.ScoreMode— opt-in competitive scoring; enables WAND pruning when the caller doesn't need exhaustive scoring.SearchResult.TotalRelation— signals when WAND pruning makesTotala lower bound rather than an exact count.Changes
WAND / MAXSCORE pruning
maxTFNormcache; per-segmentMaxTFNormcached in the TFR to halveinitWANDMaxImpactscost.ScoreFast, concrete dispatch,binary.BigEndian.Uint64ID comparisons,PutLazy, folded upper-bound sums, empty-bitmap alloc skips indisjunction:unadorned Finish()).minSegCeilingguard.Parallel segment search
ScoreModeComplete).shardK = max(count, floor).Collector & scoring
sort.Slicein heapFinal.score=none+Size=k.NormColumnBytebridge to zapx v18.prepareDocumentMatch; guardadjustDocumentMatchfor non-KNN queries.Fixes
rv.Expl.DefaultFieldTFRCacheThreshold = 4) with a fix to avoid pool donation while the caller still holds the pointer inAdvance().WANDPrunedset correctly for MAXSCORE and §7 parallel shards.Tests
TotalRelationcoverage, sharedThreshold/dmMinHeap primitives, BM25 impact table correctness,DocumentMatch.Resetcanary, collectStoreList coverage, §7 integration tests, zapx v18 expected-byte updates.Notes for reviewers
go.modcurrently pins pseudo-versions:zapx/v18 v18.0.0-2026...,bleve_index_api v1.4.1-0.2026...(branch builds), and downgradeszapx/v1717.2.0 → 17.1.8 andscorch_segment_api/v22.4.8 → 2.4.7 relative to master. These need proper tagged releases / reconciliation with master before merge.go.mod,go.sum,index/scorch/empty.go,index/scorch/optimize.go,search/collector/topn.go,search/search.go,search/searcher/search_disjunction.go,search/util.go) — largely overlap with the recently merged MB-72489: standalone query hot-path optimizations #2381 hot-path work; will be resolved as part of review.