sparse: warm-start pruning threshold for BM25 DAAT_MAXSCORE#1630
sparse: warm-start pruning threshold for BM25 DAAT_MAXSCORE#1630lyang24 wants to merge 1 commit into
Conversation
Bootstrap a safe non-zero pruning threshold before topk_ fills, so update_non_essential_lists() can demote terms early and current_score + lookup_bound pruning kicks in from the first candidate. Mechanism: scan one seed term's posting list, compute single-term BM25 contribution per matching doc, take the k-th best as warm_threshold_. Since per-term BM25 contributions are non-negative, the k-th best single-term score is a safe lower bound on the k-th best full-query score (at least k docs are guaranteed to have full_score >= S_k). Exact-recall preserving. Seed = query term with the largest per-dim max_score (high IDF, short posting list). Activated only for BM25, query.size() >= 2, k >= 2, and seed must hit >= k docs (otherwise S_m is not a valid bound on the k-th score). Falls back to current behavior on guard misses. No change to ResultHeap semantics; warm threshold is searcher-local. Signed-off-by: lyang24 <lanqingy93@gmail.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: lyang24 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@lyang24 🔍 Important: PR Classification Needed! For efficient project management and a seamless review process, it's essential to classify your PR correctly. Here's how:
For any PR outside the kind/improvement category, ensure you link to the associated issue using the format: “issue: #”. Thanks for your efforts and contribution to the community!. |
|
bench marked no gain closing |
2 participants
Bootstrap a safe non-zero pruning threshold before topk_ fills, so update_non_essential_lists() can demote terms early and current_score + lookup_bound pruning kicks in from the first candidate.
Mechanism: scan one seed term's posting list, compute single-term BM25 contribution per matching doc, take the k-th best as warm_threshold_. Since per-term BM25 contributions are non-negative, the k-th best single-term score is a safe lower bound on the k-th best full-query score (at least k docs are guaranteed to have full_score >= S_k). Exact-recall preserving.
Seed = query term with the largest per-dim max_score (high IDF, short posting list). Activated only for BM25, query.size() >= 2, k >= 2, and seed must hit >= k docs (otherwise S_m is not a valid bound on the k-th score). Falls back to current behavior on guard misses.
No change to ResultHeap semantics; warm threshold is searcher-local.