Fix MultiLeafKnnCollector global heap update bug#16404
Closed
txwei wants to merge 753 commits into
Closed
Conversation
apache#14823) `TieredMergePolicy` currently allows 10 segments per tier. With Lucene being increasingly deployed with separate indexing and search tiers that get updated via segment-based replication, I believe that it would make sense for Lucene to have more aggressive merging defaults, a price that is only paid once on the indexing tier, but that benefits all search nodes that serve queries for this index. Note that this is still a somewhat conservative default, applications with low latency requirements and low update rates will likely want to go even further, with 4 segments per tier, or even 2. `BaseMergePolicyTestCase#testSimulateAppendOnly` reports a write amplification increase from 3.4 to 3.8, while `BaseMergePolicyTestCase#testSimulateUpdates` reports a write amplification increase from 4.5 to 4.9. In exchange, the number of segments between the floor and max segment sizes decreases by about 20%. This should especially help queries that have a high per-segment overhead: PK lookups, point queries, multi-term queries and vector searches.
…he#14827) Co-authored-by: gesong.samuel <gesong.samuel@bytedance.com>
This `<code>` case causes invalid HTML in two locations of the resulting Changes.html.
Co-authored-by: gesong.samuel <gesong.samuel@bytedance.com>
* Backport Faiss-based vector format * Rebase from main --------- Co-authored-by: Kaival Parikh <kaivalp2000@gmail.com> Co-authored-by: Michael McCandless <mikemccand@apache.org>
…#14839) * assertDocValuesEquals should support sparse sorted doc_values In LuceneTestCase, assertDocValuesEquals compares the doc_values in two indices. For SortedSetDocValues, as well as other doc value types, it does not assume that every document has a value. But for SortedDocValues, it requires that every doc have a value. Remove this restriction. * add change to CHANGES.txt
If a given filter is a MatchAllQuery, we shouldn't bother going through the filtered search path. It adds unnecessary overhead through: building a bitset for filter checking checking a bit set to see if a vector matches a filter
Co-authored-by: gesong.samuel <gesong.samuel@bytedance.com> Co-authored-by: Adrien Grand <jpountz@gmail.com>
* OptimisticKnnVectorQuery that re-enters search for segments with with strong results * fix JDK23 lint * simplify OptimisticKnnVectorQuery to 2 rounds only * move OptimisticKnnVectorQuery into AbstractKnnVectorQuery * tidy * Use TopKnnCollector for second pass so we correctly collect total top K --------- Co-authored-by: Michael Sokolov <sokolovm@amazon.com>
This reverts commit 8840564.
I remember benchmarking prefix sums quite extensively, and unrolled loops
performed significantly better than their rolled on counterpart, both on micro
and macro benchmarks:
```java
private static void prefixSum(int[] arr, int len) {
for (int i = 1; i < len; ++i) {
arr[i] += arr[i-1];
}
}
```
However, I recently discovered that rewriting the loop this way performs much
better, and almost on par with the unrolled variant:
```java
private static void prefixSum(int[] arr, int len) {
int sum = 0;
for (int i = 0; i < len; ++i) {
sum += arr[i];
arr[i] = sum;
}
}
```
… to 10.4.0 (#20) * Restore WANDScorer for TOP_SCORES + minShouldMatch > 1 (apache#16176) Lucene apache#13408 (released in 10.0) dropped a guard in BooleanScorerSupplier.optionalBulkScorer that previously kept TOP_SCORES + minShouldMatch > 1 queries on WANDScorer. As a side effect, when matches are dense, dispatch now lands on BooleanScorer, which has no top-K impact pruning — causing 10–100× latency regressions on pure-should compound queries with minimumShouldMatch > 1. * Defer term collection when query term count is unknown (apache#16222) MultiTermQuery constant-score wrapper now defers term collection to ScorerSupplier#get() for queries with an unknown term count (automaton queries such as wildcard/regexp/prefix/range), instead of scanning the term dictionary while building the ScorerSupplier. This keeps the ScorerSupplier "planning" phase cheap so a parent conjunction can short-circuit (e.g. a sibling clause matching no documents) before a non-seekable scan, such as a leading wildcard, runs. If the term dictionary is large and a leading wildcard query matches very few times (<16 (BOOLEAN_REWRITE_TERM_COUNT_THRESHOLD)), using lazy instead of eager evaluation could save a lot of unnecessary term collection. * CHANGES.txt
github-actions
Bot
added
module:core/store
module:analysis
module:core/index
module:demo
module:core/search
module:highlighter
module:benchmark
module:suggest
module:spatial
module:luke
module:core/codecs
module:facet
module:grouping
module:core/FSTs
module:queryparser
module:join
module:classification
module:test-framework
module:replicator
module:expressions
module:sandbox
module:spatial3d
module:misc
module:queries
module:monitor
module:core/hnsw
module:build-infra
labels
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description