Skip to content

Commit 9919d96

Browse files
g-despotclaude
andcommitted
docs: BM25 AND operator matches per-property, not cross-object
The keyword-search operator docs stated all tokens must be "present in the object" for `and`, which reads as cross-field. Under the current default backend (BlockMaxWAND, default for collections created after v1.30), AND scopes per-property: all query tokens must appear within a single searched property. An object with one token in `title` and another in `description` does not match under AND. Corrects the wording across the concept page, the BM25 and hybrid how-tos, and the GraphQL search-operators reference (the last is indexed by kapa, which had answered this incorrectly). Verified against weaviate core (bm25_searcher_block.go, lsmkv/search_segment.go); legacy WAND's cross-object behavior is unintentional and no longer the default, so it is not documented. Source: Pylon Schema Design #5828. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 440461d commit 9919d96

4 files changed

Lines changed: 17 additions & 17 deletions

File tree

docs/weaviate/api/graphql/search-operators.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -265,7 +265,7 @@ This operator allows you to combine [BM25](#bm25) and vector search to get a "be
265265
| `vector` | no | `[float]` | optional to supply your own vector |
266266
| `properties` | no | `[string]` | list of properties to limit the BM25 search to, default all text properties |
267267
| `fusionType` | no | `string` | the type of hybrid fusion algorithm (available from `v1.20.0`) |
268-
| `bm25SearchOperator` | no | `object` | set how many of the (bm25) query tokens must be present in the target object for it to be considered a match. (available from `v1.31.0`) |
268+
| `bm25SearchOperator` | no | `object` | set how many of the (bm25) query tokens must be present within a single searched property for an object to be considered a match. (available from `v1.31.0`) |
269269

270270
* Notes:
271271
* `alpha` can be any number from 0 to 1, defaulting to 0.75.
@@ -435,11 +435,11 @@ To mitigate this effect, Weaviate automatically performs a search with a higher
435435

436436
<SearchOperators/>
437437

438-
Use `bm25SearchOperator` to set how many of the query tokens must be present in the target object for it to be considered a match in the keyword (bm25) search portion of the hybrid search. This is useful when you want to ensure that only objects with a certain number of relevant keywords are returned.
438+
Use `bm25SearchOperator` to set how many of the query tokens must be present within a single searched property for an object to be considered a match in the keyword (bm25) search portion of the hybrid search. This is useful when you want to ensure that only objects with a certain number of relevant keywords are returned.
439439

440-
The available options are `And`, or `Or`. If `Or` is set, an additional parameter `minimumOrTokensMatch` must be specified, which defines how many of the query tokens must match for the object to be considered a match.
440+
The available options are `And`, or `Or`. With `And`, all of the query tokens must appear together within a single searched property; tokens spread across different properties do not match. If `Or` is set, an additional parameter `minimumOrTokensMatch` must be specified, which defines how many of the query tokens must be present within a single searched property for the object to be considered a match.
441441

442-
If not yet, the keyword search will behave as if `Or` was set with `minimumOrTokensMatch` equal to 1.
442+
If not set, the keyword search behaves as if `Or` was set with a `minimumOrTokensMatch` of `1`.
443443

444444
## BM25
445445

@@ -458,7 +458,7 @@ The `bm25` operator supports the following variables:
458458
| --------- | -------- | ----------- |
459459
| `query` | yes | The keyword search query. |
460460
| `properties` | no | Array of properties (fields) to search in, defaulting to all properties in the collection. |
461-
| `searchOperator` | no | set how many of the query tokens must be present in the target object for it to be considered a match. (available from `v1.31.0`) |
461+
| `searchOperator` | no | set how many of the query tokens must be present within a single searched property for an object to be considered a match. (available from `v1.31.0`) |
462462

463463
:::info Boosting properties
464464
Specific properties can be boosted by a factor specified as a number after the caret sign, for example `properties: ["title^3", "summary"]`.
@@ -537,11 +537,11 @@ import GraphQLFiltersBM25FilterExample from '/_includes/code/graphql.filters.bm2
537537

538538
<SearchOperators/>
539539

540-
Use `searchOperator` to set how many of the query tokens must be present in the target object for it to be considered a match. This is useful when you want to ensure that only objects with a certain number of relevant keywords are returned.
540+
Use `searchOperator` to set how many of the query tokens must be present within a single searched property for an object to be considered a match. This is useful when you want to ensure that only objects with a certain number of relevant keywords are returned.
541541

542-
The available options are `And`, or `Or`. If `Or` is set, an additional parameter `minimumOrTokensMatch` must be specified, which defines how many of the query tokens must match for the object to be considered a match.
542+
The available options are `And`, or `Or`. With `And`, all of the query tokens must appear together within a single searched property; tokens spread across different properties do not match. If `Or` is set, an additional parameter `minimumOrTokensMatch` must be specified, which defines how many of the query tokens must be present within a single searched property for the object to be considered a match.
543543

544-
If not yet, the keyword search will behave as if `Or` was set with `minimumOrTokensMatch` equal to 1.
544+
If not set, the keyword search behaves as if `Or` was set with a `minimumOrTokensMatch` of `1`.
545545

546546
## ask
547547

docs/weaviate/concepts/search/keyword-search.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -136,15 +136,15 @@ import SearchOperators from '/_includes/feature-notes/search-operators.mdx';
136136

137137
<SearchOperators/>
138138

139-
Search operators define the minimum number of query [tokens](../../search/bm25.md#set-tokenization) that must be present in the object to be returned.
139+
Search operators define the minimum number of query [tokens](../../search/bm25.md#set-tokenization) that must be present within a single searched property for an object to be returned.
140140

141141
Conceptually, it works as though a filter is applied to the results of the BM25 score calculation. The available operators are:
142-
- `and`: All tokens must be present in the object
143-
- `or`: At least one token must be present in the object, with the minimum number of tokens being configurable (`minimumOrTokensMatch`)
142+
- `and`: All tokens must be present within a single searched property
143+
- `or`: At least one token must be present within a single searched property, with the minimum number of tokens being configurable (`minimumOrTokensMatch`)
144144

145-
As an example, a BM25 query of `computer networking guide` with the `and` operator would only return objects that contain all of the tokens `computer`, `networking`, and `guide`. In contrast, the same query with the `or` operator would return objects that contain at least one of those tokens. If the `or` operator is used with a `minimumOrTokensMatch` of `2`, then at least two of the tokens must be present in the object.
145+
As an example, a BM25 query of `computer networking guide` with the `and` operator would only return objects where all of the tokens `computer`, `networking`, and `guide` appear together within a single searched property. If the tokens are spread across different properties — for example, `computer` in `title` and `networking` in `description` — the object does not match under `and`. In contrast, the same query with the `or` operator would return objects where at least one of those tokens appears in a searched property. If the `or` operator is used with a `minimumOrTokensMatch` of `2`, then at least two of the tokens must be present within a single searched property.
146146

147-
If not specified, the default operator is `or`, with a `minimumOrTokensMatch` of `1`. This means that at least one token must be present in the object for it to be returned.
147+
If not specified, the default operator is `or`, with a `minimumOrTokensMatch` of `1`. This means that at least one token must be present in a searched property for the object to be returned.
148148

149149
import BM25OperatorsLight from '../img/bm25_operators_light.png';
150150
import BM25OperatorsDark from '../img/bm25_operators_dark.png';

docs/weaviate/search/bm25.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,7 @@ import SearchOperators from '/_includes/feature-notes/search-operators.mdx';
9999

100100
<SearchOperators/>
101101

102-
Search operators define the minimum number of query [tokens](#set-tokenization) that must be present in the object to be returned. The options are `and`, or `or` (default).
102+
Search operators define the minimum number of query [tokens](#set-tokenization) that must be present within a single searched property for an object to be returned. The options are `and`, or `or` (default).
103103

104104
### `or`
105105

@@ -142,7 +142,7 @@ With the `or` operator, the search returns objects that contain at least `minimu
142142

143143
### `and`
144144

145-
With the `and` operator, the search returns objects that contain all tokens in the search string.
145+
With the `and` operator, the search returns objects where all tokens in the search string appear together within a single searched property.
146146

147147
<Tabs className="code" groupId="languages">
148148
<TabItem value="py" label="Python">

docs/weaviate/search/hybrid.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -384,7 +384,7 @@ import SearchOperators from '/_includes/feature-notes/search-operators.mdx';
384384

385385
<SearchOperators/>
386386

387-
Keyword (BM25) search operators define the minimum number of query [tokens](#tokenization) that must be present in the object to be returned. The options are `and`, or `or` (default).
387+
Keyword (BM25) search operators define the minimum number of query [tokens](#tokenization) that must be present within a single searched property for an object to be returned. The options are `and`, or `or` (default).
388388

389389
### `or`
390390

@@ -435,7 +435,7 @@ With the `or` operator, the search returns objects that contain at least `minimu
435435

436436
### `and`
437437

438-
With the `and` operator, the search returns objects that contain all tokens in the search string.
438+
With the `and` operator, the search returns objects where all tokens in the search string appear together within a single searched property.
439439

440440
<Tabs className="code" groupId="languages">
441441
<TabItem value="py" label="Python">

0 commit comments

Comments
 (0)