Describe the bug
InMemoryDocumentStore.delete_documents() updates the global document-frequency
counter using Counter.subtract(...).
When a token's frequency reaches zero, the key is not removed from
_freq_vocab_for_idf. Instead it remains with a value of 0.
Those stale ("ghost") tokens continue participating in BM25 IDF computation,
even though no active document contains them anymore.
As documents are repeatedly inserted and deleted, the vocabulary grows with
zero-frequency entries, causing BM25 statistics to drift over time.
The result is silent retrieval quality degradation:
identical active corpora can produce different BM25 scores solely because one
store has a history of document deletions.
Error message
No exception is raised.
This is a silent scoring inconsistency that can affect BM25 ranking.
Expected behavior
Deleting a document should completely remove its contribution to the global
document-frequency statistics.
Two InMemoryDocumentStore instances containing the same active documents
should produce identical BM25 scores regardless of previous insert/delete
history.
Actual behavior
Zero-frequency tokens remain in _freq_vocab_for_idf.
During BM25 IDF computation these stale entries are still iterated over,
artificially increasing the global sum_idf used to compute the epsilon
smoothing value.
This changes BM25 scores even though the active corpus is identical.
Minimal Reproduction
# Create two identical document stores.
# Store A:
# insert active documents only
# Store B:
# insert many temporary documents
# delete them
# insert the same active documents as Store A
# Run the same BM25 query against both stores.
Although both stores contain identical active documents, the BM25 scores differ.
Observed Result
Using the same active corpus:
Fresh store:
Doc 1: 0.5459
Doc 2: 0.0993
Doc 3: 0.0682
Ranking:
1 > 2 > 3
After repeatedly inserting and deleting unrelated documents:
Polluted store:
Doc 2: 0.7515
Doc 1: 0.5459
Doc 3: 0.5163
Ranking:
2 > 1 > 3
The active documents are identical.
Only the deletion history differs.
I isolated this by comparing three stores:
- a fresh store,
- a store after repeated insert/delete operations,
- the same polluted store after manually removing zero-frequency entries.
Removing only the zero-frequency entries restores the original BM25 scores.
Root Cause
delete_documents() performs
self._freq_vocab_for_idf.subtract(Counter(freq.keys()))
Python's Counter.subtract() preserves entries whose count becomes zero.
Example:
Counter({"a": 1}).subtract(Counter({"a": 1}))
# Result
Counter({"a": 0})Later, BM25 computes IDF by iterating over
for token, frequency in self._freq_vocab_for_idf.items():
Those zero-frequency tokens are therefore still included in the IDF statistics,
even though they are no longer present in any document.
Suggested Fix
After subtraction, prune zero/negative counts.
For example:
self._freq_vocab_for_idf.subtract(Counter(freq.keys()))
self._freq_vocab_for_idf = +self._freq_vocab_for_idf
The unary + operator removes all zero and negative entries from the
Counter.
Additional context
While investigating this issue I verified that:
Counter.subtract() retains zero-count keys._freq_vocab_for_idf is not rebuilt after document deletion.- BM25 IDF statistics are recomputed from
_freq_vocab_for_idf during retrieval.
- I reproduced a ranking change on identical active corpora caused solely by deletion history.
- I also have a local regression test and a candidate fix prepared if this is considered a bug.
To Reproduce
- Create an
InMemoryDocumentStore.
- Insert many temporary documents.
- Delete them.
- Insert a fixed active corpus.
- Compare BM25 retrieval against a fresh store containing only the active corpus.
FAQ Check
System
- OS: Windows 11
- Haystack version: current
main (verified locally)
If maintainers agree this is a bug, I have already prepared a local implementation together with regression tests and would be happy to submit a PR.
Describe the bug
InMemoryDocumentStore.delete_documents()updates the global document-frequencycounter using
Counter.subtract(...).When a token's frequency reaches zero, the key is not removed from
_freq_vocab_for_idf. Instead it remains with a value of0.Those stale ("ghost") tokens continue participating in BM25 IDF computation,
even though no active document contains them anymore.
As documents are repeatedly inserted and deleted, the vocabulary grows with
zero-frequency entries, causing BM25 statistics to drift over time.
The result is silent retrieval quality degradation:
identical active corpora can produce different BM25 scores solely because one
store has a history of document deletions.
Error message
No exception is raised.
This is a silent scoring inconsistency that can affect BM25 ranking.
Expected behavior
Deleting a document should completely remove its contribution to the global
document-frequency statistics.
Two
InMemoryDocumentStoreinstances containing the same active documentsshould produce identical BM25 scores regardless of previous insert/delete
history.
Actual behavior
Zero-frequency tokens remain in
_freq_vocab_for_idf.During BM25 IDF computation these stale entries are still iterated over,
artificially increasing the global
sum_idfused to compute the epsilonsmoothing value.
This changes BM25 scores even though the active corpus is identical.
Minimal Reproduction
Although both stores contain identical active documents, the BM25 scores differ.
Observed Result
Using the same active corpus:
After repeatedly inserting and deleting unrelated documents:
The active documents are identical.
Only the deletion history differs.
I isolated this by comparing three stores:
Removing only the zero-frequency entries restores the original BM25 scores.
Root Cause
delete_documents()performsPython's
Counter.subtract()preserves entries whose count becomes zero.Example:
Later, BM25 computes IDF by iterating over
Those zero-frequency tokens are therefore still included in the IDF statistics,
even though they are no longer present in any document.
Suggested Fix
After subtraction, prune zero/negative counts.
For example:
The unary
+operator removes all zero and negative entries from theCounter.Additional context
While investigating this issue I verified that:
Counter.subtract()retains zero-count keys._freq_vocab_for_idfis not rebuilt after document deletion._freq_vocab_for_idfduring retrieval.To Reproduce
InMemoryDocumentStore.FAQ Check
System
main(verified locally)If maintainers agree this is a bug, I have already prepared a local implementation together with regression tests and would be happy to submit a PR.