Describe the bug
The >= and < metadata filter operators in haystack/utils/filters.py return wrong results when comparing datetime strings that represent the same moment in time but are formatted differently.
The cause: _greater_than_equal is implemented as _equal(...) or _greater_than(...). _greater_than parses string operands as datetimes before comparing, but _equal compares the raw strings. So for two equivalent representations of the same datetime (e.g. 2025-01-01 vs 2025-01-01T00:00:00, or the same instant expressed with different timezone offsets), the equality check fails and the strict comparison is also False, making >= return False. _less_than is defined as not _greater_than_equal(...), so it returns True for those equal datetimes.
Net effect: date range filters silently drop boundary documents in InMemoryDocumentStore (and any other component using document_matches_filter).
Error message
No error. Documents are silently excluded from (or wrongly included in) filter results.
Expected behavior
>= should return True and < should return False when the document value and the filter value parse to the same datetime, regardless of string formatting. > and <= already behave correctly because they compare parsed datetimes directly.
Additional context
The asymmetry: <= (not _greater_than) and > are correct, while >= and < (which both go through the string-based _equal) are wrong. So the bug only bites on the boundary value of range queries, which makes it easy to miss in production.
To Reproduce
from haystack.dataclasses import Document
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.utils.filters import document_matches_filter
doc = Document(content=x, meta={date: 2025-01-01T00:00:00})
f_gte = {field: meta.date, operator: >=, value: 2025-01-01}
f_lt = {field: meta.date, operator: <, value: 2025-01-01}
print(document_matches_filter(f_gte, doc)) # False, expected True
print(document_matches_filter(f_lt, doc)) # True, expected False
# Same instant, different timezone offsets
doc2 = Document(content=x, meta={date: 2025-01-01T12:00:00+02:00})
f2 = {field: meta.date, operator: >=, value: 2025-01-01T10:00:00+00:00}
print(document_matches_filter(f2, doc2)) # False, expected True
# End to end: boundary document dropped from a date range query
store = InMemoryDocumentStore()
store.write_documents([doc])
res = store.filter_documents({field: meta.date, operator: >=, value: 2025-01-01})
print(len(res)) # 0, expected 1
FAQ Check
System:
- OS: Windows 11
- Haystack version: main (2.x)
Describe the bug
The
>=and<metadata filter operators inhaystack/utils/filters.pyreturn wrong results when comparing datetime strings that represent the same moment in time but are formatted differently.The cause:
_greater_than_equalis implemented as_equal(...) or _greater_than(...)._greater_thanparses string operands as datetimes before comparing, but_equalcompares the raw strings. So for two equivalent representations of the same datetime (e.g.2025-01-01vs2025-01-01T00:00:00, or the same instant expressed with different timezone offsets), the equality check fails and the strict comparison is alsoFalse, making>=returnFalse._less_thanis defined asnot _greater_than_equal(...), so it returnsTruefor those equal datetimes.Net effect: date range filters silently drop boundary documents in
InMemoryDocumentStore(and any other component usingdocument_matches_filter).Error message
No error. Documents are silently excluded from (or wrongly included in) filter results.
Expected behavior
>=should returnTrueand<should returnFalsewhen the document value and the filter value parse to the same datetime, regardless of string formatting.>and<=already behave correctly because they compare parsed datetimes directly.Additional context
The asymmetry:
<=(not _greater_than) and>are correct, while>=and<(which both go through the string-based_equal) are wrong. So the bug only bites on the boundary value of range queries, which makes it easy to miss in production.To Reproduce
FAQ Check
System: