Is your feature request related to a problem? Please describe.
We have quite a few cases where we have ingested documents into with metadata containing arrays (e.g. tags, belonging to multiple customers, projects etc.).
Currently metadata filtering does not support filtering on array values.
Describe the solution you'd like
We have added an array_overlaps and an array_contains operator for PGVector and would like to discuss whether that's something to contribute back. We have only implemented this for PGVector
# tags array must contain one of the tags
{"field": "meta.tags", "operator": "array_overlaps", "value": ["tag1", "tag3"]}
# tags array must contain all of the tags
{"field": "meta.tags", "operator": "array_contains", "value": ["tag1", "tag2"]}
Describe alternatives you've considered
Serializing the array to a comma-separated string and using the LIKE operator (added in #1341). Feels brittle.
Additional context
The example in the documentation here might wrongly lead one to think that the in operator could be used for arrays:
https://docs.haystack.deepset.ai/docs/metadata-filtering#filters-usage
specifically the use of companies (plural) here:
{"field": "meta.companies", "operator": "in", "value": ["BMW", "Mercedes"]}
Is your feature request related to a problem? Please describe.
We have quite a few cases where we have ingested documents into with metadata containing arrays (e.g. tags, belonging to multiple customers, projects etc.).
Currently metadata filtering does not support filtering on array values.
Describe the solution you'd like
We have added an
array_overlapsand anarray_containsoperator for PGVector and would like to discuss whether that's something to contribute back. We have only implemented this for PGVectorDescribe alternatives you've considered
Serializing the array to a comma-separated string and using the
LIKEoperator (added in #1341). Feels brittle.Additional context
The example in the documentation here might wrongly lead one to think that the
inoperator could be used for arrays:https://docs.haystack.deepset.ai/docs/metadata-filtering#filters-usage
specifically the use of
companies(plural) here:{"field": "meta.companies", "operator": "in", "value": ["BMW", "Mercedes"]}