Skip to content

Support array contains and array overlaps in pgvector metadata filtering #1351

Description

@micha-f

Is your feature request related to a problem? Please describe.
We have quite a few cases where we have ingested documents into with metadata containing arrays (e.g. tags, belonging to multiple customers, projects etc.).

Currently metadata filtering does not support filtering on array values.

Describe the solution you'd like

We have added an array_overlaps and an array_contains operator for PGVector and would like to discuss whether that's something to contribute back. We have only implemented this for PGVector

# tags array must contain one of the tags
{"field": "meta.tags", "operator": "array_overlaps", "value": ["tag1", "tag3"]}

# tags array must contain all of the tags
{"field": "meta.tags", "operator": "array_contains", "value": ["tag1", "tag2"]}

Describe alternatives you've considered
Serializing the array to a comma-separated string and using the LIKE operator (added in #1341). Feels brittle.

Additional context
The example in the documentation here might wrongly lead one to think that the in operator could be used for arrays:
https://docs.haystack.deepset.ai/docs/metadata-filtering#filters-usage

specifically the use of companies (plural) here:

{"field": "meta.companies", "operator": "in", "value": ["BMW", "Mercedes"]}

Metadata

Metadata

Assignees

No one assigned

    Fields

    No fields configured for Feature.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions