From 1b1435559cb7c463bcc953baf9008d24050f4be8 Mon Sep 17 00:00:00 2001 From: jialiuyang <495120021@qq.com> Date: Sun, 28 Jun 2026 22:27:51 +0800 Subject: [PATCH 1/4] docs: Add documentation page for OpenSearchMetadataRetriever Adds the page under docs-website/docs/pipeline-components/retrievers/ and mirrors it into the version-2.30 snapshot. Updates the retrievers index and sidebar navigation in both the current and versioned configs. --- .../docs/pipeline-components/retrievers.mdx | 1 + .../opensearchmetadataretriever.mdx | 163 ++++++++++++++++++ docs-website/sidebars.js | 1 + .../pipeline-components/retrievers.mdx | 1 + .../opensearchmetadataretriever.mdx | 163 ++++++++++++++++++ .../version-2.30-sidebars.json | 1 + 6 files changed, 330 insertions(+) create mode 100644 docs-website/docs/pipeline-components/retrievers/opensearchmetadataretriever.mdx create mode 100644 docs-website/versioned_docs/version-2.30/pipeline-components/retrievers/opensearchmetadataretriever.mdx diff --git a/docs-website/docs/pipeline-components/retrievers.mdx b/docs-website/docs/pipeline-components/retrievers.mdx index 1bff1d7719e..6a1a7c6b287 100644 --- a/docs-website/docs/pipeline-components/retrievers.mdx +++ b/docs-website/docs/pipeline-components/retrievers.mdx @@ -185,6 +185,7 @@ For details on how to initialize and use a Retriever in a pipeline, see the docu | [OpenSearchBM25Retriever](retrievers/opensearchbm25retriever.mdx) | A keyword-based Retriever that fetches Documents matching a query from an OpenSearch Document Store. | | [OpenSearchEmbeddingRetriever](retrievers/opensearchembeddingretriever.mdx) | An embedding-based Retriever compatible with the OpenSearch Document Store. | | [OpenSearchHybridRetriever](retrievers/opensearchhybridretriever.mdx) | A SuperComponent that implements a Hybrid Retriever in a single component, relying on OpenSearch as the backend Document Store. | +| [OpenSearchMetadataRetriever](retrievers/opensearchmetadataretriever.mdx) | Searches and ranks the metadata fields of documents stored in an OpenSearch Document Store and returns the matching metadata values. | | [OpenSearchSQLRetriever](retrievers/opensearchsqlretriever.mdx) | Executes raw OpenSearch SQL queries against an OpenSearch Document Store and returns the raw JSON response. | | [OracleEmbeddingRetriever](retrievers/oracleembeddingretriever.mdx) | An embedding-based Retriever compatible with the Oracle Document Store. | | [OracleKeywordRetriever](retrievers/oraclekeywordretriever.mdx) | A keyword-based Retriever that fetches Documents matching a query from the Oracle Document Store. | diff --git a/docs-website/docs/pipeline-components/retrievers/opensearchmetadataretriever.mdx b/docs-website/docs/pipeline-components/retrievers/opensearchmetadataretriever.mdx new file mode 100644 index 00000000000..973c797b18b --- /dev/null +++ b/docs-website/docs/pipeline-components/retrievers/opensearchmetadataretriever.mdx @@ -0,0 +1,163 @@ +--- +title: OpenSearchMetadataRetriever +id: opensearchmetadataretriever +slug: /opensearchmetadataretriever +description: Searches and ranks the metadata fields of documents stored in an OpenSearch Document Store and returns the matching metadata values. +--- + +# OpenSearchMetadataRetriever + +Searches and ranks the metadata fields of documents stored in an OpenSearch Document Store and returns the matching metadata values. + +| | | +| --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| **Most common position in a pipeline** | Standalone, or wherever you need to look up structured metadata (categories, statuses, tags, IDs) instead of full documents | +| **Mandatory init variables** | `document_store`: An instance of `OpenSearchDocumentStore`; `metadata_fields`: List of metadata field names to search and return | +| **Mandatory run variables** | `query`: A search query string (may contain comma-separated parts) | +| **Output variables** | `metadata`: A list of dictionaries containing only the requested metadata fields | +| **API reference** | [OpenSearch](https://docs.haystack.deepset.ai/reference/integrations-opensearch) | +| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opensearch | +| **Package name** | `opensearch-haystack` | + +## Overview + +`OpenSearchMetadataRetriever` searches the metadata of documents stored in an `OpenSearchDocumentStore` and returns the matching metadata values, not the documents themselves. It is useful when the metadata is the answer: for example, listing the categories or tags that match a partial query, building a metadata autocomplete, or surfacing the structured side of an index without pulling back document content. + +Unlike the other OpenSearch retrievers (`OpenSearchBM25Retriever`, `OpenSearchEmbeddingRetriever`, `OpenSearchHybridRetriever`), this component does not return `Document` objects. The output is a list under `metadata`, where each entry is a dictionary containing only the fields you listed in `metadata_fields`. Document content and any other metadata are excluded from the result. + +The retriever supports two search modes: + +- `strict` uses prefix and wildcard matching on the configured metadata fields. +- `fuzzy` (the default) uses fuzzy matching with `dis_max` queries, allowing typos and partial matches. + +In both modes, candidate documents are scored server-side with Jaccard similarity on character n-grams (the `jaccard_n` parameter controls the n-gram size), and exact matches receive an additional boost controlled by `exact_match_weight`. Up to 1000 hits are fetched from OpenSearch, and the top `top_k` results are returned. + +Both a synchronous `run` method and an asynchronous `run_async` method are available with the same parameters. + +### Field types + +The matching engine only operates on metadata fields that OpenSearch indexes as text or keyword values. Numeric, boolean, and array-of-non-strings fields are not valid search targets, because prefix, wildcard, and full-text matching do not apply to them. Mixed-type fields, such as a list that combines strings and numbers, are also not supported. + +You can still include numeric or boolean fields in `metadata_fields` to have them returned in the output; they just will not contribute to the match. + +## Installation + +If you have Docker set up, the easiest way to run OpenSearch is to pull and run the Docker image. + +```bash +docker pull opensearchproject/opensearch:2 +docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -e "OPENSEARCH_INITIAL_ADMIN_PASSWORD=" opensearchproject/opensearch:2 +``` + +As an alternative, you can go to the [OpenSearch integration GitHub](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opensearch) and start a Docker container using the provided `docker-compose.yml`: + +```bash +docker compose up +``` + +Once you have a running OpenSearch instance, install the `opensearch-haystack` integration: + +```bash +pip install opensearch-haystack +``` + +## Usage + +### On its own + +This Retriever needs an `OpenSearchDocumentStore` with indexed documents. The example below writes three documents with simple categorical metadata and queries the `category`, `status`, and `priority` fields: + +```python +from haystack import Document +from haystack_integrations.components.retrievers.opensearch import ( + OpenSearchMetadataRetriever, +) +from haystack_integrations.document_stores.opensearch import OpenSearchDocumentStore +from haystack.document_stores.types import DuplicatePolicy + +document_store = OpenSearchDocumentStore(hosts="http://localhost:9200", index="my_index") + +documents = [ + Document( + content="Python programming guide", + meta={"category": "Python", "status": "active", "priority": 1, "author": "John Doe"}, + ), + Document( + content="Java tutorial", + meta={"category": "Java", "status": "active", "priority": 2, "author": "Jane Smith"}, + ), + Document( + content="Python advanced topics", + meta={"category": "Python", "status": "inactive", "priority": 3, "author": "John Doe"}, + ), +] + +document_store.write_documents(documents=documents, policy=DuplicatePolicy.SKIP) + +retriever = OpenSearchMetadataRetriever( + document_store=document_store, + metadata_fields=["category", "status", "priority"], + top_k=10, +) + +result = retriever.run(query="Python") + +print(result) +# { +# "metadata": [ +# {"category": "Python", "status": "active", "priority": 1}, +# {"category": "Python", "status": "inactive", "priority": 3}, +# ] +# } +``` + +Only the fields listed in `metadata_fields` appear in each result dictionary. The `author` metadata and the document content are excluded. + +### Multi-part queries + +The `query` string can contain several comma-separated parts. Each part is searched across every field listed in `metadata_fields`, and a document that matches multiple parts is ranked higher when `mode="fuzzy"` (controlled by `tie_breaker`). + +```python +result = retriever.run(query="Python, active") +# Returns the metadata of documents whose fields match both "Python" and "active". +``` + +### Strict mode + +By default the retriever runs in `fuzzy` mode, which tolerates typos and partial matches. For lookups where you only want prefix or wildcard matches and no edit-distance tolerance, switch to `strict`: + +```python +retriever = OpenSearchMetadataRetriever( + document_store=document_store, + metadata_fields=["category"], + mode="strict", +) + +result = retriever.run(query="Pyth") +# Matches "Python" through prefix matching, but not "Phyton". +``` + +The fuzzy-mode parameters (`fuzziness`, `prefix_length`, `max_expansions`, `tie_breaker`) only take effect when `mode="fuzzy"`. + +### Combining with filters + +You can narrow the candidate set before scoring by passing standard Haystack `filters` at run time. The filters are applied as a `query_string` clause inside the OpenSearch request: + +```python +result = retriever.run( + query="Python", + filters={"field": "status", "operator": "==", "value": "active"}, +) +``` + +### Asynchronous execution + +For pipelines that mix synchronous and asynchronous components, the retriever exposes `run_async` with the same signature: + +```python +result = await retriever.run_async(query="Python, active") +``` + +### Error handling + +By default, a failed OpenSearch request raises an exception. To treat a failure as an empty result instead — for example, when the retriever sits behind a forgiving API — initialize the component with `raise_on_failure=False`. The error is then logged as a warning and `metadata` is returned as an empty list. diff --git a/docs-website/sidebars.js b/docs-website/sidebars.js index 4d99fa77807..dde8286f279 100644 --- a/docs-website/sidebars.js +++ b/docs-website/sidebars.js @@ -593,6 +593,7 @@ export default { 'pipeline-components/retrievers/opensearchbm25retriever', 'pipeline-components/retrievers/opensearchembeddingretriever', 'pipeline-components/retrievers/opensearchhybridretriever', + 'pipeline-components/retrievers/opensearchmetadataretriever', 'pipeline-components/retrievers/opensearchsqlretriever', 'pipeline-components/retrievers/oracleembeddingretriever', 'pipeline-components/retrievers/oraclekeywordretriever', diff --git a/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers.mdx b/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers.mdx index cbc46c31fec..898ef650371 100644 --- a/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers.mdx +++ b/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers.mdx @@ -185,6 +185,7 @@ For details on how to initialize and use a Retriever in a pipeline, see the docu | [OpenSearchBM25Retriever](retrievers/opensearchbm25retriever.mdx) | A keyword-based Retriever that fetches Documents matching a query from an OpenSearch Document Store. | | [OpenSearchEmbeddingRetriever](retrievers/opensearchembeddingretriever.mdx) | An embedding-based Retriever compatible with the OpenSearch Document Store. | | [OpenSearchHybridRetriever](retrievers/opensearchhybridretriever.mdx) | A SuperComponent that implements a Hybrid Retriever in a single component, relying on OpenSearch as the backend Document Store. | +| [OpenSearchMetadataRetriever](retrievers/opensearchmetadataretriever.mdx) | Searches and ranks the metadata fields of documents stored in an OpenSearch Document Store and returns the matching metadata values. | | [OpenSearchSQLRetriever](retrievers/opensearchsqlretriever.mdx) | Executes raw OpenSearch SQL queries against an OpenSearch Document Store and returns the raw JSON response. | | [OracleEmbeddingRetriever](retrievers/oracleembeddingretriever.mdx) | An embedding-based Retriever compatible with the Oracle Document Store. | | [OracleKeywordRetriever](retrievers/oraclekeywordretriever.mdx) | A keyword-based Retriever that fetches Documents matching a query from the Oracle Document Store. | diff --git a/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers/opensearchmetadataretriever.mdx b/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers/opensearchmetadataretriever.mdx new file mode 100644 index 00000000000..973c797b18b --- /dev/null +++ b/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers/opensearchmetadataretriever.mdx @@ -0,0 +1,163 @@ +--- +title: OpenSearchMetadataRetriever +id: opensearchmetadataretriever +slug: /opensearchmetadataretriever +description: Searches and ranks the metadata fields of documents stored in an OpenSearch Document Store and returns the matching metadata values. +--- + +# OpenSearchMetadataRetriever + +Searches and ranks the metadata fields of documents stored in an OpenSearch Document Store and returns the matching metadata values. + +| | | +| --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| **Most common position in a pipeline** | Standalone, or wherever you need to look up structured metadata (categories, statuses, tags, IDs) instead of full documents | +| **Mandatory init variables** | `document_store`: An instance of `OpenSearchDocumentStore`; `metadata_fields`: List of metadata field names to search and return | +| **Mandatory run variables** | `query`: A search query string (may contain comma-separated parts) | +| **Output variables** | `metadata`: A list of dictionaries containing only the requested metadata fields | +| **API reference** | [OpenSearch](https://docs.haystack.deepset.ai/reference/integrations-opensearch) | +| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opensearch | +| **Package name** | `opensearch-haystack` | + +## Overview + +`OpenSearchMetadataRetriever` searches the metadata of documents stored in an `OpenSearchDocumentStore` and returns the matching metadata values, not the documents themselves. It is useful when the metadata is the answer: for example, listing the categories or tags that match a partial query, building a metadata autocomplete, or surfacing the structured side of an index without pulling back document content. + +Unlike the other OpenSearch retrievers (`OpenSearchBM25Retriever`, `OpenSearchEmbeddingRetriever`, `OpenSearchHybridRetriever`), this component does not return `Document` objects. The output is a list under `metadata`, where each entry is a dictionary containing only the fields you listed in `metadata_fields`. Document content and any other metadata are excluded from the result. + +The retriever supports two search modes: + +- `strict` uses prefix and wildcard matching on the configured metadata fields. +- `fuzzy` (the default) uses fuzzy matching with `dis_max` queries, allowing typos and partial matches. + +In both modes, candidate documents are scored server-side with Jaccard similarity on character n-grams (the `jaccard_n` parameter controls the n-gram size), and exact matches receive an additional boost controlled by `exact_match_weight`. Up to 1000 hits are fetched from OpenSearch, and the top `top_k` results are returned. + +Both a synchronous `run` method and an asynchronous `run_async` method are available with the same parameters. + +### Field types + +The matching engine only operates on metadata fields that OpenSearch indexes as text or keyword values. Numeric, boolean, and array-of-non-strings fields are not valid search targets, because prefix, wildcard, and full-text matching do not apply to them. Mixed-type fields, such as a list that combines strings and numbers, are also not supported. + +You can still include numeric or boolean fields in `metadata_fields` to have them returned in the output; they just will not contribute to the match. + +## Installation + +If you have Docker set up, the easiest way to run OpenSearch is to pull and run the Docker image. + +```bash +docker pull opensearchproject/opensearch:2 +docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -e "OPENSEARCH_INITIAL_ADMIN_PASSWORD=" opensearchproject/opensearch:2 +``` + +As an alternative, you can go to the [OpenSearch integration GitHub](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opensearch) and start a Docker container using the provided `docker-compose.yml`: + +```bash +docker compose up +``` + +Once you have a running OpenSearch instance, install the `opensearch-haystack` integration: + +```bash +pip install opensearch-haystack +``` + +## Usage + +### On its own + +This Retriever needs an `OpenSearchDocumentStore` with indexed documents. The example below writes three documents with simple categorical metadata and queries the `category`, `status`, and `priority` fields: + +```python +from haystack import Document +from haystack_integrations.components.retrievers.opensearch import ( + OpenSearchMetadataRetriever, +) +from haystack_integrations.document_stores.opensearch import OpenSearchDocumentStore +from haystack.document_stores.types import DuplicatePolicy + +document_store = OpenSearchDocumentStore(hosts="http://localhost:9200", index="my_index") + +documents = [ + Document( + content="Python programming guide", + meta={"category": "Python", "status": "active", "priority": 1, "author": "John Doe"}, + ), + Document( + content="Java tutorial", + meta={"category": "Java", "status": "active", "priority": 2, "author": "Jane Smith"}, + ), + Document( + content="Python advanced topics", + meta={"category": "Python", "status": "inactive", "priority": 3, "author": "John Doe"}, + ), +] + +document_store.write_documents(documents=documents, policy=DuplicatePolicy.SKIP) + +retriever = OpenSearchMetadataRetriever( + document_store=document_store, + metadata_fields=["category", "status", "priority"], + top_k=10, +) + +result = retriever.run(query="Python") + +print(result) +# { +# "metadata": [ +# {"category": "Python", "status": "active", "priority": 1}, +# {"category": "Python", "status": "inactive", "priority": 3}, +# ] +# } +``` + +Only the fields listed in `metadata_fields` appear in each result dictionary. The `author` metadata and the document content are excluded. + +### Multi-part queries + +The `query` string can contain several comma-separated parts. Each part is searched across every field listed in `metadata_fields`, and a document that matches multiple parts is ranked higher when `mode="fuzzy"` (controlled by `tie_breaker`). + +```python +result = retriever.run(query="Python, active") +# Returns the metadata of documents whose fields match both "Python" and "active". +``` + +### Strict mode + +By default the retriever runs in `fuzzy` mode, which tolerates typos and partial matches. For lookups where you only want prefix or wildcard matches and no edit-distance tolerance, switch to `strict`: + +```python +retriever = OpenSearchMetadataRetriever( + document_store=document_store, + metadata_fields=["category"], + mode="strict", +) + +result = retriever.run(query="Pyth") +# Matches "Python" through prefix matching, but not "Phyton". +``` + +The fuzzy-mode parameters (`fuzziness`, `prefix_length`, `max_expansions`, `tie_breaker`) only take effect when `mode="fuzzy"`. + +### Combining with filters + +You can narrow the candidate set before scoring by passing standard Haystack `filters` at run time. The filters are applied as a `query_string` clause inside the OpenSearch request: + +```python +result = retriever.run( + query="Python", + filters={"field": "status", "operator": "==", "value": "active"}, +) +``` + +### Asynchronous execution + +For pipelines that mix synchronous and asynchronous components, the retriever exposes `run_async` with the same signature: + +```python +result = await retriever.run_async(query="Python, active") +``` + +### Error handling + +By default, a failed OpenSearch request raises an exception. To treat a failure as an empty result instead — for example, when the retriever sits behind a forgiving API — initialize the component with `raise_on_failure=False`. The error is then logged as a warning and `metadata` is returned as an empty list. diff --git a/docs-website/versioned_sidebars/version-2.30-sidebars.json b/docs-website/versioned_sidebars/version-2.30-sidebars.json index 55d212b3b99..80b2beeee29 100644 --- a/docs-website/versioned_sidebars/version-2.30-sidebars.json +++ b/docs-website/versioned_sidebars/version-2.30-sidebars.json @@ -589,6 +589,7 @@ "pipeline-components/retrievers/opensearchbm25retriever", "pipeline-components/retrievers/opensearchembeddingretriever", "pipeline-components/retrievers/opensearchhybridretriever", + "pipeline-components/retrievers/opensearchmetadataretriever", "pipeline-components/retrievers/opensearchsqlretriever", "pipeline-components/retrievers/oracleembeddingretriever", "pipeline-components/retrievers/oraclekeywordretriever", From a25bc18fbf22a236b18dca724b400733a8345f69 Mon Sep 17 00:00:00 2001 From: "David S. Batista" Date: Tue, 30 Jun 2026 13:15:14 +0200 Subject: [PATCH 2/4] adding key-value-table div wrapper + changing to relative urls + rephrase the comment to avoid the false positive by codespell --- .../opensearchmetadataretriever.mdx | 34 +++++++++++++++---- .../opensearchmetadataretriever.mdx | 34 +++++++++++++++---- 2 files changed, 56 insertions(+), 12 deletions(-) diff --git a/docs-website/docs/pipeline-components/retrievers/opensearchmetadataretriever.mdx b/docs-website/docs/pipeline-components/retrievers/opensearchmetadataretriever.mdx index 973c797b18b..5e07cb6c832 100644 --- a/docs-website/docs/pipeline-components/retrievers/opensearchmetadataretriever.mdx +++ b/docs-website/docs/pipeline-components/retrievers/opensearchmetadataretriever.mdx @@ -9,16 +9,20 @@ description: Searches and ranks the metadata fields of documents stored in an Op Searches and ranks the metadata fields of documents stored in an OpenSearch Document Store and returns the matching metadata values. +
+ | | | | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | **Most common position in a pipeline** | Standalone, or wherever you need to look up structured metadata (categories, statuses, tags, IDs) instead of full documents | | **Mandatory init variables** | `document_store`: An instance of `OpenSearchDocumentStore`; `metadata_fields`: List of metadata field names to search and return | | **Mandatory run variables** | `query`: A search query string (may contain comma-separated parts) | | **Output variables** | `metadata`: A list of dictionaries containing only the requested metadata fields | -| **API reference** | [OpenSearch](https://docs.haystack.deepset.ai/reference/integrations-opensearch) | +| **API reference** | [OpenSearch](/reference/integrations-opensearch) | | **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opensearch | | **Package name** | `opensearch-haystack` | +
+ ## Overview `OpenSearchMetadataRetriever` searches the metadata of documents stored in an `OpenSearchDocumentStore` and returns the matching metadata values, not the documents themselves. It is useful when the metadata is the answer: for example, listing the categories or tags that match a partial query, building a metadata autocomplete, or surfacing the structured side of an index without pulling back document content. @@ -75,20 +79,38 @@ from haystack_integrations.components.retrievers.opensearch import ( from haystack_integrations.document_stores.opensearch import OpenSearchDocumentStore from haystack.document_stores.types import DuplicatePolicy -document_store = OpenSearchDocumentStore(hosts="http://localhost:9200", index="my_index") +document_store = OpenSearchDocumentStore( + hosts="http://localhost:9200", + index="my_index", +) documents = [ Document( content="Python programming guide", - meta={"category": "Python", "status": "active", "priority": 1, "author": "John Doe"}, + meta={ + "category": "Python", + "status": "active", + "priority": 1, + "author": "John Doe", + }, ), Document( content="Java tutorial", - meta={"category": "Java", "status": "active", "priority": 2, "author": "Jane Smith"}, + meta={ + "category": "Java", + "status": "active", + "priority": 2, + "author": "Jane Smith", + }, ), Document( content="Python advanced topics", - meta={"category": "Python", "status": "inactive", "priority": 3, "author": "John Doe"}, + meta={ + "category": "Python", + "status": "inactive", + "priority": 3, + "author": "John Doe", + }, ), ] @@ -134,7 +156,7 @@ retriever = OpenSearchMetadataRetriever( ) result = retriever.run(query="Pyth") -# Matches "Python" through prefix matching, but not "Phyton". +# Matches "Python" through prefix matching, but not transposed-letter variants. ``` The fuzzy-mode parameters (`fuzziness`, `prefix_length`, `max_expansions`, `tie_breaker`) only take effect when `mode="fuzzy"`. diff --git a/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers/opensearchmetadataretriever.mdx b/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers/opensearchmetadataretriever.mdx index 973c797b18b..5e07cb6c832 100644 --- a/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers/opensearchmetadataretriever.mdx +++ b/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers/opensearchmetadataretriever.mdx @@ -9,16 +9,20 @@ description: Searches and ranks the metadata fields of documents stored in an Op Searches and ranks the metadata fields of documents stored in an OpenSearch Document Store and returns the matching metadata values. +
+ | | | | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | **Most common position in a pipeline** | Standalone, or wherever you need to look up structured metadata (categories, statuses, tags, IDs) instead of full documents | | **Mandatory init variables** | `document_store`: An instance of `OpenSearchDocumentStore`; `metadata_fields`: List of metadata field names to search and return | | **Mandatory run variables** | `query`: A search query string (may contain comma-separated parts) | | **Output variables** | `metadata`: A list of dictionaries containing only the requested metadata fields | -| **API reference** | [OpenSearch](https://docs.haystack.deepset.ai/reference/integrations-opensearch) | +| **API reference** | [OpenSearch](/reference/integrations-opensearch) | | **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opensearch | | **Package name** | `opensearch-haystack` | +
+ ## Overview `OpenSearchMetadataRetriever` searches the metadata of documents stored in an `OpenSearchDocumentStore` and returns the matching metadata values, not the documents themselves. It is useful when the metadata is the answer: for example, listing the categories or tags that match a partial query, building a metadata autocomplete, or surfacing the structured side of an index without pulling back document content. @@ -75,20 +79,38 @@ from haystack_integrations.components.retrievers.opensearch import ( from haystack_integrations.document_stores.opensearch import OpenSearchDocumentStore from haystack.document_stores.types import DuplicatePolicy -document_store = OpenSearchDocumentStore(hosts="http://localhost:9200", index="my_index") +document_store = OpenSearchDocumentStore( + hosts="http://localhost:9200", + index="my_index", +) documents = [ Document( content="Python programming guide", - meta={"category": "Python", "status": "active", "priority": 1, "author": "John Doe"}, + meta={ + "category": "Python", + "status": "active", + "priority": 1, + "author": "John Doe", + }, ), Document( content="Java tutorial", - meta={"category": "Java", "status": "active", "priority": 2, "author": "Jane Smith"}, + meta={ + "category": "Java", + "status": "active", + "priority": 2, + "author": "Jane Smith", + }, ), Document( content="Python advanced topics", - meta={"category": "Python", "status": "inactive", "priority": 3, "author": "John Doe"}, + meta={ + "category": "Python", + "status": "inactive", + "priority": 3, + "author": "John Doe", + }, ), ] @@ -134,7 +156,7 @@ retriever = OpenSearchMetadataRetriever( ) result = retriever.run(query="Pyth") -# Matches "Python" through prefix matching, but not "Phyton". +# Matches "Python" through prefix matching, but not transposed-letter variants. ``` The fuzzy-mode parameters (`fuzziness`, `prefix_length`, `max_expansions`, `tie_breaker`) only take effect when `mode="fuzzy"`. From 0b4432ed1cc107ed63a61331b0aa2457f46481a9 Mon Sep 17 00:00:00 2001 From: "David S. Batista" Date: Tue, 30 Jun 2026 15:02:22 +0200 Subject: [PATCH 3/4] fixing position in pipeline + filter section --- .../retrievers/opensearchmetadataretriever.mdx | 5 ++--- .../retrievers/opensearchmetadataretriever.mdx | 5 ++--- 2 files changed, 4 insertions(+), 6 deletions(-) diff --git a/docs-website/docs/pipeline-components/retrievers/opensearchmetadataretriever.mdx b/docs-website/docs/pipeline-components/retrievers/opensearchmetadataretriever.mdx index 5e07cb6c832..b112098ac1f 100644 --- a/docs-website/docs/pipeline-components/retrievers/opensearchmetadataretriever.mdx +++ b/docs-website/docs/pipeline-components/retrievers/opensearchmetadataretriever.mdx @@ -13,7 +13,7 @@ Searches and ranks the metadata fields of documents stored in an OpenSearch Docu | | | | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | -| **Most common position in a pipeline** | Standalone, or wherever you need to look up structured metadata (categories, statuses, tags, IDs) instead of full documents | +| **Most common position in a pipeline** | The last component in a metadata lookup pipeline, or wherever you need other structured data from an OpenSearchDocumentStore index | | **Mandatory init variables** | `document_store`: An instance of `OpenSearchDocumentStore`; `metadata_fields`: List of metadata field names to search and return | | **Mandatory run variables** | `query`: A search query string (may contain comma-separated parts) | | **Output variables** | `metadata`: A list of dictionaries containing only the requested metadata fields | @@ -42,7 +42,6 @@ Both a synchronous `run` method and an asynchronous `run_async` method are avail The matching engine only operates on metadata fields that OpenSearch indexes as text or keyword values. Numeric, boolean, and array-of-non-strings fields are not valid search targets, because prefix, wildcard, and full-text matching do not apply to them. Mixed-type fields, such as a list that combines strings and numbers, are also not supported. -You can still include numeric or boolean fields in `metadata_fields` to have them returned in the output; they just will not contribute to the match. ## Installation @@ -163,7 +162,7 @@ The fuzzy-mode parameters (`fuzziness`, `prefix_length`, `max_expansions`, `tie_ ### Combining with filters -You can narrow the candidate set before scoring by passing standard Haystack `filters` at run time. The filters are applied as a `query_string` clause inside the OpenSearch request: +You can narrow the candidate set before scoring by passing standard Haystack `filters` at run time. The filters are applied in a `bool` `filter` context, so they exclude non-matching documents without affecting scores: ```python result = retriever.run( diff --git a/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers/opensearchmetadataretriever.mdx b/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers/opensearchmetadataretriever.mdx index 5e07cb6c832..b112098ac1f 100644 --- a/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers/opensearchmetadataretriever.mdx +++ b/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers/opensearchmetadataretriever.mdx @@ -13,7 +13,7 @@ Searches and ranks the metadata fields of documents stored in an OpenSearch Docu | | | | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | -| **Most common position in a pipeline** | Standalone, or wherever you need to look up structured metadata (categories, statuses, tags, IDs) instead of full documents | +| **Most common position in a pipeline** | The last component in a metadata lookup pipeline, or wherever you need other structured data from an OpenSearchDocumentStore index | | **Mandatory init variables** | `document_store`: An instance of `OpenSearchDocumentStore`; `metadata_fields`: List of metadata field names to search and return | | **Mandatory run variables** | `query`: A search query string (may contain comma-separated parts) | | **Output variables** | `metadata`: A list of dictionaries containing only the requested metadata fields | @@ -42,7 +42,6 @@ Both a synchronous `run` method and an asynchronous `run_async` method are avail The matching engine only operates on metadata fields that OpenSearch indexes as text or keyword values. Numeric, boolean, and array-of-non-strings fields are not valid search targets, because prefix, wildcard, and full-text matching do not apply to them. Mixed-type fields, such as a list that combines strings and numbers, are also not supported. -You can still include numeric or boolean fields in `metadata_fields` to have them returned in the output; they just will not contribute to the match. ## Installation @@ -163,7 +162,7 @@ The fuzzy-mode parameters (`fuzziness`, `prefix_length`, `max_expansions`, `tie_ ### Combining with filters -You can narrow the candidate set before scoring by passing standard Haystack `filters` at run time. The filters are applied as a `query_string` clause inside the OpenSearch request: +You can narrow the candidate set before scoring by passing standard Haystack `filters` at run time. The filters are applied in a `bool` `filter` context, so they exclude non-matching documents without affecting scores: ```python result = retriever.run( From e9d337aa837140a93d53d738d63bf277d867bea8 Mon Sep 17 00:00:00 2001 From: "David S. Batista" Date: Tue, 30 Jun 2026 15:15:54 +0200 Subject: [PATCH 4/4] removing int fields from the query examples --- .../retrievers/opensearchmetadataretriever.mdx | 10 +++++----- .../retrievers/opensearchmetadataretriever.mdx | 10 +++++----- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/docs-website/docs/pipeline-components/retrievers/opensearchmetadataretriever.mdx b/docs-website/docs/pipeline-components/retrievers/opensearchmetadataretriever.mdx index b112098ac1f..609e044b58e 100644 --- a/docs-website/docs/pipeline-components/retrievers/opensearchmetadataretriever.mdx +++ b/docs-website/docs/pipeline-components/retrievers/opensearchmetadataretriever.mdx @@ -68,7 +68,7 @@ pip install opensearch-haystack ### On its own -This Retriever needs an `OpenSearchDocumentStore` with indexed documents. The example below writes three documents with simple categorical metadata and queries the `category`, `status`, and `priority` fields: +This Retriever needs an `OpenSearchDocumentStore` with indexed documents. The example below writes three documents with simple categorical metadata and queries the `category` and `status` fields: ```python from haystack import Document @@ -117,7 +117,7 @@ document_store.write_documents(documents=documents, policy=DuplicatePolicy.SKIP) retriever = OpenSearchMetadataRetriever( document_store=document_store, - metadata_fields=["category", "status", "priority"], + metadata_fields=["category", "status"], top_k=10, ) @@ -126,8 +126,8 @@ result = retriever.run(query="Python") print(result) # { # "metadata": [ -# {"category": "Python", "status": "active", "priority": 1}, -# {"category": "Python", "status": "inactive", "priority": 3}, +# {"category": "Python", "status": "active"}, +# {"category": "Python", "status": "inactive"}, # ] # } ``` @@ -136,7 +136,7 @@ Only the fields listed in `metadata_fields` appear in each result dictionary. Th ### Multi-part queries -The `query` string can contain several comma-separated parts. Each part is searched across every field listed in `metadata_fields`, and a document that matches multiple parts is ranked higher when `mode="fuzzy"` (controlled by `tie_breaker`). +The `query` string can contain several comma-separated parts. Each part is searched across every field listed in `metadata_fields`, and a document that matches multiple parts is ranked higher (controlled by `exact_match_weight`). ```python result = retriever.run(query="Python, active") diff --git a/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers/opensearchmetadataretriever.mdx b/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers/opensearchmetadataretriever.mdx index b112098ac1f..609e044b58e 100644 --- a/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers/opensearchmetadataretriever.mdx +++ b/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers/opensearchmetadataretriever.mdx @@ -68,7 +68,7 @@ pip install opensearch-haystack ### On its own -This Retriever needs an `OpenSearchDocumentStore` with indexed documents. The example below writes three documents with simple categorical metadata and queries the `category`, `status`, and `priority` fields: +This Retriever needs an `OpenSearchDocumentStore` with indexed documents. The example below writes three documents with simple categorical metadata and queries the `category` and `status` fields: ```python from haystack import Document @@ -117,7 +117,7 @@ document_store.write_documents(documents=documents, policy=DuplicatePolicy.SKIP) retriever = OpenSearchMetadataRetriever( document_store=document_store, - metadata_fields=["category", "status", "priority"], + metadata_fields=["category", "status"], top_k=10, ) @@ -126,8 +126,8 @@ result = retriever.run(query="Python") print(result) # { # "metadata": [ -# {"category": "Python", "status": "active", "priority": 1}, -# {"category": "Python", "status": "inactive", "priority": 3}, +# {"category": "Python", "status": "active"}, +# {"category": "Python", "status": "inactive"}, # ] # } ``` @@ -136,7 +136,7 @@ Only the fields listed in `metadata_fields` appear in each result dictionary. Th ### Multi-part queries -The `query` string can contain several comma-separated parts. Each part is searched across every field listed in `metadata_fields`, and a document that matches multiple parts is ranked higher when `mode="fuzzy"` (controlled by `tie_breaker`). +The `query` string can contain several comma-separated parts. Each part is searched across every field listed in `metadata_fields`, and a document that matches multiple parts is ranked higher (controlled by `exact_match_weight`). ```python result = retriever.run(query="Python, active")