|
| 1 | +--- |
| 2 | +title: OpenSearchMetadataRetriever |
| 3 | +id: opensearchmetadataretriever |
| 4 | +slug: /opensearchmetadataretriever |
| 5 | +description: Searches and ranks the metadata fields of documents stored in an OpenSearch Document Store and returns the matching metadata values. |
| 6 | +--- |
| 7 | + |
| 8 | +# OpenSearchMetadataRetriever |
| 9 | + |
| 10 | +Searches and ranks the metadata fields of documents stored in an OpenSearch Document Store and returns the matching metadata values. |
| 11 | + |
| 12 | +<div className="key-value-table"> |
| 13 | + |
| 14 | +| | | |
| 15 | +| --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | |
| 16 | +| **Most common position in a pipeline** | The last component in a metadata lookup pipeline, or wherever you need other structured data from an OpenSearchDocumentStore index | |
| 17 | +| **Mandatory init variables** | `document_store`: An instance of `OpenSearchDocumentStore`; `metadata_fields`: List of metadata field names to search and return | |
| 18 | +| **Mandatory run variables** | `query`: A search query string (may contain comma-separated parts) | |
| 19 | +| **Output variables** | `metadata`: A list of dictionaries containing only the requested metadata fields | |
| 20 | +| **API reference** | [OpenSearch](/reference/integrations-opensearch) | |
| 21 | +| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opensearch | |
| 22 | +| **Package name** | `opensearch-haystack` | |
| 23 | + |
| 24 | +</div> |
| 25 | + |
| 26 | +## Overview |
| 27 | + |
| 28 | +`OpenSearchMetadataRetriever` searches the metadata of documents stored in an `OpenSearchDocumentStore` and returns the matching metadata values, not the documents themselves. It is useful when the metadata is the answer: for example, listing the categories or tags that match a partial query, building a metadata autocomplete, or surfacing the structured side of an index without pulling back document content. |
| 29 | + |
| 30 | +Unlike the other OpenSearch retrievers (`OpenSearchBM25Retriever`, `OpenSearchEmbeddingRetriever`, `OpenSearchHybridRetriever`), this component does not return `Document` objects. The output is a list under `metadata`, where each entry is a dictionary containing only the fields you listed in `metadata_fields`. Document content and any other metadata are excluded from the result. |
| 31 | + |
| 32 | +The retriever supports two search modes: |
| 33 | + |
| 34 | +- `strict` uses prefix and wildcard matching on the configured metadata fields. |
| 35 | +- `fuzzy` (the default) uses fuzzy matching with `dis_max` queries, allowing typos and partial matches. |
| 36 | + |
| 37 | +In both modes, candidate documents are scored server-side with Jaccard similarity on character n-grams (the `jaccard_n` parameter controls the n-gram size), and exact matches receive an additional boost controlled by `exact_match_weight`. Up to 1000 hits are fetched from OpenSearch, and the top `top_k` results are returned. |
| 38 | + |
| 39 | +Both a synchronous `run` method and an asynchronous `run_async` method are available with the same parameters. |
| 40 | + |
| 41 | +### Field types |
| 42 | + |
| 43 | +The matching engine only operates on metadata fields that OpenSearch indexes as text or keyword values. Numeric, boolean, and array-of-non-strings fields are not valid search targets, because prefix, wildcard, and full-text matching do not apply to them. Mixed-type fields, such as a list that combines strings and numbers, are also not supported. |
| 44 | + |
| 45 | + |
| 46 | +## Installation |
| 47 | + |
| 48 | +If you have Docker set up, the easiest way to run OpenSearch is to pull and run the Docker image. |
| 49 | + |
| 50 | +```bash |
| 51 | +docker pull opensearchproject/opensearch:2 |
| 52 | +docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -e "OPENSEARCH_INITIAL_ADMIN_PASSWORD=<custom-admin-password>" opensearchproject/opensearch:2 |
| 53 | +``` |
| 54 | + |
| 55 | +As an alternative, you can go to the [OpenSearch integration GitHub](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opensearch) and start a Docker container using the provided `docker-compose.yml`: |
| 56 | + |
| 57 | +```bash |
| 58 | +docker compose up |
| 59 | +``` |
| 60 | + |
| 61 | +Once you have a running OpenSearch instance, install the `opensearch-haystack` integration: |
| 62 | + |
| 63 | +```bash |
| 64 | +pip install opensearch-haystack |
| 65 | +``` |
| 66 | + |
| 67 | +## Usage |
| 68 | + |
| 69 | +### On its own |
| 70 | + |
| 71 | +This Retriever needs an `OpenSearchDocumentStore` with indexed documents. The example below writes three documents with simple categorical metadata and queries the `category` and `status` fields: |
| 72 | + |
| 73 | +```python |
| 74 | +from haystack import Document |
| 75 | +from haystack_integrations.components.retrievers.opensearch import ( |
| 76 | + OpenSearchMetadataRetriever, |
| 77 | +) |
| 78 | +from haystack_integrations.document_stores.opensearch import OpenSearchDocumentStore |
| 79 | +from haystack.document_stores.types import DuplicatePolicy |
| 80 | + |
| 81 | +document_store = OpenSearchDocumentStore( |
| 82 | + hosts="http://localhost:9200", |
| 83 | + index="my_index", |
| 84 | +) |
| 85 | + |
| 86 | +documents = [ |
| 87 | + Document( |
| 88 | + content="Python programming guide", |
| 89 | + meta={ |
| 90 | + "category": "Python", |
| 91 | + "status": "active", |
| 92 | + "priority": 1, |
| 93 | + "author": "John Doe", |
| 94 | + }, |
| 95 | + ), |
| 96 | + Document( |
| 97 | + content="Java tutorial", |
| 98 | + meta={ |
| 99 | + "category": "Java", |
| 100 | + "status": "active", |
| 101 | + "priority": 2, |
| 102 | + "author": "Jane Smith", |
| 103 | + }, |
| 104 | + ), |
| 105 | + Document( |
| 106 | + content="Python advanced topics", |
| 107 | + meta={ |
| 108 | + "category": "Python", |
| 109 | + "status": "inactive", |
| 110 | + "priority": 3, |
| 111 | + "author": "John Doe", |
| 112 | + }, |
| 113 | + ), |
| 114 | +] |
| 115 | + |
| 116 | +document_store.write_documents(documents=documents, policy=DuplicatePolicy.SKIP) |
| 117 | + |
| 118 | +retriever = OpenSearchMetadataRetriever( |
| 119 | + document_store=document_store, |
| 120 | + metadata_fields=["category", "status"], |
| 121 | + top_k=10, |
| 122 | +) |
| 123 | + |
| 124 | +result = retriever.run(query="Python") |
| 125 | + |
| 126 | +print(result) |
| 127 | +# { |
| 128 | +# "metadata": [ |
| 129 | +# {"category": "Python", "status": "active"}, |
| 130 | +# {"category": "Python", "status": "inactive"}, |
| 131 | +# ] |
| 132 | +# } |
| 133 | +``` |
| 134 | + |
| 135 | +Only the fields listed in `metadata_fields` appear in each result dictionary. The `author` metadata and the document content are excluded. |
| 136 | + |
| 137 | +### Multi-part queries |
| 138 | + |
| 139 | +The `query` string can contain several comma-separated parts. Each part is searched across every field listed in `metadata_fields`, and a document that matches multiple parts is ranked higher (controlled by `exact_match_weight`). |
| 140 | + |
| 141 | +```python |
| 142 | +result = retriever.run(query="Python, active") |
| 143 | +# Returns the metadata of documents whose fields match both "Python" and "active". |
| 144 | +``` |
| 145 | + |
| 146 | +### Strict mode |
| 147 | + |
| 148 | +By default the retriever runs in `fuzzy` mode, which tolerates typos and partial matches. For lookups where you only want prefix or wildcard matches and no edit-distance tolerance, switch to `strict`: |
| 149 | + |
| 150 | +```python |
| 151 | +retriever = OpenSearchMetadataRetriever( |
| 152 | + document_store=document_store, |
| 153 | + metadata_fields=["category"], |
| 154 | + mode="strict", |
| 155 | +) |
| 156 | + |
| 157 | +result = retriever.run(query="Pyth") |
| 158 | +# Matches "Python" through prefix matching, but not transposed-letter variants. |
| 159 | +``` |
| 160 | + |
| 161 | +The fuzzy-mode parameters (`fuzziness`, `prefix_length`, `max_expansions`, `tie_breaker`) only take effect when `mode="fuzzy"`. |
| 162 | + |
| 163 | +### Combining with filters |
| 164 | + |
| 165 | +You can narrow the candidate set before scoring by passing standard Haystack `filters` at run time. The filters are applied in a `bool` `filter` context, so they exclude non-matching documents without affecting scores: |
| 166 | + |
| 167 | +```python |
| 168 | +result = retriever.run( |
| 169 | + query="Python", |
| 170 | + filters={"field": "status", "operator": "==", "value": "active"}, |
| 171 | +) |
| 172 | +``` |
| 173 | + |
| 174 | +### Asynchronous execution |
| 175 | + |
| 176 | +For pipelines that mix synchronous and asynchronous components, the retriever exposes `run_async` with the same signature: |
| 177 | + |
| 178 | +```python |
| 179 | +result = await retriever.run_async(query="Python, active") |
| 180 | +``` |
| 181 | + |
| 182 | +### Error handling |
| 183 | + |
| 184 | +By default, a failed OpenSearch request raises an exception. To treat a failure as an empty result instead — for example, when the retriever sits behind a forgiving API — initialize the component with `raise_on_failure=False`. The error is then logged as a warning and `metadata` is returned as an empty list. |
0 commit comments