Skip to content

Commit 0a83580

Browse files
davidsbatistadavidsbatista
authored
docs: adding docs for ElasticSearchHybridRetriever (#11806)
1 parent 3935c10 commit 0a83580

6 files changed

Lines changed: 432 additions & 0 deletions

File tree

‎docs-website/docs/pipeline-components/retrievers.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,7 @@ For details on how to initialize and use a Retriever in a pipeline, see the docu
167167
| [CogneeRetriever](retrievers/cogneeretriever.mdx) | Retrieves memories from a CogneeMemoryStore and returns them as system ChatMessage objects. |
168168
| [ElasticsearchEmbeddingRetriever](retrievers/elasticsearchembeddingretriever.mdx) | An embedding-based Retriever compatible with the Elasticsearch Document Store. |
169169
| [ElasticsearchBM25Retriever](retrievers/elasticsearchbm25retriever.mdx) | A keyword-based Retriever that fetches Documents matching a query from the Elasticsearch Document Store. |
170+
| [ElasticsearchHybridRetriever](retrievers/elasticsearchhybridretriever.mdx) | A SuperComponent that combines BM25 and embedding-based retrieval from the Elasticsearch Document Store. |
170171
| [ElasticsearchSQLRetriever](retrievers/elasticsearchsqlretriever.mdx) | Executes raw Elasticsearch SQL queries against an Elasticsearch Document Store and returns the raw JSON response. |
171172
| [FAISSEmbeddingRetriever](retrievers/faissembeddingretriever.mdx) | An embedding-based Retriever compatible with the FAISSDocumentStore. |
172173
| [FalkorDBCypherRetriever](retrievers/falkordbcypherretriever.mdx) | A Retriever that executes arbitrary OpenCypher queries against a FalkorDB Document Store. |

‎docs-website/docs/pipeline-components/retrievers/elasticsearchhybridretriever.mdx‎

Lines changed: 214 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,214 @@
1+
---
2+
title: "ElasticsearchHybridRetriever"
3+
id: elasticsearchhybridretriever
4+
slug: "/elasticsearchhybridretriever"
5+
description: "This is a SuperComponent that implements a Hybrid Retriever in a single component, relying on Elasticsearch as the backend Document Store."
6+
---
7+
8+
# ElasticsearchHybridRetriever
9+
10+
This is a [SuperComponent](../../concepts/components/supercomponents.mdx) that implements a Hybrid Retriever in a single component, relying on Elasticsearch as the backend Document Store.
11+
12+
A Hybrid Retriever uses both traditional keyword-based search (BM25) and embedding-based search to retrieve documents, combining the strengths of both approaches. The Retriever then merges and re-ranks the results from both methods.
13+
14+
<div className="key-value-table">
15+
16+
| | |
17+
| --- | --- |
18+
| **Most common position in a pipeline** | 1. After a TextEmbedder and before a PromptBuilder in a RAG pipeline 2. The last component in a hybrid search pipeline 3. After a TextEmbedder and before an ExtractiveReader in an extractive QA pipeline |
19+
| **Mandatory init variables** | `document_store`: An instance of [`ElasticsearchDocumentStore`](../../document-stores/elasticsearch-document-store.mdx) <br /> <br />`embedder`: Any [Embedder](../embedders.mdx) implementing the `TextEmbedder` protocol |
20+
| **Mandatory run variables** | `query`: A query string |
21+
| **Output variables** | `documents`: A list of documents matching the query |
22+
| **API reference** | [Elasticsearch](/reference/integrations-elasticsearch) |
23+
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/elasticsearch |
24+
| **Package name** | `elasticsearch-haystack` |
25+
26+
</div>
27+
28+
## Overview
29+
30+
The `ElasticsearchHybridRetriever` combines two retrieval methods:
31+
32+
1. **BM25 Retrieval**: A keyword-based search that uses the BM25 algorithm to find documents based on term frequency and inverse document frequency. It's based on the [`ElasticsearchBM25Retriever`](elasticsearchbm25retriever.mdx) component and is suitable for finding exact matches to names, IDs, or well-defined terms.
33+
2. **Embedding-based Retrieval**: A semantic search that uses vector similarity to find documents that are semantically similar to the query. It's based on the [`ElasticsearchEmbeddingRetriever`](elasticsearchembeddingretriever.mdx) component and is suitable for semantic search.
34+
35+
The component automatically handles:
36+
37+
- Converting the query into an embedding using the provided embedder,
38+
- Running both retrieval methods in parallel,
39+
- Merging and re-ranking the results using the specified join mode (default: Reciprocal Rank Fusion).
40+
41+
### Installation
42+
43+
[Install](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html) Elasticsearch and then [start](https://www.elastic.co/guide/en/elasticsearch/reference/current/starting-elasticsearch.html) an instance. Haystack supports Elasticsearch 8.
44+
45+
If you have Docker set up, we recommend pulling the Docker image and running it.
46+
47+
```shell
48+
docker pull docker.elastic.co/elasticsearch/elasticsearch:8.11.1
49+
docker run -p 9200:9200 -e "discovery.type=single-node" -e "ES_JAVA_OPTS=-Xms1024m -Xmx1024m" -e "xpack.security.enabled=false" elasticsearch:8.11.1
50+
```
51+
52+
As an alternative, you can go to the [Elasticsearch integration GitHub](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/elasticsearch) and start a Docker container running Elasticsearch using the provided `docker-compose.yml`:
53+
54+
```shell
55+
docker compose up
56+
```
57+
58+
Once you have a running Elasticsearch instance, install the `elasticsearch-haystack` integration:
59+
60+
```shell
61+
pip install elasticsearch-haystack
62+
```
63+
64+
### Optional Parameters
65+
66+
This Retriever accepts various optional parameters. You can verify the most up-to-date list of parameters in our [API Reference](/reference/integrations-elasticsearch).
67+
68+
You can pass additional parameters to the underlying BM25 and embedding retriever components using the `top_k_bm25`, `fuzziness`, `filters_bm25`, `scale_score`, `filter_policy_bm25`, `top_k_embedding`, `filters_embedding`, `num_candidates`, and `filter_policy_embedding` parameters.
69+
70+
The `DocumentJoiner` parameters (`join_mode`, `weights`, `top_k`, and `sort_by_score`) are all exposed directly on the `ElasticsearchHybridRetriever` class.
71+
72+
## Usage
73+
74+
### On its own
75+
76+
This Retriever needs the `ElasticsearchDocumentStore` populated with documents (including embeddings) to run.
77+
78+
```python
79+
from haystack import Document
80+
from haystack_integrations.components.embedders.sentence_transformers import (
81+
SentenceTransformersTextEmbedder,
82+
SentenceTransformersDocumentEmbedder,
83+
)
84+
from haystack_integrations.components.retrievers.elasticsearch import (
85+
ElasticsearchHybridRetriever,
86+
)
87+
from haystack_integrations.document_stores.elasticsearch import (
88+
ElasticsearchDocumentStore,
89+
)
90+
91+
document_store = ElasticsearchDocumentStore(hosts="http://localhost:9200/")
92+
93+
model = "sentence-transformers/all-MiniLM-L6-v2"
94+
95+
documents = [
96+
Document(content="There are over 7,000 languages spoken around the world today."),
97+
Document(
98+
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
99+
),
100+
Document(
101+
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
102+
),
103+
]
104+
105+
doc_embedder = SentenceTransformersDocumentEmbedder(model=model)
106+
docs_with_embeddings = doc_embedder.run(documents)
107+
document_store.write_documents(docs_with_embeddings["documents"])
108+
109+
embedder = SentenceTransformersTextEmbedder(model=model)
110+
111+
retriever = ElasticsearchHybridRetriever(
112+
document_store=document_store,
113+
embedder=embedder,
114+
)
115+
116+
results = retriever.run(query="How many languages are spoken around the world today?")
117+
print(results["documents"])
118+
```
119+
120+
### In a pipeline
121+
122+
Here's a full example that uses an indexing pipeline to store documents with embeddings, and a query pipeline that uses `ElasticsearchHybridRetriever` for hybrid retrieval.
123+
124+
Set your `OPENAI_API_KEY` as an environment variable and then run the following code:
125+
126+
```python
127+
from haystack import Document, Pipeline
128+
from haystack.components.builders import ChatPromptBuilder
129+
from haystack.components.generators.chat import OpenAIChatGenerator
130+
from haystack_integrations.components.embedders.sentence_transformers import (
131+
SentenceTransformersDocumentEmbedder,
132+
SentenceTransformersTextEmbedder,
133+
)
134+
from haystack.components.writers import DocumentWriter
135+
from haystack.dataclasses import ChatMessage
136+
from haystack.document_stores.types import DuplicatePolicy
137+
from haystack_integrations.components.retrievers.elasticsearch import (
138+
ElasticsearchHybridRetriever,
139+
)
140+
from haystack_integrations.document_stores.elasticsearch import (
141+
ElasticsearchDocumentStore,
142+
)
143+
144+
document_store = ElasticsearchDocumentStore(hosts="http://localhost:9200/")
145+
146+
model = "sentence-transformers/all-MiniLM-L6-v2"
147+
148+
documents = [
149+
Document(content="There are over 7,000 languages spoken around the world today."),
150+
Document(
151+
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
152+
),
153+
Document(
154+
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
155+
),
156+
]
157+
158+
# Indexing Pipeline
159+
indexing_pipeline = Pipeline()
160+
indexing_pipeline.add_component(
161+
"doc_embedder",
162+
SentenceTransformersDocumentEmbedder(model=model),
163+
)
164+
indexing_pipeline.add_component(
165+
"doc_writer",
166+
DocumentWriter(document_store=document_store, policy=DuplicatePolicy.SKIP),
167+
)
168+
indexing_pipeline.connect("doc_embedder", "doc_writer")
169+
indexing_pipeline.run({"doc_embedder": {"documents": documents}})
170+
171+
# Query Pipeline
172+
prompt_template = [
173+
ChatMessage.from_user(
174+
"""
175+
Given these documents, answer the question.\nDocuments:
176+
{% for doc in documents %}
177+
{{ doc.content }}
178+
{% endfor %}
179+
180+
\nQuestion: {{question}}
181+
\nAnswer:
182+
""",
183+
),
184+
]
185+
186+
embedder = SentenceTransformersTextEmbedder(model=model)
187+
retriever = ElasticsearchHybridRetriever(
188+
document_store=document_store,
189+
embedder=embedder,
190+
top_k_bm25=3,
191+
top_k_embedding=3,
192+
join_mode="reciprocal_rank_fusion",
193+
)
194+
195+
query_pipeline = Pipeline()
196+
query_pipeline.add_component("retriever", retriever)
197+
query_pipeline.add_component(
198+
"prompt_builder",
199+
ChatPromptBuilder(template=prompt_template, required_variables="*"),
200+
)
201+
query_pipeline.add_component("llm", OpenAIChatGenerator())
202+
query_pipeline.connect("retriever.documents", "prompt_builder.documents")
203+
query_pipeline.connect("prompt_builder.prompt", "llm.messages")
204+
205+
question = "How many languages are spoken around the world today?"
206+
result = query_pipeline.run(
207+
{
208+
"retriever": {"query": question},
209+
"prompt_builder": {"question": question},
210+
},
211+
)
212+
213+
print(result["llm"]["replies"][0].text)
214+
```

‎docs-website/sidebars.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -575,6 +575,7 @@ export default {
575575
'pipeline-components/retrievers/chromaqueryretriever',
576576
'pipeline-components/retrievers/elasticsearchbm25retriever',
577577
'pipeline-components/retrievers/elasticsearchembeddingretriever',
578+
'pipeline-components/retrievers/elasticsearchhybridretriever',
578579
'pipeline-components/retrievers/elasticsearchsqlretriever',
579580
'pipeline-components/retrievers/faissembeddingretriever',
580581
'pipeline-components/retrievers/falkordbcypherretriever',

‎docs-website/versioned_docs/version-2.30/pipeline-components/retrievers.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,7 @@ For details on how to initialize and use a Retriever in a pipeline, see the docu
167167
| [CogneeRetriever](retrievers/cogneeretriever.mdx) | Retrieves memories from a CogneeMemoryStore and returns them as system ChatMessage objects. |
168168
| [ElasticsearchEmbeddingRetriever](retrievers/elasticsearchembeddingretriever.mdx) | An embedding-based Retriever compatible with the Elasticsearch Document Store. |
169169
| [ElasticsearchBM25Retriever](retrievers/elasticsearchbm25retriever.mdx) | A keyword-based Retriever that fetches Documents matching a query from the Elasticsearch Document Store. |
170+
| [ElasticsearchHybridRetriever](retrievers/elasticsearchhybridretriever.mdx) | A SuperComponent that combines BM25 and embedding-based retrieval from the Elasticsearch Document Store. |
170171
| [ElasticsearchSQLRetriever](retrievers/elasticsearchsqlretriever.mdx) | Executes raw Elasticsearch SQL queries against an Elasticsearch Document Store and returns the raw JSON response. |
171172
| [FAISSEmbeddingRetriever](retrievers/faissembeddingretriever.mdx) | An embedding-based Retriever compatible with the FAISSDocumentStore. |
172173
| [FalkorDBCypherRetriever](retrievers/falkordbcypherretriever.mdx) | A Retriever that executes arbitrary OpenCypher queries against a FalkorDB Document Store. |

0 commit comments

Comments
 (0)