Skip to content

Commit 26e229b

Browse files
petertothkacperlukawski
petertoth
and
kacperlukawski
authored
Add TopK integration page (#503)
* Add TopK integration page * feedback 1 * Update integrations/topk.md Co-authored-by: Kacper Łukawski <kacperlukawski@users.noreply.github.com> * Update integrations/topk.md Co-authored-by: Kacper Łukawski <kacperlukawski@users.noreply.github.com> --------- Co-authored-by: Kacper Łukawski <kacperlukawski@users.noreply.github.com>
1 parent 3432501 commit 26e229b

2 files changed

Lines changed: 284 additions & 0 deletions

File tree

‎integrations/topk.md‎

Lines changed: 262 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,262 @@
1+
---
2+
layout: integration
3+
name: TopK
4+
description: Use the TopK database with Haystack
5+
authors:
6+
- name: TopK
7+
socials:
8+
github: topk-io
9+
twitter: topk_io
10+
linkedin: https://www.linkedin.com/company/topkio/
11+
pypi: https://pypi.org/project/topk-haystack/
12+
repo: https://github.com/topk-io/topk-haystack
13+
type: Document Store
14+
report_issue: https://github.com/topk-io/topk-haystack/issues
15+
logo: /logos/topk.svg
16+
version: Haystack 2.0
17+
toc: true
18+
---
19+
20+
### **Table of Contents**
21+
22+
- [Overview](#overview)
23+
- [Installation](#installation)
24+
- [Prerequisites](#prerequisites)
25+
- [Quick start](#quick-start)
26+
- [RAG pipeline](#rag-pipeline)
27+
- [Retrievers](#retrievers)
28+
- [Multi-tenant workloads](#multi-tenant-workloads)
29+
- [Resources](#resources)
30+
- [License](#license)
31+
32+
## Overview
33+
34+
[TopK](https://topk.io) is a hosted database powering fast vector search, keyword search (BM25), hybrid search and multi-vector search.
35+
36+
This integration ships with TopK Document Store and five retrievers you can use to best fit your use case:
37+
38+
- [`TopKSemanticRetriever`](#semantic-retriever) — semantic search with server-side embedding, no embedder component needed
39+
- [`TopKBM25Retriever`](#bm25-keyword-retriever) — keyword search using BM25 scoring
40+
- [`TopKEmbeddingRetriever`](#dense-vector-retriever) — dense vector search with your own embedding model
41+
- [`TopKHybridRetriever`](#hybrid-retriever) — combines vector and BM25 scores in a single query
42+
- [`TopKMetadataRetriever`](#metadata-filtering-retriever) — filter documents by metadata fields
43+
44+
## Installation
45+
46+
```bash
47+
pip install topk-haystack
48+
```
49+
50+
## Prerequisites
51+
52+
Before you set up TopK Document Store in Haystack, you'll need:
53+
54+
- TopK API key — get one from the [TopK console](https://console.topk.io/api-key)
55+
- Region identifier — see the list of [available regions](https://docs.topk.io/regions)
56+
57+
## Quick start
58+
59+
The fastest way to build a RAG pipeline with TopK is the `TopKSemanticRetriever`. TopK handles embedding server-side — no embedder component needed:
60+
61+
```python
62+
import os
63+
from haystack import Document, Pipeline
64+
from haystack.components.writers import DocumentWriter
65+
from haystack.utils import Secret
66+
67+
from haystack_integrations.components.topk import TopKSemanticRetriever
68+
from haystack_integrations.document_stores.topk import TopKDocumentStore
69+
70+
store = TopKDocumentStore(
71+
api_key=Secret.from_env_var("TOPK_API_KEY"),
72+
region="aws-us-east-1-elastica",
73+
collection_name="my-docs",
74+
)
75+
76+
# Index
77+
indexing = Pipeline()
78+
indexing.add_component("writer", DocumentWriter(document_store=store))
79+
indexing.run({"writer": {"documents": [
80+
Document(content="Rust guarantees memory safety without a garbage collector."),
81+
Document(content="Python is known for readable syntax and scientific libraries."),
82+
]}})
83+
84+
# Query — no embedder needed
85+
retriever = TopKSemanticRetriever(document_store=store, top_k=3)
86+
pipeline = Pipeline()
87+
pipeline.add_component("retriever", retriever)
88+
result = pipeline.run({"retriever": {"query": "memory safe systems programming"}})
89+
90+
for doc in result["retriever"]["documents"]:
91+
print(f"[{doc.score:.3f}] {doc.content}")
92+
```
93+
94+
## RAG pipeline
95+
96+
```python
97+
from haystack.components.builders import ChatPromptBuilder
98+
from haystack.components.generators.chat import OpenAIChatGenerator
99+
from haystack.dataclasses import ChatMessage
100+
from haystack.utils import Secret
101+
102+
from haystack_integrations.components.topk import TopKSemanticRetriever
103+
from haystack_integrations.document_stores.topk import TopKDocumentStore
104+
105+
store = TopKDocumentStore(
106+
api_key=Secret.from_env_var("TOPK_API_KEY"),
107+
region="aws-us-east-1-elastica",
108+
collection_name="my-docs",
109+
)
110+
111+
template = [
112+
ChatMessage.from_system("Answer using only the context below.\n{% for doc in documents %}{{ doc.content }}\n{% endfor %}"),
113+
ChatMessage.from_user("{{ question }}"),
114+
]
115+
116+
rag = Pipeline()
117+
rag.add_component("retriever", TopKSemanticRetriever(document_store=store, top_k=5))
118+
rag.add_component("prompt", ChatPromptBuilder(template=template))
119+
rag.add_component("llm", OpenAIChatGenerator(model="gpt-4o-mini"))
120+
rag.connect("retriever.documents", "prompt.documents")
121+
rag.connect("prompt.prompt", "llm.messages")
122+
123+
result = rag.run({
124+
"retriever": {"query": "What makes Rust memory safe?"},
125+
"prompt": {"question": "What makes Rust memory safe?"},
126+
})
127+
print(result["llm"]["replies"][0].text)
128+
```
129+
130+
## Retrievers
131+
132+
### Semantic Retriever
133+
134+
TopK handles embedding server-side — no embedder component needed. Pass a plain text query and TopK returns semantically relevant documents:
135+
136+
```python
137+
from haystack_integrations.components.topk import TopKSemanticRetriever
138+
139+
retriever = TopKSemanticRetriever(document_store=store, top_k=5)
140+
pipeline = Pipeline()
141+
pipeline.add_component("retriever", retriever)
142+
result = pipeline.run({"retriever": {"query": "memory safe systems programming"}})
143+
```
144+
145+
### BM25 Keyword Retriever
146+
147+
```python
148+
from haystack_integrations.components.topk import TopKBM25Retriever
149+
150+
retriever = TopKBM25Retriever(document_store=store, top_k=5)
151+
pipeline = Pipeline()
152+
pipeline.add_component("retriever", retriever)
153+
result = pipeline.run({"retriever": {"query": "garbage collector memory"}})
154+
```
155+
156+
### Dense Vector Retriever
157+
158+
```python
159+
from haystack import Pipeline
160+
from haystack.components.embedders import SentenceTransformersDocumentEmbedder, SentenceTransformersTextEmbedder
161+
from haystack.components.writers import DocumentWriter
162+
from haystack.utils import Secret
163+
164+
from haystack_integrations.components.topk import TopKEmbeddingRetriever
165+
from haystack_integrations.document_stores.topk import TopKDocumentStore
166+
167+
MODEL = "sentence-transformers/all-MiniLM-L6-v2"
168+
169+
store = TopKDocumentStore(
170+
api_key=Secret.from_env_var("TOPK_API_KEY"),
171+
region="aws-us-east-1-elastica",
172+
collection_name="my-docs",
173+
embedding_dim=384, # must match the model
174+
)
175+
176+
# Indexing
177+
indexing = Pipeline()
178+
indexing.add_component("embedder", SentenceTransformersDocumentEmbedder(model=MODEL))
179+
indexing.add_component("writer", DocumentWriter(document_store=store))
180+
indexing.connect("embedder.documents", "writer.documents")
181+
182+
# Querying
183+
query_pipeline = Pipeline()
184+
query_pipeline.add_component("embedder", SentenceTransformersTextEmbedder(model=MODEL))
185+
query_pipeline.add_component("retriever", TopKEmbeddingRetriever(document_store=store, top_k=5))
186+
query_pipeline.connect("embedder.embedding", "retriever.query_embedding")
187+
188+
result = query_pipeline.run({"embedder": {"text": "type safe programming"}})
189+
```
190+
191+
### Hybrid Retriever
192+
193+
Combines dense vector similarity and BM25 keyword scoring in a single query, ranked by the sum of both scores.
194+
195+
```python
196+
from haystack import Pipeline
197+
from haystack.components.embedders import SentenceTransformersTextEmbedder
198+
from haystack_integrations.components.topk import TopKHybridRetriever
199+
200+
MODEL = "sentence-transformers/all-MiniLM-L6-v2"
201+
202+
retriever = TopKHybridRetriever(document_store=store, top_k=5)
203+
query_pipeline = Pipeline()
204+
query_pipeline.add_component("embedder", SentenceTransformersTextEmbedder(model=MODEL))
205+
query_pipeline.add_component("retriever", retriever)
206+
query_pipeline.connect("embedder.embedding", "retriever.query_embedding")
207+
208+
result = query_pipeline.run({
209+
"embedder": {"text": "concurrent network services"},
210+
"retriever": {"query": "goroutines channels"},
211+
})
212+
```
213+
214+
### Metadata Filtering Retriever
215+
216+
```python
217+
from haystack_integrations.components.topk import TopKMetadataRetriever
218+
219+
retriever = TopKMetadataRetriever(document_store=store, top_k=5)
220+
pipeline = Pipeline()
221+
pipeline.add_component("retriever", retriever)
222+
223+
result = pipeline.run({"retriever": {"filters": {
224+
"operator": "AND",
225+
"conditions": [
226+
{"field": "meta.language", "operator": "==", "value": "en"},
227+
{"field": "meta.year", "operator": ">=", "value": 2020},
228+
],
229+
}}})
230+
```
231+
232+
Supported filter operators: `==`, `!=`, `>`, `>=`, `<`, `<=`, `in`, `not in`, `AND`, `OR`, `NOT`.
233+
234+
## Multi-tenant workloads
235+
236+
Use the `partition` parameter to scope all reads and writes to a logical partition.
237+
Different partitions in the same collection are fully isolated, enabling multi-tenant
238+
workloads that scale to billions of documents.
239+
240+
```python
241+
store_a = TopKDocumentStore(
242+
api_key=Secret.from_env_var("TOPK_API_KEY"),
243+
region="aws-us-east-1-elastica",
244+
collection_name="shared",
245+
partition="tenant-a",
246+
)
247+
store_b = TopKDocumentStore(
248+
api_key=Secret.from_env_var("TOPK_API_KEY"),
249+
region="aws-us-east-1-elastica",
250+
collection_name="shared",
251+
partition="tenant-b",
252+
)
253+
```
254+
255+
## Resources
256+
257+
- [Benchmarks](https://www.topk.io/benchmarks)
258+
- [Pricing](https://www.topk.io/pricing)
259+
260+
## License
261+
262+
`topk-haystack` is distributed under the terms of the [Apache-2.0](https://spdx.org/licenses/Apache-2.0.html) license.

‎logos/topk.svg‎

Lines changed: 22 additions & 0 deletions
Loading

0 commit comments

Comments
 (0)