Skip to content

Commit 58f0578

Browse files
sjrlsjrl
authored
feat: Add docs for twelvelabs (#11769)
1 parent 777c1a5 commit 58f0578

12 files changed

Lines changed: 758 additions & 0 deletions

File tree

‎docs-website/docs/pipeline-components/converters.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,5 +38,6 @@ Use various Converters to extract data from files in different formats and cast
3838
| [PyPDFToDocument](converters/pypdftodocument.mdx) | Converts PDF files to documents. |
3939
| [TikaDocumentConverter](converters/tikadocumentconverter.mdx) | Converts various file types to documents using Apache Tika. |
4040
| [TextFileToDocument](converters/textfiletodocument.mdx) | Converts text files to documents. |
41+
| [TwelveLabsVideoConverter](converters/twelvelabsvideoconverter.mdx) | Converts videos to documents using the TwelveLabs Pegasus video-language model. |
4142
| [UnstructuredFileConverter](converters/unstructuredfileconverter.mdx) | Converts text files and directories to a document. |
4243
| [XLSXToDocument](converters/xlsxtodocument.mdx) | Converts Excel files into documents. |

‎docs-website/docs/pipeline-components/converters/twelvelabsvideoconverter.mdx‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
---
2+
title: "TwelveLabsVideoConverter"
3+
id: twelvelabsvideoconverter
4+
slug: "/twelvelabsvideoconverter"
5+
description: "`TwelveLabsVideoConverter` converts videos to Haystack Documents using the TwelveLabs Pegasus video-language model. Pegasus analyzes each video on the fly — its visuals and its own audio (via ASR) — and returns text, so each video becomes a Document whose content is the analysis."
6+
---
7+
8+
# TwelveLabsVideoConverter
9+
10+
`TwelveLabsVideoConverter` converts videos to Haystack Documents using the TwelveLabs Pegasus video-language model. Pegasus analyzes each video on the fly — its visuals **and** its own audio (via ASR) — and returns text, so each source video becomes one Document whose content is Pegasus's analysis (for example, a description plus a transcript). There is no frame extraction or separate transcription step.
11+
12+
<div className="key-value-table">
13+
14+
| | |
15+
| --- | --- |
16+
| **Most common position in a pipeline** | At the beginning of an indexing pipeline, before [PreProcessors](../preprocessors.mdx) or an embedder |
17+
| **Mandatory init variables** | `api_key`: The TwelveLabs API key. Can be set with `TWELVELABS_API_KEY` env var. |
18+
| **Mandatory run variables** | `sources`: A list of video URLs or local file paths |
19+
| **Output variables** | `documents`: A list of documents |
20+
| **API reference** | [TwelveLabs](/reference/integrations-twelvelabs) |
21+
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/twelvelabs |
22+
| **Package name** | `twelvelabs-haystack` |
23+
24+
</div>
25+
26+
## Overview
27+
28+
The `TwelveLabsVideoConverter` takes a list of video sources and produces one [`Document`](../../concepts/data-classes.mdx#document) per source, with the Document content set to Pegasus's text analysis. Sources may be publicly accessible direct video URLs or local file paths (uploaded to TwelveLabs, up to 200 MB). Sources that fail to process are skipped with a warning logged, so one bad source does not fail the whole batch.
29+
30+
Each produced Document carries metadata about the request, including `source`, `asset_id`, `analysis_id`, `model`, and `provider`. The default model is `pegasus1.5`.
31+
32+
You can steer the analysis with a custom `prompt` and tune `temperature` and `max_tokens`.
33+
34+
To start using this integration with Haystack, install the package with:
35+
36+
```shell
37+
pip install twelvelabs-haystack
38+
```
39+
40+
The component uses a `TWELVELABS_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`. To get an API key, head to [playground.twelvelabs.io](https://playground.twelvelabs.io).
41+
42+
## Usage
43+
44+
### On its own
45+
46+
```python
47+
from haystack_integrations.components.converters.twelvelabs import (
48+
TwelveLabsVideoConverter,
49+
)
50+
51+
converter = TwelveLabsVideoConverter()
52+
result = converter.run(sources=["https://example.com/clip.mp4"])
53+
54+
document = result["documents"][0]
55+
print(document.content) # Pegasus's description + transcript of the video
56+
print(document.meta) # includes source, asset_id, analysis_id, model, provider
57+
```
58+
59+
:::info
60+
We recommend setting `TWELVELABS_API_KEY` as an environment variable instead of setting it as a parameter.
61+
:::
62+
63+
### With a custom prompt
64+
65+
```python
66+
from haystack_integrations.components.converters.twelvelabs import (
67+
TwelveLabsVideoConverter,
68+
)
69+
70+
converter = TwelveLabsVideoConverter(
71+
prompt="Summarize this video in three bullet points and list any products shown.",
72+
temperature=0.2,
73+
max_tokens=1024,
74+
)
75+
result = converter.run(sources=["https://example.com/clip.mp4"])
76+
print(result["documents"][0].content)
77+
```
78+
79+
### In a pipeline
80+
81+
This indexing pipeline analyzes videos with Pegasus, embeds the resulting analysis with the [`TwelveLabsDocumentEmbedder`](../embedders/twelvelabsdocumentembedder.mdx), and writes the documents to a document store:
82+
83+
```python
84+
from haystack import Pipeline
85+
from haystack.document_stores.in_memory import InMemoryDocumentStore
86+
from haystack.components.writers import DocumentWriter
87+
from haystack_integrations.components.converters.twelvelabs import (
88+
TwelveLabsVideoConverter,
89+
)
90+
from haystack_integrations.components.embedders.twelvelabs import (
91+
TwelveLabsDocumentEmbedder,
92+
)
93+
94+
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
95+
96+
indexing_pipeline = Pipeline()
97+
indexing_pipeline.add_component("converter", TwelveLabsVideoConverter())
98+
indexing_pipeline.add_component("embedder", TwelveLabsDocumentEmbedder())
99+
indexing_pipeline.add_component("writer", DocumentWriter(document_store=document_store))
100+
indexing_pipeline.connect("converter", "embedder")
101+
indexing_pipeline.connect("embedder", "writer")
102+
103+
indexing_pipeline.run({"converter": {"sources": ["https://example.com/clip.mp4"]}})
104+
```
105+
106+
### Attaching metadata
107+
108+
Pass a single dictionary to apply metadata to all output Documents, or a list to set metadata per source:
109+
110+
```python
111+
from haystack_integrations.components.converters.twelvelabs import (
112+
TwelveLabsVideoConverter,
113+
)
114+
115+
converter = TwelveLabsVideoConverter()
116+
117+
# Same metadata for all sources
118+
result = converter.run(
119+
sources=["https://example.com/a.mp4", "https://example.com/b.mp4"],
120+
meta={"campaign": "demo"},
121+
)
122+
123+
# Per-source metadata
124+
result = converter.run(
125+
sources=["https://example.com/a.mp4", "https://example.com/b.mp4"],
126+
meta=[{"title": "Clip A"}, {"title": "Clip B"}],
127+
)
128+
```

‎docs-website/docs/pipeline-components/embedders.mdx‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,8 @@ These are the Embedders available in Haystack:
5656
| [SentenceTransformersSparseDocumentEmbedder](embedders/sentencetransformerssparsedocumentembedder.mdx) | Enriches a list of documents with their sparse embeddings using Sentence Transformers models. |
5757
| [STACKITTextEmbedder](embedders/stackittextembedder.mdx) | Enables text embedding using the STACKIT API. |
5858
| [STACKITDocumentEmbedder](embedders/stackitdocumentembedder.mdx) | Enables document embedding using the STACKIT API. |
59+
| [TwelveLabsTextEmbedder](embedders/twelvelabstextembedder.mdx) | Embeds a simple string (such as a query) with the TwelveLabs Marengo multimodal model. Requires an API key from TwelveLabs. |
60+
| [TwelveLabsDocumentEmbedder](embedders/twelvelabsdocumentembedder.mdx) | Embeds a list of documents with the TwelveLabs Marengo multimodal model. Requires an API key from TwelveLabs. |
5961
| [VertexAITextEmbedder](embedders/vertexaitextembedder.mdx) | Computes embeddings for text (such as a query) using models through VertexAI Embeddings API. **_This integration will be deprecated soon. We recommend using [GoogleGenAITextEmbedder](embedders/googlegenaitextembedder.mdx) integration instead._** |
6062
| [VertexAIDocumentEmbedder](embedders/vertexaidocumentembedder.mdx) | Computes embeddings for documents using models through VertexAI Embeddings API. **_This integration will be deprecated soon. We recommend using [GoogleGenAIDocumentEmbedder](embedders/googlegenaidocumentembedder.mdx) integration instead._** |
6163
| [VLLMTextEmbedder](embedders/vllmtextembedder.mdx) | Computes the embeddings of a string using models served with vLLM. |

‎docs-website/docs/pipeline-components/embedders/twelvelabsdocumentembedder.mdx‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
---
2+
title: "TwelveLabsDocumentEmbedder"
3+
id: twelvelabsdocumentembedder
4+
slug: "/twelvelabsdocumentembedder"
5+
description: "This component computes the embeddings of a list of documents using the TwelveLabs Marengo multimodal embedding model and stores the obtained vectors in the embedding field of each document. The vectors are necessary to perform embedding retrieval on a collection of documents."
6+
---
7+
8+
# TwelveLabsDocumentEmbedder
9+
10+
This component computes the embeddings of a list of documents using the TwelveLabs Marengo multimodal embedding model and stores the obtained vectors in the embedding field of each document. The vectors computed by this component are necessary to perform embedding retrieval on a collection of documents. At retrieval time, the vector representing the query is compared with those of the documents to find the most similar or relevant documents.
11+
12+
<div className="key-value-table">
13+
14+
| | |
15+
| --- | --- |
16+
| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline |
17+
| **Mandatory init variables** | `api_key`: The TwelveLabs API key. Can be set with `TWELVELABS_API_KEY` env var. |
18+
| **Mandatory run variables** | `documents`: A list of documents |
19+
| **Output variables** | `documents`: A list of documents (enriched with embeddings) <br /> <br />`meta`: A dictionary of metadata |
20+
| **API reference** | [TwelveLabs](/reference/integrations-twelvelabs) |
21+
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/twelvelabs |
22+
| **Package name** | `twelvelabs-haystack` |
23+
24+
</div>
25+
26+
## Overview
27+
28+
`TwelveLabsDocumentEmbedder` enriches each document with an embedding of its content. To embed a string, use the [`TwelveLabsTextEmbedder`](twelvelabstextembedder.mdx). The default model is `marengo3.0`.
29+
30+
Because Marengo embeds text, images, audio, and video into a single shared space, these embeddings support cross-modal retrieval.
31+
32+
To start using this integration with Haystack, install the package with:
33+
34+
```shell
35+
pip install twelvelabs-haystack
36+
```
37+
38+
The component uses a `TWELVELABS_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`:
39+
40+
```python
41+
from haystack.utils import Secret
42+
from haystack_integrations.components.embedders.twelvelabs import (
43+
TwelveLabsDocumentEmbedder,
44+
)
45+
46+
embedder = TwelveLabsDocumentEmbedder(api_key=Secret.from_token("<your-api-key>"))
47+
```
48+
49+
To get an API key, head to [playground.twelvelabs.io](https://playground.twelvelabs.io).
50+
51+
### Embedding Metadata
52+
53+
Text documents often come with a set of metadata. If they are distinctive and semantically meaningful, you can embed them along with the text of the document to improve retrieval.
54+
55+
You can do this by passing the relevant meta field names with `meta_fields_to_embed`:
56+
57+
```python
58+
from haystack import Document
59+
from haystack_integrations.components.embedders.twelvelabs import (
60+
TwelveLabsDocumentEmbedder,
61+
)
62+
63+
doc = Document(content="some text", meta={"title": "relevant title", "page number": 18})
64+
65+
embedder = TwelveLabsDocumentEmbedder(meta_fields_to_embed=["title"])
66+
67+
docs_w_embeddings = embedder.run(documents=[doc])["documents"]
68+
```
69+
70+
## Usage
71+
72+
### On its own
73+
74+
Here is how you can use the component on its own:
75+
76+
```python
77+
from haystack import Document
78+
from haystack_integrations.components.embedders.twelvelabs import (
79+
TwelveLabsDocumentEmbedder,
80+
)
81+
82+
doc = Document(content="a cat playing piano")
83+
84+
document_embedder = TwelveLabsDocumentEmbedder()
85+
86+
result = document_embedder.run(documents=[doc])
87+
print(result["documents"][0].embedding)
88+
89+
# [-0.043398008, -0.025287028, -0.0061081843, ...]
90+
```
91+
92+
:::info
93+
We recommend setting `TWELVELABS_API_KEY` as an environment variable instead of setting it as a parameter.
94+
:::
95+
96+
### In a pipeline
97+
98+
```python
99+
from haystack import Document, Pipeline
100+
from haystack.document_stores.in_memory import InMemoryDocumentStore
101+
from haystack.components.writers import DocumentWriter
102+
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
103+
from haystack_integrations.components.embedders.twelvelabs import (
104+
TwelveLabsDocumentEmbedder,
105+
TwelveLabsTextEmbedder,
106+
)
107+
108+
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
109+
110+
documents = [
111+
Document(content="a cat playing piano"),
112+
Document(content="a dog catching a frisbee at the beach"),
113+
Document(content="a timelapse of a city skyline at night"),
114+
]
115+
116+
indexing_pipeline = Pipeline()
117+
indexing_pipeline.add_component("embedder", TwelveLabsDocumentEmbedder())
118+
indexing_pipeline.add_component("writer", DocumentWriter(document_store=document_store))
119+
indexing_pipeline.connect("embedder", "writer")
120+
indexing_pipeline.run({"embedder": {"documents": documents}})
121+
122+
query_pipeline = Pipeline()
123+
query_pipeline.add_component("text_embedder", TwelveLabsTextEmbedder())
124+
query_pipeline.add_component(
125+
"retriever",
126+
InMemoryEmbeddingRetriever(document_store=document_store),
127+
)
128+
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
129+
130+
result = query_pipeline.run({"text_embedder": {"text": "feline making music"}})
131+
print(result["retriever"]["documents"][0].content)
132+
133+
# a cat playing piano
134+
```

‎docs-website/docs/pipeline-components/embedders/twelvelabstextembedder.mdx‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
---
2+
title: "TwelveLabsTextEmbedder"
3+
id: twelvelabstextembedder
4+
slug: "/twelvelabstextembedder"
5+
description: "This component transforms a string into a vector using the TwelveLabs Marengo multimodal embedding model. Because Marengo embeds text, images, audio, and video into one shared vector space, the resulting embeddings support cross-modal retrieval. Use this component to embed a query before searching with an embedding Retriever."
6+
---
7+
8+
# TwelveLabsTextEmbedder
9+
10+
This component transforms a string into a vector using the TwelveLabs Marengo multimodal embedding model. Because Marengo embeds text, images, audio, and video into one shared vector space, the resulting embeddings support cross-modal retrieval (for example, searching a video collection with a text query). Use this component to embed a query before searching with an embedding Retriever.
11+
12+
<div className="key-value-table">
13+
14+
| | |
15+
| --- | --- |
16+
| **Most common position in a pipeline** | Before an embedding [Retriever](../retrievers.mdx) in a query/RAG pipeline |
17+
| **Mandatory init variables** | `api_key`: The TwelveLabs API key. Can be set with `TWELVELABS_API_KEY` env var. |
18+
| **Mandatory run variables** | `text`: A string |
19+
| **Output variables** | `embedding`: A list of float numbers <br /> <br />`meta`: A dictionary of metadata |
20+
| **API reference** | [TwelveLabs](/reference/integrations-twelvelabs) |
21+
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/twelvelabs |
22+
| **Package name** | `twelvelabs-haystack` |
23+
24+
</div>
25+
26+
## Overview
27+
28+
`TwelveLabsTextEmbedder` embeds a simple string (such as a query) into a vector. For embedding lists of documents, use the [`TwelveLabsDocumentEmbedder`](twelvelabsdocumentembedder.mdx), which enriches each document with the computed embedding. The default model is `marengo3.0`.
29+
30+
Because Marengo embeds into a single shared space, embeddings produced from text are directly comparable (cosine similarity) with embeddings of images, audio, and video from the same model.
31+
32+
To start using this integration with Haystack, install the package with:
33+
34+
```shell
35+
pip install twelvelabs-haystack
36+
```
37+
38+
The component uses a `TWELVELABS_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`:
39+
40+
```python
41+
from haystack.utils import Secret
42+
from haystack_integrations.components.embedders.twelvelabs import TwelveLabsTextEmbedder
43+
44+
embedder = TwelveLabsTextEmbedder(api_key=Secret.from_token("<your-api-key>"))
45+
```
46+
47+
To get an API key, head to [playground.twelvelabs.io](https://playground.twelvelabs.io).
48+
49+
## Usage
50+
51+
### On its own
52+
53+
Here is how you can use the component on its own:
54+
55+
```python
56+
from haystack_integrations.components.embedders.twelvelabs import TwelveLabsTextEmbedder
57+
58+
text_embedder = TwelveLabsTextEmbedder()
59+
60+
result = text_embedder.run(text="a cat playing piano")
61+
print(result["embedding"])
62+
63+
# [-0.043398008, -0.025287028, -0.0061081843, ...]
64+
print(result["meta"])
65+
66+
# {'model': 'marengo3.0'}
67+
```
68+
69+
:::info
70+
We recommend setting `TWELVELABS_API_KEY` as an environment variable instead of setting it as a parameter.
71+
:::
72+
73+
### In a pipeline
74+
75+
```python
76+
from haystack import Document, Pipeline
77+
from haystack.document_stores.in_memory import InMemoryDocumentStore
78+
from haystack.components.writers import DocumentWriter
79+
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
80+
from haystack_integrations.components.embedders.twelvelabs import (
81+
TwelveLabsDocumentEmbedder,
82+
TwelveLabsTextEmbedder,
83+
)
84+
85+
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
86+
87+
documents = [
88+
Document(content="a cat playing piano"),
89+
Document(content="a dog catching a frisbee at the beach"),
90+
Document(content="a timelapse of a city skyline at night"),
91+
]
92+
93+
indexing_pipeline = Pipeline()
94+
indexing_pipeline.add_component("embedder", TwelveLabsDocumentEmbedder())
95+
indexing_pipeline.add_component("writer", DocumentWriter(document_store=document_store))
96+
indexing_pipeline.connect("embedder", "writer")
97+
indexing_pipeline.run({"embedder": {"documents": documents}})
98+
99+
query_pipeline = Pipeline()
100+
query_pipeline.add_component("text_embedder", TwelveLabsTextEmbedder())
101+
query_pipeline.add_component(
102+
"retriever",
103+
InMemoryEmbeddingRetriever(document_store=document_store),
104+
)
105+
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
106+
107+
result = query_pipeline.run({"text_embedder": {"text": "feline making music"}})
108+
print(result["retriever"]["documents"][0].content)
109+
110+
# a cat playing piano
111+
```

0 commit comments

Comments
 (0)