Skip to content

Commit 8e64b9b

Browse files
Solaris-starjulian-rischclaude
authored
docs: add TavilyFetcher page under pipeline-components fetchers (#12079)
Co-authored-by: Julian Risch <julian.risch@deepset.ai> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 9c8e299 commit 8e64b9b

6 files changed

Lines changed: 280 additions & 0 deletions

File tree

‎docs-website/docs/pipeline-components/fetchers.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,3 +15,4 @@ Fetchers retrieve content from external sources – URLs, web crawls, or cloud s
1515
| [GoogleDriveFetcher](fetchers/googledrivefetcher.mdx) | Fetches the full content of Google Drive files via the Drive API v3 and returns it as ByteStreams. |
1616
| [LinkContentFetcher](fetchers/linkcontentfetcher.mdx) | Fetches the contents of the URLs you give it so you can use them as data for your pipelines. |
1717
| [MSSharePointFetcher](fetchers/mssharepointfetcher.mdx) | Fetches the full content of Microsoft SharePoint and OneDrive items via the Microsoft Graph API and returns it as ByteStreams. |
18+
| [TavilyFetcher](fetchers/tavilyfetcher.mdx) | Extracts and parses the content of the URLs you give it with the Tavily Extract API and returns it as Documents. |

‎docs-website/docs/pipeline-components/fetchers/tavilyfetcher.mdx‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
---
2+
title: "TavilyFetcher"
3+
id: tavilyfetcher
4+
slug: "/tavilyfetcher"
5+
description: "Use Tavily Extract to fetch and parse content from URLs as Haystack Documents. Unlike web search, it retrieves content from the URLs you provide rather than discovering them via a query."
6+
---
7+
8+
# TavilyFetcher
9+
10+
Use Tavily Extract to fetch and parse content from URLs as Haystack Documents. Unlike web search, it retrieves content from the URLs you provide rather than discovering them via a query.
11+
12+
<div className="key-value-table">
13+
14+
| | |
15+
| --- | --- |
16+
| **Most common position in a pipeline** | In indexing or query pipelines as the data fetching step |
17+
| **Mandatory init variables** | `api_key`: The Tavily API key. Can be set with the `TAVILY_API_KEY` env var. |
18+
| **Mandatory run variables** | `urls`: A list of URLs (strings) to extract content from (max 20 per request) |
19+
| **Output variables** | `documents`: A list of [Documents](../../concepts/data-classes.mdx)<br />`meta`: Request-level metadata (`response_time`, `usage`, `request_id`, `failed_results`) |
20+
| **API reference** | [Tavily](/reference/integrations-tavily) |
21+
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/tavily |
22+
| **Package name** | `tavily-haystack` |
23+
24+
</div>
25+
26+
## Overview
27+
28+
`TavilyFetcher` wraps the [Tavily Extract API](https://docs.tavily.com/documentation/api-reference/endpoint/extract) to retrieve and parse web page content from one or more specified URLs. PDF URLs are also supported. Each successful URL becomes a Haystack `Document` with page content in `content` and metadata such as `url` (and optionally `images`) in `meta`.
29+
30+
This component is complementary to [`TavilyWebSearch`](../websearch/tavilywebsearch.mdx): search discovers URLs from a query, while `TavilyFetcher` extracts full content from URLs you already have.
31+
32+
### Extract parameters
33+
34+
You can control extraction behavior at initialization:
35+
36+
- `extract_depth`: `"basic"` (fast, lower cost) or `"advanced"` (more data including tables; higher latency and cost). Defaults to `"basic"`.
37+
- `include_images`: When `True`, image URLs are stored on each Document under `meta["images"]`. Defaults to `False`.
38+
- `extract_params`: Extra kwargs forwarded to the Tavily Extract API (for example `format`, `include_favicon`, `query`, `chunks_per_source`). See the [Tavily Extract API reference](https://docs.tavily.com/documentation/api-reference/endpoint/extract).
39+
40+
Of these, only `extract_params` can also be passed to `run()` to override it for a single call. Note that an `extract_params` dictionary passed to `run()` fully replaces the one set at initialization instead of being merged with it.
41+
42+
### Authorization
43+
44+
`TavilyFetcher` uses the `TAVILY_API_KEY` environment variable by default. You can also pass the key explicitly:
45+
46+
```python
47+
from haystack.utils import Secret
48+
from haystack_integrations.components.fetchers.tavily import TavilyFetcher
49+
50+
fetcher = TavilyFetcher(api_key=Secret.from_token("<your-api-key>"))
51+
```
52+
53+
To get an API key, sign up at [tavily.com](https://tavily.com).
54+
55+
### Installation
56+
57+
Install the Tavily integration with:
58+
59+
```shell
60+
pip install tavily-haystack
61+
```
62+
63+
## Usage
64+
65+
### On its own
66+
67+
```python
68+
from haystack_integrations.components.fetchers.tavily import TavilyFetcher
69+
70+
fetcher = TavilyFetcher(extract_depth="basic")
71+
72+
result = fetcher.run(urls=["https://docs.haystack.deepset.ai/docs/intro"])
73+
documents = result["documents"]
74+
meta = result["meta"]
75+
76+
for doc in documents:
77+
print(f"{doc.meta.get('url')}: {len(doc.content or '')} chars")
78+
79+
print("failed:", meta.get("failed_results"))
80+
```
81+
82+
### In a pipeline
83+
84+
Below is an example of an indexing pipeline that uses `TavilyFetcher` to extract documentation pages and store them in an `InMemoryDocumentStore`.
85+
86+
```python
87+
from haystack import Pipeline
88+
from haystack.document_stores.in_memory import InMemoryDocumentStore
89+
from haystack.components.preprocessors import DocumentSplitter
90+
from haystack.components.writers import DocumentWriter
91+
from haystack_integrations.components.fetchers.tavily import TavilyFetcher
92+
93+
document_store = InMemoryDocumentStore()
94+
95+
fetcher = TavilyFetcher(extract_depth="basic")
96+
splitter = DocumentSplitter(split_by="sentence", split_length=5)
97+
writer = DocumentWriter(document_store=document_store)
98+
99+
indexing_pipeline = Pipeline()
100+
indexing_pipeline.add_component("fetcher", fetcher)
101+
indexing_pipeline.add_component("splitter", splitter)
102+
indexing_pipeline.add_component("writer", writer)
103+
104+
indexing_pipeline.connect("fetcher.documents", "splitter.documents")
105+
indexing_pipeline.connect("splitter.documents", "writer.documents")
106+
107+
indexing_pipeline.run(
108+
data={
109+
"fetcher": {
110+
"urls": ["https://docs.haystack.deepset.ai/docs/intro"],
111+
},
112+
},
113+
)
114+
```
115+
116+
### Asynchronous execution
117+
118+
`TavilyFetcher` also supports asynchronous execution through `run_async()`:
119+
120+
```python
121+
import asyncio
122+
123+
from haystack_integrations.components.fetchers.tavily import TavilyFetcher
124+
125+
fetcher = TavilyFetcher()
126+
127+
128+
async def fetch():
129+
result = await fetcher.run_async(
130+
urls=["https://docs.haystack.deepset.ai/docs/intro"],
131+
)
132+
return result["documents"]
133+
134+
135+
documents = asyncio.run(fetch())
136+
```
137+
138+
The underlying clients are created lazily on the first call. To avoid the cold-start latency of the first call, you can call `warm_up()` explicitly.

‎docs-website/sidebars.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -415,6 +415,7 @@ export default {
415415
'pipeline-components/fetchers/googledrivefetcher',
416416
'pipeline-components/fetchers/linkcontentfetcher',
417417
'pipeline-components/fetchers/mssharepointfetcher',
418+
'pipeline-components/fetchers/tavilyfetcher',
418419
'pipeline-components/fetchers/external-integrations-fetchers',
419420
],
420421
},

‎docs-website/versioned_docs/version-3.0/pipeline-components/fetchers.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,3 +15,4 @@ Fetchers retrieve content from external sources – URLs, web crawls, or cloud s
1515
| [GoogleDriveFetcher](fetchers/googledrivefetcher.mdx) | Fetches the full content of Google Drive files via the Drive API v3 and returns it as ByteStreams. |
1616
| [LinkContentFetcher](fetchers/linkcontentfetcher.mdx) | Fetches the contents of the URLs you give it so you can use them as data for your pipelines. |
1717
| [MSSharePointFetcher](fetchers/mssharepointfetcher.mdx) | Fetches the full content of Microsoft SharePoint and OneDrive items via the Microsoft Graph API and returns it as ByteStreams. |
18+
| [TavilyFetcher](fetchers/tavilyfetcher.mdx) | Extracts and parses the content of the URLs you give it with the Tavily Extract API and returns it as Documents. |

‎docs-website/versioned_docs/version-3.0/pipeline-components/fetchers/tavilyfetcher.mdx‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
---
2+
title: "TavilyFetcher"
3+
id: tavilyfetcher
4+
slug: "/tavilyfetcher"
5+
description: "Use Tavily Extract to fetch and parse content from URLs as Haystack Documents. Unlike web search, it retrieves content from the URLs you provide rather than discovering them via a query."
6+
---
7+
8+
# TavilyFetcher
9+
10+
Use Tavily Extract to fetch and parse content from URLs as Haystack Documents. Unlike web search, it retrieves content from the URLs you provide rather than discovering them via a query.
11+
12+
<div className="key-value-table">
13+
14+
| | |
15+
| --- | --- |
16+
| **Most common position in a pipeline** | In indexing or query pipelines as the data fetching step |
17+
| **Mandatory init variables** | `api_key`: The Tavily API key. Can be set with the `TAVILY_API_KEY` env var. |
18+
| **Mandatory run variables** | `urls`: A list of URLs (strings) to extract content from (max 20 per request) |
19+
| **Output variables** | `documents`: A list of [Documents](../../concepts/data-classes.mdx)<br />`meta`: Request-level metadata (`response_time`, `usage`, `request_id`, `failed_results`) |
20+
| **API reference** | [Tavily](/reference/integrations-tavily) |
21+
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/tavily |
22+
| **Package name** | `tavily-haystack` |
23+
24+
</div>
25+
26+
## Overview
27+
28+
`TavilyFetcher` wraps the [Tavily Extract API](https://docs.tavily.com/documentation/api-reference/endpoint/extract) to retrieve and parse web page content from one or more specified URLs. PDF URLs are also supported. Each successful URL becomes a Haystack `Document` with page content in `content` and metadata such as `url` (and optionally `images`) in `meta`.
29+
30+
This component is complementary to [`TavilyWebSearch`](../websearch/tavilywebsearch.mdx): search discovers URLs from a query, while `TavilyFetcher` extracts full content from URLs you already have.
31+
32+
### Extract parameters
33+
34+
You can control extraction behavior at initialization:
35+
36+
- `extract_depth`: `"basic"` (fast, lower cost) or `"advanced"` (more data including tables; higher latency and cost). Defaults to `"basic"`.
37+
- `include_images`: When `True`, image URLs are stored on each Document under `meta["images"]`. Defaults to `False`.
38+
- `extract_params`: Extra kwargs forwarded to the Tavily Extract API (for example `format`, `include_favicon`, `query`, `chunks_per_source`). See the [Tavily Extract API reference](https://docs.tavily.com/documentation/api-reference/endpoint/extract).
39+
40+
Of these, only `extract_params` can also be passed to `run()` to override it for a single call. Note that an `extract_params` dictionary passed to `run()` fully replaces the one set at initialization instead of being merged with it.
41+
42+
### Authorization
43+
44+
`TavilyFetcher` uses the `TAVILY_API_KEY` environment variable by default. You can also pass the key explicitly:
45+
46+
```python
47+
from haystack.utils import Secret
48+
from haystack_integrations.components.fetchers.tavily import TavilyFetcher
49+
50+
fetcher = TavilyFetcher(api_key=Secret.from_token("<your-api-key>"))
51+
```
52+
53+
To get an API key, sign up at [tavily.com](https://tavily.com).
54+
55+
### Installation
56+
57+
Install the Tavily integration with:
58+
59+
```shell
60+
pip install tavily-haystack
61+
```
62+
63+
## Usage
64+
65+
### On its own
66+
67+
```python
68+
from haystack_integrations.components.fetchers.tavily import TavilyFetcher
69+
70+
fetcher = TavilyFetcher(extract_depth="basic")
71+
72+
result = fetcher.run(urls=["https://docs.haystack.deepset.ai/docs/intro"])
73+
documents = result["documents"]
74+
meta = result["meta"]
75+
76+
for doc in documents:
77+
print(f"{doc.meta.get('url')}: {len(doc.content or '')} chars")
78+
79+
print("failed:", meta.get("failed_results"))
80+
```
81+
82+
### In a pipeline
83+
84+
Below is an example of an indexing pipeline that uses `TavilyFetcher` to extract documentation pages and store them in an `InMemoryDocumentStore`.
85+
86+
```python
87+
from haystack import Pipeline
88+
from haystack.document_stores.in_memory import InMemoryDocumentStore
89+
from haystack.components.preprocessors import DocumentSplitter
90+
from haystack.components.writers import DocumentWriter
91+
from haystack_integrations.components.fetchers.tavily import TavilyFetcher
92+
93+
document_store = InMemoryDocumentStore()
94+
95+
fetcher = TavilyFetcher(extract_depth="basic")
96+
splitter = DocumentSplitter(split_by="sentence", split_length=5)
97+
writer = DocumentWriter(document_store=document_store)
98+
99+
indexing_pipeline = Pipeline()
100+
indexing_pipeline.add_component("fetcher", fetcher)
101+
indexing_pipeline.add_component("splitter", splitter)
102+
indexing_pipeline.add_component("writer", writer)
103+
104+
indexing_pipeline.connect("fetcher.documents", "splitter.documents")
105+
indexing_pipeline.connect("splitter.documents", "writer.documents")
106+
107+
indexing_pipeline.run(
108+
data={
109+
"fetcher": {
110+
"urls": ["https://docs.haystack.deepset.ai/docs/intro"],
111+
},
112+
},
113+
)
114+
```
115+
116+
### Asynchronous execution
117+
118+
`TavilyFetcher` also supports asynchronous execution through `run_async()`:
119+
120+
```python
121+
import asyncio
122+
123+
from haystack_integrations.components.fetchers.tavily import TavilyFetcher
124+
125+
fetcher = TavilyFetcher()
126+
127+
128+
async def fetch():
129+
result = await fetcher.run_async(
130+
urls=["https://docs.haystack.deepset.ai/docs/intro"],
131+
)
132+
return result["documents"]
133+
134+
135+
documents = asyncio.run(fetch())
136+
```
137+
138+
The underlying clients are created lazily on the first call. To avoid the cold-start latency of the first call, you can call `warm_up()` explicitly.

‎docs-website/versioned_sidebars/version-3.0-sidebars.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -411,6 +411,7 @@
411411
"pipeline-components/fetchers/googledrivefetcher",
412412
"pipeline-components/fetchers/linkcontentfetcher",
413413
"pipeline-components/fetchers/mssharepointfetcher",
414+
"pipeline-components/fetchers/tavilyfetcher",
414415
"pipeline-components/fetchers/external-integrations-fetchers"
415416
]
416417
},

0 commit comments

Comments
 (0)