diff --git a/docs-website/reference/integrations-api/tavily.md b/docs-website/reference/integrations-api/tavily.md index 99e82f15157..2bf711993ce 100644 --- a/docs-website/reference/integrations-api/tavily.md +++ b/docs-website/reference/integrations-api/tavily.md @@ -6,6 +6,118 @@ slug: "/integrations-tavily" --- +## haystack_integrations.components.fetchers.tavily.tavily_fetcher + +### TavilyFetcher + +A component that uses the Tavily Extract API to fetch and extract content from URLs as Haystack Documents. + +This component wraps the Tavily Extract API, which retrieves and parses web page content from +one or more specified URLs. Unlike web search, it fetches content directly from the given URLs +rather than discovering them via a query. PDF URLs are also supported for extraction. + +Tavily is an AI-powered search and extraction API optimized for LLM applications. You need a Tavily +API key from [tavily.com](https://tavily.com). + +### Usage example + +```python +from haystack_integrations.components.fetchers.tavily import TavilyFetcher +from haystack.utils import Secret + +fetcher = TavilyFetcher( + api_key=Secret.from_env_var("TAVILY_API_KEY"), + extract_depth="basic", +) +result = fetcher.run(urls=["https://haystack.deepset.ai"]) +documents = result["documents"] +meta = result["meta"] +``` + +#### __init__ + +```python +__init__( + api_key: Secret = Secret.from_env_var("TAVILY_API_KEY"), + *, + extract_depth: Literal["basic", "advanced"] = "basic", + include_images: bool = False, + extract_params: dict[str, Any] | None = None +) -> None +``` + +Initialize the TavilyFetcher component. + +**Parameters:** + +- **api_key** (Secret) – API key for Tavily. Defaults to the `TAVILY_API_KEY` environment variable. +- **extract_depth** (Literal['basic', 'advanced']) – Extraction depth: `"basic"` (fast, lower cost) or `"advanced"` (more data including + tables, higher latency and cost). Defaults to `"basic"`. +- **include_images** (bool) – If `True`, extracted image URLs are included in each Document's metadata under + the `"images"` key. Defaults to `False`. +- **extract_params** (dict\[str, Any\] | None) – Additional parameters passed to the Tavily Extract API, such as `format`, + `include_favicon`, `query`, or `chunks_per_source`. + See the [Tavily Extract API reference](https://docs.tavily.com/documentation/api-reference/endpoint/extract) + for available options. + +#### warm_up + +```python +warm_up() -> None +``` + +Initialize the Tavily sync and async clients. + +Called automatically on first use. Can be called explicitly to avoid cold-start latency. + +#### run + +```python +run( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + +#### run_async + +```python +run_async( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Asynchronously fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + ## haystack_integrations.components.websearch.tavily.tavily_websearch ### TavilyWebSearch diff --git a/docs-website/reference_versioned_docs/version-2.18/integrations-api/tavily.md b/docs-website/reference_versioned_docs/version-2.18/integrations-api/tavily.md index 99e82f15157..2bf711993ce 100644 --- a/docs-website/reference_versioned_docs/version-2.18/integrations-api/tavily.md +++ b/docs-website/reference_versioned_docs/version-2.18/integrations-api/tavily.md @@ -6,6 +6,118 @@ slug: "/integrations-tavily" --- +## haystack_integrations.components.fetchers.tavily.tavily_fetcher + +### TavilyFetcher + +A component that uses the Tavily Extract API to fetch and extract content from URLs as Haystack Documents. + +This component wraps the Tavily Extract API, which retrieves and parses web page content from +one or more specified URLs. Unlike web search, it fetches content directly from the given URLs +rather than discovering them via a query. PDF URLs are also supported for extraction. + +Tavily is an AI-powered search and extraction API optimized for LLM applications. You need a Tavily +API key from [tavily.com](https://tavily.com). + +### Usage example + +```python +from haystack_integrations.components.fetchers.tavily import TavilyFetcher +from haystack.utils import Secret + +fetcher = TavilyFetcher( + api_key=Secret.from_env_var("TAVILY_API_KEY"), + extract_depth="basic", +) +result = fetcher.run(urls=["https://haystack.deepset.ai"]) +documents = result["documents"] +meta = result["meta"] +``` + +#### __init__ + +```python +__init__( + api_key: Secret = Secret.from_env_var("TAVILY_API_KEY"), + *, + extract_depth: Literal["basic", "advanced"] = "basic", + include_images: bool = False, + extract_params: dict[str, Any] | None = None +) -> None +``` + +Initialize the TavilyFetcher component. + +**Parameters:** + +- **api_key** (Secret) – API key for Tavily. Defaults to the `TAVILY_API_KEY` environment variable. +- **extract_depth** (Literal['basic', 'advanced']) – Extraction depth: `"basic"` (fast, lower cost) or `"advanced"` (more data including + tables, higher latency and cost). Defaults to `"basic"`. +- **include_images** (bool) – If `True`, extracted image URLs are included in each Document's metadata under + the `"images"` key. Defaults to `False`. +- **extract_params** (dict\[str, Any\] | None) – Additional parameters passed to the Tavily Extract API, such as `format`, + `include_favicon`, `query`, or `chunks_per_source`. + See the [Tavily Extract API reference](https://docs.tavily.com/documentation/api-reference/endpoint/extract) + for available options. + +#### warm_up + +```python +warm_up() -> None +``` + +Initialize the Tavily sync and async clients. + +Called automatically on first use. Can be called explicitly to avoid cold-start latency. + +#### run + +```python +run( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + +#### run_async + +```python +run_async( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Asynchronously fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + ## haystack_integrations.components.websearch.tavily.tavily_websearch ### TavilyWebSearch diff --git a/docs-website/reference_versioned_docs/version-2.19/integrations-api/tavily.md b/docs-website/reference_versioned_docs/version-2.19/integrations-api/tavily.md index 99e82f15157..2bf711993ce 100644 --- a/docs-website/reference_versioned_docs/version-2.19/integrations-api/tavily.md +++ b/docs-website/reference_versioned_docs/version-2.19/integrations-api/tavily.md @@ -6,6 +6,118 @@ slug: "/integrations-tavily" --- +## haystack_integrations.components.fetchers.tavily.tavily_fetcher + +### TavilyFetcher + +A component that uses the Tavily Extract API to fetch and extract content from URLs as Haystack Documents. + +This component wraps the Tavily Extract API, which retrieves and parses web page content from +one or more specified URLs. Unlike web search, it fetches content directly from the given URLs +rather than discovering them via a query. PDF URLs are also supported for extraction. + +Tavily is an AI-powered search and extraction API optimized for LLM applications. You need a Tavily +API key from [tavily.com](https://tavily.com). + +### Usage example + +```python +from haystack_integrations.components.fetchers.tavily import TavilyFetcher +from haystack.utils import Secret + +fetcher = TavilyFetcher( + api_key=Secret.from_env_var("TAVILY_API_KEY"), + extract_depth="basic", +) +result = fetcher.run(urls=["https://haystack.deepset.ai"]) +documents = result["documents"] +meta = result["meta"] +``` + +#### __init__ + +```python +__init__( + api_key: Secret = Secret.from_env_var("TAVILY_API_KEY"), + *, + extract_depth: Literal["basic", "advanced"] = "basic", + include_images: bool = False, + extract_params: dict[str, Any] | None = None +) -> None +``` + +Initialize the TavilyFetcher component. + +**Parameters:** + +- **api_key** (Secret) – API key for Tavily. Defaults to the `TAVILY_API_KEY` environment variable. +- **extract_depth** (Literal['basic', 'advanced']) – Extraction depth: `"basic"` (fast, lower cost) or `"advanced"` (more data including + tables, higher latency and cost). Defaults to `"basic"`. +- **include_images** (bool) – If `True`, extracted image URLs are included in each Document's metadata under + the `"images"` key. Defaults to `False`. +- **extract_params** (dict\[str, Any\] | None) – Additional parameters passed to the Tavily Extract API, such as `format`, + `include_favicon`, `query`, or `chunks_per_source`. + See the [Tavily Extract API reference](https://docs.tavily.com/documentation/api-reference/endpoint/extract) + for available options. + +#### warm_up + +```python +warm_up() -> None +``` + +Initialize the Tavily sync and async clients. + +Called automatically on first use. Can be called explicitly to avoid cold-start latency. + +#### run + +```python +run( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + +#### run_async + +```python +run_async( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Asynchronously fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + ## haystack_integrations.components.websearch.tavily.tavily_websearch ### TavilyWebSearch diff --git a/docs-website/reference_versioned_docs/version-2.20/integrations-api/tavily.md b/docs-website/reference_versioned_docs/version-2.20/integrations-api/tavily.md index 99e82f15157..2bf711993ce 100644 --- a/docs-website/reference_versioned_docs/version-2.20/integrations-api/tavily.md +++ b/docs-website/reference_versioned_docs/version-2.20/integrations-api/tavily.md @@ -6,6 +6,118 @@ slug: "/integrations-tavily" --- +## haystack_integrations.components.fetchers.tavily.tavily_fetcher + +### TavilyFetcher + +A component that uses the Tavily Extract API to fetch and extract content from URLs as Haystack Documents. + +This component wraps the Tavily Extract API, which retrieves and parses web page content from +one or more specified URLs. Unlike web search, it fetches content directly from the given URLs +rather than discovering them via a query. PDF URLs are also supported for extraction. + +Tavily is an AI-powered search and extraction API optimized for LLM applications. You need a Tavily +API key from [tavily.com](https://tavily.com). + +### Usage example + +```python +from haystack_integrations.components.fetchers.tavily import TavilyFetcher +from haystack.utils import Secret + +fetcher = TavilyFetcher( + api_key=Secret.from_env_var("TAVILY_API_KEY"), + extract_depth="basic", +) +result = fetcher.run(urls=["https://haystack.deepset.ai"]) +documents = result["documents"] +meta = result["meta"] +``` + +#### __init__ + +```python +__init__( + api_key: Secret = Secret.from_env_var("TAVILY_API_KEY"), + *, + extract_depth: Literal["basic", "advanced"] = "basic", + include_images: bool = False, + extract_params: dict[str, Any] | None = None +) -> None +``` + +Initialize the TavilyFetcher component. + +**Parameters:** + +- **api_key** (Secret) – API key for Tavily. Defaults to the `TAVILY_API_KEY` environment variable. +- **extract_depth** (Literal['basic', 'advanced']) – Extraction depth: `"basic"` (fast, lower cost) or `"advanced"` (more data including + tables, higher latency and cost). Defaults to `"basic"`. +- **include_images** (bool) – If `True`, extracted image URLs are included in each Document's metadata under + the `"images"` key. Defaults to `False`. +- **extract_params** (dict\[str, Any\] | None) – Additional parameters passed to the Tavily Extract API, such as `format`, + `include_favicon`, `query`, or `chunks_per_source`. + See the [Tavily Extract API reference](https://docs.tavily.com/documentation/api-reference/endpoint/extract) + for available options. + +#### warm_up + +```python +warm_up() -> None +``` + +Initialize the Tavily sync and async clients. + +Called automatically on first use. Can be called explicitly to avoid cold-start latency. + +#### run + +```python +run( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + +#### run_async + +```python +run_async( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Asynchronously fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + ## haystack_integrations.components.websearch.tavily.tavily_websearch ### TavilyWebSearch diff --git a/docs-website/reference_versioned_docs/version-2.21/integrations-api/tavily.md b/docs-website/reference_versioned_docs/version-2.21/integrations-api/tavily.md index 99e82f15157..2bf711993ce 100644 --- a/docs-website/reference_versioned_docs/version-2.21/integrations-api/tavily.md +++ b/docs-website/reference_versioned_docs/version-2.21/integrations-api/tavily.md @@ -6,6 +6,118 @@ slug: "/integrations-tavily" --- +## haystack_integrations.components.fetchers.tavily.tavily_fetcher + +### TavilyFetcher + +A component that uses the Tavily Extract API to fetch and extract content from URLs as Haystack Documents. + +This component wraps the Tavily Extract API, which retrieves and parses web page content from +one or more specified URLs. Unlike web search, it fetches content directly from the given URLs +rather than discovering them via a query. PDF URLs are also supported for extraction. + +Tavily is an AI-powered search and extraction API optimized for LLM applications. You need a Tavily +API key from [tavily.com](https://tavily.com). + +### Usage example + +```python +from haystack_integrations.components.fetchers.tavily import TavilyFetcher +from haystack.utils import Secret + +fetcher = TavilyFetcher( + api_key=Secret.from_env_var("TAVILY_API_KEY"), + extract_depth="basic", +) +result = fetcher.run(urls=["https://haystack.deepset.ai"]) +documents = result["documents"] +meta = result["meta"] +``` + +#### __init__ + +```python +__init__( + api_key: Secret = Secret.from_env_var("TAVILY_API_KEY"), + *, + extract_depth: Literal["basic", "advanced"] = "basic", + include_images: bool = False, + extract_params: dict[str, Any] | None = None +) -> None +``` + +Initialize the TavilyFetcher component. + +**Parameters:** + +- **api_key** (Secret) – API key for Tavily. Defaults to the `TAVILY_API_KEY` environment variable. +- **extract_depth** (Literal['basic', 'advanced']) – Extraction depth: `"basic"` (fast, lower cost) or `"advanced"` (more data including + tables, higher latency and cost). Defaults to `"basic"`. +- **include_images** (bool) – If `True`, extracted image URLs are included in each Document's metadata under + the `"images"` key. Defaults to `False`. +- **extract_params** (dict\[str, Any\] | None) – Additional parameters passed to the Tavily Extract API, such as `format`, + `include_favicon`, `query`, or `chunks_per_source`. + See the [Tavily Extract API reference](https://docs.tavily.com/documentation/api-reference/endpoint/extract) + for available options. + +#### warm_up + +```python +warm_up() -> None +``` + +Initialize the Tavily sync and async clients. + +Called automatically on first use. Can be called explicitly to avoid cold-start latency. + +#### run + +```python +run( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + +#### run_async + +```python +run_async( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Asynchronously fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + ## haystack_integrations.components.websearch.tavily.tavily_websearch ### TavilyWebSearch diff --git a/docs-website/reference_versioned_docs/version-2.22/integrations-api/tavily.md b/docs-website/reference_versioned_docs/version-2.22/integrations-api/tavily.md index 99e82f15157..2bf711993ce 100644 --- a/docs-website/reference_versioned_docs/version-2.22/integrations-api/tavily.md +++ b/docs-website/reference_versioned_docs/version-2.22/integrations-api/tavily.md @@ -6,6 +6,118 @@ slug: "/integrations-tavily" --- +## haystack_integrations.components.fetchers.tavily.tavily_fetcher + +### TavilyFetcher + +A component that uses the Tavily Extract API to fetch and extract content from URLs as Haystack Documents. + +This component wraps the Tavily Extract API, which retrieves and parses web page content from +one or more specified URLs. Unlike web search, it fetches content directly from the given URLs +rather than discovering them via a query. PDF URLs are also supported for extraction. + +Tavily is an AI-powered search and extraction API optimized for LLM applications. You need a Tavily +API key from [tavily.com](https://tavily.com). + +### Usage example + +```python +from haystack_integrations.components.fetchers.tavily import TavilyFetcher +from haystack.utils import Secret + +fetcher = TavilyFetcher( + api_key=Secret.from_env_var("TAVILY_API_KEY"), + extract_depth="basic", +) +result = fetcher.run(urls=["https://haystack.deepset.ai"]) +documents = result["documents"] +meta = result["meta"] +``` + +#### __init__ + +```python +__init__( + api_key: Secret = Secret.from_env_var("TAVILY_API_KEY"), + *, + extract_depth: Literal["basic", "advanced"] = "basic", + include_images: bool = False, + extract_params: dict[str, Any] | None = None +) -> None +``` + +Initialize the TavilyFetcher component. + +**Parameters:** + +- **api_key** (Secret) – API key for Tavily. Defaults to the `TAVILY_API_KEY` environment variable. +- **extract_depth** (Literal['basic', 'advanced']) – Extraction depth: `"basic"` (fast, lower cost) or `"advanced"` (more data including + tables, higher latency and cost). Defaults to `"basic"`. +- **include_images** (bool) – If `True`, extracted image URLs are included in each Document's metadata under + the `"images"` key. Defaults to `False`. +- **extract_params** (dict\[str, Any\] | None) – Additional parameters passed to the Tavily Extract API, such as `format`, + `include_favicon`, `query`, or `chunks_per_source`. + See the [Tavily Extract API reference](https://docs.tavily.com/documentation/api-reference/endpoint/extract) + for available options. + +#### warm_up + +```python +warm_up() -> None +``` + +Initialize the Tavily sync and async clients. + +Called automatically on first use. Can be called explicitly to avoid cold-start latency. + +#### run + +```python +run( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + +#### run_async + +```python +run_async( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Asynchronously fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + ## haystack_integrations.components.websearch.tavily.tavily_websearch ### TavilyWebSearch diff --git a/docs-website/reference_versioned_docs/version-2.23/integrations-api/tavily.md b/docs-website/reference_versioned_docs/version-2.23/integrations-api/tavily.md index 99e82f15157..2bf711993ce 100644 --- a/docs-website/reference_versioned_docs/version-2.23/integrations-api/tavily.md +++ b/docs-website/reference_versioned_docs/version-2.23/integrations-api/tavily.md @@ -6,6 +6,118 @@ slug: "/integrations-tavily" --- +## haystack_integrations.components.fetchers.tavily.tavily_fetcher + +### TavilyFetcher + +A component that uses the Tavily Extract API to fetch and extract content from URLs as Haystack Documents. + +This component wraps the Tavily Extract API, which retrieves and parses web page content from +one or more specified URLs. Unlike web search, it fetches content directly from the given URLs +rather than discovering them via a query. PDF URLs are also supported for extraction. + +Tavily is an AI-powered search and extraction API optimized for LLM applications. You need a Tavily +API key from [tavily.com](https://tavily.com). + +### Usage example + +```python +from haystack_integrations.components.fetchers.tavily import TavilyFetcher +from haystack.utils import Secret + +fetcher = TavilyFetcher( + api_key=Secret.from_env_var("TAVILY_API_KEY"), + extract_depth="basic", +) +result = fetcher.run(urls=["https://haystack.deepset.ai"]) +documents = result["documents"] +meta = result["meta"] +``` + +#### __init__ + +```python +__init__( + api_key: Secret = Secret.from_env_var("TAVILY_API_KEY"), + *, + extract_depth: Literal["basic", "advanced"] = "basic", + include_images: bool = False, + extract_params: dict[str, Any] | None = None +) -> None +``` + +Initialize the TavilyFetcher component. + +**Parameters:** + +- **api_key** (Secret) – API key for Tavily. Defaults to the `TAVILY_API_KEY` environment variable. +- **extract_depth** (Literal['basic', 'advanced']) – Extraction depth: `"basic"` (fast, lower cost) or `"advanced"` (more data including + tables, higher latency and cost). Defaults to `"basic"`. +- **include_images** (bool) – If `True`, extracted image URLs are included in each Document's metadata under + the `"images"` key. Defaults to `False`. +- **extract_params** (dict\[str, Any\] | None) – Additional parameters passed to the Tavily Extract API, such as `format`, + `include_favicon`, `query`, or `chunks_per_source`. + See the [Tavily Extract API reference](https://docs.tavily.com/documentation/api-reference/endpoint/extract) + for available options. + +#### warm_up + +```python +warm_up() -> None +``` + +Initialize the Tavily sync and async clients. + +Called automatically on first use. Can be called explicitly to avoid cold-start latency. + +#### run + +```python +run( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + +#### run_async + +```python +run_async( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Asynchronously fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + ## haystack_integrations.components.websearch.tavily.tavily_websearch ### TavilyWebSearch diff --git a/docs-website/reference_versioned_docs/version-2.24/integrations-api/tavily.md b/docs-website/reference_versioned_docs/version-2.24/integrations-api/tavily.md index 99e82f15157..2bf711993ce 100644 --- a/docs-website/reference_versioned_docs/version-2.24/integrations-api/tavily.md +++ b/docs-website/reference_versioned_docs/version-2.24/integrations-api/tavily.md @@ -6,6 +6,118 @@ slug: "/integrations-tavily" --- +## haystack_integrations.components.fetchers.tavily.tavily_fetcher + +### TavilyFetcher + +A component that uses the Tavily Extract API to fetch and extract content from URLs as Haystack Documents. + +This component wraps the Tavily Extract API, which retrieves and parses web page content from +one or more specified URLs. Unlike web search, it fetches content directly from the given URLs +rather than discovering them via a query. PDF URLs are also supported for extraction. + +Tavily is an AI-powered search and extraction API optimized for LLM applications. You need a Tavily +API key from [tavily.com](https://tavily.com). + +### Usage example + +```python +from haystack_integrations.components.fetchers.tavily import TavilyFetcher +from haystack.utils import Secret + +fetcher = TavilyFetcher( + api_key=Secret.from_env_var("TAVILY_API_KEY"), + extract_depth="basic", +) +result = fetcher.run(urls=["https://haystack.deepset.ai"]) +documents = result["documents"] +meta = result["meta"] +``` + +#### __init__ + +```python +__init__( + api_key: Secret = Secret.from_env_var("TAVILY_API_KEY"), + *, + extract_depth: Literal["basic", "advanced"] = "basic", + include_images: bool = False, + extract_params: dict[str, Any] | None = None +) -> None +``` + +Initialize the TavilyFetcher component. + +**Parameters:** + +- **api_key** (Secret) – API key for Tavily. Defaults to the `TAVILY_API_KEY` environment variable. +- **extract_depth** (Literal['basic', 'advanced']) – Extraction depth: `"basic"` (fast, lower cost) or `"advanced"` (more data including + tables, higher latency and cost). Defaults to `"basic"`. +- **include_images** (bool) – If `True`, extracted image URLs are included in each Document's metadata under + the `"images"` key. Defaults to `False`. +- **extract_params** (dict\[str, Any\] | None) – Additional parameters passed to the Tavily Extract API, such as `format`, + `include_favicon`, `query`, or `chunks_per_source`. + See the [Tavily Extract API reference](https://docs.tavily.com/documentation/api-reference/endpoint/extract) + for available options. + +#### warm_up + +```python +warm_up() -> None +``` + +Initialize the Tavily sync and async clients. + +Called automatically on first use. Can be called explicitly to avoid cold-start latency. + +#### run + +```python +run( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + +#### run_async + +```python +run_async( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Asynchronously fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + ## haystack_integrations.components.websearch.tavily.tavily_websearch ### TavilyWebSearch diff --git a/docs-website/reference_versioned_docs/version-2.25/integrations-api/tavily.md b/docs-website/reference_versioned_docs/version-2.25/integrations-api/tavily.md index 99e82f15157..2bf711993ce 100644 --- a/docs-website/reference_versioned_docs/version-2.25/integrations-api/tavily.md +++ b/docs-website/reference_versioned_docs/version-2.25/integrations-api/tavily.md @@ -6,6 +6,118 @@ slug: "/integrations-tavily" --- +## haystack_integrations.components.fetchers.tavily.tavily_fetcher + +### TavilyFetcher + +A component that uses the Tavily Extract API to fetch and extract content from URLs as Haystack Documents. + +This component wraps the Tavily Extract API, which retrieves and parses web page content from +one or more specified URLs. Unlike web search, it fetches content directly from the given URLs +rather than discovering them via a query. PDF URLs are also supported for extraction. + +Tavily is an AI-powered search and extraction API optimized for LLM applications. You need a Tavily +API key from [tavily.com](https://tavily.com). + +### Usage example + +```python +from haystack_integrations.components.fetchers.tavily import TavilyFetcher +from haystack.utils import Secret + +fetcher = TavilyFetcher( + api_key=Secret.from_env_var("TAVILY_API_KEY"), + extract_depth="basic", +) +result = fetcher.run(urls=["https://haystack.deepset.ai"]) +documents = result["documents"] +meta = result["meta"] +``` + +#### __init__ + +```python +__init__( + api_key: Secret = Secret.from_env_var("TAVILY_API_KEY"), + *, + extract_depth: Literal["basic", "advanced"] = "basic", + include_images: bool = False, + extract_params: dict[str, Any] | None = None +) -> None +``` + +Initialize the TavilyFetcher component. + +**Parameters:** + +- **api_key** (Secret) – API key for Tavily. Defaults to the `TAVILY_API_KEY` environment variable. +- **extract_depth** (Literal['basic', 'advanced']) – Extraction depth: `"basic"` (fast, lower cost) or `"advanced"` (more data including + tables, higher latency and cost). Defaults to `"basic"`. +- **include_images** (bool) – If `True`, extracted image URLs are included in each Document's metadata under + the `"images"` key. Defaults to `False`. +- **extract_params** (dict\[str, Any\] | None) – Additional parameters passed to the Tavily Extract API, such as `format`, + `include_favicon`, `query`, or `chunks_per_source`. + See the [Tavily Extract API reference](https://docs.tavily.com/documentation/api-reference/endpoint/extract) + for available options. + +#### warm_up + +```python +warm_up() -> None +``` + +Initialize the Tavily sync and async clients. + +Called automatically on first use. Can be called explicitly to avoid cold-start latency. + +#### run + +```python +run( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + +#### run_async + +```python +run_async( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Asynchronously fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + ## haystack_integrations.components.websearch.tavily.tavily_websearch ### TavilyWebSearch diff --git a/docs-website/reference_versioned_docs/version-2.26/integrations-api/tavily.md b/docs-website/reference_versioned_docs/version-2.26/integrations-api/tavily.md index 99e82f15157..2bf711993ce 100644 --- a/docs-website/reference_versioned_docs/version-2.26/integrations-api/tavily.md +++ b/docs-website/reference_versioned_docs/version-2.26/integrations-api/tavily.md @@ -6,6 +6,118 @@ slug: "/integrations-tavily" --- +## haystack_integrations.components.fetchers.tavily.tavily_fetcher + +### TavilyFetcher + +A component that uses the Tavily Extract API to fetch and extract content from URLs as Haystack Documents. + +This component wraps the Tavily Extract API, which retrieves and parses web page content from +one or more specified URLs. Unlike web search, it fetches content directly from the given URLs +rather than discovering them via a query. PDF URLs are also supported for extraction. + +Tavily is an AI-powered search and extraction API optimized for LLM applications. You need a Tavily +API key from [tavily.com](https://tavily.com). + +### Usage example + +```python +from haystack_integrations.components.fetchers.tavily import TavilyFetcher +from haystack.utils import Secret + +fetcher = TavilyFetcher( + api_key=Secret.from_env_var("TAVILY_API_KEY"), + extract_depth="basic", +) +result = fetcher.run(urls=["https://haystack.deepset.ai"]) +documents = result["documents"] +meta = result["meta"] +``` + +#### __init__ + +```python +__init__( + api_key: Secret = Secret.from_env_var("TAVILY_API_KEY"), + *, + extract_depth: Literal["basic", "advanced"] = "basic", + include_images: bool = False, + extract_params: dict[str, Any] | None = None +) -> None +``` + +Initialize the TavilyFetcher component. + +**Parameters:** + +- **api_key** (Secret) – API key for Tavily. Defaults to the `TAVILY_API_KEY` environment variable. +- **extract_depth** (Literal['basic', 'advanced']) – Extraction depth: `"basic"` (fast, lower cost) or `"advanced"` (more data including + tables, higher latency and cost). Defaults to `"basic"`. +- **include_images** (bool) – If `True`, extracted image URLs are included in each Document's metadata under + the `"images"` key. Defaults to `False`. +- **extract_params** (dict\[str, Any\] | None) – Additional parameters passed to the Tavily Extract API, such as `format`, + `include_favicon`, `query`, or `chunks_per_source`. + See the [Tavily Extract API reference](https://docs.tavily.com/documentation/api-reference/endpoint/extract) + for available options. + +#### warm_up + +```python +warm_up() -> None +``` + +Initialize the Tavily sync and async clients. + +Called automatically on first use. Can be called explicitly to avoid cold-start latency. + +#### run + +```python +run( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + +#### run_async + +```python +run_async( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Asynchronously fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + ## haystack_integrations.components.websearch.tavily.tavily_websearch ### TavilyWebSearch diff --git a/docs-website/reference_versioned_docs/version-2.27/integrations-api/tavily.md b/docs-website/reference_versioned_docs/version-2.27/integrations-api/tavily.md index 99e82f15157..2bf711993ce 100644 --- a/docs-website/reference_versioned_docs/version-2.27/integrations-api/tavily.md +++ b/docs-website/reference_versioned_docs/version-2.27/integrations-api/tavily.md @@ -6,6 +6,118 @@ slug: "/integrations-tavily" --- +## haystack_integrations.components.fetchers.tavily.tavily_fetcher + +### TavilyFetcher + +A component that uses the Tavily Extract API to fetch and extract content from URLs as Haystack Documents. + +This component wraps the Tavily Extract API, which retrieves and parses web page content from +one or more specified URLs. Unlike web search, it fetches content directly from the given URLs +rather than discovering them via a query. PDF URLs are also supported for extraction. + +Tavily is an AI-powered search and extraction API optimized for LLM applications. You need a Tavily +API key from [tavily.com](https://tavily.com). + +### Usage example + +```python +from haystack_integrations.components.fetchers.tavily import TavilyFetcher +from haystack.utils import Secret + +fetcher = TavilyFetcher( + api_key=Secret.from_env_var("TAVILY_API_KEY"), + extract_depth="basic", +) +result = fetcher.run(urls=["https://haystack.deepset.ai"]) +documents = result["documents"] +meta = result["meta"] +``` + +#### __init__ + +```python +__init__( + api_key: Secret = Secret.from_env_var("TAVILY_API_KEY"), + *, + extract_depth: Literal["basic", "advanced"] = "basic", + include_images: bool = False, + extract_params: dict[str, Any] | None = None +) -> None +``` + +Initialize the TavilyFetcher component. + +**Parameters:** + +- **api_key** (Secret) – API key for Tavily. Defaults to the `TAVILY_API_KEY` environment variable. +- **extract_depth** (Literal['basic', 'advanced']) – Extraction depth: `"basic"` (fast, lower cost) or `"advanced"` (more data including + tables, higher latency and cost). Defaults to `"basic"`. +- **include_images** (bool) – If `True`, extracted image URLs are included in each Document's metadata under + the `"images"` key. Defaults to `False`. +- **extract_params** (dict\[str, Any\] | None) – Additional parameters passed to the Tavily Extract API, such as `format`, + `include_favicon`, `query`, or `chunks_per_source`. + See the [Tavily Extract API reference](https://docs.tavily.com/documentation/api-reference/endpoint/extract) + for available options. + +#### warm_up + +```python +warm_up() -> None +``` + +Initialize the Tavily sync and async clients. + +Called automatically on first use. Can be called explicitly to avoid cold-start latency. + +#### run + +```python +run( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + +#### run_async + +```python +run_async( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Asynchronously fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + ## haystack_integrations.components.websearch.tavily.tavily_websearch ### TavilyWebSearch diff --git a/docs-website/reference_versioned_docs/version-2.28/integrations-api/tavily.md b/docs-website/reference_versioned_docs/version-2.28/integrations-api/tavily.md index 99e82f15157..2bf711993ce 100644 --- a/docs-website/reference_versioned_docs/version-2.28/integrations-api/tavily.md +++ b/docs-website/reference_versioned_docs/version-2.28/integrations-api/tavily.md @@ -6,6 +6,118 @@ slug: "/integrations-tavily" --- +## haystack_integrations.components.fetchers.tavily.tavily_fetcher + +### TavilyFetcher + +A component that uses the Tavily Extract API to fetch and extract content from URLs as Haystack Documents. + +This component wraps the Tavily Extract API, which retrieves and parses web page content from +one or more specified URLs. Unlike web search, it fetches content directly from the given URLs +rather than discovering them via a query. PDF URLs are also supported for extraction. + +Tavily is an AI-powered search and extraction API optimized for LLM applications. You need a Tavily +API key from [tavily.com](https://tavily.com). + +### Usage example + +```python +from haystack_integrations.components.fetchers.tavily import TavilyFetcher +from haystack.utils import Secret + +fetcher = TavilyFetcher( + api_key=Secret.from_env_var("TAVILY_API_KEY"), + extract_depth="basic", +) +result = fetcher.run(urls=["https://haystack.deepset.ai"]) +documents = result["documents"] +meta = result["meta"] +``` + +#### __init__ + +```python +__init__( + api_key: Secret = Secret.from_env_var("TAVILY_API_KEY"), + *, + extract_depth: Literal["basic", "advanced"] = "basic", + include_images: bool = False, + extract_params: dict[str, Any] | None = None +) -> None +``` + +Initialize the TavilyFetcher component. + +**Parameters:** + +- **api_key** (Secret) – API key for Tavily. Defaults to the `TAVILY_API_KEY` environment variable. +- **extract_depth** (Literal['basic', 'advanced']) – Extraction depth: `"basic"` (fast, lower cost) or `"advanced"` (more data including + tables, higher latency and cost). Defaults to `"basic"`. +- **include_images** (bool) – If `True`, extracted image URLs are included in each Document's metadata under + the `"images"` key. Defaults to `False`. +- **extract_params** (dict\[str, Any\] | None) – Additional parameters passed to the Tavily Extract API, such as `format`, + `include_favicon`, `query`, or `chunks_per_source`. + See the [Tavily Extract API reference](https://docs.tavily.com/documentation/api-reference/endpoint/extract) + for available options. + +#### warm_up + +```python +warm_up() -> None +``` + +Initialize the Tavily sync and async clients. + +Called automatically on first use. Can be called explicitly to avoid cold-start latency. + +#### run + +```python +run( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + +#### run_async + +```python +run_async( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Asynchronously fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + ## haystack_integrations.components.websearch.tavily.tavily_websearch ### TavilyWebSearch diff --git a/docs-website/reference_versioned_docs/version-2.29/integrations-api/tavily.md b/docs-website/reference_versioned_docs/version-2.29/integrations-api/tavily.md index 99e82f15157..2bf711993ce 100644 --- a/docs-website/reference_versioned_docs/version-2.29/integrations-api/tavily.md +++ b/docs-website/reference_versioned_docs/version-2.29/integrations-api/tavily.md @@ -6,6 +6,118 @@ slug: "/integrations-tavily" --- +## haystack_integrations.components.fetchers.tavily.tavily_fetcher + +### TavilyFetcher + +A component that uses the Tavily Extract API to fetch and extract content from URLs as Haystack Documents. + +This component wraps the Tavily Extract API, which retrieves and parses web page content from +one or more specified URLs. Unlike web search, it fetches content directly from the given URLs +rather than discovering them via a query. PDF URLs are also supported for extraction. + +Tavily is an AI-powered search and extraction API optimized for LLM applications. You need a Tavily +API key from [tavily.com](https://tavily.com). + +### Usage example + +```python +from haystack_integrations.components.fetchers.tavily import TavilyFetcher +from haystack.utils import Secret + +fetcher = TavilyFetcher( + api_key=Secret.from_env_var("TAVILY_API_KEY"), + extract_depth="basic", +) +result = fetcher.run(urls=["https://haystack.deepset.ai"]) +documents = result["documents"] +meta = result["meta"] +``` + +#### __init__ + +```python +__init__( + api_key: Secret = Secret.from_env_var("TAVILY_API_KEY"), + *, + extract_depth: Literal["basic", "advanced"] = "basic", + include_images: bool = False, + extract_params: dict[str, Any] | None = None +) -> None +``` + +Initialize the TavilyFetcher component. + +**Parameters:** + +- **api_key** (Secret) – API key for Tavily. Defaults to the `TAVILY_API_KEY` environment variable. +- **extract_depth** (Literal['basic', 'advanced']) – Extraction depth: `"basic"` (fast, lower cost) or `"advanced"` (more data including + tables, higher latency and cost). Defaults to `"basic"`. +- **include_images** (bool) – If `True`, extracted image URLs are included in each Document's metadata under + the `"images"` key. Defaults to `False`. +- **extract_params** (dict\[str, Any\] | None) – Additional parameters passed to the Tavily Extract API, such as `format`, + `include_favicon`, `query`, or `chunks_per_source`. + See the [Tavily Extract API reference](https://docs.tavily.com/documentation/api-reference/endpoint/extract) + for available options. + +#### warm_up + +```python +warm_up() -> None +``` + +Initialize the Tavily sync and async clients. + +Called automatically on first use. Can be called explicitly to avoid cold-start latency. + +#### run + +```python +run( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + +#### run_async + +```python +run_async( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Asynchronously fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + ## haystack_integrations.components.websearch.tavily.tavily_websearch ### TavilyWebSearch diff --git a/docs-website/reference_versioned_docs/version-2.30/integrations-api/tavily.md b/docs-website/reference_versioned_docs/version-2.30/integrations-api/tavily.md index 99e82f15157..2bf711993ce 100644 --- a/docs-website/reference_versioned_docs/version-2.30/integrations-api/tavily.md +++ b/docs-website/reference_versioned_docs/version-2.30/integrations-api/tavily.md @@ -6,6 +6,118 @@ slug: "/integrations-tavily" --- +## haystack_integrations.components.fetchers.tavily.tavily_fetcher + +### TavilyFetcher + +A component that uses the Tavily Extract API to fetch and extract content from URLs as Haystack Documents. + +This component wraps the Tavily Extract API, which retrieves and parses web page content from +one or more specified URLs. Unlike web search, it fetches content directly from the given URLs +rather than discovering them via a query. PDF URLs are also supported for extraction. + +Tavily is an AI-powered search and extraction API optimized for LLM applications. You need a Tavily +API key from [tavily.com](https://tavily.com). + +### Usage example + +```python +from haystack_integrations.components.fetchers.tavily import TavilyFetcher +from haystack.utils import Secret + +fetcher = TavilyFetcher( + api_key=Secret.from_env_var("TAVILY_API_KEY"), + extract_depth="basic", +) +result = fetcher.run(urls=["https://haystack.deepset.ai"]) +documents = result["documents"] +meta = result["meta"] +``` + +#### __init__ + +```python +__init__( + api_key: Secret = Secret.from_env_var("TAVILY_API_KEY"), + *, + extract_depth: Literal["basic", "advanced"] = "basic", + include_images: bool = False, + extract_params: dict[str, Any] | None = None +) -> None +``` + +Initialize the TavilyFetcher component. + +**Parameters:** + +- **api_key** (Secret) – API key for Tavily. Defaults to the `TAVILY_API_KEY` environment variable. +- **extract_depth** (Literal['basic', 'advanced']) – Extraction depth: `"basic"` (fast, lower cost) or `"advanced"` (more data including + tables, higher latency and cost). Defaults to `"basic"`. +- **include_images** (bool) – If `True`, extracted image URLs are included in each Document's metadata under + the `"images"` key. Defaults to `False`. +- **extract_params** (dict\[str, Any\] | None) – Additional parameters passed to the Tavily Extract API, such as `format`, + `include_favicon`, `query`, or `chunks_per_source`. + See the [Tavily Extract API reference](https://docs.tavily.com/documentation/api-reference/endpoint/extract) + for available options. + +#### warm_up + +```python +warm_up() -> None +``` + +Initialize the Tavily sync and async clients. + +Called automatically on first use. Can be called explicitly to avoid cold-start latency. + +#### run + +```python +run( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + +#### run_async + +```python +run_async( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Asynchronously fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + ## haystack_integrations.components.websearch.tavily.tavily_websearch ### TavilyWebSearch diff --git a/docs-website/reference_versioned_docs/version-2.31/integrations-api/tavily.md b/docs-website/reference_versioned_docs/version-2.31/integrations-api/tavily.md index 99e82f15157..2bf711993ce 100644 --- a/docs-website/reference_versioned_docs/version-2.31/integrations-api/tavily.md +++ b/docs-website/reference_versioned_docs/version-2.31/integrations-api/tavily.md @@ -6,6 +6,118 @@ slug: "/integrations-tavily" --- +## haystack_integrations.components.fetchers.tavily.tavily_fetcher + +### TavilyFetcher + +A component that uses the Tavily Extract API to fetch and extract content from URLs as Haystack Documents. + +This component wraps the Tavily Extract API, which retrieves and parses web page content from +one or more specified URLs. Unlike web search, it fetches content directly from the given URLs +rather than discovering them via a query. PDF URLs are also supported for extraction. + +Tavily is an AI-powered search and extraction API optimized for LLM applications. You need a Tavily +API key from [tavily.com](https://tavily.com). + +### Usage example + +```python +from haystack_integrations.components.fetchers.tavily import TavilyFetcher +from haystack.utils import Secret + +fetcher = TavilyFetcher( + api_key=Secret.from_env_var("TAVILY_API_KEY"), + extract_depth="basic", +) +result = fetcher.run(urls=["https://haystack.deepset.ai"]) +documents = result["documents"] +meta = result["meta"] +``` + +#### __init__ + +```python +__init__( + api_key: Secret = Secret.from_env_var("TAVILY_API_KEY"), + *, + extract_depth: Literal["basic", "advanced"] = "basic", + include_images: bool = False, + extract_params: dict[str, Any] | None = None +) -> None +``` + +Initialize the TavilyFetcher component. + +**Parameters:** + +- **api_key** (Secret) – API key for Tavily. Defaults to the `TAVILY_API_KEY` environment variable. +- **extract_depth** (Literal['basic', 'advanced']) – Extraction depth: `"basic"` (fast, lower cost) or `"advanced"` (more data including + tables, higher latency and cost). Defaults to `"basic"`. +- **include_images** (bool) – If `True`, extracted image URLs are included in each Document's metadata under + the `"images"` key. Defaults to `False`. +- **extract_params** (dict\[str, Any\] | None) – Additional parameters passed to the Tavily Extract API, such as `format`, + `include_favicon`, `query`, or `chunks_per_source`. + See the [Tavily Extract API reference](https://docs.tavily.com/documentation/api-reference/endpoint/extract) + for available options. + +#### warm_up + +```python +warm_up() -> None +``` + +Initialize the Tavily sync and async clients. + +Called automatically on first use. Can be called explicitly to avoid cold-start latency. + +#### run + +```python +run( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + +#### run_async + +```python +run_async( + urls: list[str], extract_params: dict[str, Any] | None = None +) -> dict[str, Any] +``` + +Asynchronously fetch and extract content from the given URLs using the Tavily Extract API. + +**Parameters:** + +- **urls** (list\[str\]) – List of URLs to extract content from. Maximum 20 URLs per request. +- **extract_params** (dict\[str, Any\] | None) – Optional per-run override of extract parameters. + If provided, fully replaces the init-time `extract_params`. + +**Returns:** + +- dict\[str, Any\] – A dictionary with: +- `documents`: List of Documents containing extracted page content. + Each Document's `meta` includes `"url"` and, if `include_images` is True, `"images"`. +- `meta`: Request-level metadata containing `"response_time"`, `"usage"`, + `"request_id"`, and `"failed_results"` for URLs that could not be processed. + ## haystack_integrations.components.websearch.tavily.tavily_websearch ### TavilyWebSearch