Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs-website/docs/pipeline-components/connectors.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ These are Haystack integrations that connect your pipelines to services by exter
| [GitHubRepoViewer](connectors/githubrepoviewer.mdx) | Enables navigating and fetching content from GitHub repositories through the GitHub API. |
| [JinaReaderConnector](connectors/jinareaderconnector.mdx) | Use Jina AI’s Reader API with Haystack. |
| [LangfuseConnector](connectors/langfuseconnector.mdx) | Enables tracing in Haystack pipelines using Langfuse. |
| [OAuthTokenResolver](connectors/oauthtokenresolver.mdx) | Resolves an OAuth access token at runtime and emits it for downstream components. |
| [OpenAPIConnector](connectors/openapiconnector.mdx) | Acts as an interface between the Haystack ecosystem and OpenAPI services, using explicit input arguments. |
| [OpenAPIServiceConnector](connectors/openapiserviceconnector.mdx) | Acts as an interface between the Haystack ecosystem and OpenAPI services. |
| [OpenTelemetryConnector](connectors/opentelemetryconnector.mdx) | Enables tracing in Haystack pipelines using OpenTelemetry. |
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
---
title: "OAuthTokenResolver"
id: oauthtokenresolver
slug: "/oauthtokenresolver"
description: "Resolves an OAuth access token at pipeline runtime and emits it for downstream components such as the SharePoint and Google Drive retrievers and fetchers."
---

# OAuthTokenResolver

Resolves an OAuth access token at pipeline runtime and emits it for downstream components such as the SharePoint and Google Drive retrievers and fetchers.

<div className="key-value-table">

| | |
| --- | --- |
| **Most common position in a pipeline** | At the start of a pipeline, feeding `access_token` into downstream components such as [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx) or [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx) |
| **Mandatory init variables** | `token_source`: The strategy that resolves the access token, for example `OAuthRefreshTokenSource` |
| **Mandatory run variables** | None for config-only sources. `subject_token`: a controller-injected per-request credential, mandatory only when the source requires it (for example `OAuthTokenExchangeSource`) |
| **Output variables** | `access_token`: A bearer token string |
| **API reference** | [OAuth](/reference/integrations-oauth) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/oauth |
| **Package name** | `oauth-haystack` |

</div>

## Overview

`OAuthTokenResolver` resolves an OAuth access token when the pipeline runs and emits it on the `access_token` output socket. Downstream components – such as [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx), [`MSSharePointFetcher`](../fetchers/mssharepointfetcher.mdx), [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx), and [`GoogleDriveFetcher`](../fetchers/googledrivefetcher.mdx) – consume the token through a normal connection and never need to know how it was obtained.

The resolver itself is a thin wrapper. The actual work of getting a token is delegated to a pluggable **token source** that decides *where* the token comes from. This separation lets you swap authentication strategies (refresh-token grant, per-request token exchange, or a static long-lived token) without changing the rest of your pipeline.

### Token sources

You pass a token source to the resolver through the `token_source` parameter. All sources are importable from `haystack_integrations.utils.oauth`.

| Source | Use it when | Per-request input |
| --- | --- | --- |
| `OAuthRefreshTokenSource` | You have a single, fixed identity backed by a stored refresh token and want the source to exchange it for short-lived access tokens and cache them. | None |
| `OAuthTokenExchangeSource` | You serve multiple users (or run multiple replicas) and want to exchange an incoming per-request user assertion for a downstream token, with no persistent storage. Implements RFC 8693 token exchange and Microsoft's on-behalf-of flow. | `subject_token` |
| `OAuthStaticTokenSource` | Your provider issues a non-expiring token that you manage out of band (for example Slack or Notion). | None |

When the configured source needs a per-request credential (`OAuthTokenExchangeSource` sets `requires_subject_token = True`), the resolver declares a **mandatory** `subject_token` run input. This is a controller-injected credential – for example an incoming user assertion – not a value chosen by an end user. For config-only sources (`OAuthRefreshTokenSource`, `OAuthStaticTokenSource`), the resolver declares no run input and acts as a source node.

:::info[Scopes are provider-specific]

The OAuth scopes you request depend on the downstream service. For Microsoft Graph, that means scopes such as `https://graph.microsoft.com/Files.Read.All`; for Google Drive, scopes such as `https://www.googleapis.com/auth/drive.readonly`. Always consult your identity provider's documentation for the exact scope values.

:::

### Installation

Install the OAuth integration with:

```shell
pip install oauth-haystack
```

## Usage

### On its own

Resolve a token with a stored refresh token using `OAuthRefreshTokenSource`. The refresh token is read from an environment variable through the [Secret API](../../concepts/secret-management.mdx):

```python
from haystack.utils import Secret
from haystack_integrations.components.connectors.oauth import OAuthTokenResolver
from haystack_integrations.utils.oauth import OAuthRefreshTokenSource

resolver = OAuthTokenResolver(
token_source=OAuthRefreshTokenSource(
token_url="https://login.microsoftonline.com/common/oauth2/v2.0/token",
client_id="aaa-bbb-ccc",
refresh_token=Secret.from_env_var("MS_REFRESH_TOKEN"),
scopes=[
"https://graph.microsoft.com/Files.Read.All",
"offline_access",
],
),
)

access_token = resolver.run()["access_token"]
```

For a provider that issues long-lived, non-expiring tokens, use `OAuthStaticTokenSource` instead:

```python
from haystack.utils import Secret
from haystack_integrations.components.connectors.oauth import OAuthTokenResolver
from haystack_integrations.utils.oauth import OAuthStaticTokenSource

resolver = OAuthTokenResolver(
token_source=OAuthStaticTokenSource(token=Secret.from_env_var("SERVICE_TOKEN")),
)

access_token = resolver.run()["access_token"]
```

For multi-user backends, use `OAuthTokenExchangeSource`. The resolver then requires a per-request `subject_token`:

```python
from haystack_integrations.components.connectors.oauth import OAuthTokenResolver
from haystack_integrations.utils.oauth import OAuthTokenExchangeSource

resolver = OAuthTokenResolver(
token_source=OAuthTokenExchangeSource(
token_url="https://login.microsoftonline.com/<tenant>/oauth2/v2.0/token",
client_id="aaa-bbb-ccc",
subject_token_param="assertion",
grant_type="urn:ietf:params:oauth:grant-type:jwt-bearer",
scopes=["https://graph.microsoft.com/Files.Read.All"],
extra_token_params={"requested_token_use": "on_behalf_of"},
),
)

# `subject_token` is the incoming per-request user assertion, injected by your application.
access_token = resolver.run(subject_token="<incoming-user-assertion>")["access_token"]
```

### In a pipeline

In a pipeline, connect the resolver's `access_token` output to the `access_token` input of one or more downstream components. The example below wires the resolver into a [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx) so that searching SharePoint requires only a query at runtime:

```python
from haystack import Pipeline
from haystack.utils import Secret
from haystack_integrations.components.connectors.oauth import OAuthTokenResolver
from haystack_integrations.utils.oauth import OAuthRefreshTokenSource
from haystack_integrations.components.retrievers.microsoft_sharepoint import (
MSSharePointRetriever,
)

pipeline = Pipeline()
pipeline.add_component(
"resolver",
OAuthTokenResolver(
token_source=OAuthRefreshTokenSource(
token_url="https://login.microsoftonline.com/common/oauth2/v2.0/token",
client_id="aaa-bbb-ccc",
refresh_token=Secret.from_env_var("MS_REFRESH_TOKEN"),
scopes=[
"https://graph.microsoft.com/Files.Read.All",
"https://graph.microsoft.com/Sites.Read.All",
"offline_access",
],
),
),
)
pipeline.add_component("retriever", MSSharePointRetriever(top_k=5))
pipeline.connect("resolver.access_token", "retriever.access_token")

result = pipeline.run({"retriever": {"query": "quarterly roadmap"}})
documents = result["retriever"]["documents"]
```

A single `access_token` output can be connected to several downstream inputs. For a full retrieve-then-fetch pipeline that feeds the same token to both a retriever and a fetcher, see the [`MSSharePointFetcher`](../fetchers/mssharepointfetcher.mdx) and [`GoogleDriveFetcher`](../fetchers/googledrivefetcher.mdx) pages.
9 changes: 6 additions & 3 deletions docs-website/docs/pipeline-components/fetchers.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,16 @@
title: "Fetchers"
id: fetchers
slug: "/fetchers"
description: "Currently, there's one Fetcher in Haystack: LinkContentFetcher. It fetches the contents of the URLs you give it."
description: "Fetchers retrieve content from external sources – URLs, web crawls, or cloud storage such as SharePoint and Google Drive – so you can use it as data for your pipelines."
---

# Fetchers

Currently, there's one Fetcher in Haystack: LinkContentFetcher. It fetches the contents of the URLs you give it.
Fetchers retrieve content from external sources – URLs, web crawls, or cloud storage such as SharePoint and Google Drive – so you can use it as data for your pipelines.

| Component | Description |
| --- | --- |
| [LinkContentFetcher](fetchers/linkcontentfetcher.mdx) | Fetches the contents of the URLs you give it so you can use them as data for your pipelines. |
| [FirecrawlCrawler](fetchers/firecrawlcrawler.mdx) | Crawls websites with Firecrawl, following links to discover subpages, and returns them as Documents. |
| [GoogleDriveFetcher](fetchers/googledrivefetcher.mdx) | Fetches the full content of Google Drive files via the Drive API v3 and returns it as ByteStreams. |
| [LinkContentFetcher](fetchers/linkcontentfetcher.mdx) | Fetches the contents of the URLs you give it so you can use them as data for your pipelines. |
| [MSSharePointFetcher](fetchers/mssharepointfetcher.mdx) | Fetches the full content of Microsoft SharePoint and OneDrive items via the Microsoft Graph API and returns it as ByteStreams. |
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
---
title: "GoogleDriveFetcher"
id: googledrivefetcher
slug: "/googledrivefetcher"
description: "Fetches the full content of Google Drive files via the Drive API v3 and returns it as ByteStreams."
---

# GoogleDriveFetcher

Fetches the full content of Google Drive files via the Drive API v3 and returns it as ByteStreams.

<div className="key-value-table">

| | |
| --- | --- |
| **Most common position in a pipeline** | After [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx), before a Router or File Converters |
| **Mandatory init variables** | None |
| **Mandatory run variables** | `access_token`: A delegated Google OAuth bearer token, typically wired from an upstream `OAuthTokenResolver` <br /> <br />`targets`: A list of `Document`s (from `GoogleDriveRetriever`) or raw Google Drive file ids / URLs |
| **Output variables** | `streams`: A list of [ByteStreams](../../concepts/data-classes.mdx) holding the fetched content |
| **API reference** | [Google Drive](/reference/integrations-google-drive) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/google_drive |
| **Package name** | `google-drive-haystack` |

</div>

## Overview

`GoogleDriveFetcher` downloads the full content of Google Drive files through the [Drive API v3](https://developers.google.com/drive/api/reference/rest/v3) and returns `ByteStream` objects, ready for a downstream converter.

It complements [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx), which returns only metadata (and optionally exported text). Wire the retriever's `documents` (or a list of file ids / Drive URLs) into the fetcher to download the underlying content. The fetcher dispatches on each file's mime type:

- **Binary files** (PDF, DOCX, images, ...) are downloaded as-is via `files.get?alt=media`.
- **Native Google Docs/Sheets/Slides** are exported with `files.export`, by default to the Office formats (DOCX/XLSX/PPTX), configurable via `export_mime_types`.
- **Folders** and other non-downloadable Google types (Forms, Sites, ...) are skipped.

Each `ByteStream`'s `meta` carries `file_id`, `web_url`, `file_name`, and `content_type`. Because the output is a list of `ByteStream`s of mixed types, the typical next step is a [`FileTypeRouter`](../routers/filetyperouter.mdx) that dispatches each stream to the right converter ([`PyPDFToDocument`](../converters/pypdftodocument.mdx), [`DOCXToDocument`](../converters/docxtodocument.mdx), [`XLSXToDocument`](../converters/xlsxtodocument.mdx), or [`PPTXToDocument`](../converters/pptxtodocument.mdx)).

### Authentication

The fetcher takes a per-user `access_token` as a run input. The token must carry a delegated Google OAuth scope that allows reading file content, for example `https://www.googleapis.com/auth/drive.readonly`. Typically you wire it from an upstream [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx), which emits a plain string. A `Secret` is also accepted and resolved internally.

### Error handling and concurrency

- `raise_on_failure` (default `True`): when `False`, a failed fetch is logged and the file is skipped, so the remaining files are still returned.
- `max_retries` (default `3`): retries on throttled (HTTP 429) and transient server errors.
- `max_concurrent_requests` (default `5`): bounds the number of files fetched concurrently by `run_async` to avoid tripping Drive rate limits. It has no effect on the synchronous `run`, which fetches files one at a time.
- `export_mime_types`: overrides the default native-Google-to-Office export mapping. Drive caps a single export at 10 MB.

### Installation

Install the Google Drive integration with:

```shell
pip install google-drive-haystack
```

## Usage

### On its own

`access_token` below is a per-user delegated Google OAuth bearer token. You can pass either raw file ids / Drive URLs or the `Document`s produced by `GoogleDriveRetriever`.

```python
from haystack_integrations.components.fetchers.google_drive import GoogleDriveFetcher

fetcher = GoogleDriveFetcher()

result = fetcher.run(
access_token="my-delegated-google-token",
targets=[
"https://drive.google.com/file/d/1AbCdEfGhIjKlMnOpQrStUvWxYz/view",
],
)

for stream in result["streams"]:
print(stream.meta["file_name"], stream.meta["content_type"])
```

### In a pipeline

The following query pipeline ties the whole integration together: an [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx) provides a token, [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx) searches Drive, `GoogleDriveFetcher` downloads the matching files, and a [`FileTypeRouter`](../routers/filetyperouter.mdx) sends each `ByteStream` to the right converter. Note that the resolver's single `access_token` output feeds both the retriever and the fetcher.

```python
from haystack import Pipeline
from haystack.utils import Secret
from haystack.components.routers import FileTypeRouter
from haystack.components.converters import PyPDFToDocument, DOCXToDocument

from haystack_integrations.components.connectors.oauth import OAuthTokenResolver
from haystack_integrations.utils.oauth import OAuthRefreshTokenSource
from haystack_integrations.components.retrievers.google_drive import (
GoogleDriveRetriever,
)
from haystack_integrations.components.fetchers.google_drive import GoogleDriveFetcher

pipeline = Pipeline()
pipeline.add_component(
"resolver",
OAuthTokenResolver(
token_source=OAuthRefreshTokenSource(
token_url="https://oauth2.googleapis.com/token",
client_id="aaa-bbb-ccc",
refresh_token=Secret.from_env_var("GOOGLE_REFRESH_TOKEN"),
scopes=["https://www.googleapis.com/auth/drive.readonly"],
),
),
)
pipeline.add_component("retriever", GoogleDriveRetriever(top_k=5))
pipeline.add_component("fetcher", GoogleDriveFetcher())
pipeline.add_component(
"router",
FileTypeRouter(
mime_types=[
"application/pdf",
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
],
),
)
pipeline.add_component("pdf_converter", PyPDFToDocument())
pipeline.add_component("docx_converter", DOCXToDocument())

# The same token feeds both the retriever and the fetcher.
pipeline.connect("resolver.access_token", "retriever.access_token")
pipeline.connect("resolver.access_token", "fetcher.access_token")

# The retrieved documents become the fetcher's targets.
pipeline.connect("retriever.documents", "fetcher.targets")

# Route each fetched ByteStream to the matching converter.
pipeline.connect("fetcher.streams", "router.sources")
pipeline.connect("router.application/pdf", "pdf_converter.sources")
pipeline.connect(
"router.application/vnd.openxmlformats-officedocument.wordprocessingml.document",
"docx_converter.sources",
)

result = pipeline.run({"retriever": {"query": "quarterly roadmap"}})
```
Loading
Loading