Skip to content

Commit 71513e8

Browse files
bogdankosticbogdankostic
authored
docs: Add documentation for Google Drive, Microsoft SharePoint and OAuth integrations (#11772)
1 parent 24cb0ec commit 71513e8

18 files changed

Lines changed: 1340 additions & 6 deletions

File tree

‎docs-website/docs/pipeline-components/connectors.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ These are Haystack integrations that connect your pipelines to services by exter
2020
| [GitHubRepoViewer](connectors/githubrepoviewer.mdx) | Enables navigating and fetching content from GitHub repositories through the GitHub API. |
2121
| [JinaReaderConnector](connectors/jinareaderconnector.mdx) | Use Jina AI’s Reader API with Haystack. |
2222
| [LangfuseConnector](connectors/langfuseconnector.mdx) | Enables tracing in Haystack pipelines using Langfuse. |
23+
| [OAuthTokenResolver](connectors/oauthtokenresolver.mdx) | Resolves an OAuth access token at runtime and emits it for downstream components. |
2324
| [OpenAPIConnector](connectors/openapiconnector.mdx) | Acts as an interface between the Haystack ecosystem and OpenAPI services, using explicit input arguments. |
2425
| [OpenAPIServiceConnector](connectors/openapiserviceconnector.mdx) | Acts as an interface between the Haystack ecosystem and OpenAPI services. |
2526
| [OpenTelemetryConnector](connectors/opentelemetryconnector.mdx) | Enables tracing in Haystack pipelines using OpenTelemetry. |

‎docs-website/docs/pipeline-components/connectors/oauthtokenresolver.mdx‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
---
2+
title: "OAuthTokenResolver"
3+
id: oauthtokenresolver
4+
slug: "/oauthtokenresolver"
5+
description: "Resolves an OAuth access token at pipeline runtime and emits it for downstream components such as the SharePoint and Google Drive retrievers and fetchers."
6+
---
7+
8+
# OAuthTokenResolver
9+
10+
Resolves an OAuth access token at pipeline runtime and emits it for downstream components such as the SharePoint and Google Drive retrievers and fetchers.
11+
12+
<div className="key-value-table">
13+
14+
| | |
15+
| --- | --- |
16+
| **Most common position in a pipeline** | At the start of a pipeline, feeding `access_token` into downstream components such as [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx) or [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx) |
17+
| **Mandatory init variables** | `token_source`: The strategy that resolves the access token, for example `OAuthRefreshTokenSource` |
18+
| **Mandatory run variables** | None for config-only sources. `subject_token`: a controller-injected per-request credential, mandatory only when the source requires it (for example `OAuthTokenExchangeSource`) |
19+
| **Output variables** | `access_token`: A bearer token string |
20+
| **API reference** | [OAuth](/reference/integrations-oauth) |
21+
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/oauth |
22+
| **Package name** | `oauth-haystack` |
23+
24+
</div>
25+
26+
## Overview
27+
28+
`OAuthTokenResolver` resolves an OAuth access token when the pipeline runs and emits it on the `access_token` output socket. Downstream components – such as [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx), [`MSSharePointFetcher`](../fetchers/mssharepointfetcher.mdx), [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx), and [`GoogleDriveFetcher`](../fetchers/googledrivefetcher.mdx) – consume the token through a normal connection and never need to know how it was obtained.
29+
30+
The resolver itself is a thin wrapper. The actual work of getting a token is delegated to a pluggable **token source** that decides *where* the token comes from. This separation lets you swap authentication strategies (refresh-token grant, per-request token exchange, or a static long-lived token) without changing the rest of your pipeline.
31+
32+
### Token sources
33+
34+
You pass a token source to the resolver through the `token_source` parameter. All sources are importable from `haystack_integrations.utils.oauth`.
35+
36+
| Source | Use it when | Per-request input |
37+
| --- | --- | --- |
38+
| `OAuthRefreshTokenSource` | You have a single, fixed identity backed by a stored refresh token and want the source to exchange it for short-lived access tokens and cache them. | None |
39+
| `OAuthTokenExchangeSource` | You serve multiple users (or run multiple replicas) and want to exchange an incoming per-request user assertion for a downstream token, with no persistent storage. Implements RFC 8693 token exchange and Microsoft's on-behalf-of flow. | `subject_token` |
40+
| `OAuthStaticTokenSource` | Your provider issues a non-expiring token that you manage out of band (for example Slack or Notion). | None |
41+
42+
When the configured source needs a per-request credential (`OAuthTokenExchangeSource` sets `requires_subject_token = True`), the resolver declares a **mandatory** `subject_token` run input. This is a controller-injected credential – for example an incoming user assertion – not a value chosen by an end user. For config-only sources (`OAuthRefreshTokenSource`, `OAuthStaticTokenSource`), the resolver declares no run input and acts as a source node.
43+
44+
:::info[Scopes are provider-specific]
45+
46+
The OAuth scopes you request depend on the downstream service. For Microsoft Graph, that means scopes such as `https://graph.microsoft.com/Files.Read.All`; for Google Drive, scopes such as `https://www.googleapis.com/auth/drive.readonly`. Always consult your identity provider's documentation for the exact scope values.
47+
48+
:::
49+
50+
### Installation
51+
52+
Install the OAuth integration with:
53+
54+
```shell
55+
pip install oauth-haystack
56+
```
57+
58+
## Usage
59+
60+
### On its own
61+
62+
Resolve a token with a stored refresh token using `OAuthRefreshTokenSource`. The refresh token is read from an environment variable through the [Secret API](../../concepts/secret-management.mdx):
63+
64+
```python
65+
from haystack.utils import Secret
66+
from haystack_integrations.components.connectors.oauth import OAuthTokenResolver
67+
from haystack_integrations.utils.oauth import OAuthRefreshTokenSource
68+
69+
resolver = OAuthTokenResolver(
70+
token_source=OAuthRefreshTokenSource(
71+
token_url="https://login.microsoftonline.com/common/oauth2/v2.0/token",
72+
client_id="aaa-bbb-ccc",
73+
refresh_token=Secret.from_env_var("MS_REFRESH_TOKEN"),
74+
scopes=[
75+
"https://graph.microsoft.com/Files.Read.All",
76+
"offline_access",
77+
],
78+
),
79+
)
80+
81+
access_token = resolver.run()["access_token"]
82+
```
83+
84+
For a provider that issues long-lived, non-expiring tokens, use `OAuthStaticTokenSource` instead:
85+
86+
```python
87+
from haystack.utils import Secret
88+
from haystack_integrations.components.connectors.oauth import OAuthTokenResolver
89+
from haystack_integrations.utils.oauth import OAuthStaticTokenSource
90+
91+
resolver = OAuthTokenResolver(
92+
token_source=OAuthStaticTokenSource(token=Secret.from_env_var("SERVICE_TOKEN")),
93+
)
94+
95+
access_token = resolver.run()["access_token"]
96+
```
97+
98+
For multi-user backends, use `OAuthTokenExchangeSource`. The resolver then requires a per-request `subject_token`:
99+
100+
```python
101+
from haystack_integrations.components.connectors.oauth import OAuthTokenResolver
102+
from haystack_integrations.utils.oauth import OAuthTokenExchangeSource
103+
104+
resolver = OAuthTokenResolver(
105+
token_source=OAuthTokenExchangeSource(
106+
token_url="https://login.microsoftonline.com/<tenant>/oauth2/v2.0/token",
107+
client_id="aaa-bbb-ccc",
108+
subject_token_param="assertion",
109+
grant_type="urn:ietf:params:oauth:grant-type:jwt-bearer",
110+
scopes=["https://graph.microsoft.com/Files.Read.All"],
111+
extra_token_params={"requested_token_use": "on_behalf_of"},
112+
),
113+
)
114+
115+
# `subject_token` is the incoming per-request user assertion, injected by your application.
116+
access_token = resolver.run(subject_token="<incoming-user-assertion>")["access_token"]
117+
```
118+
119+
### In a pipeline
120+
121+
In a pipeline, connect the resolver's `access_token` output to the `access_token` input of one or more downstream components. The example below wires the resolver into a [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx) so that searching SharePoint requires only a query at runtime:
122+
123+
```python
124+
from haystack import Pipeline
125+
from haystack.utils import Secret
126+
from haystack_integrations.components.connectors.oauth import OAuthTokenResolver
127+
from haystack_integrations.utils.oauth import OAuthRefreshTokenSource
128+
from haystack_integrations.components.retrievers.microsoft_sharepoint import (
129+
MSSharePointRetriever,
130+
)
131+
132+
pipeline = Pipeline()
133+
pipeline.add_component(
134+
"resolver",
135+
OAuthTokenResolver(
136+
token_source=OAuthRefreshTokenSource(
137+
token_url="https://login.microsoftonline.com/common/oauth2/v2.0/token",
138+
client_id="aaa-bbb-ccc",
139+
refresh_token=Secret.from_env_var("MS_REFRESH_TOKEN"),
140+
scopes=[
141+
"https://graph.microsoft.com/Files.Read.All",
142+
"https://graph.microsoft.com/Sites.Read.All",
143+
"offline_access",
144+
],
145+
),
146+
),
147+
)
148+
pipeline.add_component("retriever", MSSharePointRetriever(top_k=5))
149+
pipeline.connect("resolver.access_token", "retriever.access_token")
150+
151+
result = pipeline.run({"retriever": {"query": "quarterly roadmap"}})
152+
documents = result["retriever"]["documents"]
153+
```
154+
155+
A single `access_token` output can be connected to several downstream inputs. For a full retrieve-then-fetch pipeline that feeds the same token to both a retriever and a fetcher, see the [`MSSharePointFetcher`](../fetchers/mssharepointfetcher.mdx) and [`GoogleDriveFetcher`](../fetchers/googledrivefetcher.mdx) pages.

‎docs-website/docs/pipeline-components/fetchers.mdx‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,16 @@
22
title: "Fetchers"
33
id: fetchers
44
slug: "/fetchers"
5-
description: "Currently, there's one Fetcher in Haystack: LinkContentFetcher. It fetches the contents of the URLs you give it."
5+
description: "Fetchers retrieve content from external sources – URLs, web crawls, or cloud storage such as SharePoint and Google Drive – so you can use it as data for your pipelines."
66
---
77

88
# Fetchers
99

10-
Currently, there's one Fetcher in Haystack: LinkContentFetcher. It fetches the contents of the URLs you give it.
10+
Fetchers retrieve content from external sources – URLs, web crawls, or cloud storage such as SharePoint and Google Drive – so you can use it as data for your pipelines.
1111

1212
| Component | Description |
1313
| --- | --- |
14-
| [LinkContentFetcher](fetchers/linkcontentfetcher.mdx) | Fetches the contents of the URLs you give it so you can use them as data for your pipelines. |
14+
| [FirecrawlCrawler](fetchers/firecrawlcrawler.mdx) | Crawls websites with Firecrawl, following links to discover subpages, and returns them as Documents. |
15+
| [GoogleDriveFetcher](fetchers/googledrivefetcher.mdx) | Fetches the full content of Google Drive files via the Drive API v3 and returns it as ByteStreams. |
16+
| [LinkContentFetcher](fetchers/linkcontentfetcher.mdx) | Fetches the contents of the URLs you give it so you can use them as data for your pipelines. |
17+
| [MSSharePointFetcher](fetchers/mssharepointfetcher.mdx) | Fetches the full content of Microsoft SharePoint and OneDrive items via the Microsoft Graph API and returns it as ByteStreams. |

‎docs-website/docs/pipeline-components/fetchers/googledrivefetcher.mdx‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
---
2+
title: "GoogleDriveFetcher"
3+
id: googledrivefetcher
4+
slug: "/googledrivefetcher"
5+
description: "Fetches the full content of Google Drive files via the Drive API v3 and returns it as ByteStreams."
6+
---
7+
8+
# GoogleDriveFetcher
9+
10+
Fetches the full content of Google Drive files via the Drive API v3 and returns it as ByteStreams.
11+
12+
<div className="key-value-table">
13+
14+
| | |
15+
| --- | --- |
16+
| **Most common position in a pipeline** | After [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx), before a Router or File Converters |
17+
| **Mandatory init variables** | None |
18+
| **Mandatory run variables** | `access_token`: A delegated Google OAuth bearer token, typically wired from an upstream `OAuthTokenResolver` <br /> <br />`targets`: A list of `Document`s (from `GoogleDriveRetriever`) or raw Google Drive file ids / URLs |
19+
| **Output variables** | `streams`: A list of [ByteStreams](../../concepts/data-classes.mdx) holding the fetched content |
20+
| **API reference** | [Google Drive](/reference/integrations-google-drive) |
21+
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/google_drive |
22+
| **Package name** | `google-drive-haystack` |
23+
24+
</div>
25+
26+
## Overview
27+
28+
`GoogleDriveFetcher` downloads the full content of Google Drive files through the [Drive API v3](https://developers.google.com/drive/api/reference/rest/v3) and returns `ByteStream` objects, ready for a downstream converter.
29+
30+
It complements [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx), which returns only metadata (and optionally exported text). Wire the retriever's `documents` (or a list of file ids / Drive URLs) into the fetcher to download the underlying content. The fetcher dispatches on each file's mime type:
31+
32+
- **Binary files** (PDF, DOCX, images, ...) are downloaded as-is via `files.get?alt=media`.
33+
- **Native Google Docs/Sheets/Slides** are exported with `files.export`, by default to the Office formats (DOCX/XLSX/PPTX), configurable via `export_mime_types`.
34+
- **Folders** and other non-downloadable Google types (Forms, Sites, ...) are skipped.
35+
36+
Each `ByteStream`'s `meta` carries `file_id`, `web_url`, `file_name`, and `content_type`. Because the output is a list of `ByteStream`s of mixed types, the typical next step is a [`FileTypeRouter`](../routers/filetyperouter.mdx) that dispatches each stream to the right converter ([`PyPDFToDocument`](../converters/pypdftodocument.mdx), [`DOCXToDocument`](../converters/docxtodocument.mdx), [`XLSXToDocument`](../converters/xlsxtodocument.mdx), or [`PPTXToDocument`](../converters/pptxtodocument.mdx)).
37+
38+
### Authentication
39+
40+
The fetcher takes a per-user `access_token` as a run input. The token must carry a delegated Google OAuth scope that allows reading file content, for example `https://www.googleapis.com/auth/drive.readonly`. Typically you wire it from an upstream [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx), which emits a plain string. A `Secret` is also accepted and resolved internally.
41+
42+
### Error handling and concurrency
43+
44+
- `raise_on_failure` (default `True`): when `False`, a failed fetch is logged and the file is skipped, so the remaining files are still returned.
45+
- `max_retries` (default `3`): retries on throttled (HTTP 429) and transient server errors.
46+
- `max_concurrent_requests` (default `5`): bounds the number of files fetched concurrently by `run_async` to avoid tripping Drive rate limits. It has no effect on the synchronous `run`, which fetches files one at a time.
47+
- `export_mime_types`: overrides the default native-Google-to-Office export mapping. Drive caps a single export at 10 MB.
48+
49+
### Installation
50+
51+
Install the Google Drive integration with:
52+
53+
```shell
54+
pip install google-drive-haystack
55+
```
56+
57+
## Usage
58+
59+
### On its own
60+
61+
`access_token` below is a per-user delegated Google OAuth bearer token. You can pass either raw file ids / Drive URLs or the `Document`s produced by `GoogleDriveRetriever`.
62+
63+
```python
64+
from haystack_integrations.components.fetchers.google_drive import GoogleDriveFetcher
65+
66+
fetcher = GoogleDriveFetcher()
67+
68+
result = fetcher.run(
69+
access_token="my-delegated-google-token",
70+
targets=[
71+
"https://drive.google.com/file/d/1AbCdEfGhIjKlMnOpQrStUvWxYz/view",
72+
],
73+
)
74+
75+
for stream in result["streams"]:
76+
print(stream.meta["file_name"], stream.meta["content_type"])
77+
```
78+
79+
### In a pipeline
80+
81+
The following query pipeline ties the whole integration together: an [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx) provides a token, [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx) searches Drive, `GoogleDriveFetcher` downloads the matching files, and a [`FileTypeRouter`](../routers/filetyperouter.mdx) sends each `ByteStream` to the right converter. Note that the resolver's single `access_token` output feeds both the retriever and the fetcher.
82+
83+
```python
84+
from haystack import Pipeline
85+
from haystack.utils import Secret
86+
from haystack.components.routers import FileTypeRouter
87+
from haystack.components.converters import PyPDFToDocument, DOCXToDocument
88+
89+
from haystack_integrations.components.connectors.oauth import OAuthTokenResolver
90+
from haystack_integrations.utils.oauth import OAuthRefreshTokenSource
91+
from haystack_integrations.components.retrievers.google_drive import (
92+
GoogleDriveRetriever,
93+
)
94+
from haystack_integrations.components.fetchers.google_drive import GoogleDriveFetcher
95+
96+
pipeline = Pipeline()
97+
pipeline.add_component(
98+
"resolver",
99+
OAuthTokenResolver(
100+
token_source=OAuthRefreshTokenSource(
101+
token_url="https://oauth2.googleapis.com/token",
102+
client_id="aaa-bbb-ccc",
103+
refresh_token=Secret.from_env_var("GOOGLE_REFRESH_TOKEN"),
104+
scopes=["https://www.googleapis.com/auth/drive.readonly"],
105+
),
106+
),
107+
)
108+
pipeline.add_component("retriever", GoogleDriveRetriever(top_k=5))
109+
pipeline.add_component("fetcher", GoogleDriveFetcher())
110+
pipeline.add_component(
111+
"router",
112+
FileTypeRouter(
113+
mime_types=[
114+
"application/pdf",
115+
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
116+
],
117+
),
118+
)
119+
pipeline.add_component("pdf_converter", PyPDFToDocument())
120+
pipeline.add_component("docx_converter", DOCXToDocument())
121+
122+
# The same token feeds both the retriever and the fetcher.
123+
pipeline.connect("resolver.access_token", "retriever.access_token")
124+
pipeline.connect("resolver.access_token", "fetcher.access_token")
125+
126+
# The retrieved documents become the fetcher's targets.
127+
pipeline.connect("retriever.documents", "fetcher.targets")
128+
129+
# Route each fetched ByteStream to the matching converter.
130+
pipeline.connect("fetcher.streams", "router.sources")
131+
pipeline.connect("router.application/pdf", "pdf_converter.sources")
132+
pipeline.connect(
133+
"router.application/vnd.openxmlformats-officedocument.wordprocessingml.document",
134+
"docx_converter.sources",
135+
)
136+
137+
result = pipeline.run({"retriever": {"query": "quarterly roadmap"}})
138+
```

0 commit comments

Comments
 (0)