diff --git a/docs-website/docs/pipeline-components/connectors.mdx b/docs-website/docs/pipeline-components/connectors.mdx index 0341bb0dcda..3ffb6ecd69c 100644 --- a/docs-website/docs/pipeline-components/connectors.mdx +++ b/docs-website/docs/pipeline-components/connectors.mdx @@ -20,6 +20,7 @@ These are Haystack integrations that connect your pipelines to services by exter | [GitHubRepoViewer](connectors/githubrepoviewer.mdx) | Enables navigating and fetching content from GitHub repositories through the GitHub API. | | [JinaReaderConnector](connectors/jinareaderconnector.mdx) | Use Jina AI’s Reader API with Haystack. | | [LangfuseConnector](connectors/langfuseconnector.mdx) | Enables tracing in Haystack pipelines using Langfuse. | +| [OAuthTokenResolver](connectors/oauthtokenresolver.mdx) | Resolves an OAuth access token at runtime and emits it for downstream components. | | [OpenAPIConnector](connectors/openapiconnector.mdx) | Acts as an interface between the Haystack ecosystem and OpenAPI services, using explicit input arguments. | | [OpenAPIServiceConnector](connectors/openapiserviceconnector.mdx) | Acts as an interface between the Haystack ecosystem and OpenAPI services. | | [OpenTelemetryConnector](connectors/opentelemetryconnector.mdx) | Enables tracing in Haystack pipelines using OpenTelemetry. | diff --git a/docs-website/docs/pipeline-components/connectors/oauthtokenresolver.mdx b/docs-website/docs/pipeline-components/connectors/oauthtokenresolver.mdx new file mode 100644 index 00000000000..ac1ec63fdc0 --- /dev/null +++ b/docs-website/docs/pipeline-components/connectors/oauthtokenresolver.mdx @@ -0,0 +1,155 @@ +--- +title: "OAuthTokenResolver" +id: oauthtokenresolver +slug: "/oauthtokenresolver" +description: "Resolves an OAuth access token at pipeline runtime and emits it for downstream components such as the SharePoint and Google Drive retrievers and fetchers." +--- + +# OAuthTokenResolver + +Resolves an OAuth access token at pipeline runtime and emits it for downstream components such as the SharePoint and Google Drive retrievers and fetchers. + +
+ +| | | +| --- | --- | +| **Most common position in a pipeline** | At the start of a pipeline, feeding `access_token` into downstream components such as [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx) or [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx) | +| **Mandatory init variables** | `token_source`: The strategy that resolves the access token, for example `OAuthRefreshTokenSource` | +| **Mandatory run variables** | None for config-only sources. `subject_token`: a controller-injected per-request credential, mandatory only when the source requires it (for example `OAuthTokenExchangeSource`) | +| **Output variables** | `access_token`: A bearer token string | +| **API reference** | [OAuth](/reference/integrations-oauth) | +| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/oauth | +| **Package name** | `oauth-haystack` | + +
+ +## Overview + +`OAuthTokenResolver` resolves an OAuth access token when the pipeline runs and emits it on the `access_token` output socket. Downstream components – such as [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx), [`MSSharePointFetcher`](../fetchers/mssharepointfetcher.mdx), [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx), and [`GoogleDriveFetcher`](../fetchers/googledrivefetcher.mdx) – consume the token through a normal connection and never need to know how it was obtained. + +The resolver itself is a thin wrapper. The actual work of getting a token is delegated to a pluggable **token source** that decides *where* the token comes from. This separation lets you swap authentication strategies (refresh-token grant, per-request token exchange, or a static long-lived token) without changing the rest of your pipeline. + +### Token sources + +You pass a token source to the resolver through the `token_source` parameter. All sources are importable from `haystack_integrations.utils.oauth`. + +| Source | Use it when | Per-request input | +| --- | --- | --- | +| `OAuthRefreshTokenSource` | You have a single, fixed identity backed by a stored refresh token and want the source to exchange it for short-lived access tokens and cache them. | None | +| `OAuthTokenExchangeSource` | You serve multiple users (or run multiple replicas) and want to exchange an incoming per-request user assertion for a downstream token, with no persistent storage. Implements RFC 8693 token exchange and Microsoft's on-behalf-of flow. | `subject_token` | +| `OAuthStaticTokenSource` | Your provider issues a non-expiring token that you manage out of band (for example Slack or Notion). | None | + +When the configured source needs a per-request credential (`OAuthTokenExchangeSource` sets `requires_subject_token = True`), the resolver declares a **mandatory** `subject_token` run input. This is a controller-injected credential – for example an incoming user assertion – not a value chosen by an end user. For config-only sources (`OAuthRefreshTokenSource`, `OAuthStaticTokenSource`), the resolver declares no run input and acts as a source node. + +:::info[Scopes are provider-specific] + +The OAuth scopes you request depend on the downstream service. For Microsoft Graph, that means scopes such as `https://graph.microsoft.com/Files.Read.All`; for Google Drive, scopes such as `https://www.googleapis.com/auth/drive.readonly`. Always consult your identity provider's documentation for the exact scope values. + +::: + +### Installation + +Install the OAuth integration with: + +```shell +pip install oauth-haystack +``` + +## Usage + +### On its own + +Resolve a token with a stored refresh token using `OAuthRefreshTokenSource`. The refresh token is read from an environment variable through the [Secret API](../../concepts/secret-management.mdx): + +```python +from haystack.utils import Secret +from haystack_integrations.components.connectors.oauth import OAuthTokenResolver +from haystack_integrations.utils.oauth import OAuthRefreshTokenSource + +resolver = OAuthTokenResolver( + token_source=OAuthRefreshTokenSource( + token_url="https://login.microsoftonline.com/common/oauth2/v2.0/token", + client_id="aaa-bbb-ccc", + refresh_token=Secret.from_env_var("MS_REFRESH_TOKEN"), + scopes=[ + "https://graph.microsoft.com/Files.Read.All", + "offline_access", + ], + ), +) + +access_token = resolver.run()["access_token"] +``` + +For a provider that issues long-lived, non-expiring tokens, use `OAuthStaticTokenSource` instead: + +```python +from haystack.utils import Secret +from haystack_integrations.components.connectors.oauth import OAuthTokenResolver +from haystack_integrations.utils.oauth import OAuthStaticTokenSource + +resolver = OAuthTokenResolver( + token_source=OAuthStaticTokenSource(token=Secret.from_env_var("SERVICE_TOKEN")), +) + +access_token = resolver.run()["access_token"] +``` + +For multi-user backends, use `OAuthTokenExchangeSource`. The resolver then requires a per-request `subject_token`: + +```python +from haystack_integrations.components.connectors.oauth import OAuthTokenResolver +from haystack_integrations.utils.oauth import OAuthTokenExchangeSource + +resolver = OAuthTokenResolver( + token_source=OAuthTokenExchangeSource( + token_url="https://login.microsoftonline.com//oauth2/v2.0/token", + client_id="aaa-bbb-ccc", + subject_token_param="assertion", + grant_type="urn:ietf:params:oauth:grant-type:jwt-bearer", + scopes=["https://graph.microsoft.com/Files.Read.All"], + extra_token_params={"requested_token_use": "on_behalf_of"}, + ), +) + +# `subject_token` is the incoming per-request user assertion, injected by your application. +access_token = resolver.run(subject_token="")["access_token"] +``` + +### In a pipeline + +In a pipeline, connect the resolver's `access_token` output to the `access_token` input of one or more downstream components. The example below wires the resolver into a [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx) so that searching SharePoint requires only a query at runtime: + +```python +from haystack import Pipeline +from haystack.utils import Secret +from haystack_integrations.components.connectors.oauth import OAuthTokenResolver +from haystack_integrations.utils.oauth import OAuthRefreshTokenSource +from haystack_integrations.components.retrievers.microsoft_sharepoint import ( + MSSharePointRetriever, +) + +pipeline = Pipeline() +pipeline.add_component( + "resolver", + OAuthTokenResolver( + token_source=OAuthRefreshTokenSource( + token_url="https://login.microsoftonline.com/common/oauth2/v2.0/token", + client_id="aaa-bbb-ccc", + refresh_token=Secret.from_env_var("MS_REFRESH_TOKEN"), + scopes=[ + "https://graph.microsoft.com/Files.Read.All", + "https://graph.microsoft.com/Sites.Read.All", + "offline_access", + ], + ), + ), +) +pipeline.add_component("retriever", MSSharePointRetriever(top_k=5)) +pipeline.connect("resolver.access_token", "retriever.access_token") + +result = pipeline.run({"retriever": {"query": "quarterly roadmap"}}) +documents = result["retriever"]["documents"] +``` + +A single `access_token` output can be connected to several downstream inputs. For a full retrieve-then-fetch pipeline that feeds the same token to both a retriever and a fetcher, see the [`MSSharePointFetcher`](../fetchers/mssharepointfetcher.mdx) and [`GoogleDriveFetcher`](../fetchers/googledrivefetcher.mdx) pages. diff --git a/docs-website/docs/pipeline-components/fetchers.mdx b/docs-website/docs/pipeline-components/fetchers.mdx index e06b360a158..9a3596da83f 100644 --- a/docs-website/docs/pipeline-components/fetchers.mdx +++ b/docs-website/docs/pipeline-components/fetchers.mdx @@ -2,13 +2,16 @@ title: "Fetchers" id: fetchers slug: "/fetchers" -description: "Currently, there's one Fetcher in Haystack: LinkContentFetcher. It fetches the contents of the URLs you give it." +description: "Fetchers retrieve content from external sources – URLs, web crawls, or cloud storage such as SharePoint and Google Drive – so you can use it as data for your pipelines." --- # Fetchers -Currently, there's one Fetcher in Haystack: LinkContentFetcher. It fetches the contents of the URLs you give it. +Fetchers retrieve content from external sources – URLs, web crawls, or cloud storage such as SharePoint and Google Drive – so you can use it as data for your pipelines. | Component | Description | | --- | --- | -| [LinkContentFetcher](fetchers/linkcontentfetcher.mdx) | Fetches the contents of the URLs you give it so you can use them as data for your pipelines. | \ No newline at end of file +| [FirecrawlCrawler](fetchers/firecrawlcrawler.mdx) | Crawls websites with Firecrawl, following links to discover subpages, and returns them as Documents. | +| [GoogleDriveFetcher](fetchers/googledrivefetcher.mdx) | Fetches the full content of Google Drive files via the Drive API v3 and returns it as ByteStreams. | +| [LinkContentFetcher](fetchers/linkcontentfetcher.mdx) | Fetches the contents of the URLs you give it so you can use them as data for your pipelines. | +| [MSSharePointFetcher](fetchers/mssharepointfetcher.mdx) | Fetches the full content of Microsoft SharePoint and OneDrive items via the Microsoft Graph API and returns it as ByteStreams. | diff --git a/docs-website/docs/pipeline-components/fetchers/googledrivefetcher.mdx b/docs-website/docs/pipeline-components/fetchers/googledrivefetcher.mdx new file mode 100644 index 00000000000..899d6b24796 --- /dev/null +++ b/docs-website/docs/pipeline-components/fetchers/googledrivefetcher.mdx @@ -0,0 +1,138 @@ +--- +title: "GoogleDriveFetcher" +id: googledrivefetcher +slug: "/googledrivefetcher" +description: "Fetches the full content of Google Drive files via the Drive API v3 and returns it as ByteStreams." +--- + +# GoogleDriveFetcher + +Fetches the full content of Google Drive files via the Drive API v3 and returns it as ByteStreams. + +
+ +| | | +| --- | --- | +| **Most common position in a pipeline** | After [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx), before a Router or File Converters | +| **Mandatory init variables** | None | +| **Mandatory run variables** | `access_token`: A delegated Google OAuth bearer token, typically wired from an upstream `OAuthTokenResolver`

`targets`: A list of `Document`s (from `GoogleDriveRetriever`) or raw Google Drive file ids / URLs | +| **Output variables** | `streams`: A list of [ByteStreams](../../concepts/data-classes.mdx) holding the fetched content | +| **API reference** | [Google Drive](/reference/integrations-google-drive) | +| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/google_drive | +| **Package name** | `google-drive-haystack` | + +
+ +## Overview + +`GoogleDriveFetcher` downloads the full content of Google Drive files through the [Drive API v3](https://developers.google.com/drive/api/reference/rest/v3) and returns `ByteStream` objects, ready for a downstream converter. + +It complements [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx), which returns only metadata (and optionally exported text). Wire the retriever's `documents` (or a list of file ids / Drive URLs) into the fetcher to download the underlying content. The fetcher dispatches on each file's mime type: + +- **Binary files** (PDF, DOCX, images, ...) are downloaded as-is via `files.get?alt=media`. +- **Native Google Docs/Sheets/Slides** are exported with `files.export`, by default to the Office formats (DOCX/XLSX/PPTX), configurable via `export_mime_types`. +- **Folders** and other non-downloadable Google types (Forms, Sites, ...) are skipped. + +Each `ByteStream`'s `meta` carries `file_id`, `web_url`, `file_name`, and `content_type`. Because the output is a list of `ByteStream`s of mixed types, the typical next step is a [`FileTypeRouter`](../routers/filetyperouter.mdx) that dispatches each stream to the right converter ([`PyPDFToDocument`](../converters/pypdftodocument.mdx), [`DOCXToDocument`](../converters/docxtodocument.mdx), [`XLSXToDocument`](../converters/xlsxtodocument.mdx), or [`PPTXToDocument`](../converters/pptxtodocument.mdx)). + +### Authentication + +The fetcher takes a per-user `access_token` as a run input. The token must carry a delegated Google OAuth scope that allows reading file content, for example `https://www.googleapis.com/auth/drive.readonly`. Typically you wire it from an upstream [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx), which emits a plain string. A `Secret` is also accepted and resolved internally. + +### Error handling and concurrency + +- `raise_on_failure` (default `True`): when `False`, a failed fetch is logged and the file is skipped, so the remaining files are still returned. +- `max_retries` (default `3`): retries on throttled (HTTP 429) and transient server errors. +- `max_concurrent_requests` (default `5`): bounds the number of files fetched concurrently by `run_async` to avoid tripping Drive rate limits. It has no effect on the synchronous `run`, which fetches files one at a time. +- `export_mime_types`: overrides the default native-Google-to-Office export mapping. Drive caps a single export at 10 MB. + +### Installation + +Install the Google Drive integration with: + +```shell +pip install google-drive-haystack +``` + +## Usage + +### On its own + +`access_token` below is a per-user delegated Google OAuth bearer token. You can pass either raw file ids / Drive URLs or the `Document`s produced by `GoogleDriveRetriever`. + +```python +from haystack_integrations.components.fetchers.google_drive import GoogleDriveFetcher + +fetcher = GoogleDriveFetcher() + +result = fetcher.run( + access_token="my-delegated-google-token", + targets=[ + "https://drive.google.com/file/d/1AbCdEfGhIjKlMnOpQrStUvWxYz/view", + ], +) + +for stream in result["streams"]: + print(stream.meta["file_name"], stream.meta["content_type"]) +``` + +### In a pipeline + +The following query pipeline ties the whole integration together: an [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx) provides a token, [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx) searches Drive, `GoogleDriveFetcher` downloads the matching files, and a [`FileTypeRouter`](../routers/filetyperouter.mdx) sends each `ByteStream` to the right converter. Note that the resolver's single `access_token` output feeds both the retriever and the fetcher. + +```python +from haystack import Pipeline +from haystack.utils import Secret +from haystack.components.routers import FileTypeRouter +from haystack.components.converters import PyPDFToDocument, DOCXToDocument + +from haystack_integrations.components.connectors.oauth import OAuthTokenResolver +from haystack_integrations.utils.oauth import OAuthRefreshTokenSource +from haystack_integrations.components.retrievers.google_drive import ( + GoogleDriveRetriever, +) +from haystack_integrations.components.fetchers.google_drive import GoogleDriveFetcher + +pipeline = Pipeline() +pipeline.add_component( + "resolver", + OAuthTokenResolver( + token_source=OAuthRefreshTokenSource( + token_url="https://oauth2.googleapis.com/token", + client_id="aaa-bbb-ccc", + refresh_token=Secret.from_env_var("GOOGLE_REFRESH_TOKEN"), + scopes=["https://www.googleapis.com/auth/drive.readonly"], + ), + ), +) +pipeline.add_component("retriever", GoogleDriveRetriever(top_k=5)) +pipeline.add_component("fetcher", GoogleDriveFetcher()) +pipeline.add_component( + "router", + FileTypeRouter( + mime_types=[ + "application/pdf", + "application/vnd.openxmlformats-officedocument.wordprocessingml.document", + ], + ), +) +pipeline.add_component("pdf_converter", PyPDFToDocument()) +pipeline.add_component("docx_converter", DOCXToDocument()) + +# The same token feeds both the retriever and the fetcher. +pipeline.connect("resolver.access_token", "retriever.access_token") +pipeline.connect("resolver.access_token", "fetcher.access_token") + +# The retrieved documents become the fetcher's targets. +pipeline.connect("retriever.documents", "fetcher.targets") + +# Route each fetched ByteStream to the matching converter. +pipeline.connect("fetcher.streams", "router.sources") +pipeline.connect("router.application/pdf", "pdf_converter.sources") +pipeline.connect( + "router.application/vnd.openxmlformats-officedocument.wordprocessingml.document", + "docx_converter.sources", +) + +result = pipeline.run({"retriever": {"query": "quarterly roadmap"}}) +``` diff --git a/docs-website/docs/pipeline-components/fetchers/mssharepointfetcher.mdx b/docs-website/docs/pipeline-components/fetchers/mssharepointfetcher.mdx new file mode 100644 index 00000000000..3989d6bc041 --- /dev/null +++ b/docs-website/docs/pipeline-components/fetchers/mssharepointfetcher.mdx @@ -0,0 +1,147 @@ +--- +title: "MSSharePointFetcher" +id: mssharepointfetcher +slug: "/mssharepointfetcher" +description: "Fetches the full content of Microsoft SharePoint and OneDrive items via the Microsoft Graph API and returns it as ByteStreams." +--- + +# MSSharePointFetcher + +Fetches the full content of Microsoft SharePoint and OneDrive items via the Microsoft Graph API and returns it as ByteStreams. + +
+ +| | | +| --- | --- | +| **Most common position in a pipeline** | After [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx), before a Router or File Converters | +| **Mandatory init variables** | None | +| **Mandatory run variables** | `access_token`: A delegated Microsoft Graph bearer token, typically wired from an upstream `OAuthTokenResolver`

`targets`: A list of `Document`s (from `MSSharePointRetriever`) or raw SharePoint/OneDrive `web_url` strings | +| **Output variables** | `streams`: A list of [ByteStreams](../../concepts/data-classes.mdx) holding the fetched content | +| **API reference** | [Microsoft SharePoint](/reference/integrations-microsoft-sharepoint) | +| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/microsoft_sharepoint | +| **Package name** | `microsoft-sharepoint-haystack` | + +
+ +## Overview + +`MSSharePointFetcher` downloads the full content of Microsoft SharePoint and OneDrive items through the [Microsoft Graph API](https://learn.microsoft.com/en-us/graph/use-the-api) and returns `ByteStream` objects, ready for a downstream converter. + +It complements [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx), which returns only Search snippets and metadata. Wire the retriever's `documents` (or a list of `web_url`s) into the fetcher to download the underlying content. The fetcher dispatches on the entity type of each hit: + +- **Files** (`driveItem`) are downloaded as their raw bytes (PDF, DOCX, ...). +- **List items** (`listItem`) are returned as a JSON `ByteStream` of the item's column values (`fields`). +- **SharePoint pages** (`sitePage`) are returned as an HTML `ByteStream` built from the page's web parts. + +Each `ByteStream`'s `meta` carries `url`, `file_name`, `content_type`, and a normalized `entity_type` (`driveItem`, `listItem`, or `sitePage`). Everything is resolved through the Microsoft Graph `shares` endpoint (plus the Pages API for pages), so only the `web_url` already exposed by the retriever is needed. + +Because the output is a list of `ByteStream`s of mixed types, the typical next step is a [`FileTypeRouter`](../routers/filetyperouter.mdx) that dispatches each stream to the right converter ([`PyPDFToDocument`](../converters/pypdftodocument.mdx), [`DOCXToDocument`](../converters/docxtodocument.mdx), [`HTMLToDocument`](../converters/htmltodocument.mdx), or a JSON converter). + +### Authentication + +The fetcher takes a per-user `access_token` as a run input. The token must carry **delegated** Microsoft Graph permissions (for example `Files.Read.All` for files and `Sites.Read.All` for list items and pages). Typically you wire it from an upstream [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx), which emits a plain string. A `Secret` is also accepted and resolved internally. + +### Error handling and concurrency + +- `raise_on_failure` (default `True`): when `False`, a failed fetch is logged and the item is skipped, so the remaining items are still returned. +- `max_retries` (default `3`): retries on throttled (HTTP 429) and transient server errors. +- `max_concurrent_requests` (default `5`): bounds the number of items fetched concurrently by `run_async` to avoid tripping Microsoft Graph rate limits. It has no effect on the synchronous `run`, which fetches items one at a time. + +### Installation + +Install the Microsoft SharePoint integration with: + +```shell +pip install microsoft-sharepoint-haystack +``` + +## Usage + +### On its own + +`access_token` below is a per-user delegated Microsoft Graph bearer token. You can pass either raw `web_url` strings or the `Document`s produced by `MSSharePointRetriever`. + +```python +from haystack_integrations.components.fetchers.microsoft_sharepoint import ( + MSSharePointFetcher, +) + +fetcher = MSSharePointFetcher() + +result = fetcher.run( + access_token="my-delegated-graph-token", + targets=[ + "https://contoso.sharepoint.com/sites/contoso-team/contoso-designs.docx", + ], +) + +for stream in result["streams"]: + print(stream.meta["file_name"], stream.meta["content_type"]) +``` + +### In a pipeline + +The following query pipeline ties the whole integration together: an [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx) provides a token, [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx) searches SharePoint, `MSSharePointFetcher` downloads the matching items, and a [`FileTypeRouter`](../routers/filetyperouter.mdx) sends each `ByteStream` to the right converter. Note that the resolver's single `access_token` output feeds both the retriever and the fetcher. + +```python +from haystack import Pipeline +from haystack.utils import Secret +from haystack.components.routers import FileTypeRouter +from haystack.components.converters import PyPDFToDocument, DOCXToDocument + +from haystack_integrations.components.connectors.oauth import OAuthTokenResolver +from haystack_integrations.utils.oauth import OAuthRefreshTokenSource +from haystack_integrations.components.retrievers.microsoft_sharepoint import ( + MSSharePointRetriever, +) +from haystack_integrations.components.fetchers.microsoft_sharepoint import ( + MSSharePointFetcher, +) + +pipeline = Pipeline() +pipeline.add_component( + "resolver", + OAuthTokenResolver( + token_source=OAuthRefreshTokenSource( + token_url="https://login.microsoftonline.com/common/oauth2/v2.0/token", + client_id="aaa-bbb-ccc", + refresh_token=Secret.from_env_var("MS_REFRESH_TOKEN"), + scopes=[ + "https://graph.microsoft.com/Files.Read.All", + "https://graph.microsoft.com/Sites.Read.All", + "offline_access", + ], + ), + ), +) +pipeline.add_component("retriever", MSSharePointRetriever(top_k=5)) +pipeline.add_component("fetcher", MSSharePointFetcher()) +pipeline.add_component( + "router", + FileTypeRouter( + mime_types=[ + "application/pdf", + "application/vnd.openxmlformats-officedocument.wordprocessingml.document", + ], + ), +) +pipeline.add_component("pdf_converter", PyPDFToDocument()) +pipeline.add_component("docx_converter", DOCXToDocument()) + +# The same token feeds both the retriever and the fetcher. +pipeline.connect("resolver.access_token", "retriever.access_token") +pipeline.connect("resolver.access_token", "fetcher.access_token") + +# The retrieved documents become the fetcher's targets. +pipeline.connect("retriever.documents", "fetcher.targets") + +# Route each fetched ByteStream to the matching converter. +pipeline.connect("fetcher.streams", "router.sources") +pipeline.connect("router.application/pdf", "pdf_converter.sources") +pipeline.connect( + "router.application/vnd.openxmlformats-officedocument.wordprocessingml.document", + "docx_converter.sources", +) + +result = pipeline.run({"retriever": {"query": "quarterly roadmap"}}) +``` diff --git a/docs-website/docs/pipeline-components/retrievers.mdx b/docs-website/docs/pipeline-components/retrievers.mdx index f200649ae11..1bff1d7719e 100644 --- a/docs-website/docs/pipeline-components/retrievers.mdx +++ b/docs-website/docs/pipeline-components/retrievers.mdx @@ -171,6 +171,7 @@ For details on how to initialize and use a Retriever in a pipeline, see the docu | [FAISSEmbeddingRetriever](retrievers/faissembeddingretriever.mdx) | An embedding-based Retriever compatible with the FAISSDocumentStore. | | [FalkorDBCypherRetriever](retrievers/falkordbcypherretriever.mdx) | A Retriever that executes arbitrary OpenCypher queries against a FalkorDB Document Store. | | [FalkorDBEmbeddingRetriever](retrievers/falkordbembeddingretriever.mdx) | An embedding-based Retriever compatible with the FalkorDB Document Store. | +| [GoogleDriveRetriever](retrievers/googledriveretriever.mdx) | Retrieves files from Google Drive via the Drive API v3 search endpoint. | | [InMemoryBM25Retriever](retrievers/inmemorybm25retriever.mdx) | A keyword-based Retriever compatible with the InMemoryDocumentStore. | | [InMemoryEmbeddingRetriever](retrievers/inmemoryembeddingretriever.mdx) | An embedding-based Retriever compatible with the InMemoryDocumentStore. | | [FilterRetriever](retrievers/filterretriever.mdx) | A special Retriever to be used with any Document Store to get the Documents that match specific filters. | @@ -180,6 +181,7 @@ For details on how to initialize and use a Retriever in a pipeline, see the docu | [MultiRetriever](retrievers/multiretriever.mdx) | Runs multiple text retrievers in parallel and combines their deduplicated results. Experimental. | | [MongoDBAtlasEmbeddingRetriever](retrievers/mongodbatlasembeddingretriever.mdx) | An embedding Retriever compatible with the MongoDB Atlas Document Store. | | [MongoDBAtlasFullTextRetriever](retrievers/mongodbatlasfulltextretriever.mdx) | A full-text search Retriever compatible with the MongoDB Atlas Document Store. | +| [MSSharePointRetriever](retrievers/mssharepointretriever.mdx) | Retrieves content from Microsoft SharePoint and OneDrive via the Microsoft Search (Graph) API. | | [OpenSearchBM25Retriever](retrievers/opensearchbm25retriever.mdx) | A keyword-based Retriever that fetches Documents matching a query from an OpenSearch Document Store. | | [OpenSearchEmbeddingRetriever](retrievers/opensearchembeddingretriever.mdx) | An embedding-based Retriever compatible with the OpenSearch Document Store. | | [OpenSearchHybridRetriever](retrievers/opensearchhybridretriever.mdx) | A SuperComponent that implements a Hybrid Retriever in a single component, relying on OpenSearch as the backend Document Store. | diff --git a/docs-website/docs/pipeline-components/retrievers/googledriveretriever.mdx b/docs-website/docs/pipeline-components/retrievers/googledriveretriever.mdx new file mode 100644 index 00000000000..474ec142974 --- /dev/null +++ b/docs-website/docs/pipeline-components/retrievers/googledriveretriever.mdx @@ -0,0 +1,106 @@ +--- +title: "GoogleDriveRetriever" +id: googledriveretriever +slug: "/googledriveretriever" +description: "Retrieves files from Google Drive via the Drive API v3 search endpoint." +--- + +# GoogleDriveRetriever + +Retrieves files from Google Drive via the Drive API v3 search endpoint. + +
+ +| | | +| --- | --- | +| **Most common position in a pipeline** | At the start of a query pipeline, after an [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx) that provides the `access_token` | +| **Mandatory init variables** | None | +| **Mandatory run variables** | `query`: The search query string

`access_token`: A delegated Google OAuth bearer token, typically wired from an upstream `OAuthTokenResolver` | +| **Output variables** | `documents`: A list of [Documents](../../concepts/data-classes.mdx) holding file metadata (and optionally exported text) | +| **API reference** | [Google Drive](/reference/integrations-google-drive) | +| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/google_drive | +| **Package name** | `google-drive-haystack` | + +
+ +## Overview + +`GoogleDriveRetriever` runs a full-text search over a user's Google Drive (and optionally shared drives) through the [Drive API v3](https://developers.google.com/drive/api/reference/rest/v3/files/list) `files.list` endpoint and maps each matching file to a Haystack `Document`. + +By default, each `Document` carries resource metadata (`file_name`, `file_id`, `web_url`, `mime_type`, `file_extension`, author, and timestamps) and uses the file `description` or `name` as `content`, because the Drive search API does not return a text snippet. Set `include_content=True` to additionally export native Google Docs/Sheets/Slides to text and use that as the `Document` content. Binary files (PDF, DOCX, ...) are never downloaded by the retriever. + +To download the full content of the matching files, compose it with [`GoogleDriveFetcher`](../fetchers/googledrivefetcher.mdx) on the returned `web_url`/`file_id`, followed by a converter. + +### Authentication + +The retriever takes a per-user `access_token` as a run input. The token must carry a delegated Google OAuth scope that allows search, for example `https://www.googleapis.com/auth/drive.readonly`. The metadata-only `drive.metadata.readonly` scope cannot search file content or export documents. Typically you wire the token from an upstream [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx), which emits a plain string. A `Secret` is also accepted and resolved internally. + +### Scoping and filtering the search + +- `query_filter`: an optional Drive query clause AND-ed with the full-text search term, for example `"mimeType != 'application/vnd.google-apps.folder'"` or `"'' in parents"`. +- `include_shared_drives`: when `True`, the search spans shared drives as well as the user's My Drive. +- `order_by`: an optional Drive `orderBy` expression, for example `"modifiedTime desc"`. + +### Installation + +Install the Google Drive integration with: + +```shell +pip install google-drive-haystack +``` + +## Usage + +### On its own + +`access_token` below is a per-user delegated Google OAuth bearer token. In production you would obtain it from an [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx) rather than pasting it in. + +```python +from haystack_integrations.components.retrievers.google_drive import ( + GoogleDriveRetriever, +) + +retriever = GoogleDriveRetriever(top_k=5) + +result = retriever.run( + query="quarterly roadmap", + access_token="my-delegated-google-token", +) + +for doc in result["documents"]: + print(doc.meta["file_name"], "-", doc.meta["web_url"]) +``` + +### In a pipeline + +The following pipeline obtains a token from an `OAuthTokenResolver` and feeds it into the retriever, so that running the pipeline requires only the query: + +```python +from haystack import Pipeline +from haystack.utils import Secret +from haystack_integrations.components.connectors.oauth import OAuthTokenResolver +from haystack_integrations.utils.oauth import OAuthRefreshTokenSource +from haystack_integrations.components.retrievers.google_drive import ( + GoogleDriveRetriever, +) + +pipeline = Pipeline() +pipeline.add_component( + "resolver", + OAuthTokenResolver( + token_source=OAuthRefreshTokenSource( + token_url="https://oauth2.googleapis.com/token", + client_id="aaa-bbb-ccc", + refresh_token=Secret.from_env_var("GOOGLE_REFRESH_TOKEN"), + scopes=["https://www.googleapis.com/auth/drive.readonly"], + ), + ), +) +pipeline.add_component("retriever", GoogleDriveRetriever(top_k=5)) +pipeline.connect("resolver.access_token", "retriever.access_token") + +result = pipeline.run({"retriever": {"query": "quarterly roadmap"}}) +documents = result["retriever"]["documents"] +``` + +To download and convert the full content of the retrieved files, connect the retriever's `documents` output to a [`GoogleDriveFetcher`](../fetchers/googledrivefetcher.mdx). See that page for an end-to-end retrieve-fetch-convert example. diff --git a/docs-website/docs/pipeline-components/retrievers/mssharepointretriever.mdx b/docs-website/docs/pipeline-components/retrievers/mssharepointretriever.mdx new file mode 100644 index 00000000000..b34f8f7301b --- /dev/null +++ b/docs-website/docs/pipeline-components/retrievers/mssharepointretriever.mdx @@ -0,0 +1,110 @@ +--- +title: "MSSharePointRetriever" +id: mssharepointretriever +slug: "/mssharepointretriever" +description: "Retrieves content from Microsoft SharePoint and OneDrive via the Microsoft Search (Graph) API." +--- + +# MSSharePointRetriever + +Retrieves content from Microsoft SharePoint and OneDrive via the Microsoft Search (Graph) API. + +
+ +| | | +| --- | --- | +| **Most common position in a pipeline** | At the start of a query pipeline, after an [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx) that provides the `access_token` | +| **Mandatory init variables** | None | +| **Mandatory run variables** | `query`: The search query string

`access_token`: A delegated Microsoft Graph bearer token, typically wired from an upstream `OAuthTokenResolver` | +| **Output variables** | `documents`: A list of [Documents](../../concepts/data-classes.mdx) holding the search snippets and resource metadata | +| **API reference** | [Microsoft SharePoint](/reference/integrations-microsoft-sharepoint) | +| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/microsoft_sharepoint | +| **Package name** | `microsoft-sharepoint-haystack` | + +
+ +## Overview + +`MSSharePointRetriever` searches a user's Microsoft SharePoint and OneDrive content through the [Microsoft Search (Graph) API](https://learn.microsoft.com/en-us/graph/api/resources/search-api-overview). Given a query, it calls `POST /search/query` and maps each hit to a Haystack `Document` whose `content` is the search snippet and whose `meta` carries the resource metadata: `file_name`, `web_url`, `entity_type`, `created_date_time`, `last_modified_date_time`, `created_by`, `last_modified_by`, `mime_type`, and `file_extension`. It also stores the SharePoint identifiers a downstream fetcher needs to read list items and pages by ID (`site_id`, `list_id`, `list_item_id`, `list_item_unique_id`). + +The retriever does **not** download or convert the underlying files – it only returns Search snippets and metadata. To download the full content of the hits, compose it with [`MSSharePointFetcher`](../fetchers/mssharepointfetcher.mdx) followed by a converter. + +### Authentication + +The retriever takes a per-user `access_token` as a run input. The token must carry **delegated** Microsoft Graph permissions (for example `Files.Read.All`, plus `Sites.Read.All` for site and list scoping); the Search API supports delegated permissions only. Typically you wire the token from an upstream [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx), which emits a plain string. A `Secret` is also accepted and resolved internally. + +### Scoping and filtering the search + +You can narrow what is searched in several ways: + +- `entity_types`: which Microsoft Search entity types to query. Defaults to `["driveItem", "listItem"]`, which covers files, folders, SharePoint pages and news, and list items. Other valid values are `"list"` and `"site"`. +- KQL operators embedded directly in the query, for example `filetype:docx`, `author:"Jane Doe"`, or `path:"https://contoso.sharepoint.com/sites/Team"`. See the [Keyword Query Language (KQL) syntax reference](https://learn.microsoft.com/en-us/sharepoint/dev/general-development/keyword-query-language-kql-syntax-reference). +- `query_template`: a reusable template such as `'{searchTerms} path:"https://contoso.sharepoint.com/sites/Team"'`, where the literal `{searchTerms}` placeholder is replaced by the run-time query. + +### Installation + +Install the Microsoft SharePoint integration with: + +```shell +pip install microsoft-sharepoint-haystack +``` + +## Usage + +### On its own + +`access_token` below is a per-user delegated Microsoft Graph bearer token. In production you would obtain it from an [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx) rather than pasting it in. + +```python +from haystack_integrations.components.retrievers.microsoft_sharepoint import ( + MSSharePointRetriever, +) + +retriever = MSSharePointRetriever(top_k=5) + +result = retriever.run( + query="quarterly roadmap", + access_token="my-delegated-graph-token", +) + +for doc in result["documents"]: + print(doc.meta["file_name"], "-", doc.meta["web_url"]) +``` + +### In a pipeline + +The following pipeline obtains a token from an `OAuthTokenResolver` and feeds it into the retriever, so that running the pipeline requires only the query: + +```python +from haystack import Pipeline +from haystack.utils import Secret +from haystack_integrations.components.connectors.oauth import OAuthTokenResolver +from haystack_integrations.utils.oauth import OAuthRefreshTokenSource +from haystack_integrations.components.retrievers.microsoft_sharepoint import ( + MSSharePointRetriever, +) + +pipeline = Pipeline() +pipeline.add_component( + "resolver", + OAuthTokenResolver( + token_source=OAuthRefreshTokenSource( + token_url="https://login.microsoftonline.com/common/oauth2/v2.0/token", + client_id="aaa-bbb-ccc", + refresh_token=Secret.from_env_var("MS_REFRESH_TOKEN"), + scopes=[ + "https://graph.microsoft.com/Files.Read.All", + "https://graph.microsoft.com/Sites.Read.All", + "offline_access", + ], + ), + ), +) +pipeline.add_component("retriever", MSSharePointRetriever(top_k=5)) +pipeline.connect("resolver.access_token", "retriever.access_token") + +result = pipeline.run({"retriever": {"query": "quarterly roadmap"}}) +documents = result["retriever"]["documents"] +``` + +To download and convert the full content of the retrieved hits, connect the retriever's `documents` output to a [`MSSharePointFetcher`](../fetchers/mssharepointfetcher.mdx). See that page for an end-to-end retrieve-fetch-convert example. diff --git a/docs-website/sidebars.js b/docs-website/sidebars.js index 803fe0f66e1..c533dbb17bc 100644 --- a/docs-website/sidebars.js +++ b/docs-website/sidebars.js @@ -223,6 +223,7 @@ export default { 'pipeline-components/connectors/githubrepoviewer', 'pipeline-components/connectors/jinareaderconnector', 'pipeline-components/connectors/langfuseconnector', + 'pipeline-components/connectors/oauthtokenresolver', 'pipeline-components/connectors/openapiconnector', 'pipeline-components/connectors/openapiserviceconnector', 'pipeline-components/connectors/opentelemetryconnector', @@ -383,7 +384,9 @@ export default { }, items: [ 'pipeline-components/fetchers/firecrawlcrawler', + 'pipeline-components/fetchers/googledrivefetcher', 'pipeline-components/fetchers/linkcontentfetcher', + 'pipeline-components/fetchers/mssharepointfetcher', 'pipeline-components/fetchers/external-integrations-fetchers', ], }, @@ -573,12 +576,14 @@ export default { 'pipeline-components/retrievers/falkordbcypherretriever', 'pipeline-components/retrievers/falkordbembeddingretriever', 'pipeline-components/retrievers/filterretriever', + 'pipeline-components/retrievers/googledriveretriever', 'pipeline-components/retrievers/inmemorybm25retriever', 'pipeline-components/retrievers/inmemoryembeddingretriever', 'pipeline-components/retrievers/cogneeretriever', 'pipeline-components/retrievers/mem0memoryretriever', 'pipeline-components/retrievers/mongodbatlasembeddingretriever', 'pipeline-components/retrievers/mongodbatlasfulltextretriever', + 'pipeline-components/retrievers/mssharepointretriever', 'pipeline-components/retrievers/multiqueryembeddingretriever', 'pipeline-components/retrievers/multiquerytextretriever', 'pipeline-components/retrievers/multiretriever', diff --git a/docs-website/versioned_docs/version-2.30/pipeline-components/connectors.mdx b/docs-website/versioned_docs/version-2.30/pipeline-components/connectors.mdx index 0341bb0dcda..3ffb6ecd69c 100644 --- a/docs-website/versioned_docs/version-2.30/pipeline-components/connectors.mdx +++ b/docs-website/versioned_docs/version-2.30/pipeline-components/connectors.mdx @@ -20,6 +20,7 @@ These are Haystack integrations that connect your pipelines to services by exter | [GitHubRepoViewer](connectors/githubrepoviewer.mdx) | Enables navigating and fetching content from GitHub repositories through the GitHub API. | | [JinaReaderConnector](connectors/jinareaderconnector.mdx) | Use Jina AI’s Reader API with Haystack. | | [LangfuseConnector](connectors/langfuseconnector.mdx) | Enables tracing in Haystack pipelines using Langfuse. | +| [OAuthTokenResolver](connectors/oauthtokenresolver.mdx) | Resolves an OAuth access token at runtime and emits it for downstream components. | | [OpenAPIConnector](connectors/openapiconnector.mdx) | Acts as an interface between the Haystack ecosystem and OpenAPI services, using explicit input arguments. | | [OpenAPIServiceConnector](connectors/openapiserviceconnector.mdx) | Acts as an interface between the Haystack ecosystem and OpenAPI services. | | [OpenTelemetryConnector](connectors/opentelemetryconnector.mdx) | Enables tracing in Haystack pipelines using OpenTelemetry. | diff --git a/docs-website/versioned_docs/version-2.30/pipeline-components/connectors/oauthtokenresolver.mdx b/docs-website/versioned_docs/version-2.30/pipeline-components/connectors/oauthtokenresolver.mdx new file mode 100644 index 00000000000..ac1ec63fdc0 --- /dev/null +++ b/docs-website/versioned_docs/version-2.30/pipeline-components/connectors/oauthtokenresolver.mdx @@ -0,0 +1,155 @@ +--- +title: "OAuthTokenResolver" +id: oauthtokenresolver +slug: "/oauthtokenresolver" +description: "Resolves an OAuth access token at pipeline runtime and emits it for downstream components such as the SharePoint and Google Drive retrievers and fetchers." +--- + +# OAuthTokenResolver + +Resolves an OAuth access token at pipeline runtime and emits it for downstream components such as the SharePoint and Google Drive retrievers and fetchers. + +
+ +| | | +| --- | --- | +| **Most common position in a pipeline** | At the start of a pipeline, feeding `access_token` into downstream components such as [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx) or [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx) | +| **Mandatory init variables** | `token_source`: The strategy that resolves the access token, for example `OAuthRefreshTokenSource` | +| **Mandatory run variables** | None for config-only sources. `subject_token`: a controller-injected per-request credential, mandatory only when the source requires it (for example `OAuthTokenExchangeSource`) | +| **Output variables** | `access_token`: A bearer token string | +| **API reference** | [OAuth](/reference/integrations-oauth) | +| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/oauth | +| **Package name** | `oauth-haystack` | + +
+ +## Overview + +`OAuthTokenResolver` resolves an OAuth access token when the pipeline runs and emits it on the `access_token` output socket. Downstream components – such as [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx), [`MSSharePointFetcher`](../fetchers/mssharepointfetcher.mdx), [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx), and [`GoogleDriveFetcher`](../fetchers/googledrivefetcher.mdx) – consume the token through a normal connection and never need to know how it was obtained. + +The resolver itself is a thin wrapper. The actual work of getting a token is delegated to a pluggable **token source** that decides *where* the token comes from. This separation lets you swap authentication strategies (refresh-token grant, per-request token exchange, or a static long-lived token) without changing the rest of your pipeline. + +### Token sources + +You pass a token source to the resolver through the `token_source` parameter. All sources are importable from `haystack_integrations.utils.oauth`. + +| Source | Use it when | Per-request input | +| --- | --- | --- | +| `OAuthRefreshTokenSource` | You have a single, fixed identity backed by a stored refresh token and want the source to exchange it for short-lived access tokens and cache them. | None | +| `OAuthTokenExchangeSource` | You serve multiple users (or run multiple replicas) and want to exchange an incoming per-request user assertion for a downstream token, with no persistent storage. Implements RFC 8693 token exchange and Microsoft's on-behalf-of flow. | `subject_token` | +| `OAuthStaticTokenSource` | Your provider issues a non-expiring token that you manage out of band (for example Slack or Notion). | None | + +When the configured source needs a per-request credential (`OAuthTokenExchangeSource` sets `requires_subject_token = True`), the resolver declares a **mandatory** `subject_token` run input. This is a controller-injected credential – for example an incoming user assertion – not a value chosen by an end user. For config-only sources (`OAuthRefreshTokenSource`, `OAuthStaticTokenSource`), the resolver declares no run input and acts as a source node. + +:::info[Scopes are provider-specific] + +The OAuth scopes you request depend on the downstream service. For Microsoft Graph, that means scopes such as `https://graph.microsoft.com/Files.Read.All`; for Google Drive, scopes such as `https://www.googleapis.com/auth/drive.readonly`. Always consult your identity provider's documentation for the exact scope values. + +::: + +### Installation + +Install the OAuth integration with: + +```shell +pip install oauth-haystack +``` + +## Usage + +### On its own + +Resolve a token with a stored refresh token using `OAuthRefreshTokenSource`. The refresh token is read from an environment variable through the [Secret API](../../concepts/secret-management.mdx): + +```python +from haystack.utils import Secret +from haystack_integrations.components.connectors.oauth import OAuthTokenResolver +from haystack_integrations.utils.oauth import OAuthRefreshTokenSource + +resolver = OAuthTokenResolver( + token_source=OAuthRefreshTokenSource( + token_url="https://login.microsoftonline.com/common/oauth2/v2.0/token", + client_id="aaa-bbb-ccc", + refresh_token=Secret.from_env_var("MS_REFRESH_TOKEN"), + scopes=[ + "https://graph.microsoft.com/Files.Read.All", + "offline_access", + ], + ), +) + +access_token = resolver.run()["access_token"] +``` + +For a provider that issues long-lived, non-expiring tokens, use `OAuthStaticTokenSource` instead: + +```python +from haystack.utils import Secret +from haystack_integrations.components.connectors.oauth import OAuthTokenResolver +from haystack_integrations.utils.oauth import OAuthStaticTokenSource + +resolver = OAuthTokenResolver( + token_source=OAuthStaticTokenSource(token=Secret.from_env_var("SERVICE_TOKEN")), +) + +access_token = resolver.run()["access_token"] +``` + +For multi-user backends, use `OAuthTokenExchangeSource`. The resolver then requires a per-request `subject_token`: + +```python +from haystack_integrations.components.connectors.oauth import OAuthTokenResolver +from haystack_integrations.utils.oauth import OAuthTokenExchangeSource + +resolver = OAuthTokenResolver( + token_source=OAuthTokenExchangeSource( + token_url="https://login.microsoftonline.com//oauth2/v2.0/token", + client_id="aaa-bbb-ccc", + subject_token_param="assertion", + grant_type="urn:ietf:params:oauth:grant-type:jwt-bearer", + scopes=["https://graph.microsoft.com/Files.Read.All"], + extra_token_params={"requested_token_use": "on_behalf_of"}, + ), +) + +# `subject_token` is the incoming per-request user assertion, injected by your application. +access_token = resolver.run(subject_token="")["access_token"] +``` + +### In a pipeline + +In a pipeline, connect the resolver's `access_token` output to the `access_token` input of one or more downstream components. The example below wires the resolver into a [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx) so that searching SharePoint requires only a query at runtime: + +```python +from haystack import Pipeline +from haystack.utils import Secret +from haystack_integrations.components.connectors.oauth import OAuthTokenResolver +from haystack_integrations.utils.oauth import OAuthRefreshTokenSource +from haystack_integrations.components.retrievers.microsoft_sharepoint import ( + MSSharePointRetriever, +) + +pipeline = Pipeline() +pipeline.add_component( + "resolver", + OAuthTokenResolver( + token_source=OAuthRefreshTokenSource( + token_url="https://login.microsoftonline.com/common/oauth2/v2.0/token", + client_id="aaa-bbb-ccc", + refresh_token=Secret.from_env_var("MS_REFRESH_TOKEN"), + scopes=[ + "https://graph.microsoft.com/Files.Read.All", + "https://graph.microsoft.com/Sites.Read.All", + "offline_access", + ], + ), + ), +) +pipeline.add_component("retriever", MSSharePointRetriever(top_k=5)) +pipeline.connect("resolver.access_token", "retriever.access_token") + +result = pipeline.run({"retriever": {"query": "quarterly roadmap"}}) +documents = result["retriever"]["documents"] +``` + +A single `access_token` output can be connected to several downstream inputs. For a full retrieve-then-fetch pipeline that feeds the same token to both a retriever and a fetcher, see the [`MSSharePointFetcher`](../fetchers/mssharepointfetcher.mdx) and [`GoogleDriveFetcher`](../fetchers/googledrivefetcher.mdx) pages. diff --git a/docs-website/versioned_docs/version-2.30/pipeline-components/fetchers.mdx b/docs-website/versioned_docs/version-2.30/pipeline-components/fetchers.mdx index e06b360a158..9a3596da83f 100644 --- a/docs-website/versioned_docs/version-2.30/pipeline-components/fetchers.mdx +++ b/docs-website/versioned_docs/version-2.30/pipeline-components/fetchers.mdx @@ -2,13 +2,16 @@ title: "Fetchers" id: fetchers slug: "/fetchers" -description: "Currently, there's one Fetcher in Haystack: LinkContentFetcher. It fetches the contents of the URLs you give it." +description: "Fetchers retrieve content from external sources – URLs, web crawls, or cloud storage such as SharePoint and Google Drive – so you can use it as data for your pipelines." --- # Fetchers -Currently, there's one Fetcher in Haystack: LinkContentFetcher. It fetches the contents of the URLs you give it. +Fetchers retrieve content from external sources – URLs, web crawls, or cloud storage such as SharePoint and Google Drive – so you can use it as data for your pipelines. | Component | Description | | --- | --- | -| [LinkContentFetcher](fetchers/linkcontentfetcher.mdx) | Fetches the contents of the URLs you give it so you can use them as data for your pipelines. | \ No newline at end of file +| [FirecrawlCrawler](fetchers/firecrawlcrawler.mdx) | Crawls websites with Firecrawl, following links to discover subpages, and returns them as Documents. | +| [GoogleDriveFetcher](fetchers/googledrivefetcher.mdx) | Fetches the full content of Google Drive files via the Drive API v3 and returns it as ByteStreams. | +| [LinkContentFetcher](fetchers/linkcontentfetcher.mdx) | Fetches the contents of the URLs you give it so you can use them as data for your pipelines. | +| [MSSharePointFetcher](fetchers/mssharepointfetcher.mdx) | Fetches the full content of Microsoft SharePoint and OneDrive items via the Microsoft Graph API and returns it as ByteStreams. | diff --git a/docs-website/versioned_docs/version-2.30/pipeline-components/fetchers/googledrivefetcher.mdx b/docs-website/versioned_docs/version-2.30/pipeline-components/fetchers/googledrivefetcher.mdx new file mode 100644 index 00000000000..899d6b24796 --- /dev/null +++ b/docs-website/versioned_docs/version-2.30/pipeline-components/fetchers/googledrivefetcher.mdx @@ -0,0 +1,138 @@ +--- +title: "GoogleDriveFetcher" +id: googledrivefetcher +slug: "/googledrivefetcher" +description: "Fetches the full content of Google Drive files via the Drive API v3 and returns it as ByteStreams." +--- + +# GoogleDriveFetcher + +Fetches the full content of Google Drive files via the Drive API v3 and returns it as ByteStreams. + +
+ +| | | +| --- | --- | +| **Most common position in a pipeline** | After [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx), before a Router or File Converters | +| **Mandatory init variables** | None | +| **Mandatory run variables** | `access_token`: A delegated Google OAuth bearer token, typically wired from an upstream `OAuthTokenResolver`

`targets`: A list of `Document`s (from `GoogleDriveRetriever`) or raw Google Drive file ids / URLs | +| **Output variables** | `streams`: A list of [ByteStreams](../../concepts/data-classes.mdx) holding the fetched content | +| **API reference** | [Google Drive](/reference/integrations-google-drive) | +| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/google_drive | +| **Package name** | `google-drive-haystack` | + +
+ +## Overview + +`GoogleDriveFetcher` downloads the full content of Google Drive files through the [Drive API v3](https://developers.google.com/drive/api/reference/rest/v3) and returns `ByteStream` objects, ready for a downstream converter. + +It complements [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx), which returns only metadata (and optionally exported text). Wire the retriever's `documents` (or a list of file ids / Drive URLs) into the fetcher to download the underlying content. The fetcher dispatches on each file's mime type: + +- **Binary files** (PDF, DOCX, images, ...) are downloaded as-is via `files.get?alt=media`. +- **Native Google Docs/Sheets/Slides** are exported with `files.export`, by default to the Office formats (DOCX/XLSX/PPTX), configurable via `export_mime_types`. +- **Folders** and other non-downloadable Google types (Forms, Sites, ...) are skipped. + +Each `ByteStream`'s `meta` carries `file_id`, `web_url`, `file_name`, and `content_type`. Because the output is a list of `ByteStream`s of mixed types, the typical next step is a [`FileTypeRouter`](../routers/filetyperouter.mdx) that dispatches each stream to the right converter ([`PyPDFToDocument`](../converters/pypdftodocument.mdx), [`DOCXToDocument`](../converters/docxtodocument.mdx), [`XLSXToDocument`](../converters/xlsxtodocument.mdx), or [`PPTXToDocument`](../converters/pptxtodocument.mdx)). + +### Authentication + +The fetcher takes a per-user `access_token` as a run input. The token must carry a delegated Google OAuth scope that allows reading file content, for example `https://www.googleapis.com/auth/drive.readonly`. Typically you wire it from an upstream [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx), which emits a plain string. A `Secret` is also accepted and resolved internally. + +### Error handling and concurrency + +- `raise_on_failure` (default `True`): when `False`, a failed fetch is logged and the file is skipped, so the remaining files are still returned. +- `max_retries` (default `3`): retries on throttled (HTTP 429) and transient server errors. +- `max_concurrent_requests` (default `5`): bounds the number of files fetched concurrently by `run_async` to avoid tripping Drive rate limits. It has no effect on the synchronous `run`, which fetches files one at a time. +- `export_mime_types`: overrides the default native-Google-to-Office export mapping. Drive caps a single export at 10 MB. + +### Installation + +Install the Google Drive integration with: + +```shell +pip install google-drive-haystack +``` + +## Usage + +### On its own + +`access_token` below is a per-user delegated Google OAuth bearer token. You can pass either raw file ids / Drive URLs or the `Document`s produced by `GoogleDriveRetriever`. + +```python +from haystack_integrations.components.fetchers.google_drive import GoogleDriveFetcher + +fetcher = GoogleDriveFetcher() + +result = fetcher.run( + access_token="my-delegated-google-token", + targets=[ + "https://drive.google.com/file/d/1AbCdEfGhIjKlMnOpQrStUvWxYz/view", + ], +) + +for stream in result["streams"]: + print(stream.meta["file_name"], stream.meta["content_type"]) +``` + +### In a pipeline + +The following query pipeline ties the whole integration together: an [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx) provides a token, [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx) searches Drive, `GoogleDriveFetcher` downloads the matching files, and a [`FileTypeRouter`](../routers/filetyperouter.mdx) sends each `ByteStream` to the right converter. Note that the resolver's single `access_token` output feeds both the retriever and the fetcher. + +```python +from haystack import Pipeline +from haystack.utils import Secret +from haystack.components.routers import FileTypeRouter +from haystack.components.converters import PyPDFToDocument, DOCXToDocument + +from haystack_integrations.components.connectors.oauth import OAuthTokenResolver +from haystack_integrations.utils.oauth import OAuthRefreshTokenSource +from haystack_integrations.components.retrievers.google_drive import ( + GoogleDriveRetriever, +) +from haystack_integrations.components.fetchers.google_drive import GoogleDriveFetcher + +pipeline = Pipeline() +pipeline.add_component( + "resolver", + OAuthTokenResolver( + token_source=OAuthRefreshTokenSource( + token_url="https://oauth2.googleapis.com/token", + client_id="aaa-bbb-ccc", + refresh_token=Secret.from_env_var("GOOGLE_REFRESH_TOKEN"), + scopes=["https://www.googleapis.com/auth/drive.readonly"], + ), + ), +) +pipeline.add_component("retriever", GoogleDriveRetriever(top_k=5)) +pipeline.add_component("fetcher", GoogleDriveFetcher()) +pipeline.add_component( + "router", + FileTypeRouter( + mime_types=[ + "application/pdf", + "application/vnd.openxmlformats-officedocument.wordprocessingml.document", + ], + ), +) +pipeline.add_component("pdf_converter", PyPDFToDocument()) +pipeline.add_component("docx_converter", DOCXToDocument()) + +# The same token feeds both the retriever and the fetcher. +pipeline.connect("resolver.access_token", "retriever.access_token") +pipeline.connect("resolver.access_token", "fetcher.access_token") + +# The retrieved documents become the fetcher's targets. +pipeline.connect("retriever.documents", "fetcher.targets") + +# Route each fetched ByteStream to the matching converter. +pipeline.connect("fetcher.streams", "router.sources") +pipeline.connect("router.application/pdf", "pdf_converter.sources") +pipeline.connect( + "router.application/vnd.openxmlformats-officedocument.wordprocessingml.document", + "docx_converter.sources", +) + +result = pipeline.run({"retriever": {"query": "quarterly roadmap"}}) +``` diff --git a/docs-website/versioned_docs/version-2.30/pipeline-components/fetchers/mssharepointfetcher.mdx b/docs-website/versioned_docs/version-2.30/pipeline-components/fetchers/mssharepointfetcher.mdx new file mode 100644 index 00000000000..3989d6bc041 --- /dev/null +++ b/docs-website/versioned_docs/version-2.30/pipeline-components/fetchers/mssharepointfetcher.mdx @@ -0,0 +1,147 @@ +--- +title: "MSSharePointFetcher" +id: mssharepointfetcher +slug: "/mssharepointfetcher" +description: "Fetches the full content of Microsoft SharePoint and OneDrive items via the Microsoft Graph API and returns it as ByteStreams." +--- + +# MSSharePointFetcher + +Fetches the full content of Microsoft SharePoint and OneDrive items via the Microsoft Graph API and returns it as ByteStreams. + +
+ +| | | +| --- | --- | +| **Most common position in a pipeline** | After [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx), before a Router or File Converters | +| **Mandatory init variables** | None | +| **Mandatory run variables** | `access_token`: A delegated Microsoft Graph bearer token, typically wired from an upstream `OAuthTokenResolver`

`targets`: A list of `Document`s (from `MSSharePointRetriever`) or raw SharePoint/OneDrive `web_url` strings | +| **Output variables** | `streams`: A list of [ByteStreams](../../concepts/data-classes.mdx) holding the fetched content | +| **API reference** | [Microsoft SharePoint](/reference/integrations-microsoft-sharepoint) | +| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/microsoft_sharepoint | +| **Package name** | `microsoft-sharepoint-haystack` | + +
+ +## Overview + +`MSSharePointFetcher` downloads the full content of Microsoft SharePoint and OneDrive items through the [Microsoft Graph API](https://learn.microsoft.com/en-us/graph/use-the-api) and returns `ByteStream` objects, ready for a downstream converter. + +It complements [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx), which returns only Search snippets and metadata. Wire the retriever's `documents` (or a list of `web_url`s) into the fetcher to download the underlying content. The fetcher dispatches on the entity type of each hit: + +- **Files** (`driveItem`) are downloaded as their raw bytes (PDF, DOCX, ...). +- **List items** (`listItem`) are returned as a JSON `ByteStream` of the item's column values (`fields`). +- **SharePoint pages** (`sitePage`) are returned as an HTML `ByteStream` built from the page's web parts. + +Each `ByteStream`'s `meta` carries `url`, `file_name`, `content_type`, and a normalized `entity_type` (`driveItem`, `listItem`, or `sitePage`). Everything is resolved through the Microsoft Graph `shares` endpoint (plus the Pages API for pages), so only the `web_url` already exposed by the retriever is needed. + +Because the output is a list of `ByteStream`s of mixed types, the typical next step is a [`FileTypeRouter`](../routers/filetyperouter.mdx) that dispatches each stream to the right converter ([`PyPDFToDocument`](../converters/pypdftodocument.mdx), [`DOCXToDocument`](../converters/docxtodocument.mdx), [`HTMLToDocument`](../converters/htmltodocument.mdx), or a JSON converter). + +### Authentication + +The fetcher takes a per-user `access_token` as a run input. The token must carry **delegated** Microsoft Graph permissions (for example `Files.Read.All` for files and `Sites.Read.All` for list items and pages). Typically you wire it from an upstream [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx), which emits a plain string. A `Secret` is also accepted and resolved internally. + +### Error handling and concurrency + +- `raise_on_failure` (default `True`): when `False`, a failed fetch is logged and the item is skipped, so the remaining items are still returned. +- `max_retries` (default `3`): retries on throttled (HTTP 429) and transient server errors. +- `max_concurrent_requests` (default `5`): bounds the number of items fetched concurrently by `run_async` to avoid tripping Microsoft Graph rate limits. It has no effect on the synchronous `run`, which fetches items one at a time. + +### Installation + +Install the Microsoft SharePoint integration with: + +```shell +pip install microsoft-sharepoint-haystack +``` + +## Usage + +### On its own + +`access_token` below is a per-user delegated Microsoft Graph bearer token. You can pass either raw `web_url` strings or the `Document`s produced by `MSSharePointRetriever`. + +```python +from haystack_integrations.components.fetchers.microsoft_sharepoint import ( + MSSharePointFetcher, +) + +fetcher = MSSharePointFetcher() + +result = fetcher.run( + access_token="my-delegated-graph-token", + targets=[ + "https://contoso.sharepoint.com/sites/contoso-team/contoso-designs.docx", + ], +) + +for stream in result["streams"]: + print(stream.meta["file_name"], stream.meta["content_type"]) +``` + +### In a pipeline + +The following query pipeline ties the whole integration together: an [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx) provides a token, [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx) searches SharePoint, `MSSharePointFetcher` downloads the matching items, and a [`FileTypeRouter`](../routers/filetyperouter.mdx) sends each `ByteStream` to the right converter. Note that the resolver's single `access_token` output feeds both the retriever and the fetcher. + +```python +from haystack import Pipeline +from haystack.utils import Secret +from haystack.components.routers import FileTypeRouter +from haystack.components.converters import PyPDFToDocument, DOCXToDocument + +from haystack_integrations.components.connectors.oauth import OAuthTokenResolver +from haystack_integrations.utils.oauth import OAuthRefreshTokenSource +from haystack_integrations.components.retrievers.microsoft_sharepoint import ( + MSSharePointRetriever, +) +from haystack_integrations.components.fetchers.microsoft_sharepoint import ( + MSSharePointFetcher, +) + +pipeline = Pipeline() +pipeline.add_component( + "resolver", + OAuthTokenResolver( + token_source=OAuthRefreshTokenSource( + token_url="https://login.microsoftonline.com/common/oauth2/v2.0/token", + client_id="aaa-bbb-ccc", + refresh_token=Secret.from_env_var("MS_REFRESH_TOKEN"), + scopes=[ + "https://graph.microsoft.com/Files.Read.All", + "https://graph.microsoft.com/Sites.Read.All", + "offline_access", + ], + ), + ), +) +pipeline.add_component("retriever", MSSharePointRetriever(top_k=5)) +pipeline.add_component("fetcher", MSSharePointFetcher()) +pipeline.add_component( + "router", + FileTypeRouter( + mime_types=[ + "application/pdf", + "application/vnd.openxmlformats-officedocument.wordprocessingml.document", + ], + ), +) +pipeline.add_component("pdf_converter", PyPDFToDocument()) +pipeline.add_component("docx_converter", DOCXToDocument()) + +# The same token feeds both the retriever and the fetcher. +pipeline.connect("resolver.access_token", "retriever.access_token") +pipeline.connect("resolver.access_token", "fetcher.access_token") + +# The retrieved documents become the fetcher's targets. +pipeline.connect("retriever.documents", "fetcher.targets") + +# Route each fetched ByteStream to the matching converter. +pipeline.connect("fetcher.streams", "router.sources") +pipeline.connect("router.application/pdf", "pdf_converter.sources") +pipeline.connect( + "router.application/vnd.openxmlformats-officedocument.wordprocessingml.document", + "docx_converter.sources", +) + +result = pipeline.run({"retriever": {"query": "quarterly roadmap"}}) +``` diff --git a/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers.mdx b/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers.mdx index 5d317e42f98..cbc46c31fec 100644 --- a/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers.mdx +++ b/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers.mdx @@ -171,6 +171,7 @@ For details on how to initialize and use a Retriever in a pipeline, see the docu | [FAISSEmbeddingRetriever](retrievers/faissembeddingretriever.mdx) | An embedding-based Retriever compatible with the FAISSDocumentStore. | | [FalkorDBCypherRetriever](retrievers/falkordbcypherretriever.mdx) | A Retriever that executes arbitrary OpenCypher queries against a FalkorDB Document Store. | | [FalkorDBEmbeddingRetriever](retrievers/falkordbembeddingretriever.mdx) | An embedding-based Retriever compatible with the FalkorDB Document Store. | +| [GoogleDriveRetriever](retrievers/googledriveretriever.mdx) | Retrieves files from Google Drive via the Drive API v3 search endpoint. | | [InMemoryBM25Retriever](retrievers/inmemorybm25retriever.mdx) | A keyword-based Retriever compatible with the InMemoryDocumentStore. | | [InMemoryEmbeddingRetriever](retrievers/inmemoryembeddingretriever.mdx) | An embedding-based Retriever compatible with the InMemoryDocumentStore. | | [FilterRetriever](retrievers/filterretriever.mdx) | A special Retriever to be used with any Document Store to get the Documents that match specific filters. | @@ -180,6 +181,7 @@ For details on how to initialize and use a Retriever in a pipeline, see the docu | [MultiRetriever](retrievers/multiretriever.mdx) | Runs multiple text retrievers in parallel and combines their deduplicated results. Experimental. | | [MongoDBAtlasEmbeddingRetriever](retrievers/mongodbatlasembeddingretriever.mdx) | An embedding Retriever compatible with the MongoDB Atlas Document Store. | | [MongoDBAtlasFullTextRetriever](retrievers/mongodbatlasfulltextretriever.mdx) | A full-text search Retriever compatible with the MongoDB Atlas Document Store. | +| [MSSharePointRetriever](retrievers/mssharepointretriever.mdx) | Retrieves content from Microsoft SharePoint and OneDrive via the Microsoft Search (Graph) API. | | [OpenSearchBM25Retriever](retrievers/opensearchbm25retriever.mdx) | A keyword-based Retriever that fetches Documents matching a query from an OpenSearch Document Store. | | [OpenSearchEmbeddingRetriever](retrievers/opensearchembeddingretriever.mdx) | An embedding-based Retriever compatible with the OpenSearch Document Store. | | [OpenSearchHybridRetriever](retrievers/opensearchhybridretriever.mdx) | A SuperComponent that implements a Hybrid Retriever in a single component, relying on OpenSearch as the backend Document Store. | diff --git a/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers/googledriveretriever.mdx b/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers/googledriveretriever.mdx new file mode 100644 index 00000000000..474ec142974 --- /dev/null +++ b/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers/googledriveretriever.mdx @@ -0,0 +1,106 @@ +--- +title: "GoogleDriveRetriever" +id: googledriveretriever +slug: "/googledriveretriever" +description: "Retrieves files from Google Drive via the Drive API v3 search endpoint." +--- + +# GoogleDriveRetriever + +Retrieves files from Google Drive via the Drive API v3 search endpoint. + +
+ +| | | +| --- | --- | +| **Most common position in a pipeline** | At the start of a query pipeline, after an [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx) that provides the `access_token` | +| **Mandatory init variables** | None | +| **Mandatory run variables** | `query`: The search query string

`access_token`: A delegated Google OAuth bearer token, typically wired from an upstream `OAuthTokenResolver` | +| **Output variables** | `documents`: A list of [Documents](../../concepts/data-classes.mdx) holding file metadata (and optionally exported text) | +| **API reference** | [Google Drive](/reference/integrations-google-drive) | +| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/google_drive | +| **Package name** | `google-drive-haystack` | + +
+ +## Overview + +`GoogleDriveRetriever` runs a full-text search over a user's Google Drive (and optionally shared drives) through the [Drive API v3](https://developers.google.com/drive/api/reference/rest/v3/files/list) `files.list` endpoint and maps each matching file to a Haystack `Document`. + +By default, each `Document` carries resource metadata (`file_name`, `file_id`, `web_url`, `mime_type`, `file_extension`, author, and timestamps) and uses the file `description` or `name` as `content`, because the Drive search API does not return a text snippet. Set `include_content=True` to additionally export native Google Docs/Sheets/Slides to text and use that as the `Document` content. Binary files (PDF, DOCX, ...) are never downloaded by the retriever. + +To download the full content of the matching files, compose it with [`GoogleDriveFetcher`](../fetchers/googledrivefetcher.mdx) on the returned `web_url`/`file_id`, followed by a converter. + +### Authentication + +The retriever takes a per-user `access_token` as a run input. The token must carry a delegated Google OAuth scope that allows search, for example `https://www.googleapis.com/auth/drive.readonly`. The metadata-only `drive.metadata.readonly` scope cannot search file content or export documents. Typically you wire the token from an upstream [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx), which emits a plain string. A `Secret` is also accepted and resolved internally. + +### Scoping and filtering the search + +- `query_filter`: an optional Drive query clause AND-ed with the full-text search term, for example `"mimeType != 'application/vnd.google-apps.folder'"` or `"'' in parents"`. +- `include_shared_drives`: when `True`, the search spans shared drives as well as the user's My Drive. +- `order_by`: an optional Drive `orderBy` expression, for example `"modifiedTime desc"`. + +### Installation + +Install the Google Drive integration with: + +```shell +pip install google-drive-haystack +``` + +## Usage + +### On its own + +`access_token` below is a per-user delegated Google OAuth bearer token. In production you would obtain it from an [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx) rather than pasting it in. + +```python +from haystack_integrations.components.retrievers.google_drive import ( + GoogleDriveRetriever, +) + +retriever = GoogleDriveRetriever(top_k=5) + +result = retriever.run( + query="quarterly roadmap", + access_token="my-delegated-google-token", +) + +for doc in result["documents"]: + print(doc.meta["file_name"], "-", doc.meta["web_url"]) +``` + +### In a pipeline + +The following pipeline obtains a token from an `OAuthTokenResolver` and feeds it into the retriever, so that running the pipeline requires only the query: + +```python +from haystack import Pipeline +from haystack.utils import Secret +from haystack_integrations.components.connectors.oauth import OAuthTokenResolver +from haystack_integrations.utils.oauth import OAuthRefreshTokenSource +from haystack_integrations.components.retrievers.google_drive import ( + GoogleDriveRetriever, +) + +pipeline = Pipeline() +pipeline.add_component( + "resolver", + OAuthTokenResolver( + token_source=OAuthRefreshTokenSource( + token_url="https://oauth2.googleapis.com/token", + client_id="aaa-bbb-ccc", + refresh_token=Secret.from_env_var("GOOGLE_REFRESH_TOKEN"), + scopes=["https://www.googleapis.com/auth/drive.readonly"], + ), + ), +) +pipeline.add_component("retriever", GoogleDriveRetriever(top_k=5)) +pipeline.connect("resolver.access_token", "retriever.access_token") + +result = pipeline.run({"retriever": {"query": "quarterly roadmap"}}) +documents = result["retriever"]["documents"] +``` + +To download and convert the full content of the retrieved files, connect the retriever's `documents` output to a [`GoogleDriveFetcher`](../fetchers/googledrivefetcher.mdx). See that page for an end-to-end retrieve-fetch-convert example. diff --git a/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers/mssharepointretriever.mdx b/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers/mssharepointretriever.mdx new file mode 100644 index 00000000000..b34f8f7301b --- /dev/null +++ b/docs-website/versioned_docs/version-2.30/pipeline-components/retrievers/mssharepointretriever.mdx @@ -0,0 +1,110 @@ +--- +title: "MSSharePointRetriever" +id: mssharepointretriever +slug: "/mssharepointretriever" +description: "Retrieves content from Microsoft SharePoint and OneDrive via the Microsoft Search (Graph) API." +--- + +# MSSharePointRetriever + +Retrieves content from Microsoft SharePoint and OneDrive via the Microsoft Search (Graph) API. + +
+ +| | | +| --- | --- | +| **Most common position in a pipeline** | At the start of a query pipeline, after an [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx) that provides the `access_token` | +| **Mandatory init variables** | None | +| **Mandatory run variables** | `query`: The search query string

`access_token`: A delegated Microsoft Graph bearer token, typically wired from an upstream `OAuthTokenResolver` | +| **Output variables** | `documents`: A list of [Documents](../../concepts/data-classes.mdx) holding the search snippets and resource metadata | +| **API reference** | [Microsoft SharePoint](/reference/integrations-microsoft-sharepoint) | +| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/microsoft_sharepoint | +| **Package name** | `microsoft-sharepoint-haystack` | + +
+ +## Overview + +`MSSharePointRetriever` searches a user's Microsoft SharePoint and OneDrive content through the [Microsoft Search (Graph) API](https://learn.microsoft.com/en-us/graph/api/resources/search-api-overview). Given a query, it calls `POST /search/query` and maps each hit to a Haystack `Document` whose `content` is the search snippet and whose `meta` carries the resource metadata: `file_name`, `web_url`, `entity_type`, `created_date_time`, `last_modified_date_time`, `created_by`, `last_modified_by`, `mime_type`, and `file_extension`. It also stores the SharePoint identifiers a downstream fetcher needs to read list items and pages by ID (`site_id`, `list_id`, `list_item_id`, `list_item_unique_id`). + +The retriever does **not** download or convert the underlying files – it only returns Search snippets and metadata. To download the full content of the hits, compose it with [`MSSharePointFetcher`](../fetchers/mssharepointfetcher.mdx) followed by a converter. + +### Authentication + +The retriever takes a per-user `access_token` as a run input. The token must carry **delegated** Microsoft Graph permissions (for example `Files.Read.All`, plus `Sites.Read.All` for site and list scoping); the Search API supports delegated permissions only. Typically you wire the token from an upstream [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx), which emits a plain string. A `Secret` is also accepted and resolved internally. + +### Scoping and filtering the search + +You can narrow what is searched in several ways: + +- `entity_types`: which Microsoft Search entity types to query. Defaults to `["driveItem", "listItem"]`, which covers files, folders, SharePoint pages and news, and list items. Other valid values are `"list"` and `"site"`. +- KQL operators embedded directly in the query, for example `filetype:docx`, `author:"Jane Doe"`, or `path:"https://contoso.sharepoint.com/sites/Team"`. See the [Keyword Query Language (KQL) syntax reference](https://learn.microsoft.com/en-us/sharepoint/dev/general-development/keyword-query-language-kql-syntax-reference). +- `query_template`: a reusable template such as `'{searchTerms} path:"https://contoso.sharepoint.com/sites/Team"'`, where the literal `{searchTerms}` placeholder is replaced by the run-time query. + +### Installation + +Install the Microsoft SharePoint integration with: + +```shell +pip install microsoft-sharepoint-haystack +``` + +## Usage + +### On its own + +`access_token` below is a per-user delegated Microsoft Graph bearer token. In production you would obtain it from an [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx) rather than pasting it in. + +```python +from haystack_integrations.components.retrievers.microsoft_sharepoint import ( + MSSharePointRetriever, +) + +retriever = MSSharePointRetriever(top_k=5) + +result = retriever.run( + query="quarterly roadmap", + access_token="my-delegated-graph-token", +) + +for doc in result["documents"]: + print(doc.meta["file_name"], "-", doc.meta["web_url"]) +``` + +### In a pipeline + +The following pipeline obtains a token from an `OAuthTokenResolver` and feeds it into the retriever, so that running the pipeline requires only the query: + +```python +from haystack import Pipeline +from haystack.utils import Secret +from haystack_integrations.components.connectors.oauth import OAuthTokenResolver +from haystack_integrations.utils.oauth import OAuthRefreshTokenSource +from haystack_integrations.components.retrievers.microsoft_sharepoint import ( + MSSharePointRetriever, +) + +pipeline = Pipeline() +pipeline.add_component( + "resolver", + OAuthTokenResolver( + token_source=OAuthRefreshTokenSource( + token_url="https://login.microsoftonline.com/common/oauth2/v2.0/token", + client_id="aaa-bbb-ccc", + refresh_token=Secret.from_env_var("MS_REFRESH_TOKEN"), + scopes=[ + "https://graph.microsoft.com/Files.Read.All", + "https://graph.microsoft.com/Sites.Read.All", + "offline_access", + ], + ), + ), +) +pipeline.add_component("retriever", MSSharePointRetriever(top_k=5)) +pipeline.connect("resolver.access_token", "retriever.access_token") + +result = pipeline.run({"retriever": {"query": "quarterly roadmap"}}) +documents = result["retriever"]["documents"] +``` + +To download and convert the full content of the retrieved hits, connect the retriever's `documents` output to a [`MSSharePointFetcher`](../fetchers/mssharepointfetcher.mdx). See that page for an end-to-end retrieve-fetch-convert example. diff --git a/docs-website/versioned_sidebars/version-2.30-sidebars.json b/docs-website/versioned_sidebars/version-2.30-sidebars.json index f21f3bd1991..b97958b2d30 100644 --- a/docs-website/versioned_sidebars/version-2.30-sidebars.json +++ b/docs-website/versioned_sidebars/version-2.30-sidebars.json @@ -219,6 +219,7 @@ "pipeline-components/connectors/githubrepoviewer", "pipeline-components/connectors/jinareaderconnector", "pipeline-components/connectors/langfuseconnector", + "pipeline-components/connectors/oauthtokenresolver", "pipeline-components/connectors/openapiconnector", "pipeline-components/connectors/openapiserviceconnector", "pipeline-components/connectors/opentelemetryconnector", @@ -379,7 +380,9 @@ }, "items": [ "pipeline-components/fetchers/firecrawlcrawler", + "pipeline-components/fetchers/googledrivefetcher", "pipeline-components/fetchers/linkcontentfetcher", + "pipeline-components/fetchers/mssharepointfetcher", "pipeline-components/fetchers/external-integrations-fetchers" ] }, @@ -569,12 +572,14 @@ "pipeline-components/retrievers/falkordbcypherretriever", "pipeline-components/retrievers/falkordbembeddingretriever", "pipeline-components/retrievers/filterretriever", + "pipeline-components/retrievers/googledriveretriever", "pipeline-components/retrievers/inmemorybm25retriever", "pipeline-components/retrievers/inmemoryembeddingretriever", "pipeline-components/retrievers/cogneeretriever", "pipeline-components/retrievers/mem0memoryretriever", "pipeline-components/retrievers/mongodbatlasembeddingretriever", "pipeline-components/retrievers/mongodbatlasfulltextretriever", + "pipeline-components/retrievers/mssharepointretriever", "pipeline-components/retrievers/multiqueryembeddingretriever", "pipeline-components/retrievers/multiquerytextretriever", "pipeline-components/retrievers/multiretriever",