From e77f12140f839762694a45a61c0b930d59661c08 Mon Sep 17 00:00:00 2001 From: Julian Risch Date: Sun, 5 Jul 2026 14:39:18 +0200 Subject: [PATCH 1/5] docs: document AsyncPipeline task cancellation, sync streaming callbacks in run_async, and SearchableToolset duplicate-name check Co-Authored-By: Claude Fable 5 --- .../docs/concepts/pipelines/asyncpipeline.mdx | 8 +++++ .../choosing-the-right-generator.mdx | 29 +++++++++++++++++++ docs-website/docs/tools/searchabletoolset.mdx | 2 ++ 3 files changed, 39 insertions(+) diff --git a/docs-website/docs/concepts/pipelines/asyncpipeline.mdx b/docs-website/docs/concepts/pipelines/asyncpipeline.mdx index b291245ae1c..40cb984629f 100644 --- a/docs-website/docs/concepts/pipelines/asyncpipeline.mdx +++ b/docs-website/docs/concepts/pipelines/asyncpipeline.mdx @@ -36,6 +36,8 @@ Executes the pipeline synchronously with the provided input data. This method is Executes the pipeline in an asynchronous manner, allowing non-blocking execution. This method is ideal when integrating the pipeline into an async workflow, enabling smooth operation within larger async applications or services. +When streaming, components accept a sync `streaming_callback` in `run_async` too — see [Sync and Async Callbacks](../../pipeline-components/generators/guides-to-generators/choosing-the-right-generator.mdx#sync-and-async-callbacks) for details. + #### Asynchronous Generator (`run_async_generator`) Allows step-by-step execution by yielding partial outputs as components complete their tasks. This is particularly useful for monitoring progress, debugging, and handling outputs incrementally. It differs from `run_async`, which executes the pipeline in a single async call. @@ -46,6 +48,12 @@ In an `AsyncPipeline`, components such as A and B will run in parallel _only if You can control the maximum number of components that run simultaneously using the `concurrency_limit` parameter to ensure controlled resource usage. +### Error Handling and Task Cancellation + +If a component raises an error while sibling components are still running concurrently, the `AsyncPipeline` cancels and drains those in-flight tasks before re-raising the original error, so no tasks keep running in the background. The same cleanup applies when you stop iterating `run_async_generator` early (for example, by breaking out of the loop and closing the generator) or when the run itself is cancelled. + +Note that cancellation only interrupts components that run natively async. Sync components are offloaded to a worker thread, which cannot be interrupted and runs to completion in the background. Their outputs are discarded, so the pipeline state stays consistent, but the component's side effects still complete. + You can find more details in our [API Reference](/reference/pipeline-api#asyncpipeline), or directly in the pipeline's [GitHub code](https://github.com/deepset-ai/haystack/blob/main/haystack/core/pipeline/async_pipeline.py). ## Example diff --git a/docs-website/docs/pipeline-components/generators/guides-to-generators/choosing-the-right-generator.mdx b/docs-website/docs/pipeline-components/generators/guides-to-generators/choosing-the-right-generator.mdx index 5bfed612450..1c5ec086743 100644 --- a/docs-website/docs/pipeline-components/generators/guides-to-generators/choosing-the-right-generator.mdx +++ b/docs-website/docs/pipeline-components/generators/guides-to-generators/choosing-the-right-generator.mdx @@ -56,6 +56,35 @@ generator = SomeGenerator(streaming_callback=print_streaming_chunk) # For ChatGenerators, pass a list[ChatMessage]. For text generators, pass a prompt string. ``` +### Sync and Async Callbacks + +Streaming-capable components (such as `OpenAIChatGenerator`, `HuggingFaceAPIChatGenerator`, `HuggingFaceLocalChatGenerator`, and `Agent`) accept both sync and async streaming callbacks in async contexts (`run_async` or an [AsyncPipeline](../../../concepts/pipelines/asyncpipeline.mdx)). When you pass a sync callback to `run_async`, a warning is logged because the callback runs synchronously on the event loop and may block it, but the run proceeds and streams as expected. Async callbacks remain preferred for performance in async contexts. + +```python +import asyncio + +from haystack.components.generators.chat import OpenAIChatGenerator +from haystack.dataclasses import ChatMessage, StreamingChunk + + +def print_chunk(chunk: StreamingChunk) -> None: + print(chunk.content, end="", flush=True) + + +async def main(): + llm = OpenAIChatGenerator() + # a sync callback in an async context logs a warning and streams as expected + await llm.run_async( + [ChatMessage.from_user("Tell me about Italy")], + streaming_callback=print_chunk, + ) + + +asyncio.run(main()) +``` + +The reverse is not supported: passing an async callback to the sync `run` method raises an error, since a coroutine cannot be awaited from sync code. + ### Custom Callback If you need custom rendering, write your own callback. Handle the four chunk types in order: diff --git a/docs-website/docs/tools/searchabletoolset.mdx b/docs-website/docs/tools/searchabletoolset.mdx index d6c971da6dc..9b71e5d0f1a 100644 --- a/docs-website/docs/tools/searchabletoolset.mdx +++ b/docs-website/docs/tools/searchabletoolset.mdx @@ -46,6 +46,8 @@ subsequent iterations. `SearchableToolset` does not support adding new tools after initialization or merging with other toolsets. Use `catalog` to provide all tools upfront. ::: +All tool names in the catalog must be unique. `warm_up()` raises a `ValueError` if the catalog contains tools with duplicate names, since a search hit could otherwise resolve to the wrong tool. + ## Usage ### Basic usage with an Agent From c5a02df91ee06737ae8334027b2d32dc917a59fc Mon Sep 17 00:00:00 2001 From: Julian Risch Date: Sun, 5 Jul 2026 22:36:28 +0200 Subject: [PATCH 2/5] docs: move SearchableToolset duplicate-name note to the PR covering the other SearchableToolset docs Co-Authored-By: Claude Fable 5 --- docs-website/docs/tools/searchabletoolset.mdx | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs-website/docs/tools/searchabletoolset.mdx b/docs-website/docs/tools/searchabletoolset.mdx index 9b71e5d0f1a..d6c971da6dc 100644 --- a/docs-website/docs/tools/searchabletoolset.mdx +++ b/docs-website/docs/tools/searchabletoolset.mdx @@ -46,8 +46,6 @@ subsequent iterations. `SearchableToolset` does not support adding new tools after initialization or merging with other toolsets. Use `catalog` to provide all tools upfront. ::: -All tool names in the catalog must be unique. `warm_up()` raises a `ValueError` if the catalog contains tools with duplicate names, since a search hit could otherwise resolve to the wrong tool. - ## Usage ### Basic usage with an Agent From 3497cb900ee4fb7095183c9aba8e0bec0c354898 Mon Sep 17 00:00:00 2001 From: Julian Risch Date: Wed, 8 Jul 2026 21:38:21 +0200 Subject: [PATCH 3/5] docs: move error-handling and cancellation docs to Pipelines page The AsyncPipeline page is being removed (AsyncPipeline merged into Pipeline in 3.0), so the new content lands on the Pipelines concepts page instead, and the streaming-callbacks guide links there. Co-Authored-By: Claude Fable 5 --- docs-website/docs/concepts/pipelines.mdx | 8 ++++++++ docs-website/docs/concepts/pipelines/asyncpipeline.mdx | 8 -------- .../guides-to-generators/choosing-the-right-generator.mdx | 2 +- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/docs-website/docs/concepts/pipelines.mdx b/docs-website/docs/concepts/pipelines.mdx index f3311f8b19f..b469aac160f 100644 --- a/docs-website/docs/concepts/pipelines.mdx +++ b/docs-website/docs/concepts/pipelines.mdx @@ -48,6 +48,14 @@ The AsyncPipeline enables parallel execution of Haystack components when their d Find out more in our [AsyncPipeline](pipelines/asyncpipeline.mdx) documentation. +When streaming, components accept a sync `streaming_callback` in `run_async` too — see [Sync and Async Callbacks](../pipeline-components/generators/guides-to-generators/choosing-the-right-generator.mdx#sync-and-async-callbacks) for details. + +#### Error Handling and Task Cancellation + +If a component raises an error while sibling components are still running concurrently, the pipeline cancels and drains those in-flight tasks before re-raising the original error, so no tasks keep running in the background. The same cleanup applies when you stop iterating `run_async_generator` early (for example, by breaking out of the loop and closing the generator) or when the run itself is cancelled. + +Note that cancellation only interrupts components that run natively async. Sync components are offloaded to a worker thread, which cannot be interrupted and runs to completion in the background. Their outputs are discarded, so the pipeline state stays consistent, but the component's side effects still complete. + ## SuperComponents To simplify your code, we have introduced [SuperComponents](components/supercomponents.mdx) that allow you to wrap complete pipelines and reuse them as a single component. Check out their documentation page for the details and examples. diff --git a/docs-website/docs/concepts/pipelines/asyncpipeline.mdx b/docs-website/docs/concepts/pipelines/asyncpipeline.mdx index 46aab021c1c..e113b331bac 100644 --- a/docs-website/docs/concepts/pipelines/asyncpipeline.mdx +++ b/docs-website/docs/concepts/pipelines/asyncpipeline.mdx @@ -36,8 +36,6 @@ Executes the pipeline synchronously with the provided input data. This method is Executes the pipeline in an asynchronous manner, allowing non-blocking execution. This method is ideal when integrating the pipeline into an async workflow, enabling smooth operation within larger async applications or services. -When streaming, components accept a sync `streaming_callback` in `run_async` too — see [Sync and Async Callbacks](../../pipeline-components/generators/guides-to-generators/choosing-the-right-generator.mdx#sync-and-async-callbacks) for details. - #### Asynchronous Generator (`run_async_generator`) Allows step-by-step execution by yielding partial outputs as components complete their tasks. This is particularly useful for monitoring progress, debugging, and handling outputs incrementally. It differs from `run_async`, which executes the pipeline in a single async call. @@ -48,12 +46,6 @@ In an `AsyncPipeline`, components such as A and B will run in parallel _only if You can control the maximum number of components that run simultaneously using the `concurrency_limit` parameter to ensure controlled resource usage. -### Error Handling and Task Cancellation - -If a component raises an error while sibling components are still running concurrently, the `AsyncPipeline` cancels and drains those in-flight tasks before re-raising the original error, so no tasks keep running in the background. The same cleanup applies when you stop iterating `run_async_generator` early (for example, by breaking out of the loop and closing the generator) or when the run itself is cancelled. - -Note that cancellation only interrupts components that run natively async. Sync components are offloaded to a worker thread, which cannot be interrupted and runs to completion in the background. Their outputs are discarded, so the pipeline state stays consistent, but the component's side effects still complete. - You can find more details in our [API Reference](/reference/pipeline-api#pipeline), or directly in the pipeline's [GitHub code](https://github.com/deepset-ai/haystack/blob/main/haystack/core/pipeline/pipeline.py). ## Example diff --git a/docs-website/docs/pipeline-components/generators/guides-to-generators/choosing-the-right-generator.mdx b/docs-website/docs/pipeline-components/generators/guides-to-generators/choosing-the-right-generator.mdx index 1c5ec086743..0a16758a00a 100644 --- a/docs-website/docs/pipeline-components/generators/guides-to-generators/choosing-the-right-generator.mdx +++ b/docs-website/docs/pipeline-components/generators/guides-to-generators/choosing-the-right-generator.mdx @@ -58,7 +58,7 @@ generator = SomeGenerator(streaming_callback=print_streaming_chunk) ### Sync and Async Callbacks -Streaming-capable components (such as `OpenAIChatGenerator`, `HuggingFaceAPIChatGenerator`, `HuggingFaceLocalChatGenerator`, and `Agent`) accept both sync and async streaming callbacks in async contexts (`run_async` or an [AsyncPipeline](../../../concepts/pipelines/asyncpipeline.mdx)). When you pass a sync callback to `run_async`, a warning is logged because the callback runs synchronously on the event loop and may block it, but the run proceeds and streams as expected. Async callbacks remain preferred for performance in async contexts. +Streaming-capable components (such as `OpenAIChatGenerator`, `HuggingFaceAPIChatGenerator`, `HuggingFaceLocalChatGenerator`, and `Agent`) accept both sync and async streaming callbacks in async contexts (`run_async` or [async pipeline execution](../../../concepts/pipelines.mdx)). When you pass a sync callback to `run_async`, a warning is logged because the callback runs synchronously on the event loop and may block it, but the run proceeds and streams as expected. Async callbacks remain preferred for performance in async contexts. ```python import asyncio From c2063fe5e561068a38a3712705b83ac411a0e351 Mon Sep 17 00:00:00 2001 From: Julian Risch Date: Wed, 8 Jul 2026 21:44:53 +0200 Subject: [PATCH 4/5] docs: hand pipelines.mdx changes over to #11871 All pipelines.mdx edits are consolidated in #11871 to avoid cross-PR conflicts; this PR now only adds the Sync and Async Callbacks section. Co-Authored-By: Claude Fable 5 --- docs-website/docs/concepts/pipelines.mdx | 8 -------- 1 file changed, 8 deletions(-) diff --git a/docs-website/docs/concepts/pipelines.mdx b/docs-website/docs/concepts/pipelines.mdx index b469aac160f..f3311f8b19f 100644 --- a/docs-website/docs/concepts/pipelines.mdx +++ b/docs-website/docs/concepts/pipelines.mdx @@ -48,14 +48,6 @@ The AsyncPipeline enables parallel execution of Haystack components when their d Find out more in our [AsyncPipeline](pipelines/asyncpipeline.mdx) documentation. -When streaming, components accept a sync `streaming_callback` in `run_async` too — see [Sync and Async Callbacks](../pipeline-components/generators/guides-to-generators/choosing-the-right-generator.mdx#sync-and-async-callbacks) for details. - -#### Error Handling and Task Cancellation - -If a component raises an error while sibling components are still running concurrently, the pipeline cancels and drains those in-flight tasks before re-raising the original error, so no tasks keep running in the background. The same cleanup applies when you stop iterating `run_async_generator` early (for example, by breaking out of the loop and closing the generator) or when the run itself is cancelled. - -Note that cancellation only interrupts components that run natively async. Sync components are offloaded to a worker thread, which cannot be interrupted and runs to completion in the background. Their outputs are discarded, so the pipeline state stays consistent, but the component's side effects still complete. - ## SuperComponents To simplify your code, we have introduced [SuperComponents](components/supercomponents.mdx) that allow you to wrap complete pipelines and reuse them as a single component. Check out their documentation page for the details and examples. From a1554b95448ee6a45a1fb49787020a8610dd51a5 Mon Sep 17 00:00:00 2001 From: Julian Risch Date: Thu, 9 Jul 2026 09:55:26 +0200 Subject: [PATCH 5/5] refer to TransformersChatGenerator --- .../guides-to-generators/choosing-the-right-generator.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs-website/docs/pipeline-components/generators/guides-to-generators/choosing-the-right-generator.mdx b/docs-website/docs/pipeline-components/generators/guides-to-generators/choosing-the-right-generator.mdx index 0a16758a00a..9c23d09b932 100644 --- a/docs-website/docs/pipeline-components/generators/guides-to-generators/choosing-the-right-generator.mdx +++ b/docs-website/docs/pipeline-components/generators/guides-to-generators/choosing-the-right-generator.mdx @@ -58,7 +58,7 @@ generator = SomeGenerator(streaming_callback=print_streaming_chunk) ### Sync and Async Callbacks -Streaming-capable components (such as `OpenAIChatGenerator`, `HuggingFaceAPIChatGenerator`, `HuggingFaceLocalChatGenerator`, and `Agent`) accept both sync and async streaming callbacks in async contexts (`run_async` or [async pipeline execution](../../../concepts/pipelines.mdx)). When you pass a sync callback to `run_async`, a warning is logged because the callback runs synchronously on the event loop and may block it, but the run proceeds and streams as expected. Async callbacks remain preferred for performance in async contexts. +Streaming-capable components (such as `OpenAIChatGenerator`, `HuggingFaceAPIChatGenerator`, `TransformersChatGenerator`, and `Agent`) accept both sync and async streaming callbacks in async contexts (`run_async` or [async pipeline execution](../../../concepts/pipelines.mdx)). When you pass a sync callback to `run_async`, a warning is logged because the callback runs synchronously on the event loop and may block it, but the run proceeds and streams as expected. Async callbacks remain preferred for performance in async contexts. ```python import asyncio @@ -222,7 +222,7 @@ On-premise models mean that you host open models on your machine or infrastructu #### Local Experimentation -- GPU: [`HuggingFaceLocalChatGenerator`](../huggingfacelocalchatgenerator.mdx) is based on the Hugging Face Transformers library. This is good for experimentation when you have some GPU resources (for example, in Colab). If GPU resources are limited, alternative quantization options like bitsandbytes, GPTQ, and AWQ are supported. For more performant solutions in production use cases, refer to the options below. +- GPU: [`TransformersChatGenerator`](../transformerschatgenerator.mdx) is based on the Hugging Face Transformers library. This is good for experimentation when you have some GPU resources (for example, in Colab). If GPU resources are limited, alternative quantization options like bitsandbytes, GPTQ, and AWQ are supported. For more performant solutions in production use cases, refer to the options below. - CPU (+ GPU if available): [`LlamaCppChatGenerator`](../llamacppchatgenerator.mdx) uses the Llama.cpp library – a project written in C/C++ for efficient inference of LLMs. In particular, it employs the quantized GGUF format, suitable for running these models on standard machines (even without GPUs). If GPU resources are available, some model layers can be offloaded to GPU for enhanced speed. - CPU (+ GPU if available): [`OllamaChatGenerator`](../ollamachatgenerator.mdx) is based on the Ollama project, acting like Docker for LLMs. It provides a simple way to package and deploy these models. Internally based on the Llama.cpp library, it offers a more streamlined process for running on various platforms.