Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 4 additions & 58 deletions docs-website/docs/concepts/pipelines/pipeline-breakpoints.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,21 +2,19 @@
title: "Pipeline Breakpoints"
id: pipeline-breakpoints
slug: "/pipeline-breakpoints"
description: "Learn how to pause and resume Haystack pipeline or Agent execution using breakpoints to debug, inspect, and continue workflows from saved snapshots."
description: "Learn how to pause and resume Haystack pipeline execution using breakpoints to debug, inspect, and continue workflows from saved snapshots."
---

# Pipeline Breakpoints

Learn how to pause and resume Haystack pipeline or Agent execution using breakpoints to debug, inspect, and continue workflows from saved snapshots.
Learn how to pause and resume Haystack pipeline execution using breakpoints to debug, inspect, and continue workflows from saved snapshots.

## Introduction

Haystack pipelines support breakpoints for debugging complex execution flows. A `Breakpoint` allows you to pause the execution at specific components, inspect the pipeline state, and resume execution from saved snapshots. This feature works for any regular component as well as an `Agent` component.

You can set a `Breakpoint` on any component in a pipeline with a specific visit count. When triggered, the system stops the execution of the `Pipeline` and captures a snapshot of the current pipeline state. The state can be saved to a JSON file when snapshot file saving is enabled, see [Snapshot file saving](#snapshot-file-saving) below. You can inspect and modify the snapshot and use it to resume execution from the exact point where it stopped.

You can also set breakpoints on an Agent, specifically on the `ChatGenerator` component or on any of the `Tool` specified in the `ToolInvoker` component .

## Setting a `Breakpoint` on a Regular Component

Create a `Breakpoint` by specifying the component name and the visit count at which to trigger it. This is useful for pipelines with loops. The default `visit_count` value is 0.
Expand Down Expand Up @@ -49,7 +47,7 @@ To access the pipeline state during the breakpoint we can both catch the excepti

## Using a custom snapshot callback

You can pass a `snapshot_callback` to `Pipeline.run()` or `Agent.run()` to handle snapshots yourself instead of saving to a file. When a breakpoint is triggered or a snapshot is created on error, the callback is invoked with the `PipelineSnapshot` object. This is useful for saving snapshots to a database, sending them to a remote service, or custom logging.
You can pass a `snapshot_callback` to `Pipeline.run()` to handle snapshots yourself instead of saving to a file. When a breakpoint is triggered or a snapshot is created on error, the callback is invoked with the `PipelineSnapshot` object. This is useful for saving snapshots to a database, sending them to a remote service, or custom logging.

```python
from haystack.core.errors import BreakpointException
Expand All @@ -72,7 +70,7 @@ except BreakpointException as e:
print(f"Breakpoint triggered: {e.component}")
```

When `snapshot_callback` is provided, file-saving is skipped and the callback is responsible for handling the snapshot. The same parameter is available on `Agent.run()` for agent breakpoints.
When `snapshot_callback` is provided, file-saving is skipped and the callback is responsible for handling the snapshot.

## Snapshot file saving

Expand Down Expand Up @@ -111,58 +109,6 @@ result = pipeline.run(data={}, pipeline_snapshot=snapshot)
print(result["llm"]["replies"])
```

## Setting a Breakpoint on an Agent

You can also set breakpoints in an Agent component. An Agent supports two types of breakpoints:

1. **Chat Generator Breakpoint**: Pauses before LLM calls.
2. **Tool Invoker Breakpoint**: Pauses before any tool execution.

A `ChatGenerator` breakpoint is defined as shown below. You need to define a `Breakpoint` as for a pipeline breakpoint and then an `AgentBreakpoint` where you pass the breakpoint defined before and the name of Agent component.

```python
from haystack.dataclasses.breakpoints import AgentBreakpoint, Breakpoint, ToolBreakpoint

# Break at chat generator (LLM calls)
chat_bp = Breakpoint(component_name="chat_generator", visit_count=0)
agent_breakpoint = AgentBreakpoint(break_point=chat_bp, agent_name="my_agent")
```

To set a breakpoint on a Tool in an Agent, do the following:

First, define a `ToolBreakpoint` specifying the `ToolInvoker` component whose name is `tool_invoker` and then the tool associated with the breakpoint, in this case – a `weather_tool` .

Then, define an `AgentBreakpoint` passing the `ToolBreakpoint` defined before as the breakpoint.

```python
from haystack.dataclasses.breakpoints import AgentBreakpoint, Breakpoint, ToolBreakpoint

# Break at tool invoker (tool calls)
tool_bp = ToolBreakpoint(
component_name="tool_invoker",
visit_count=0,
tool_name="weather_tool", # Specific tool, or None for any tool
)
agent_breakpoint = AgentBreakpoint(break_point=tool_bp, agent_name="my_agent")
```

### Resuming Agent Execution

When an Agent breakpoint is triggered, you can resume execution using the saved snapshot. Similar to the regular component in a pipeline, pass the JSON file with the snapshot to the `run()` method of the pipeline.

```python
from haystack.core.pipeline.breakpoint import load_pipeline_snapshot

# Load the snapshot
snapshot_file = "./agent_debug/agent_chat_generator_2025_07_11_23_23.json"
snapshot = load_pipeline_snapshot(snapshot_file)

# Resume pipeline execution
result = pipeline.run(data={}, pipeline_snapshot=snapshot)
print("Pipeline resumed successfully")
print(f"Final result: {result}")
```

## Error Recovery with Snapshots

Pipelines automatically create a snapshot of the last valid state if a run fails. The snapshot contains inputs, visit counts, and intermediate outputs up to the failure. You can inspect it, fix the issue, and resume execution from that checkpoint instead of restarting the whole run.
Expand Down
23 changes: 7 additions & 16 deletions docs-website/docs/development/tracing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -23,26 +23,17 @@ Instrumented applications typically send traces to a trace collector or a tracin
| [LoggingTracer](tracing/logging-tracer.mdx) | Inspect the data flowing through your pipeline in real time through logs, with no backend setup. |
| [Custom Tracer](tracing/custom-tracer.mdx) | Connect any tracing backend by implementing the `Tracer` interface. |

## Disabling Auto Tracing
## Enabling and Disabling Tracing

Haystack automatically detects and enables tracing under the following circumstances:
Haystack never enables tracing automatically. To enable it, either call `haystack.tracing.enable_tracing(...)` with the tracer of your choice, or add a tracing connector component such as the [`OpenTelemetryConnector`](tracing/opentelemetry.mdx) or the [`DatadogConnector`](tracing/datadog.mdx) to your pipeline.

- If `opentelemetry-sdk` is installed and configured for OpenTelemetry. Note that this auto-enabling is deprecated and will be removed in Haystack 3.0 – use the [`OpenTelemetryConnector`](tracing/opentelemetry.mdx) to enable OpenTelemetry tracing instead.
- If `ddtrace` is installed for Datadog. Note that this auto-enabling is deprecated and will be removed in Haystack 3.0 – use the [`DatadogConnector`](tracing/datadog.mdx) to enable Datadog tracing instead.
To disable an enabled tracer:

To disable this behavior, there are two options:
```python
from haystack.tracing import disable_tracing

- Set the environment variable `HAYSTACK_AUTO_TRACE_ENABLED` to `false` when running your Haystack application

— or —

- Disable tracing in Python:

```python
from haystack.tracing import disable_tracing

disable_tracing()
```
disable_tracing()
```

## Content Tracing

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Here's a table of key concepts and their approximate equivalents between the two
| Multi-Agent Collaboration (LangChain) | Multi-Agent System | Using [`ComponentTool`](../tools/componenttool.mdx), agents can use other agents as tools, enabling [multi-agent architectures](https://haystack.deepset.ai/tutorials/45_creating_a_multi_agent_system) within one framework. |
| Model Context Protocol `load_mcp_tools` `MultiServerMCPClient` | Model Context Protocol - `MCPTool`, `MCPToolset`, `StdioServerInfo`, `StreamableHttpServerInfo` | Haystack provides [various MCP primitives](https://haystack.deepset.ai/integrations/mcp) for connecting multiple MCP servers and organizing MCP toolsets. |
| Memory (State, short-term, long-term) | Memory (Agent State, short-term, long-term) | Agent [State](../pipeline-components/agents-1/state.mdx) provides a structured way to share data between tools and store intermediate results during agent execution. For short-term memory, Haystack offers a [ChatMessage Store](/reference/experimental-chatmessage-store-api) to persist chat history. More memory options are coming soon. |
| Time travel (Checkpoints) | Breakpoints (Breakpoint, AgentBreakpoint, ToolBreakpoint, Snapshot) | [Breakpoints](../concepts/pipelines/pipeline-breakpoints.mdx) let you pause, inspect, modify, and resume a pipeline, agent, or tool for debugging or iterative development. |
| Time travel (Checkpoints) | Breakpoints (Breakpoint, PipelineSnapshot) | [Breakpoints](../concepts/pipelines/pipeline-breakpoints.mdx) let you pause, inspect, modify, and resume a pipeline for debugging or iterative development. |
| Human-in-the-Loop (Interrupts / Commands) | Human-in-the-loop (`ConfirmationHook` with confirmation strategies) | Haystack applies [confirmation strategies](https://haystack.deepset.ai/tutorials/47_human_in_the_loop_agent) through a `ConfirmationHook` registered under the Agent's `before_tool` [hook point](../pipeline-components/agents-1/hooks.mdx) to pause or block the execution to gather user feedback |

## Ecosystem and Tooling Mapping: LangChain → Haystack
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ def get_current_weather(
tool = create_tool_from_function(get_current_weather)
```

Create a new instance of `GoogleAIGeminiChatGenerator` to set the tools and a [ToolInvoker](../tools/toolinvoker.mdx) to invoke the tools.
Create a new instance of `GoogleAIGeminiChatGenerator` to set the tools and a `ToolInvoker` to invoke the tools.

```python
import os
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ At this point the model has only *requested* a call. Nothing has been executed y

### 3. Actually Invoke the Tool

To execute the requested tool calls, use the [`ToolInvoker`](../../tools/toolinvoker.mdx) component. It takes a list of [`ChatMessage`](../../../concepts/data-classes/chatmessage.mdx) objects containing tool calls, runs the corresponding tools, and returns new `ChatMessage` objects (with role `tool`) carrying the results as `ToolCallResult`s. To get a final natural-language answer, feed those results back to the ChatGenerator.
To execute the requested tool calls, use the `ToolInvoker` component. It takes a list of [`ChatMessage`](../../../concepts/data-classes/chatmessage.mdx) objects containing tool calls, runs the corresponding tools, and returns new `ChatMessage` objects (with role `tool`) carrying the results as `ToolCallResult`s. To get a final natural-language answer, feed those results back to the ChatGenerator.

```python
from haystack.components.tools import ToolInvoker
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ def get_current_weather(
tool = create_tool_from_function(get_current_weather)
```

Create a new instance of `VertexAIGeminiChatGenerator` to set the tools and a [ToolInvoker](../tools/toolinvoker.mdx) to invoke the tools.:
Create a new instance of `VertexAIGeminiChatGenerator` to set the tools and a `ToolInvoker` to invoke the tools.:

```python
from haystack_integrations.components.generators.google_vertex import (
Expand Down
214 changes: 0 additions & 214 deletions docs-website/docs/pipeline-components/tools/toolinvoker.mdx

This file was deleted.

Loading