Agent snapshots can become non-resumable when snapshot fallback stores empty tool_invoker/chat_generator payloads

[shaun0927]

Describe the bug
_create_agent_snapshot() now avoids masking the original runtime error by falling back to {} when serializing chat_generator or tool_invoker inputs fails. However, that fallback can produce a saved agent snapshot that is no longer resumable.

I reproduced this on the current main branch by passing a non-serializable runtime-only callback (lambda) as streaming_callback. The agent still saves a snapshot file, but the snapshot contains empty {} payloads for chat_generator / tool_invoker, and agent.run(..., snapshot=...) then fails during resume.

This looks like a follow-up regression to the robustness fix in #11108: the original runtime error is preserved, but the saved snapshot violates the resume contract.

Error message
On resume:

DeserializationError: Invalid format of passed serialized payload. Expected a dictionary with keys 'serialization_schema' and 'serialized_data'. Got: {}

Warnings emitted when the snapshot is created:

Failed to serialize the agent's chat_generator inputs. The inputs in the snapshot will be replaced with an empty dictionary. Error: Serialization of lambdas is not supported.
Failed to serialize the agent's tool_invoker inputs. The inputs in the snapshot will be replaced with an empty dictionary. Error: Serialization of lambdas is not supported.

Expected behavior
One of the following:

1. The snapshot remains resumable by omitting only the non-serializable runtime-only fields.
2. Or Haystack clearly marks the snapshot as non-resumable instead of saving structurally invalid {} payloads.

Additional context

• Related issue/PR: Update _create_agent_snapshot to be more robust towards errors just like _create_pipeline_snapshot #10642 / feat: Enhance agent snapshot serialization with error handling for non-serializable inputs #11108
• I also verified that the normal serializable snapshot path still resumes correctly; the problem is specific to the fallback {} path.
• I did not find a dedicated issue for this downstream resume failure.

To Reproduce
Minimal repro outline:

1. Create an Agent with a ToolBreakpoint snapshot path.
2. Pass a non-serializable runtime callback, for example streaming_callback=lambda chunk: None.
3. Trigger a breakpoint or tool failure so the agent writes a snapshot.
4. Load the saved snapshot with load_pipeline_snapshot(...).
5. Call agent.run(..., snapshot=loaded.agent_snapshot).
6. Resume fails with DeserializationError because chat_generator / tool_invoker were saved as {}.

A concrete version of the repro (validated locally on current main) is:

from pathlib import Path
import os

from haystack import component
from haystack.components.agents import Agent
from haystack.core.errors import BreakpointException
from haystack.core.pipeline.breakpoint import load_pipeline_snapshot
from haystack.dataclasses import ChatMessage, ToolCall
from haystack.dataclasses.breakpoints import AgentBreakpoint, ToolBreakpoint
from haystack.tools import Tool, Toolset


@component
class MockChatGenerator:
    def __init__(self):
        self._counter = 0
        self.responses = [
            ChatMessage.from_assistant(tool_calls=[ToolCall(tool_name="weather_tool", arguments={"location": "Berlin"})]),
            ChatMessage.from_assistant(text="The weather in Berlin is sunny."),
        ]

    def to_dict(self):
        return {"type": "MockChatGenerator", "data": {}}

    @classmethod
    def from_dict(cls, data):
        return cls()

    @component.output_types(replies=list[ChatMessage])
    def run(self, messages, tools: list[Tool] | Toolset | None = None, **kwargs):
        result = self.responses[self._counter]
        self._counter += 1
        return {"replies": [result]}


def weather_tool(location: str):
    return {"weather": "sunny", "location": location}


os.environ["HAYSTACK_PIPELINE_SNAPSHOT_SAVE_ENABLED"] = "true"
debug_path = Path("debug_snapshots")

agent = Agent(
    chat_generator=MockChatGenerator(),
    tools=[
        Tool(
            name="weather_tool",
            description="Weather tool",
            parameters={"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]},
            function=weather_tool,
        )
    ],
)

breakpoint = AgentBreakpoint(
    break_point=ToolBreakpoint(component_name="tool_invoker", tool_name="weather_tool", snapshot_file_path=str(debug_path)),
    agent_name="test_agent",
)

try:
    agent.run(
        messages=[ChatMessage.from_user("What's the weather in Berlin?")],
        break_point=breakpoint,
        streaming_callback=lambda chunk: None,
    )
except BreakpointException:
    pass

snapshot_file = max(debug_path.glob("test_agent_tool_invoker_*.json"), key=lambda p: p.stat().st_ctime)
loaded = load_pipeline_snapshot(snapshot_file)
agent.run(messages=[ChatMessage.from_user("ignored")], snapshot=loaded.agent_snapshot)

FAQ Check

• Have you had a look at our new FAQ page?

System:

• OS: macOS (darwin arm64)
• GPU/CPU: CPU
• Haystack version (commit or version number): 07e8c9bad898a466f0a91c97d817ba3e6ba13dc8 (main as of 2026-04-17)
• DocumentStore: N/A
• Reader: N/A
• Retriever: N/A

[shaun0927]

I opened #11127 with a narrow follow-up fix for this.

The approach keeps the original #11108 behavior of preserving the runtime error, but makes the agent snapshot fallback field-aware so non-serializable runtime-only inputs (for example a lambda callback) are omitted without dropping resumable fields like messages, state, and tools. The PR also adds a regression test that resumes from a tool-invoker snapshot created with a non-serializable runtime callback.

[shaun0927]

I re-validated this locally on the latest upstream main I fetched, at 8dd36b366a5974c41eed4bc9be937fa7d2bcd9b9.

Repro command:

cd /Users/jh0927/Workspace/haystack-autopilot-audit
hatch -e test run python /Users/jh0927/Workspace/haystack_snapshot_resume_probe.py

Observed behavior on current main:

• the tool-invoker breakpoint snapshot is saved
• both chat_generator and tool_invoker fall back to {} when the runtime-only lambda callback is not serializable
• resume then fails with:

DeserializationError: Invalid format of passed serialized payload. Expected a dictionary with keys 'serialization_schema' and 'serialized_data'. Got: {}

High-signal output from the local repro:

initial_run=breakpoint component=tool_invoker
snapshot_file_count=1
tool_inputs_repr={}
resume_status=error type=DeserializationError message=Invalid format of passed serialized payload. Expected a dictionary with keys 'serialization_schema' and 'serialized_data'. Got: {}

I also re-ran the existing normal resume tests on the same main checkout, and those still pass:

cd /Users/jh0927/Workspace/haystack-autopilot-audit
hatch -e test run pytest test/components/agents/test_agent_breakpoints.py -k 'test_resume_from_tool_invoker and not new_breakpoint and not non_serializable' -q

Result:

2 passed, 20 deselected in 13.34s

So this still looks isolated to the fallback path triggered by non-serializable runtime-only inputs, not to agent snapshot resume in general.

I also validated PR #11127 locally. On that branch, the snapshot omits only streaming_callback, preserves tool_invoker.state, and resume succeeds.