From 8a6a343e9f05cb29eb69d55cbbb0916353f1394e Mon Sep 17 00:00:00 2001 From: Julian Risch Date: Sun, 5 Jul 2026 15:06:12 +0200 Subject: [PATCH 1/4] docs: document Agent run metadata, runtime exit_conditions, and MockChatGenerator Co-Authored-By: Claude Fable 5 --- .../pipeline-components/agents-1/agent.mdx | 21 +++- .../pipeline-components/agents-1/state.mdx | 4 + .../docs/pipeline-components/generators.mdx | 1 + .../generators/mockchatgenerator.mdx | 109 ++++++++++++++++++ docs-website/docs/tools/searchabletoolset.mdx | 6 + docs-website/sidebars.js | 1 + 6 files changed, 140 insertions(+), 2 deletions(-) create mode 100644 docs-website/docs/pipeline-components/generators/mockchatgenerator.mdx diff --git a/docs-website/docs/pipeline-components/agents-1/agent.mdx b/docs-website/docs/pipeline-components/agents-1/agent.mdx index fef0f17e649..5e57458389b 100644 --- a/docs-website/docs/pipeline-components/agents-1/agent.mdx +++ b/docs-website/docs/pipeline-components/agents-1/agent.mdx @@ -38,16 +38,33 @@ The `Agent` returns a dictionary containing: - `messages`: the full conversation history, - `last_message`: the final `ChatMessage` from the agent, +- `step_count`: the number of steps the agent ran, +- `token_usage`: aggregated token usage summed across every LLM call in the run, +- `tool_call_counts`: how many times each tool was invoked, keyed by tool name, - Additional dynamic keys based on `state_schema`. +### Run Metadata + +The `step_count`, `token_usage`, and `tool_call_counts` outputs are populated automatically during a run. They are added to the agent's `state_schema` behind the scenes, so tools registered with `inputs_from_state` can read them mid-run. They are outputs only — they cannot be passed as inputs to `run()` or `run_async()`, and using them as keys in your own `state_schema` raises a `ValueError`. See [State](./state.mdx#schema-definition) for details. + +```python +response = agent.run(messages=[ChatMessage.from_user("What is 7 * (4 + 2)?")]) + +print(response["step_count"]) # e.g. 2 +print( + response["token_usage"], +) # e.g. {"prompt_tokens": 512, "completion_tokens": 86, ...} +print(response["tool_call_counts"]) # e.g. {"calculator": 1} +``` + ## Parameters `chat_generator` is the only mandatory parameter — an instance of a Chat Generator that supports tools. All other parameters are optional. -- `tools`: A list of tool or toolset instances the agent can call. Supported types: [`Tool`](../../tools/tool.mdx), [`ComponentTool`](../../tools/componenttool.mdx), [`PipelineTool`](../../tools/pipelinetool.mdx), [`MCPTool`](../../tools/mcptool.mdx), [`Toolset`](../../tools/toolset.mdx), [`MCPToolset`](../../tools/mcptoolset.mdx), [`SearchableToolset`](../../tools/searchabletoolset.mdx). +- `tools`: A list of tool or toolset instances the agent can call. Supported types: [`Tool`](../../tools/tool.mdx), [`ComponentTool`](../../tools/componenttool.mdx), [`PipelineTool`](../../tools/pipelinetool.mdx), [`MCPTool`](../../tools/mcptool.mdx), [`Toolset`](../../tools/toolset.mdx), [`MCPToolset`](../../tools/mcptoolset.mdx), [`SearchableToolset`](../../tools/searchabletoolset.mdx). Tool names must be unique; duplicate names are detected at the start of each agent step, before the chat generator is called. - `system_prompt`: A plain string or Jinja2 template used as the system message for every run. If the template contains Jinja2 variables, those variables become additional inputs to `run()`. - `user_prompt`: A Jinja2 template appended to the user-provided messages on each run. Template variables become additional inputs to `run()`. Use `required_variables` to enforce which variables must be provided. -- `exit_conditions`: List of conditions that cause the agent to stop. Use `”text”` to stop when the LLM replies without a tool call, or a tool name to stop once that tool has been executed. Defaults to `[“text”]`. +- `exit_conditions`: List of conditions that cause the agent to stop. Use `”text”` to stop when the LLM replies without a tool call, or a tool name to stop once that tool has been executed. Defaults to `[“text”]`. Exit conditions are evaluated at runtime rather than validated at initialization, so a condition can name a tool that is only loaded later — for example, a tool passed at runtime via `run(tools=...)` or one discovered by a [`SearchableToolset`](../../tools/searchabletoolset.mdx). - `state_schema`: Defines the agent's runtime state — a dict mapping key names to type configs (e.g. `{“docs”: {“type”: list[Document]}}`). Tools can read from and write to state keys via `inputs_from_state` and `outputs_to_state`. See [State](./state.mdx) for full details. - `streaming_callback`: A callback invoked for each streamed token. Use the built-in `print_streaming_chunk` for console output. - `max_agent_steps`: Maximum number of LLM + tool call iterations before the agent stops. Defaults to `100`. diff --git a/docs-website/docs/pipeline-components/agents-1/state.mdx b/docs-website/docs/pipeline-components/agents-1/state.mdx index 1a124636dcc..f153a5374d7 100644 --- a/docs-website/docs/pipeline-components/agents-1/state.mdx +++ b/docs-website/docs/pipeline-components/agents-1/state.mdx @@ -67,6 +67,10 @@ The schema defines what data can be stored and how values are updated. Each sche If you don't specify a handler, State automatically assigns a default based on the type. +:::info Reserved keys +The `Agent` manages some state keys itself and rejects them in a user-provided `state_schema` with a `ValueError`. This includes the run-metadata keys `step_count`, `token_usage`, and `tool_call_counts`, which the Agent populates automatically during a run: tools can read them mid-run via `inputs_from_state`, and they are returned in the result dictionary. If one of your state keys clashes, rename it (for example, `my_token_usage`). +::: + ### Default Handlers State provides two built-in merge behaviors (importable from `haystack.components.agents.state`): diff --git a/docs-website/docs/pipeline-components/generators.mdx b/docs-website/docs/pipeline-components/generators.mdx index 9b4a48db07f..113b05fc1eb 100644 --- a/docs-website/docs/pipeline-components/generators.mdx +++ b/docs-website/docs/pipeline-components/generators.mdx @@ -38,6 +38,7 @@ Generators are responsible for generating text after you give them a prompt. The | [LlamaStackChatGenerator](generators/llamastackchatgenerator.mdx) | Enables chat completions using an LLM model made available via Llama Stack server | ✅ | | [MetaLlamaChatGenerator](generators/metallamachatgenerator.mdx) | Enables chat completion with any model hosted available with Meta Llama API. | ✅ | | [MistralChatGenerator](generators/mistralchatgenerator.mdx) | Enables chat completion using Mistral's text generation models. | ✅ | +| [MockChatGenerator](generators/mockchatgenerator.mdx) | Returns predefined responses without calling any API — a deterministic, zero-cost stand-in for real Chat Generators in tests and prototypes. | ✅ | | [NvidiaChatGenerator](generators/nvidiachatgenerator.mdx) | Enables chat completion using Nvidia-hosted models. | ✅ | | [NvidiaGenerator](generators/nvidiagenerator.mdx) | Provides an interface for generating text using LLMs self-hosted with NVIDIA NIM or models hosted on the NVIDIA API catalog. | ❌ | | [OllamaChatGenerator](generators/ollamachatgenerator.mdx) | Enables chat completion using an LLM running on Ollama. | ✅ | diff --git a/docs-website/docs/pipeline-components/generators/mockchatgenerator.mdx b/docs-website/docs/pipeline-components/generators/mockchatgenerator.mdx new file mode 100644 index 00000000000..87a58f03454 --- /dev/null +++ b/docs-website/docs/pipeline-components/generators/mockchatgenerator.mdx @@ -0,0 +1,109 @@ +--- +title: "MockChatGenerator" +id: mockchatgenerator +slug: "/mockchatgenerator" +description: "A Chat Generator that returns predefined responses without calling any API, for tests and quick prototypes." +--- + +# MockChatGenerator + +A Chat Generator that returns predefined responses without calling any API, for tests and quick prototypes. + +
+ +| | | +| --- | --- | +| **Most common position in a pipeline** | In place of a real Chat Generator, in tests and prototypes | +| **Mandatory init variables** | None | +| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects | +| **Output variables** | `replies`: A list of generated `ChatMessage` objects | +| **API reference** | [Generators](/reference/generators-api) | +| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/generators/chat/mock.py | +| **Package name** | `haystack-ai` | + +
+ +## Overview + +`MockChatGenerator` is a deterministic, zero-cost drop-in replacement for real Chat Generators such as `OpenAIChatGenerator`. It implements the full Chat Generator interface — `run`, `run_async`, streaming callbacks, and serialization — but never contacts an external service, which makes it ideal for unit tests, smoke tests, and quick prototypes. + +The response is selected based on how the component is configured: + +- **Fixed response**: Pass a single string or `ChatMessage` via `responses`. The same reply is returned on every call. A `ChatMessage` passed as a response must have the `assistant` role. +- **Cycling responses**: Pass a list of strings and/or `ChatMessage` objects via `responses`. Each call returns the next item, wrapping around to the start once the list is exhausted. This is useful to drive multi-step flows such as Agents, where the first call returns a tool call and a later call returns the final answer. +- **Dynamic response**: Pass a `response_fn` callable that receives the input messages and returns the reply as a string or an assistant `ChatMessage`. Use this when the reply should depend on the input. To support serialization, pass a named function — lambdas and nested functions cannot be serialized. +- **Echo (default)**: With no configuration, the component echoes back the text of the last message that has text content, so it is usable out of the box. + +`responses` and `response_fn` are mutually exclusive. + +Further optional parameters: + +- `model`: The model name reported in the response metadata. Purely cosmetic — no model is loaded. Defaults to `"mock-model"`. +- `meta`: Additional metadata merged into the `meta` of every returned `ChatMessage`. A per-response `ChatMessage`'s own metadata takes precedence. +- `streaming_callback`: An optional callback invoked with `StreamingChunk` objects reconstructed from the predefined response. It lets the mock exercise streaming code paths without a real model. + +## Usage + +### On its own + +```python +from haystack.components.generators.chat import MockChatGenerator +from haystack.dataclasses import ChatMessage + +# Fixed response +generator = MockChatGenerator(responses="Hello, this is a mock response.") +result = generator.run([ChatMessage.from_user("Hi!")]) +print(result["replies"][0].text) # "Hello, this is a mock response." + +# Echo mode (default): returns the last message with text content +generator = MockChatGenerator() +result = generator.run([ChatMessage.from_user("Repeat after me")]) +print(result["replies"][0].text) # "Repeat after me" +``` + +### Driving an Agent + +Pass `ChatMessage` objects (rather than plain strings) to return tool calls or reasoning content. With cycling responses, you can script a full agent loop without a real model: + +```python +from haystack.components.agents import Agent +from haystack.components.generators.chat import MockChatGenerator +from haystack.dataclasses import ChatMessage, ToolCall +from haystack.tools import tool + + +@tool +def search(query: str) -> str: + """Search for information.""" + return f"Results for: {query}" + + +generator = MockChatGenerator( + responses=[ + ChatMessage.from_assistant( + tool_calls=[ToolCall(tool_name="search", arguments={"query": "Haystack"})], + ), + "Here is the final answer.", + ], +) + +agent = Agent(chat_generator=generator, tools=[search]) +result = agent.run(messages=[ChatMessage.from_user("Tell me about Haystack")]) +print(result["last_message"].text) # "Here is the final answer." +``` + +### Input-dependent responses + +```python +from haystack.components.generators.chat import MockChatGenerator +from haystack.dataclasses import ChatMessage + + +def shout_back(messages: list[ChatMessage]) -> str: + return messages[-1].text.upper() + + +generator = MockChatGenerator(response_fn=shout_back) +result = generator.run([ChatMessage.from_user("hello")]) +print(result["replies"][0].text) # "HELLO" +``` diff --git a/docs-website/docs/tools/searchabletoolset.mdx b/docs-website/docs/tools/searchabletoolset.mdx index d6c971da6dc..15dbfd5bb24 100644 --- a/docs-website/docs/tools/searchabletoolset.mdx +++ b/docs-website/docs/tools/searchabletoolset.mdx @@ -46,6 +46,12 @@ subsequent iterations. `SearchableToolset` does not support adding new tools after initialization or merging with other toolsets. Use `catalog` to provide all tools upfront. ::: +### Warm-up + +`SearchableToolset` builds its search index during `warm_up()`. When used with an [`Agent`](../pipeline-components/agents-1/agent.mdx), constructing the Agent does not trigger this — warm-up happens when you call `Agent.warm_up()` or automatically at run time. + +The Agent evaluates its `exit_conditions` at runtime, so an exit condition can name any tool in the catalog, even one the agent has not discovered yet. + ## Usage ### Basic usage with an Agent diff --git a/docs-website/sidebars.js b/docs-website/sidebars.js index 4d99fa77807..8cd8ee4550f 100644 --- a/docs-website/sidebars.js +++ b/docs-website/sidebars.js @@ -436,6 +436,7 @@ export default { 'pipeline-components/generators/llamastackchatgenerator', 'pipeline-components/generators/metallamachatgenerator', 'pipeline-components/generators/mistralchatgenerator', + 'pipeline-components/generators/mockchatgenerator', 'pipeline-components/generators/nvidiachatgenerator', 'pipeline-components/generators/nvidiagenerator', 'pipeline-components/generators/ollamachatgenerator', From 9eb4f03f2d2f8831868a8a7e04d32d38b7768840 Mon Sep 17 00:00:00 2001 From: Julian Risch Date: Sun, 5 Jul 2026 22:43:56 +0200 Subject: [PATCH 2/4] docs: absorb SearchableToolset duplicate-name note and extend reserved state keys Co-Authored-By: Claude Fable 5 --- docs-website/docs/pipeline-components/agents-1/state.mdx | 7 ++++++- docs-website/docs/tools/searchabletoolset.mdx | 2 ++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/docs-website/docs/pipeline-components/agents-1/state.mdx b/docs-website/docs/pipeline-components/agents-1/state.mdx index f153a5374d7..9c089f8d7a6 100644 --- a/docs-website/docs/pipeline-components/agents-1/state.mdx +++ b/docs-website/docs/pipeline-components/agents-1/state.mdx @@ -68,7 +68,12 @@ The schema defines what data can be stored and how values are updated. Each sche If you don't specify a handler, State automatically assigns a default based on the type. :::info Reserved keys -The `Agent` manages some state keys itself and rejects them in a user-provided `state_schema` with a `ValueError`. This includes the run-metadata keys `step_count`, `token_usage`, and `tool_call_counts`, which the Agent populates automatically during a run: tools can read them mid-run via `inputs_from_state`, and they are returned in the result dictionary. If one of your state keys clashes, rename it (for example, `my_token_usage`). +The `Agent` manages some state keys itself and rejects them in a user-provided `state_schema` with a `ValueError`: + +- The run-metadata keys `step_count`, `token_usage`, and `tool_call_counts`, which the Agent populates automatically during a run: tools can read them mid-run via `inputs_from_state`, and they are returned in the result dictionary. +- The hook-facing keys `continue_run` (set by an `on_exit` hook to keep the Agent running), `tools` (the tools available in the current step, for hooks to inspect), and `hook_context` (the request-scoped resources passed to `Agent.run(hook_context={...})`). + +If one of your state keys clashes, rename it (for example, `my_token_usage`). ::: ### Default Handlers diff --git a/docs-website/docs/tools/searchabletoolset.mdx b/docs-website/docs/tools/searchabletoolset.mdx index 15dbfd5bb24..2c1341be2f5 100644 --- a/docs-website/docs/tools/searchabletoolset.mdx +++ b/docs-website/docs/tools/searchabletoolset.mdx @@ -50,6 +50,8 @@ subsequent iterations. `SearchableToolset` builds its search index during `warm_up()`. When used with an [`Agent`](../pipeline-components/agents-1/agent.mdx), constructing the Agent does not trigger this — warm-up happens when you call `Agent.warm_up()` or automatically at run time. +All tool names in the catalog must be unique: `warm_up()` raises a `ValueError` if the catalog contains tools with duplicate names, since a search hit could otherwise resolve to the wrong tool. + The Agent evaluates its `exit_conditions` at runtime, so an exit condition can name any tool in the catalog, even one the agent has not discovered yet. ## Usage From 881f4e97cde85b98b052dd794fa820a25bb36cb3 Mon Sep 17 00:00:00 2001 From: Julian Risch Date: Tue, 7 Jul 2026 11:01:36 +0200 Subject: [PATCH 3/4] Refine overview of MockChatGenerator Removed redundant wording and improved clarity in the overview section of MockChatGenerator documentation. --- .../pipeline-components/generators/mockchatgenerator.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs-website/docs/pipeline-components/generators/mockchatgenerator.mdx b/docs-website/docs/pipeline-components/generators/mockchatgenerator.mdx index 87a58f03454..123744a90bc 100644 --- a/docs-website/docs/pipeline-components/generators/mockchatgenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/mockchatgenerator.mdx @@ -25,20 +25,20 @@ A Chat Generator that returns predefined responses without calling any API, for ## Overview -`MockChatGenerator` is a deterministic, zero-cost drop-in replacement for real Chat Generators such as `OpenAIChatGenerator`. It implements the full Chat Generator interface — `run`, `run_async`, streaming callbacks, and serialization — but never contacts an external service, which makes it ideal for unit tests, smoke tests, and quick prototypes. +`MockChatGenerator` is a deterministic, zero-cost drop-in replacement for real Chat Generators such as `OpenAIChatGenerator`. It implements `run`, `run_async`, streaming callbacks, and serialization but never contacts an external service, which makes it ideal for unit tests, smoke tests, and quick prototypes. The response is selected based on how the component is configured: - **Fixed response**: Pass a single string or `ChatMessage` via `responses`. The same reply is returned on every call. A `ChatMessage` passed as a response must have the `assistant` role. - **Cycling responses**: Pass a list of strings and/or `ChatMessage` objects via `responses`. Each call returns the next item, wrapping around to the start once the list is exhausted. This is useful to drive multi-step flows such as Agents, where the first call returns a tool call and a later call returns the final answer. -- **Dynamic response**: Pass a `response_fn` callable that receives the input messages and returns the reply as a string or an assistant `ChatMessage`. Use this when the reply should depend on the input. To support serialization, pass a named function — lambdas and nested functions cannot be serialized. +- **Dynamic response**: Pass a `response_fn` callable that receives the input messages and returns the reply as a string or an assistant `ChatMessage`. Use this when the reply should depend on the input. To support serialization, pass a named function. - **Echo (default)**: With no configuration, the component echoes back the text of the last message that has text content, so it is usable out of the box. `responses` and `response_fn` are mutually exclusive. Further optional parameters: -- `model`: The model name reported in the response metadata. Purely cosmetic — no model is loaded. Defaults to `"mock-model"`. +- `model`: The model name reported in the response metadata. Defaults to `"mock-model"`. - `meta`: Additional metadata merged into the `meta` of every returned `ChatMessage`. A per-response `ChatMessage`'s own metadata takes precedence. - `streaming_callback`: An optional callback invoked with `StreamingChunk` objects reconstructed from the predefined response. It lets the mock exercise streaming code paths without a real model. From ba754791a628c69027d4ab2ef36b720cf53b9e25 Mon Sep 17 00:00:00 2001 From: Julian Risch Date: Tue, 7 Jul 2026 12:36:09 +0200 Subject: [PATCH 4/4] Remove "e.g." from agent.mdx example code --- docs-website/docs/pipeline-components/agents-1/agent.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs-website/docs/pipeline-components/agents-1/agent.mdx b/docs-website/docs/pipeline-components/agents-1/agent.mdx index 5e57458389b..3b1fac63b59 100644 --- a/docs-website/docs/pipeline-components/agents-1/agent.mdx +++ b/docs-website/docs/pipeline-components/agents-1/agent.mdx @@ -50,11 +50,11 @@ The `step_count`, `token_usage`, and `tool_call_counts` outputs are populated au ```python response = agent.run(messages=[ChatMessage.from_user("What is 7 * (4 + 2)?")]) -print(response["step_count"]) # e.g. 2 +print(response["step_count"]) # 2 print( response["token_usage"], -) # e.g. {"prompt_tokens": 512, "completion_tokens": 86, ...} -print(response["tool_call_counts"]) # e.g. {"calculator": 1} +) # {"prompt_tokens": 512, "completion_tokens": 86, ...} +print(response["tool_call_counts"]) # {"calculator": 1} ``` ## Parameters