Skip to content

Commit d037854

Browse files
julian-rischclaudedavidsbatista
authored
docs: docs: document Agent run metadata, runtime exit_conditions, SearchableToolset notes, and MockChatGenerator (#11873)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: David S. Batista <dsbatista@gmail.com>
1 parent 9c25765 commit d037854

6 files changed

Lines changed: 147 additions & 2 deletions

File tree

docs-website/docs/pipeline-components/agents-1/agent.mdx

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,16 +38,33 @@ The `Agent` returns a dictionary containing:
3838

3939
- `messages`: the full conversation history,
4040
- `last_message`: the final `ChatMessage` from the agent,
41+
- `step_count`: the number of steps the agent ran,
42+
- `token_usage`: aggregated token usage summed across every LLM call in the run,
43+
- `tool_call_counts`: how many times each tool was invoked, keyed by tool name,
4144
- Additional dynamic keys based on `state_schema`.
4245

46+
### Run Metadata
47+
48+
The `step_count`, `token_usage`, and `tool_call_counts` outputs are populated automatically during a run. They are added to the agent's `state_schema` behind the scenes, so tools registered with `inputs_from_state` can read them mid-run. They are outputs only — they cannot be passed as inputs to `run()` or `run_async()`, and using them as keys in your own `state_schema` raises a `ValueError`. See [State](./state.mdx#schema-definition) for details.
49+
50+
```python
51+
response = agent.run(messages=[ChatMessage.from_user("What is 7 * (4 + 2)?")])
52+
53+
print(response["step_count"]) # 2
54+
print(
55+
response["token_usage"],
56+
) # {"prompt_tokens": 512, "completion_tokens": 86, ...}
57+
print(response["tool_call_counts"]) # {"calculator": 1}
58+
```
59+
4360
## Parameters
4461

4562
`chat_generator` is the only mandatory parameter — an instance of a Chat Generator that supports tools. All other parameters are optional.
4663

47-
- `tools`: A list of tool or toolset instances the agent can call. Supported types: [`Tool`](../../tools/tool.mdx), [`ComponentTool`](../../tools/componenttool.mdx), [`PipelineTool`](../../tools/pipelinetool.mdx), [`MCPTool`](../../tools/mcptool.mdx), [`Toolset`](../../tools/toolset.mdx), [`MCPToolset`](../../tools/mcptoolset.mdx), [`SearchableToolset`](../../tools/searchabletoolset.mdx).
64+
- `tools`: A list of tool or toolset instances the agent can call. Supported types: [`Tool`](../../tools/tool.mdx), [`ComponentTool`](../../tools/componenttool.mdx), [`PipelineTool`](../../tools/pipelinetool.mdx), [`MCPTool`](../../tools/mcptool.mdx), [`Toolset`](../../tools/toolset.mdx), [`MCPToolset`](../../tools/mcptoolset.mdx), [`SearchableToolset`](../../tools/searchabletoolset.mdx). Tool names must be unique; duplicate names are detected at the start of each agent step, before the chat generator is called.
4865
- `system_prompt`: A plain string or Jinja2 template used as the system message for every run. If the template contains Jinja2 variables, those variables become additional inputs to `run()`.
4966
- `user_prompt`: A Jinja2 template appended to the user-provided messages on each run. Template variables become additional inputs to `run()`. Use `required_variables` to enforce which variables must be provided.
50-
- `exit_conditions`: List of conditions that cause the agent to stop. Use `”text”` to stop when the LLM replies without a tool call, or a tool name to stop once that tool has been executed. Defaults to `[“text”]`.
67+
- `exit_conditions`: List of conditions that cause the agent to stop. Use `”text”` to stop when the LLM replies without a tool call, or a tool name to stop once that tool has been executed. Defaults to `[“text”]`. Exit conditions are evaluated at runtime rather than validated at initialization, so a condition can name a tool that is only loaded later — for example, a tool passed at runtime via `run(tools=...)` or one discovered by a [`SearchableToolset`](../../tools/searchabletoolset.mdx).
5168
- `state_schema`: Defines the agent's runtime state — a dict mapping key names to type configs (e.g. `{“docs”: {“type”: list[Document]}}`). Tools can read from and write to state keys via `inputs_from_state` and `outputs_to_state`. See [State](./state.mdx) for full details.
5269
- `streaming_callback`: A callback invoked for each streamed token. Use the built-in `print_streaming_chunk` for console output.
5370
- `max_agent_steps`: Maximum number of LLM + tool call iterations before the agent stops. Defaults to `100`.

docs-website/docs/pipeline-components/agents-1/state.mdx

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,15 @@ The schema defines what data can be stored and how values are updated. Each sche
6767

6868
If you don't specify a handler, State automatically assigns a default based on the type.
6969

70+
:::info Reserved keys
71+
The `Agent` manages some state keys itself and rejects them in a user-provided `state_schema` with a `ValueError`:
72+
73+
- The run-metadata keys `step_count`, `token_usage`, and `tool_call_counts`, which the Agent populates automatically during a run: tools can read them mid-run via `inputs_from_state`, and they are returned in the result dictionary.
74+
- The hook-facing keys `continue_run` (set by an `on_exit` hook to keep the Agent running), `tools` (the tools available in the current step, for hooks to inspect), and `hook_context` (the request-scoped resources passed to `Agent.run(hook_context={...})`).
75+
76+
If one of your state keys clashes, rename it (for example, `my_token_usage`).
77+
:::
78+
7079
### Default Handlers
7180

7281
State provides two built-in merge behaviors (importable from `haystack.components.agents.state`):

docs-website/docs/pipeline-components/generators.mdx

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ Generators are responsible for generating text after you give them a prompt. The
3535
| [LlamaStackChatGenerator](generators/llamastackchatgenerator.mdx) | Enables chat completions using an LLM model made available via Llama Stack server ||
3636
| [MetaLlamaChatGenerator](generators/metallamachatgenerator.mdx) | Enables chat completion with any model hosted available with Meta Llama API. ||
3737
| [MistralChatGenerator](generators/mistralchatgenerator.mdx) | Enables chat completion using Mistral's text generation models. ||
38+
| [MockChatGenerator](generators/mockchatgenerator.mdx) | Returns predefined responses without calling any API — a deterministic, zero-cost stand-in for real Chat Generators in tests and prototypes. ||
3839
| [NvidiaChatGenerator](generators/nvidiachatgenerator.mdx) | Enables chat completion using Nvidia-hosted models. ||
3940
| [NvidiaGenerator](generators/nvidiagenerator.mdx) | Provides an interface for generating text using LLMs self-hosted with NVIDIA NIM or models hosted on the NVIDIA API catalog. ||
4041
| [OllamaChatGenerator](generators/ollamachatgenerator.mdx) | Enables chat completion using an LLM running on Ollama. ||
Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
---
2+
title: "MockChatGenerator"
3+
id: mockchatgenerator
4+
slug: "/mockchatgenerator"
5+
description: "A Chat Generator that returns predefined responses without calling any API, for tests and quick prototypes."
6+
---
7+
8+
# MockChatGenerator
9+
10+
A Chat Generator that returns predefined responses without calling any API, for tests and quick prototypes.
11+
12+
<div className="key-value-table">
13+
14+
| | |
15+
| --- | --- |
16+
| **Most common position in a pipeline** | In place of a real Chat Generator, in tests and prototypes |
17+
| **Mandatory init variables** | None |
18+
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects |
19+
| **Output variables** | `replies`: A list of generated `ChatMessage` objects |
20+
| **API reference** | [Generators](/reference/generators-api) |
21+
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/generators/chat/mock.py |
22+
| **Package name** | `haystack-ai` |
23+
24+
</div>
25+
26+
## Overview
27+
28+
`MockChatGenerator` is a deterministic, zero-cost drop-in replacement for real Chat Generators such as `OpenAIChatGenerator`. It implements `run`, `run_async`, streaming callbacks, and serialization but never contacts an external service, which makes it ideal for unit tests, smoke tests, and quick prototypes.
29+
30+
The response is selected based on how the component is configured:
31+
32+
- **Fixed response**: Pass a single string or `ChatMessage` via `responses`. The same reply is returned on every call. A `ChatMessage` passed as a response must have the `assistant` role.
33+
- **Cycling responses**: Pass a list of strings and/or `ChatMessage` objects via `responses`. Each call returns the next item, wrapping around to the start once the list is exhausted. This is useful to drive multi-step flows such as Agents, where the first call returns a tool call and a later call returns the final answer.
34+
- **Dynamic response**: Pass a `response_fn` callable that receives the input messages and returns the reply as a string or an assistant `ChatMessage`. Use this when the reply should depend on the input. To support serialization, pass a named function.
35+
- **Echo (default)**: With no configuration, the component echoes back the text of the last message that has text content, so it is usable out of the box.
36+
37+
`responses` and `response_fn` are mutually exclusive.
38+
39+
Further optional parameters:
40+
41+
- `model`: The model name reported in the response metadata. Defaults to `"mock-model"`.
42+
- `meta`: Additional metadata merged into the `meta` of every returned `ChatMessage`. A per-response `ChatMessage`'s own metadata takes precedence.
43+
- `streaming_callback`: An optional callback invoked with `StreamingChunk` objects reconstructed from the predefined response. It lets the mock exercise streaming code paths without a real model.
44+
45+
## Usage
46+
47+
### On its own
48+
49+
```python
50+
from haystack.components.generators.chat import MockChatGenerator
51+
from haystack.dataclasses import ChatMessage
52+
53+
# Fixed response
54+
generator = MockChatGenerator(responses="Hello, this is a mock response.")
55+
result = generator.run([ChatMessage.from_user("Hi!")])
56+
print(result["replies"][0].text) # "Hello, this is a mock response."
57+
58+
# Echo mode (default): returns the last message with text content
59+
generator = MockChatGenerator()
60+
result = generator.run([ChatMessage.from_user("Repeat after me")])
61+
print(result["replies"][0].text) # "Repeat after me"
62+
```
63+
64+
### Driving an Agent
65+
66+
Pass `ChatMessage` objects (rather than plain strings) to return tool calls or reasoning content. With cycling responses, you can script a full agent loop without a real model:
67+
68+
```python
69+
from haystack.components.agents import Agent
70+
from haystack.components.generators.chat import MockChatGenerator
71+
from haystack.dataclasses import ChatMessage, ToolCall
72+
from haystack.tools import tool
73+
74+
75+
@tool
76+
def search(query: str) -> str:
77+
"""Search for information."""
78+
return f"Results for: {query}"
79+
80+
81+
generator = MockChatGenerator(
82+
responses=[
83+
ChatMessage.from_assistant(
84+
tool_calls=[ToolCall(tool_name="search", arguments={"query": "Haystack"})],
85+
),
86+
"Here is the final answer.",
87+
],
88+
)
89+
90+
agent = Agent(chat_generator=generator, tools=[search])
91+
result = agent.run(messages=[ChatMessage.from_user("Tell me about Haystack")])
92+
print(result["last_message"].text) # "Here is the final answer."
93+
```
94+
95+
### Input-dependent responses
96+
97+
```python
98+
from haystack.components.generators.chat import MockChatGenerator
99+
from haystack.dataclasses import ChatMessage
100+
101+
102+
def shout_back(messages: list[ChatMessage]) -> str:
103+
return messages[-1].text.upper()
104+
105+
106+
generator = MockChatGenerator(response_fn=shout_back)
107+
result = generator.run([ChatMessage.from_user("hello")])
108+
print(result["replies"][0].text) # "HELLO"
109+
```

docs-website/docs/tools/searchabletoolset.mdx

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,14 @@ subsequent iterations.
4646
`SearchableToolset` does not support adding new tools after initialization or merging with other toolsets. Use `catalog` to provide all tools upfront.
4747
:::
4848

49+
### Warm-up
50+
51+
`SearchableToolset` builds its search index during `warm_up()`. When used with an [`Agent`](../pipeline-components/agents-1/agent.mdx), constructing the Agent does not trigger this — warm-up happens when you call `Agent.warm_up()` or automatically at run time.
52+
53+
All tool names in the catalog must be unique: `warm_up()` raises a `ValueError` if the catalog contains tools with duplicate names, since a search hit could otherwise resolve to the wrong tool.
54+
55+
The Agent evaluates its `exit_conditions` at runtime, so an exit condition can name any tool in the catalog, even one the agent has not discovered yet.
56+
4957
## Usage
5058

5159
### Basic usage with an Agent

docs-website/sidebars.js

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -433,6 +433,7 @@ export default {
433433
'pipeline-components/generators/llamastackchatgenerator',
434434
'pipeline-components/generators/metallamachatgenerator',
435435
'pipeline-components/generators/mistralchatgenerator',
436+
'pipeline-components/generators/mockchatgenerator',
436437
'pipeline-components/generators/nvidiachatgenerator',
437438
'pipeline-components/generators/nvidiagenerator',
438439
'pipeline-components/generators/ollamachatgenerator',

0 commit comments

Comments
 (0)