|
| 1 | +--- |
| 2 | +title: "Tool Result Offloading" |
| 3 | +id: tool-result-offloading |
| 4 | +slug: "/tool-result-offloading" |
| 5 | +description: "Tool result offloading writes large tool results to a store and replaces them in the conversation with a compact pointer, keeping the Agent's context window small." |
| 6 | +--- |
| 7 | + |
| 8 | +# Tool Result Offloading |
| 9 | + |
| 10 | +Tool result offloading writes selected tool results to a store and replaces them in the conversation with a compact pointer — a reference plus a short preview — so the next LLM call sees a reference instead of the full result. |
| 11 | +This keeps the context window small when tools return large outputs (web pages, file contents, query results), and it is a step towards letting an Agent operate on offloaded results with follow-up tools, such as a file-reading tool that opens the referenced files. |
| 12 | + |
| 13 | +<div className="key-value-table"> |
| 14 | + |
| 15 | +| | | |
| 16 | +| --- | --- | |
| 17 | +| **Configured on** | The [`Agent`](./agent.mdx) component, as a `ToolResultOffloadHook` registered under the `after_tool` [hook point](./hooks.mdx) | |
| 18 | +| **Key classes** | `ToolResultOffloadHook`, `FileSystemToolResultStore`, `AlwaysOffload`, `NeverOffload`, `OffloadOverChars` | |
| 19 | +| **Import path** | `haystack.hooks.tool_result_offloading` | |
| 20 | +| **API reference** | [Hooks](/reference/hooks-api) | |
| 21 | +| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/hooks/tool_result_offloading/ | |
| 22 | +| **Package name** | `haystack-ai` | |
| 23 | + |
| 24 | +</div> |
| 25 | + |
| 26 | +## Overview |
| 27 | + |
| 28 | +Tool result offloading is one application of the Agent's general [hooks](./hooks.mdx) mechanism: a `ToolResultOffloadHook` registered under the `after_tool` hook point runs after each step's tools execute and rewrites the freshly produced tool-result messages in the Agent's [`State`](./state.mdx). It only considers the current step's results; earlier conversation history is left untouched. |
| 29 | + |
| 30 | +The system is composed of these layers: |
| 31 | + |
| 32 | +- **`ToolResultOffloadHook`** - the `after_tool` hook that applies your offload strategies to fresh tool results. Its `offload_strategies` mapping accepts a single tool name, a tuple of tool names, or the wildcard `"*"` that applies to any tool without a more specific entry. |
| 33 | +- **Policy** - decides *whether* a given result is offloaded. Built-in policies: `AlwaysOffload`, `NeverOffload`, `OffloadOverChars`. |
| 34 | +- **Store** - decides *where* the full result lives. The built-in `FileSystemToolResultStore` writes results to the local file system. |
| 35 | + |
| 36 | +When a result is offloaded, the hook writes the full text to the store and rebuilds the message with a one-line pointer in its place: |
| 37 | + |
| 38 | +``` |
| 39 | +Tool result offloaded to '/abs/path/tool_results/2_search_call-123.txt' (18234 characters). Preview: Fusion startups reported... |
| 40 | +``` |
| 41 | + |
| 42 | +The pointer carries the store reference, the original length, and a preview of the first `preview_chars` characters (200 by default, configurable on the hook), so the model knows roughly what was offloaded and where to find it. |
| 43 | + |
| 44 | +## Usage |
| 45 | + |
| 46 | +### Basic setup |
| 47 | + |
| 48 | +The example below offloads any tool result longer than 4,000 characters to files under a local `tool_results` directory: |
| 49 | + |
| 50 | +```python |
| 51 | +from typing import Annotated |
| 52 | + |
| 53 | +from haystack.components.agents import Agent |
| 54 | +from haystack.components.generators.chat import OpenAIChatGenerator |
| 55 | +from haystack.dataclasses import ChatMessage |
| 56 | +from haystack.hooks.tool_result_offloading import ( |
| 57 | + FileSystemToolResultStore, |
| 58 | + OffloadOverChars, |
| 59 | + ToolResultOffloadHook, |
| 60 | +) |
| 61 | +from haystack.tools import tool |
| 62 | + |
| 63 | + |
| 64 | +@tool |
| 65 | +def search(query: Annotated[str, "The search query"]) -> str: |
| 66 | + """Search the web and return the (potentially large) results.""" |
| 67 | + # Placeholder: would call a real search API |
| 68 | + return f"... large result for {query} ..." |
| 69 | + |
| 70 | + |
| 71 | +offload_hook = ToolResultOffloadHook( |
| 72 | + store=FileSystemToolResultStore(root="tool_results"), |
| 73 | + offload_strategies={"*": OffloadOverChars(4000)}, |
| 74 | +) |
| 75 | + |
| 76 | +agent = Agent( |
| 77 | + chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"), |
| 78 | + tools=[search], |
| 79 | + hooks={"after_tool": [offload_hook]}, |
| 80 | +) |
| 81 | + |
| 82 | +result = agent.run(messages=[ChatMessage.from_user("Summarize today's tech news")]) |
| 83 | +``` |
| 84 | + |
| 85 | +### Configuring what gets offloaded per tool |
| 86 | + |
| 87 | +Each key in `offload_strategies` may be a single tool name, a tuple of tool names sharing one policy, or the wildcard `"*"`. More specific keys win over `"*"`, and a tool with no matching key (and no `"*"`) is never offloaded: |
| 88 | + |
| 89 | +```python |
| 90 | +from haystack.hooks.tool_result_offloading import ( |
| 91 | + AlwaysOffload, |
| 92 | + FileSystemToolResultStore, |
| 93 | + NeverOffload, |
| 94 | + OffloadOverChars, |
| 95 | + ToolResultOffloadHook, |
| 96 | +) |
| 97 | + |
| 98 | +offload_hook = ToolResultOffloadHook( |
| 99 | + store=FileSystemToolResultStore(root="tool_results"), |
| 100 | + offload_strategies={ |
| 101 | + "web_search": AlwaysOffload(), # force offload |
| 102 | + "get_time": NeverOffload(), # opt out of the wildcard default |
| 103 | + ("read_file", "list_dir"): OffloadOverChars(4000), # tuple key: shared policy |
| 104 | + "*": OffloadOverChars(8000), # default for any unlisted tool |
| 105 | + }, |
| 106 | +) |
| 107 | +``` |
| 108 | + |
| 109 | +### What is offloaded |
| 110 | + |
| 111 | +The hook only offloads **successful, text** tool results: |
| 112 | + |
| 113 | +- Error results — including rejections produced by a `before_tool` [Human-in-the-Loop](./human-in-the-loop.mdx) hook — are always left in context, so the model sees what went wrong. |
| 114 | +- Non-text results (image or file content) are left in context; supporting only text is a deliberate choice for now. A warning is logged when a non-text result has a matching offload policy. |
| 115 | +- Each result is offloaded at most once, even though the hook runs on every tool step. This also means two offload hooks registered under `after_tool` won't offload each other's pointers. |
| 116 | + |
| 117 | +## Policies |
| 118 | + |
| 119 | +Policies control *whether* a result is offloaded. |
| 120 | + |
| 121 | +| Policy | Behavior | |
| 122 | +| --- | --- | |
| 123 | +| `AlwaysOffload` | Offload every result of the tool it is assigned to | |
| 124 | +| `NeverOffload` | Never offload - keep the full result in context (useful to opt a tool out of a wildcard default) | |
| 125 | +| `OffloadOverChars(threshold)` | Offload only when the result is longer than `threshold` characters | |
| 126 | + |
| 127 | +### Custom policy |
| 128 | + |
| 129 | +Subclass the `OffloadPolicy` protocol from `haystack.hooks.tool_result_offloading` for custom conditions. A policy needs a `should_offload` method, which receives the tool name, the result text, and the Agent's live [`State`](./state.mdx), so it can also decide based on run context: |
| 130 | + |
| 131 | +```python |
| 132 | +from haystack.components.agents.state import State |
| 133 | +from haystack.hooks.tool_result_offloading import OffloadPolicy |
| 134 | + |
| 135 | + |
| 136 | +class OffloadLateSteps(OffloadPolicy): |
| 137 | + """Offload results only once the run is several steps deep and context pressure builds up.""" |
| 138 | + |
| 139 | + def should_offload(self, tool_name: str, result: str, state: State) -> bool: |
| 140 | + return state.data.get("step_count", 0) >= 3 and len(result) > 1000 |
| 141 | +``` |
| 142 | + |
| 143 | +The protocol provides default `to_dict` / `from_dict` implementations, so a policy like this one, whose constructor takes no arguments, is serializable as-is. A policy with constructor arguments should implement both methods itself, following `OffloadOverChars` as an example. |
| 144 | + |
| 145 | +## Stores |
| 146 | + |
| 147 | +### `FileSystemToolResultStore` |
| 148 | + |
| 149 | +`FileSystemToolResultStore(root=...)` writes each offloaded result to a file under its root directory and returns the absolute file path as the reference. The directory is created on first write. Store keys are derived from the step count, tool name, and tool call ID (for example `2_search_call-123.txt`), so results from different tools and steps do not collide. A key that would resolve outside the root directory is rejected. |
| 150 | + |
| 151 | +### Custom store |
| 152 | + |
| 153 | +Subclass the `ToolResultStore` protocol to target other backends, such as object storage or an isolated sandbox file system. A store needs two methods: `write(key=..., content=...)` persists the content and returns an opaque reference string, and `read(reference)` resolves that reference back to the content: |
| 154 | + |
| 155 | +```python |
| 156 | +from haystack.hooks.tool_result_offloading import ToolResultStore |
| 157 | + |
| 158 | + |
| 159 | +class InMemoryToolResultStore(ToolResultStore): |
| 160 | + """Keep offloaded results in a dict - useful for tests.""" |
| 161 | + |
| 162 | + def __init__(self) -> None: |
| 163 | + self._data: dict[str, str] = {} |
| 164 | + |
| 165 | + def write(self, *, key: str, content: str) -> str: |
| 166 | + self._data[key] = content |
| 167 | + return key |
| 168 | + |
| 169 | + def read(self, reference: str) -> str: |
| 170 | + return self._data[reference] |
| 171 | +``` |
| 172 | + |
| 173 | +Like `OffloadPolicy`, the protocol provides default `to_dict` / `from_dict` implementations covering stores whose constructor takes no arguments; implement both methods for stores with constructor arguments. |
| 174 | + |
| 175 | +### Per-run stores via `hook_context` |
| 176 | + |
| 177 | +The constructor `store` is shared by every run - fine for single-user or local use. In a multi-user server, give each run its own isolated store (for example, a per-session directory) by passing it in the Agent's generic `hook_context` run argument under the key `RESULT_STORE_CONTEXT_KEY`. It overrides the constructor store for that run: |
| 178 | + |
| 179 | +```python |
| 180 | +from haystack.hooks.tool_result_offloading import ( |
| 181 | + RESULT_STORE_CONTEXT_KEY, |
| 182 | + FileSystemToolResultStore, |
| 183 | +) |
| 184 | + |
| 185 | +per_request_store = FileSystemToolResultStore(root=f"tool_results/{session_id}") |
| 186 | +result = agent.run( |
| 187 | + messages=[ChatMessage.from_user("...")], |
| 188 | + hook_context={RESULT_STORE_CONTEXT_KEY: per_request_store}, |
| 189 | +) |
| 190 | +``` |
| 191 | + |
| 192 | +Isolating the store per run keeps concurrent users from colliding on store keys or reading each other's offloaded results — especially important when a file-reading tool is scoped to the store. The hook itself keeps no mutable state, so a single instance is safe to share across concurrent runs. |
| 193 | + |
| 194 | +## Letting the Agent read offloaded results back |
| 195 | + |
| 196 | +The pointer left in the conversation tells the model where the full result lives, but the model can only act on it if the Agent has a tool that can read from the store. With `FileSystemToolResultStore`, that can be a simple file-reading tool: |
| 197 | + |
| 198 | +```python |
| 199 | +from typing import Annotated |
| 200 | + |
| 201 | +from haystack.tools import tool |
| 202 | + |
| 203 | + |
| 204 | +@tool |
| 205 | +def read_offloaded_result( |
| 206 | + path: Annotated[str, "Absolute path of an offloaded tool result"], |
| 207 | +) -> str: |
| 208 | + """Read back the full content of an offloaded tool result.""" |
| 209 | + return FileSystemToolResultStore(root="tool_results").read(path) |
| 210 | +``` |
| 211 | + |
| 212 | +With this tool available, the Agent can work with a compact conversation and selectively re-read only the offloaded results it actually needs — instead of carrying every full result in context on every LLM call. |
| 213 | + |
| 214 | +## Serialization |
| 215 | + |
| 216 | +`ToolResultOffloadHook` implements `to_dict` / `from_dict`, so an Agent using it can be serialized as long as the configured store and policies are serializable too. The built-in store and policies all are; for custom ones, see the notes in [Policies](#custom-policy) and [Stores](#custom-store) above. |
| 217 | + |
| 218 | +## Additional References |
| 219 | + |
| 220 | +📖 Related docs: |
| 221 | + |
| 222 | +- [Hooks](./hooks.mdx) — the general mechanism behind this feature, including the `after_tool` hook point |
| 223 | +- [Human in the Loop](./human-in-the-loop.mdx) — another ready-made hook, intercepting tool calls for human review |
| 224 | +- [State](./state.mdx) — the live run state hooks and policies receive |
0 commit comments