Skip to content

fix: Fix usage of SearchableToolset with Agent when selecting a subset of tools to be active#11564

Open
sjrl wants to merge 8 commits into
v3from
toolset-runtime-tool-selection
Open

fix: Fix usage of SearchableToolset with Agent when selecting a subset of tools to be active#11564
sjrl wants to merge 8 commits into
v3from
toolset-runtime-tool-selection

Conversation

@sjrl

@sjrl sjrl commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Related Issues

Proposed Changes:

Before trying to restrict a SearchableToolset to a subset of its catalog by name at Agent runtime did not work:

from typing import Annotated

from haystack.components.agents import Agent
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.tools import SearchableToolset, tool

@tool
def get_weather(city: Annotated[str, "The city to look up"]) -> str:
    """Get the current weather for a city."""
    return f"It's sunny in {city}."

@tool
def get_stock_price(symbol: Annotated[str, "The ticker symbol, e.g. 'AAPL'"]) -> str:
    """Get the latest stock price for a ticker symbol."""
    return f"{symbol}: $123.45"

@tool
def search_news(query: Annotated[str, "What to search for"]) -> str:
    """Search recent news headlines."""
    return f"Top story for '{query}': ..."

# We set search_threshold=2 so SearchableToolset is put into search mode.
toolset = SearchableToolset(catalog=[get_weather, get_stock_price, search_news], search_threshold=2)
agent = Agent(chat_generator=OpenAIChatGenerator(), tools=toolset)

# We restrict this run to only use get_weather and get_stock_price
result = agent.run(
    messages=[ChatMessage.from_user("What's the weather in Milan and the price of AAPL?")],
    tools=["get_weather", "get_stock_price"],
)

Before:

  • the call raised ValueError: The following tool names are not valid: {'get_weather', 'get_stock_price'}. Valid tool names are: {'search_tools'}
  • Previously SearchableToolset only exposed its search_tools bootstrap tool (not its catalog) when flattened, so catalog tools couldn't be selected by name.
  • And passing tools=["search_tools"] instead would have collapsed the toolset into a single item list of just search_tools which disabled discovery.

After: the SearchableToolset is kept intact, and when a user restricts the tools to get_weather and get_stock_price search and lazy-loading still work over the restricted subset.

How this is fixed: instead of flattening the toolset into a static list of tools, the Agent now keeps the Toolset intact and applies the name selection to it:

  1. Validation against the full catalog. Requested names are validated against Toolset.get_selectable_tools(). SearchableToolset overrides this to return its entire catalog, so catalog tools are now resolvable by name (previously only its search_tools tool was, since iterating the toolset only exposes that bootstrap tool plus already-discovered tools).
  2. The toolset stays live. Rather than replacing the toolset with a static list[Tool], the Agent registers the requested names as a per-run selection on the live Toolset (via Toolset._selected_tool_names) and keeps that object as the run's tools.
  3. Iteration honors the selection. When the Agent re-reads its tools each step, the Toolset yields only the selected tools. For SearchableToolset this means the search_tools tool is still exposed and search/discovery is restricted to the selected catalog tools — so search and lazy-loading keep working over the subset instead of being disabled.
  4. Toolset state is reset after the run. When the run finishes, the Agent calls Toolset.reset(), which clears the per-run selection. SearchableToolset.reset() additionally clears its discovered tools, so the selection does not leak into later runs.

Changes Made:

  • Toolset gained two methods: get_selectable_tools() returns every tool available for name-based
    selection (ignoring any active selection restriction, e.g. the one used by SearchableToolset), and
    reset() clears a Toolset's per-run state. Subclasses can override reset() to reset additional state.
  • SearchableToolset now exposes its full catalog of tools via SearchableToolset.get_selectable_tools().
  • Runtime tool-name selection via Agent.run(tools=["tool_a", "tool_b"]) now resolves correctly when a
    SearchableToolset is configured. Previously the SearchableToolset was flattened into a single-item
    list (just its search tool), which broke its search and lazy-loading behavior.
  • SearchableToolset now clears its discovered tools at the end of each Agent run (via
    SearchableToolset.reset()). Discovered tools no longer persist across runs, so each run starts fresh.

How did you test it?

New tests

Notes for the reviewer

I did check locally that these changes work with our existing MCPToolset integration without any need for changes there.

Checklist

  • I have read the contributors guidelines and the code of conduct.
  • I have updated the related issue with new insights and changes.
  • I have added unit tests and updated the docstrings.
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
  • I have documented my code.
  • I have added a release note file, following the contributors guidelines.
  • I have run pre-commit hooks and fixed any issue.

@sjrl sjrl requested a review from a team as a code owner June 9, 2026 13:29
@sjrl sjrl requested review from julian-risch and removed request for a team June 9, 2026 13:29
@vercel

vercel Bot commented Jun 9, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
haystack-docs Ignored Ignored Preview Jun 10, 2026 12:54pm

Request Review

@github-actions github-actions Bot added topic:tests type:documentation Improvements on the docs labels Jun 9, 2026
@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  haystack/components/agents
  agent.py
  haystack/tools
  searchable_toolset.py 188, 211
  toolset.py 193
Project Total  

This report was generated by python-coverage-comment-action

@sjrl sjrl self-assigned this Jun 9, 2026
@claude

claude Bot commented Jun 9, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

Comment thread haystack/tools/toolset.py Outdated
sjrl and others added 2 commits June 10, 2026 13:38
Co-authored-by: Julian Risch <julian.risch@deepset.ai>
@sjrl

sjrl commented Jun 10, 2026

Copy link
Copy Markdown
Contributor Author

@julian-risch based on our convo offline I've gone ahead and updated this PR to create a shallow copy of Toolset at Agent runtime using a new Toolset.spawn() method. This should address the concurrency worry.

@sjrl sjrl requested a review from julian-risch June 10, 2026 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

topic:tests type:documentation Improvements on the docs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants