[FEATURE] Support Anthropic Tool Search (deferred tool loading)

[skovy]

Scope check

• This is core LLM communication (not application logic)
• This benefits most users (not just my use case)
• This can't be solved in application code with current RubyLLM
• I read the Contributing Guide

Due diligence

• I searched existing issues
• I checked the documentation

What problem does this solve?

Anthropic's tool search tool lets Claude dynamically discover and load tools on-demand from a large catalog, instead of including all tool definitions in every request. This solves two scaling problems:

1. Context bloat — tool definitions consume tokens fast. A typical multi-server/MCP setup can eat ~55k tokens in definitions before Claude does any work. Tool search reduces this by 85%+, loading only the 3–5 tools Claude actually needs per request.
2. Tool selection accuracy — Claude's ability to pick the right tool degrades past 30–50 tools. On-demand loading keeps accuracy high even across thousands.

This matters for anyone building MCP-powered agents or systems with many tools. Without this, every tool definition is sent in full on every request, which is expensive and hurts quality at scale.

Currently RubyLLM sends all tools eagerly in Providers::Anthropic::Chat#build_payload and Providers::Anthropic::Tools#function_for. There's no way to mark tools as deferred or handle the new response types.

Proposed solution

1. Deferred tool definitions

Allow marking tools with defer_loading: true so they're included in the API payload but not loaded into Claude's context:

chat.with_tool(MyTool, defer_loading: true)

2. Tool search tool registration

Support adding the tool search tool as a special tool type (two variants):

# Regex variant — Claude constructs Python re.search() patterns
chat.with_tool_search(:regex)

# BM25 variant — Claude uses natural language queries
chat.with_tool_search(:bm25)

These map to tool_search_tool_regex_20251119 and tool_search_tool_bm25_20251119 respectively.

3. Response parsing

Handle the new block types in the Anthropic streaming and non-streaming response parsers:

• server_tool_use — Claude invoking the tool search tool (similar pattern to [FEATURE] Anthropic Server Tools (web_search, web_fetch) - Native Support + Streaming Fix #567's server tools)
• tool_search_tool_result — search results containing tool_reference blocks
• tool_reference — pointers to discovered tools (auto-expanded by the API)

4. Custom tool search via tool_reference in tool results

Support returning tool_reference blocks from user-defined tools for custom search implementations (e.g., embeddings-based):

# In a custom tool's execute method, return tool references
{ tool_references: [{ type: "tool_reference", tool_name: "discovered_tool" }] }

Why this belongs in RubyLLM

This is core Anthropic API communication — it changes how tool definitions are sent in the request payload and introduces new response block types that must be parsed by the provider. Application code can't:

• Add defer_loading: true to tool definitions in the API payload
• Register the special tool_search_tool_* tool types (they have no input schema, just a type + name)
• Parse server_tool_use, tool_search_tool_result, and tool_reference response blocks
• Maintain prompt cache compatibility (deferred tools are appended inline, not in the system prompt prefix)

This follows the same pattern as extended thinking (#551) and server tools (#567) — provider-specific API features that require changes to payload construction and response parsing.