Add tool search abstraction (defer_loading + search_tools)

Translates Anthropic's native deferred tool loading into a
Ruby-idiomatic API that works across providers.

Developers opt in via a `deferred` class DSL on RubyLLM::Tool, or by
passing `defer: true` to `Chat#with_tool` / `#with_tools`. When any
tool is deferred, a built-in `search_tools` function is exposed to
the model so it can load schemas on demand. On Anthropic, this
forwards `defer_loading: true` and appends the native
`tool_search_tool_bm25_20251119` primitive; on OpenAI/Gemini/Bedrock
(and OpenAI/Gemini-compatible providers), deferred tools are emitted
as name+description stubs and a pure-Ruby BM25 ranker drives the
client-side search.

Zero new runtime dependencies. Fully non-breaking: absent `defer:` or
`deferred`, behaviour is unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: swistaczek

docs/_advanced/upgrading.md

@@ -21,6 +21,16 @@ redirect_from:
 {:toc}
 
 ---
+# Upgrade to 1.15
+
+## How to Upgrade
+
+1.15 adds tool search in a fully additive way. No generator, no migration — upgrade the gem and continue using RubyLLM as before.
+
+## What's New in 1.15
+
+- **Tool Search** — `RubyLLM::Chat#with_tool` and `#with_tools` now accept a `defer:` keyword argument, and `RubyLLM::Tool` exposes a class-level `deferred` DSL. Marking a tool as deferred keeps its JSON schema out of the system-prompt prefix; the model loads it on demand via a built-in `search_tools` function. On Anthropic, this translates to the native `defer_loading: true` + `tool_search_tool_bm25_20251119`. On other providers (OpenAI, Gemini, Bedrock, and their OpenAI/Gemini-compatible descendants), the behavior is emulated client-side using a pure-Ruby BM25 ranker. If you don't use `defer:` or `deferred`, nothing changes. See [Tool Search]({% link _core_features/tool-search.md %}).
+
 # Upgrade to 1.14
 
 ## How to Upgrade

docs/_core_features/tool-search.md

@@ -0,0 +1,181 @@
+---
+layout: default
+title: Tool Search
+nav_order: 9
+description: Scale to hundreds of tools without blowing up your token budget. Defer tool schemas and let the model search for what it needs.
+redirect_from:
+  - /guides/tool-search
+---
+
+# {{ page.title }}
+{: .d-inline-block .no_toc }
+
+New in 1.15
+{: .label .label-green }
+
+{{ page.description }}
+{: .fs-6 .fw-300 }
+
+## Table of contents
+{: .no_toc .text-delta }
+
+1. TOC
+{:toc}
+
+---
+
+After reading this guide, you will know:
+
+*   When tool search helps (and when it doesn't).
+*   How to mark tools as deferred.
+*   How the model discovers and loads deferred tools at runtime.
+*   How providers differ: Anthropic's native path vs. client-side emulation.
+*   How to plug in a custom search function (e.g. embeddings).
+
+## When to use it
+
+When a `RubyLLM::Chat` is wired to many tools — especially across one or more MCP servers — every tool's full JSON Schema ships in the system-prompt prefix on every turn. Three real costs follow:
+
+1. **Token bloat.** Hundreds of tools can add tens of thousands of tokens per request.
+2. **Prompt-cache eviction.** Adding or removing tools changes the prefix and invalidates the cache.
+3. **Selection accuracy.** Models choose worse tools when the menu is long.
+
+Tool search solves this by withholding the schemas of "deferred" tools from the prompt prefix and exposing a built-in `search_tools` function the model can call to load the schemas it actually needs.
+
+Reach for it when:
+
+*   You use MCP servers with more than ~20 tools.
+*   You have a handful of rarely-needed heavy tools (deep-research, large schemas) bloating your prefix.
+*   You run on Anthropic and want to preserve prompt cache across long conversations.
+
+Skip it when:
+
+*   You have fewer than a dozen tools and no prompt-cache concerns.
+*   All your tools are always relevant to every turn.
+
+## Marking tools deferred
+
+### Per-call (best for MCP bulk registration)
+
+```ruby
+chat = RubyLLM.chat(model: "claude-sonnet-4-6")
+chat.with_tools(*mcp_client.tools, defer: true)
+```
+
+### Per-class (best for tools that are intrinsically heavy)
+
+```ruby
+class DeepResearchTool < RubyLLM::Tool
+  description "Runs a multi-step web research task."
+  deferred
+  param :query, desc: "Research question"
+  def execute(query:)
+    # ...
+  end
+end
+
+chat.with_tool(DeepResearchTool)
+```
+
+The per-call value wins when both are specified, so you can override:
+
+```ruby
+chat.with_tool(DeepResearchTool, defer: false) # force it onto the active list
+```
+
+## How the model discovers deferred tools
+
+When the catalog contains at least one deferred tool, `RubyLLM::Chat` automatically adds a built-in `search_tools` function to the active tool list. The model sees a compact description like:
+
+> **search_tools** — Search and load deferred tools by keyword, or load specific tools by name using `select:Name1,Name2`.
+
+The typical flow:
+
+```
+user      : "Delete the stale S3 bucket"
+assistant : tool_call(search_tools, query: "delete s3 bucket")
+tool      : { loaded: [:s3_delete_bucket], descriptions: {...} }
+assistant : tool_call(s3_delete_bucket, bucket: "stale-bucket-2023")
+tool      : "deleted"
+assistant : "Done — stale-bucket-2023 has been deleted."
+```
+
+Once a tool is promoted by `search_tools`, it stays active for the rest of the conversation.
+
+## The `select:` shortcut
+
+For deterministic loading — "when event X happens, load tools Y and Z" — prefix the query with `select:` and list the names exactly. No ranking, no LLM reasoning required:
+
+```ruby
+chat.tools[:search_tools].execute(query: "select:s3_delete_bucket,s3_list_buckets")
+```
+
+You can also drive `select:` entirely from application code when you know which tools a task needs, skipping the discovery turn.
+
+## Anthropic: native server-side search
+
+On Anthropic models, ruby_llm forwards `defer_loading: true` to the API on every deferred tool and appends the native `tool_search_tool_bm25_20251119` server-side search primitive. The practical win: deferred tool schemas never enter the cached system-prompt prefix, so prompt caching stays intact across turns.
+
+## Other providers: client-side emulation
+
+On OpenAI, Gemini, Bedrock, and providers that inherit from them (Azure, DeepSeek, Mistral, OpenRouter, Perplexity, xAI, Ollama, GPUStack, Vertex AI), ruby_llm emits deferred tools as name-plus-description stubs with an empty parameters schema. The model cannot invoke a stub directly — it must call `search_tools` first to load the full schema, at which point the tool becomes callable normally.
+
+The default client-side ranker is a pure-Ruby BM25 implementation over `"#{name} #{description}"`. No external gem, no embedding infrastructure.
+
+## Custom search
+
+If BM25 over name + description is not enough — for example, you want to rank by your own tool metadata or use embeddings — supply your own ranker:
+
+```ruby
+# Per-chat
+chat.with_tool_search do |query, candidates, max:|
+  MyEmbeddingIndex.rank(candidates, query, k: max) # returns Array<Symbol>
+end
+
+# Or, globally, once per process
+RubyLLM.configure do |c|
+  c.tool_search_function = lambda do |query, candidates, max:|
+    MyEmbeddingIndex.rank(candidates, query, k: max)
+  end
+end
+```
+
+The block receives the query string, the hash of candidate tools (keyed by name symbol), and `max:`. It must return an array of tool-name symbols ordered by relevance.
+
+## Kill switch
+
+Tool search is on by default. To disable it globally — in which case `defer: true` and the `deferred` DSL become no-ops and a one-time warning is logged — set:
+
+```ruby
+RubyLLM.configure { |c| c.tool_search_enabled = false }
+```
+
+## Observing activation
+
+```ruby
+chat.on_tool_search do |query:, results:|
+  Rails.logger.info("tool_search: #{query} → #{results.join(', ')}")
+end
+
+chat.tool_catalog.loaded_tools # => #<Set: {:s3_delete_bucket}>
+chat.tool_catalog.deferred_tools.keys # all tools still hidden
+```
+
+## Configuration reference
+
+| Location | Setting | Default | Purpose |
+|---|---|---|---|
+| `RubyLLM.config` | `tool_search_enabled` | `true` | Global on/off kill switch |
+| `RubyLLM.config` | `tool_search_function` | `nil` (uses BM25) | Global default ranker |
+| `RubyLLM::Tool` | `deferred` (class DSL) | not set | Marks every instance deferred by default |
+| `RubyLLM::Chat#with_tool` | `defer:` kwarg | `nil` (inherits class) | Per-registration override |
+| `RubyLLM::Chat#with_tools` | `defer:` kwarg | `nil` | Bulk per-registration override |
+| `RubyLLM::Chat#with_tool_search` | block | — | Per-chat ranker |
+| `RubyLLM::Chat#on_tool_search` | block | — | Activation callback |
+| `RubyLLM::Chat#tool_catalog` | reader | — | Inspect deferred / loaded sets |
+
+## Further reading
+
+*   [Tools guide]({% link _core_features/tools.md %})
+*   [Agents guide]({% link _core_features/agents.md %})
+*   [Anthropic tool search tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool)

docs/_core_features/tools.md

@@ -509,6 +509,8 @@ end
 
 For MCP server integration, check out the community-maintained [`ruby_llm-mcp`](https://github.com/patvice/ruby_llm-mcp) gem.
 
+When a chat is wired to many tools — especially across MCP servers — see [Tool Search]({% link _core_features/tool-search.md %}) for how to defer tool schemas and let the model load only the ones it needs.
+
 ## Debugging Tools
 
 Set the `RUBYLLM_DEBUG` environment variable to see detailed logging, including tool calls and results.

lib/ruby_llm.rb

@@ -19,6 +19,7 @@
   'UI' => 'UI',
   'api' => 'API',
   'bedrock' => 'Bedrock',
+  'bm25' => 'BM25',
   'deepseek' => 'DeepSeek',
   'gpustack' => 'GPUStack',
   'llm' => 'LLM',

lib/ruby_llm/chat.rb

@@ -5,7 +5,7 @@ module RubyLLM
   class Chat
     include Enumerable
 
-    attr_reader :model, :messages, :tools, :tool_prefs, :params, :headers, :schema
+    attr_reader :model, :messages, :tools, :tool_prefs, :params, :headers, :schema, :tool_catalog
 
     def initialize(model: nil, provider: nil, assume_model_exists: false, context: nil)
       if assume_model_exists && !provider
@@ -19,6 +19,7 @@ def initialize(model: nil, provider: nil, assume_model_exists: false, context: n
       @temperature = nil
       @messages = []
       @tools = {}
+      @tool_catalog = ToolCatalog.new
       @tool_prefs = { choice: nil, calls: nil }
       @params = {}
       @headers = {}
@@ -28,7 +29,8 @@ def initialize(model: nil, provider: nil, assume_model_exists: false, context: n
         new_message: nil,
         end_message: nil,
         tool_call: nil,
-        tool_result: nil
+        tool_result: nil,
+        tool_search: nil
       }
     end
 
@@ -51,22 +53,27 @@ def with_instructions(instructions, append: false, replace: nil)
       self
     end
 
-    def with_tool(tool, choice: nil, calls: nil)
-      unless tool.nil?
-        tool_instance = tool.is_a?(Class) ? tool.new : tool
-        @tools[tool_instance.name.to_sym] = tool_instance
-      end
+    def with_tool(tool, defer: nil, choice: nil, calls: nil)
+      register_tool(tool, defer: defer) unless tool.nil?
       update_tool_options(choice:, calls:)
       self
     end
 
-    def with_tools(*tools, replace: false, choice: nil, calls: nil)
-      @tools.clear if replace
-      tools.compact.each { |tool| with_tool tool }
+    def with_tools(*tools, replace: false, defer: nil, choice: nil, calls: nil)
+      if replace
+        @tools.clear
+        @tool_catalog = ToolCatalog.new
+      end
+      tools.compact.each { |tool| with_tool tool, defer: defer }
       update_tool_options(choice:, calls:)
       self
     end
 
+    def with_tool_search(&block)
+      @tool_catalog.search_function = block
+      self
+    end
+
     def with_model(model_id, provider: nil, assume_exists: false)
       @model, @provider = Models.resolve(model_id, provider:, assume_exists:, config: @config)
       @connection = @provider.connection
@@ -132,14 +139,19 @@ def on_tool_result(&block)
       self
     end
 
+    def on_tool_search(&block)
+      @on[:tool_search] = block
+      self
+    end
+
     def each(&)
       messages.each(&)
     end
 
     def complete(&) # rubocop:disable Metrics/PerceivedComplexity
       response = @provider.complete(
         messages,
-        tools: @tools,
+        tools: effective_tools,
         tool_prefs: @tool_prefs,
         temperature: @temperature,
         model: @model,
@@ -176,6 +188,15 @@ def add_message(message_or_attributes)
       message
     end
 
+    def register_loaded_tool(tool)
+      @tools[tool.name.to_sym] = tool
+      self
+    end
+
+    def emit_tool_search(query:, results:)
+      @on[:tool_search]&.call(Tool::SearchEvent.new(query, results))
+    end
+
     def reset_messages!
       @messages.clear
     end
@@ -329,6 +350,49 @@ def content_like?(object)
       object.is_a?(Content) || object.is_a?(Content::Raw)
     end
 
+    def effective_tools
+      active = @tools.transform_values { |t| Tool::Registration.new(t, deferred: false) }
+      return active if @tool_catalog.empty?
+
+      deferred = @tool_catalog.available.transform_values { |t| Tool::Registration.new(t, deferred: true) }
+      deferred.merge(active)
+    end
+
+    def register_tool(tool, defer:)
+      tool_instance = tool.is_a?(Class) ? tool.new : tool
+
+      if defer_requested?(tool_instance, defer)
+        @tool_catalog.add(tool_instance)
+        ensure_search_tool!
+      else
+        @tools[tool_instance.name.to_sym] = tool_instance
+      end
+    end
+
+    def defer_requested?(tool, explicit)
+      requested = explicit.nil? ? tool.deferred? : explicit == true
+      return false unless requested
+      return true if @config.tool_search_enabled
+
+      warn_tool_search_disabled_once
+      false
+    end
+
+    def ensure_search_tool!
+      return if @tools.key?(:search_tools)
+
+      @tools[:search_tools] = Tools::SearchTools.new(self)
+    end
+
+    def warn_tool_search_disabled_once
+      return if @tool_search_warning_emitted
+
+      RubyLLM.logger.warn(
+        'tool_search_enabled is false; ignoring defer: true / deferred DSL for the rest of this chat'
+      )
+      @tool_search_warning_emitted = true
+    end
+
     def append_system_instruction(instructions)
       system_messages, non_system_messages = @messages.partition { |msg| msg.role == :system }
       system_messages << Message.new(role: :system, content: instructions)

lib/ruby_llm/configuration.rb

@@ -56,6 +56,9 @@ def defaults = @defaults ||= {}
     option :log_stream_debug, -> { ENV['RUBYLLM_STREAM_DEBUG'] == 'true' }
     option :log_regexp_timeout, -> { Regexp.respond_to?(:timeout) ? (Regexp.timeout || 1.0) : nil }
 
+    option :tool_search_enabled, true
+    option :tool_search_function, nil
+
     def initialize
       self.class.send(:defaults).each do |key, default|
         value = default.respond_to?(:call) ? instance_exec(&default) : default

lib/ruby_llm/error.rb

@@ -22,6 +22,7 @@ class ConfigurationError < StandardError; end
   class PromptNotFoundError < StandardError; end
   class InvalidRoleError < StandardError; end
   class InvalidToolChoiceError < StandardError; end
+  class InvalidToolConfigurationError < StandardError; end
   class ModelNotFoundError < StandardError; end
   class UnsupportedAttachmentError < StandardError; end
 

lib/ruby_llm/providers/anthropic/chat.rb

@@ -65,7 +65,7 @@ def build_base_payload(chat_messages, model, stream, thinking)
 
         def add_optional_fields(payload, system_content:, tools:, tool_prefs:, temperature:, schema: nil) # rubocop:disable Metrics/ParameterLists
           if tools.any?
-            payload[:tools] = tools.values.map { |t| Tools.function_for(t) }
+            payload[:tools] = Tools.format_tools(tools)
             unless tool_prefs[:choice].nil? && tool_prefs[:calls].nil?
               payload[:tool_choice] = Tools.build_tool_choice(tool_prefs)
             end