[BUG] Anthropic with_schema silently truncates at literal double quotes in generated strings (output_config constrained decoding)

[christiangenco]

Basic checks

• I searched existing issues; the closest is [FEATURE] Support Anthropic's new structured outputs support #501, the feature request that added this mechanism
• I can reproduce this consistently (4/4 runs, script below)
• This is a RubyLLM bug, not my application code. The faulty decoding is on Anthropic's side, but RubyLLM routes every Anthropic schema chat into it unconditionally with no opt-out, so every release since 1.13.0 is affected.

What's broken?

Since 1.13.0 (#501), with_schema on Anthropic sends the schema as native structured outputs: output_config.format.json_schema (lib/ruby_llm/providers/anthropic/chat.rb, build_output_config). Under that constrained decoding, when the model generates a string value whose content calls for a literal double quote (a nickname like Robert J. McAllister Jr. ("Bobby Mac"), or output that quotes its sources), the model emits the " unescaped instead of as \". The grammar reads it as the end of the JSON string, the object closes legally, and the API returns HTTP 200, stop_reason: "end_turn", and a valid but truncated object:

{"digest":"## Who they are\nRobert J. McAllister Jr. ("}

Nothing raises. response.content parses fine. And because the truncation point is determined by the content, retries return the same truncated result every time.

We hit this in production: a pipeline over 603 records deterministically truncated exactly the 2 whose source text contained a straight-quoted nickname, returning about 30 output tokens instead of 1,300.

The same messages and schema sent as a forced tool call (tool_choice: {type: "tool", ...}) return the full output with the quotes intact. Tool-input JSON is a format the model escapes correctly.

How to reproduce

# ANTHROPIC_API_KEY=... ruby repro.rb
require "ruby_llm"

RubyLLM.configure { |c| c.anthropic_api_key = ENV.fetch("ANTHROPIC_API_KEY") }

SCHEMA = {
  type: "object",
  properties: {
    digest: { type: "string", description: "Dense narrative digest in the section structure described in the instructions." }
  },
  required: ["digest"],
  additionalProperties: false
}.freeze

INSTRUCTIONS = <<~TXT.freeze
  You're reading the full corpus of one expert and producing a dense narrative
  digest. Ground every claim in the corpus. Quote source material when you can.
  Don't invent.

  The digest should be 300-500 words in this section structure:

  ## Who they are
  One paragraph: name (with nickname exactly as the corpus styles it), current
  role, total years of relevant experience, retired or active.

  ## Direct experience
  Bullet list. Each bullet a single specific position with dates and named orgs.

  ## Notable claims with sources
  Bullet list of striking quotes, each tagged with source ("Resume", "Call 2024-09").
TXT

CORPUS = <<~TXT.freeze
  === RESUME ===
  Robert J. McAllister Jr. ("Bobby Mac")
  Founder & Principal, McAllister Packaging Advisors (2023-Present)

  Thirty years in corrugated packaging operations. Plant Manager, Great Lakes
  Container (1995-2004): led two plant turnarounds, took OEE from 61% to 84%.
  VP of Operations, Midwest Box (2004-2012): ran nine plants, completed an ERP
  migration on time and under budget. CEO, Coastal Container Corp (2012-2023):
  grew revenue from $180M to $310M; sold to Hargrove Capital in 2023.

  === CALL SUMMARY 2024-09 ===
  Expert introduced himself as "Bobby Mac" and said "everyone from the plant
  floor to the board calls me that." Stated: "I have personally walked every
  one of the 40+ box plants in the Midwest worth buying." Currently advises
  private equity firms on packaging acquisitions; active, not retired.

  === BIO ===
  Robert J. McAllister Jr. ("Bobby Mac") is the Founder and Principal of
  McAllister Packaging Advisors, an operations advisory firm serving private
  equity investors in the corrugated packaging sector.
TXT

chat = RubyLLM.chat(model: "claude-sonnet-4-6")
              .with_instructions(INSTRUCTIONS)
              .with_schema(SCHEMA)
response = chat.ask("Corpus follows:\n\n#{CORPUS}")

digest = response.content["digest"].to_s
puts "output_tokens: #{response.output_tokens}"
puts "digest.length: #{digest.length}"
puts digest.inspect

The ingredients that make it fire reliably: markdown-shaped output inside the JSON string, instructions to quote source material, and source text containing straight double quotes for the model to copy. A bare "write a bio with a quoted nickname" prompt gets escaped correctly, but the digest shape above truncates every time, and it's a common shape for real pipelines.

Expected behavior

A digest of roughly 1,300 chars with the nickname intact, which is what the same request returns through a forced tool call.

What actually happened

Identical across 4 runs (2026-06-11):

output_tokens: 26
digest.length: 42
"## Who they are\nRobert J. McAllister Jr. ("

Generation ends at the unescaped quote. No error, no stop_reason hint, no parse failure. The content is just silently missing.

Raw API side-by-side (same system, messages, schema; no RubyLLM involved)

output_config.format.json_schema (what RubyLLM sends):

request-id:  req_011CbwUVumNfXBjo93b3bEBi
stop_reason: end_turn
output_tokens: 26
text content: {"digest":"## Who they are\nRobert J. McAllister Jr. ("}

Forced tool call with the identical schema as input_schema:

request-id:  req_011CbwUW7Nt3gzSEpqiAjmV8
stop_reason: tool_use
output_tokens: 406
input.digest: 1,397 chars, complete, opens:
  ## Who they are
  Robert J. McAllister Jr., known universally as "Bobby Mac" — "everyone from the plant floor to the boa…

So the model handles the same content correctly when the schema rides on a tool; the truncation is specific to the output_config decoding path.

Environment

• RubyLLM 1.14.1 (also code-verified the same build_output_config path in 1.16.0); introduced in 1.13.0
• Ruby 3.4.8, macOS
• Provider: anthropic (direct API), model claude-sonnet-4-6, non-streaming

Suggested direction

We've reported the underlying unescaped-quote behavior to Anthropic as well, but since RubyLLM pins every Anthropic schema chat to output_config the gem stays affected until the API changes. Two options, happy to PR either:

1. Send Anthropic schemas as a forced synthetic tool instead of output_config. This is what we run in production now (patch below), and it covers streaming too. One caveat: a forced tool_choice can't coexist with real tools the model should remain free to call, so it would need to apply only when no other tools are registered, or keep output_config for the mixed case.
2. Or minimally, an opt-out, e.g. with_schema(schema, mode: :native | :tool) or a provider config flag, so apps that hit this can route around output_config without monkey-patching.

Workaround we're running in production (Rails initializer, RubyLLM 1.14.1)

# Reroutes Anthropic schema chats to a forced structured_output tool and
# converts the tool_use response back into JSON text, so Chat#complete's
# parsing and persistence are untouched.
module RubyLLMAnthropicStructuredOutputViaTool
  TOOL_NAME = "structured_output".freeze

  def render_payload(messages, tools:, temperature:, model:, stream: false, schema: nil, thinking: nil, tool_prefs: nil)
    payload = super(messages, tools:, temperature:, model:, stream:, schema: nil, thinking:, tool_prefs:)
    return payload unless schema

    input_schema = RubyLLM::Utils.deep_dup(schema[:schema])
    input_schema.delete(:strict)
    input_schema.delete("strict")

    payload[:tools] = Array(payload[:tools]) + [ {
      name: TOOL_NAME,
      description: "Record the structured response. Always call this exactly once.",
      input_schema: input_schema
    } ]
    payload[:tool_choice] = { type: "tool", name: TOOL_NAME }
    payload
  end

  # Non-streaming: rewrite the body before ruby_llm parses it.
  def parse_completion_response(response)
    blocks = response.body["content"]
    if blocks.is_a?(Array)
      structured = blocks.find { |b| b["type"] == "tool_use" && b["name"] == TOOL_NAME }
      response.body["content"] = [ { "type" => "text", "text" => JSON.generate(structured["input"]) } ] if structured
    end
    super
  end

  # Streaming: rebuild the accumulated synthetic tool call as text content.
  def stream_response(connection, payload, additional_headers = {}, &)
    message = super
    structured = message.tool_calls&.values&.find { |tc| tc.name == TOOL_NAME }
    return message unless structured

    RubyLLM::Message.new(
      role: :assistant,
      content: JSON.generate(structured.arguments),
      thinking: message.thinking,
      tokens: message.tokens,
      model_id: message.model_id,
      raw: message.raw
    )
  end
end

RubyLLM::Providers::Anthropic.prepend(RubyLLMAnthropicStructuredOutputViaTool)

[hschne]

FYI we hit this bug as well. Thank you for the workaround, 1. works well for us.