Skip to content

[FEATURE] Surface Anthropic compaction blocks #763

Description

@fvaleye

Scope check

  • This is core LLM communication (not application logic)
  • This benefits most users (not just my use case)
  • This can't be solved in application code with current RubyLLM
  • I read the Contributing Guide

Due diligence

  • I searched existing issues
  • I checked the documentation

What problem does this solve?

Anthropic shipped a server-side context-compaction feature (compact_20260112, docs) that summarizes earlier conversation turns when input tokens cross a configurable threshold. The summary is returned in the response as a dedicated compaction content block, separate from the assistant's normal text reply.

When streaming, the response stream contains a new event type:

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"compaction"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"compaction_delta","summary":[...]}}

lib/ruby_llm/providers/anthropic/streaming.rb#build_chunk only recognizes text_delta, thinking_delta, and signature_delta:

def extract_content_delta(data, delta_type)
  return data.dig('delta', 'text') if delta_type == 'text_delta'
  nil
end

So compaction_delta events are silently dropped.

Result: using RubyLLM::Chat#ask against claude-sonnet-4-6 or claude-opus-4-7 with context_management.edits = [{ type: 'compact_20260112', ... }] have no way to:

  1. Detect that compaction actually fired server-side
  2. Read the compaction summary content (needed for follow-up turns per Anthropic's recommended pattern: "Keep the original messages in your list and let the API handle removing the compacted content". But the summary itself is what needs to round-trip back to keep Anthropic from re-running compaction every turn)
  3. Persist the summary for observability

Proposed solution

Mirror how thinking is handled. Concretely:
1.

  • Recognize compaction_delta (and any compaction_* deltas) in build_chunk / extract_*_delta helpers
  • Add compaction: accumulation alongside content/thinking
  • Add extract_compaction_content(content_blocks) that selects c['type'] == 'compaction'
  • Pass to Message.new via a new keyword
  • Add compaction attribute similar to thinking
  • Default nil for non-Anthropic providers

4: Update format_message to round-trip compaction blocks back to Anthropic on subsequent turns (so the gem also handles the cost-saving "skip re-compaction" path Anthropic's docs describe).

This is symmetric with the thinking-block plumbing already present, so the change should be local and additive.

Why this belongs in RubyLLM

  • It's a first-class feature of the Anthropic Messages API.
  • The data is only available at the streaming layer: compaction_delta events live in the SSE stream; without gem support, an application has to either fork the gem or re-implement Anthropic SSE parsing externally.
  • It's not actually Anthropic-specific in concept. OpenAI shipped an equivalent server-side compaction feature on its Responses API (docs) with the same shape: a context_management config, a token threshold, and a server-returned compaction item meant to round-trip on subsequent turns. So a generic compaction accessor on RubyLLM::Message will eventually be populated by multiple providers, same evolution thinking went through.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions