Skip to content

feat(sandbox): add LiteLLMCompaction capability sized via litellm#3409

Closed
scalabreseGD wants to merge 4 commits into
openai:mainfrom
scalabreseGD:add-litellm-compaction
Closed

feat(sandbox): add LiteLLMCompaction capability sized via litellm#3409
scalabreseGD wants to merge 4 commits into
openai:mainfrom
scalabreseGD:add-litellm-compaction

Conversation

@scalabreseGD
Copy link
Copy Markdown

feat(sandbox): add LiteLLMCompaction capability sized via litellm

Summary

Adds agents.extensions.sandbox.LiteLLMCompaction, a drop-in subclass of agents.sandbox.capabilities.Compaction whose default policy is sized from litellm.get_model_info() rather than the built-in OpenAI-only context-window registry.

Why

The existing Compaction capability ships a private _MODEL_CONTEXT_WINDOWS registry that catalogues OpenAI families only. When a litellm-routed model (Anthropic, Bedrock, Vertex, custom proxy aliases, ...) is used:

  • CompactionModelInfo.maybe_for_model(model) misses the lookup.
  • Compaction.sampling_params therefore cannot pick a DynamicCompactionPolicy and falls back to a hard-coded 240k StaticCompactionPolicy.
  • A 1M-context model and a 200k model end up being compacted at the same point, regardless of their actual input window.

What this PR adds

A new optional capability under the existing openai-agents[litellm] extra:

  • LiteLLMCompaction.for_model(model, *, threshold=0.8) — resolves the cap via litellm.get_model_info() and builds a DynamicCompactionPolicy sized to the reported max_input_tokens. Falls back to a conservative 200k window with a WARNING when litellm has no entry for the model (brand-new beta IDs, custom proxy aliases, Bedrock embedding entries with no max_input_tokens), so the run still benefits from compaction instead of failing.
  • LiteLLMCompaction.for_context_window(context_window, *, threshold=0.8) — builds the same policy from an explicitly-provided cap, for callers whose config layer already applies per-org ceilings or test overrides that should clamp the compaction threshold below the litellm-reported window.

The class is a pure pydantic subclass of Compaction, so process_context, sampling_params, the type: "compaction" serialization tag, and direct LiteLLMCompaction(policy=...) construction all behave identically to the parent.

Compatibility

Purely additive — no existing released APIs change. The module is gated behind the existing openai-agents[litellm] optional extra via raise_optional_dependency_error (same pattern as RedisSession, DaprSession, MongoDBSession), and is exported from agents.extensions.sandbox.__init__ only when the extra is installed.

Docs

  • New API reference page at docs/ref/extensions/sandbox/litellm_compaction.md, registered under Extensions > Sandbox capabilities > LiteLLM compaction in mkdocs.yml.
  • docs/sandbox/guide.md now calls out the OpenAI-only registry limitation right after the built-in capabilities table and points readers at LiteLLMCompaction for litellm-routed deployments.

Translated docs under docs/ja/, docs/ko/, and docs/zh/ are intentionally untouched (they are generated).

Usage

from agents.extensions.sandbox import LiteLLMCompaction

# Cap derived from litellm's registry.
capability = LiteLLMCompaction.for_model("anthropic/claude-3-5-sonnet")

# Cap derived from an externally-resolved value (per-org ceiling, test override, ...).
capability = LiteLLMCompaction.for_context_window(my_app_config.max_input_tokens)

Test plan

New tests at tests/extensions/sandbox/test_litellm_compaction.py (8 tests, all passing locally) cover:

  • for_context_window builds the right DynamicCompactionPolicy with the default (0.8) and custom thresholds.
  • sampling_params emits the correct compact_threshold end-to-end (context_window * threshold).
  • for_model calls litellm.get_model_info and uses max_input_tokens.
  • for_model falls back to the 200k default both when litellm raises and when max_input_tokens is None (Bedrock-embedding-style entries).
  • Direct LiteLLMCompaction(policy=StaticCompactionPolicy(...)) construction still works (parent contract preserved).

make format, make lint, and mypy on the touched files all pass cleanly. The pre-existing mypy/pyright errors in function_schema.py, openai_responses.py, mount_lifecycle.py, and examples/tools/computer_use.py reproduce on main and are unrelated to this change.

Issue number

N/A

Checks

  • I've added new tests
  • I've added/updated the relevant documentation
  • I've run make lint and make format
  • I've made sure tests pass

Add `agents.extensions.sandbox.LiteLLMCompaction`, a drop-in subclass
of `agents.sandbox.capabilities.Compaction` whose default policy is
sized from `litellm.get_model_info()` rather than the OpenAI-only
context-window registry.
The default `Compaction` registry catalogues OpenAI families only, so
litellm-routed models (Anthropic, Bedrock, Vertex, custom proxy
aliases, ...) miss the lookup and fall back to a hard-coded 240k
`StaticCompactionPolicy` regardless of the model's actual input
window — compacting a 1M-context model at the same point as a 200k
model.
`LiteLLMCompaction` exposes two classmethods:
- `for_model(model, *, threshold=0.8)` resolves the cap via
  `litellm.get_model_info()`. Falls back to a 200k window with a
  WARNING when litellm has no entry (new beta IDs, custom proxy
  aliases, Bedrock embedding entries with no `max_input_tokens`).
- `for_context_window(context_window, *, threshold=0.8)` builds the
  same policy from an explicitly-provided cap, for callers whose
  config layer already applies per-org ceilings or test overrides.
Purely additive — no released APIs change. Gated behind the existing
`openai-agents[litellm]` optional extra via
`raise_optional_dependency_error`, and exported from
`agents.extensions.sandbox` only when the extra is installed.
Add `agents.extensions.sandbox.LiteLLMCompaction`, a drop-in subclass
of `agents.sandbox.capabilities.Compaction` whose default policy is
sized from `litellm.get_model_info()` rather than the OpenAI-only
context-window registry.
The default `Compaction` registry catalogues OpenAI families only, so
litellm-routed models (Anthropic, Bedrock, Vertex, custom proxy
aliases, ...) miss the lookup and fall back to a hard-coded 240k
`StaticCompactionPolicy` regardless of the model's actual input
window — compacting a 1M-context model at the same point as a 200k
model.
`LiteLLMCompaction` exposes two classmethods:
- `for_model(model, *, threshold=0.8)` resolves the cap via
  `litellm.get_model_info()`. Falls back to a 200k window with a
  WARNING when litellm has no entry (new beta IDs, custom proxy
  aliases, Bedrock embedding entries with no `max_input_tokens`).
- `for_context_window(context_window, *, threshold=0.8)` builds the
  same policy from an explicitly-provided cap, for callers whose
  config layer already applies per-org ceilings or test overrides.
Purely additive — no released APIs change. Gated behind the existing
`openai-agents[litellm]` optional extra via
`raise_optional_dependency_error`, and exported from
`agents.extensions.sandbox` only when the extra is installed.
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 21a06c23ac

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/agents/extensions/sandbox/__init__.py Outdated
@seratch
Copy link
Copy Markdown
Member

seratch commented May 15, 2026

Thanks for suggesting this. However, we don't plan to add this module within this SDK. If you need it for your projects, please feel free to share the code as your own package/module.

@seratch seratch closed this May 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants