Skip to content

fix(embeddings): truncate oversized litellm-sdk inputs before embedding (#2501)#2516

Open
nicoloboschi wants to merge 1 commit into
mainfrom
fix/embed-truncate-litellm-sdk-2501
Open

fix(embeddings): truncate oversized litellm-sdk inputs before embedding (#2501)#2516
nicoloboschi wants to merge 1 commit into
mainfrom
fix/embed-truncate-litellm-sdk-2501

Conversation

@nicoloboschi

Copy link
Copy Markdown
Collaborator

Fixes #2501 (partial — the embedding-overflow half).

Problem

A pinned mental model in delta refresh mode accumulates content that grows monotonically across refresh cycles. Once it exceeds the configured embedding model's fixed input-token limit (e.g. Bedrock Titan V2's hard 8,192-token cap), refresh_mental_model fails permanently — every retry hits the same ContextWindowExceededError because the stored content never shrinks. The only recovery today is manually calling POST .../mental-models/{id}/clear.

The embedding call chain (LiteLLMSDKEmbeddings.encode()litellm.embedding()) had zero length check or truncation.

Fix

Add an opt-in HINDSIGHT_API_EMBEDDINGS_LITELLM_SDK_MAX_INPUT_TOKENS cap. When set, encode() truncates each input to that many cl100k_base tokens before calling litellm.embedding() — mirroring the existing reranker HINDSIGHT_API_RERANKER_LITELLM_MAX_TOKENS_PER_DOC / _truncate_to_tokens pattern the issue references.

  • Truncation emits a log.warning naming the model and the largest original token count, so it's never silent.
  • Off by default → no behavior change or data loss for large-context embedding models. Titan users set it to the model's real limit (e.g. 8192) with a little headroom (tiktoken only approximates the provider's tokenizer).
  • Server-level static config (embeddings aren't per-bank overridable).

Scope

This addresses the embedding call overflow (gap #2 in the issue) — the direct cause of the permanent failure — per the "just truncate the vector indexing for now" steer. The deeper root cause (gap #1: unbounded delta-document growth in apply_operations()) is not addressed here and is worth a follow-up if we want to cap persisted document size.

Tests

tests/test_litellm_sdk_embeddings.py:

  • oversized input is truncated to the cap and a warning is logged
  • warning does not fire when inputs already fit
  • text passes through verbatim when the cap is unset (default)

All 29 tests pass; ruff + ty + lint.sh clean.

…ng (#2501)

Mental-model content in delta-refresh mode can grow past an embedding
model's fixed input-token limit (e.g. Bedrock Titan V2's hard 8192 cap),
after which every refresh fails permanently with ContextWindowExceededError
and no recovery path.

Add an opt-in `HINDSIGHT_API_EMBEDDINGS_LITELLM_SDK_MAX_INPUT_TOKENS` cap.
When set, `LiteLLMSDKEmbeddings.encode()` truncates each input to that many
cl100k_base tokens before calling litellm.embedding(), mirroring the existing
reranker `max_tokens_per_doc` pattern. Truncation emits a log.warning naming
the model and largest original token count so it isn't silent.

Off by default (no behavior change / data loss for large-context models);
Titan users set it to the model's real limit with a little headroom.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mental-model content grows unbounded via delta-refresh, permanently overflows embedding model's input token limit (no truncation before embed)

1 participant