fix(embeddings): truncate oversized litellm-sdk inputs before embedding (#2501)#2516
Open
nicoloboschi wants to merge 1 commit into
Open
fix(embeddings): truncate oversized litellm-sdk inputs before embedding (#2501)#2516nicoloboschi wants to merge 1 commit into
nicoloboschi wants to merge 1 commit into
Conversation
…ng (#2501) Mental-model content in delta-refresh mode can grow past an embedding model's fixed input-token limit (e.g. Bedrock Titan V2's hard 8192 cap), after which every refresh fails permanently with ContextWindowExceededError and no recovery path. Add an opt-in `HINDSIGHT_API_EMBEDDINGS_LITELLM_SDK_MAX_INPUT_TOKENS` cap. When set, `LiteLLMSDKEmbeddings.encode()` truncates each input to that many cl100k_base tokens before calling litellm.embedding(), mirroring the existing reranker `max_tokens_per_doc` pattern. Truncation emits a log.warning naming the model and largest original token count so it isn't silent. Off by default (no behavior change / data loss for large-context models); Titan users set it to the model's real limit with a little headroom.
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2501 (partial — the embedding-overflow half).
Problem
A pinned mental model in
deltarefresh mode accumulatescontentthat grows monotonically across refresh cycles. Once it exceeds the configured embedding model's fixed input-token limit (e.g. Bedrock Titan V2's hard 8,192-token cap),refresh_mental_modelfails permanently — every retry hits the sameContextWindowExceededErrorbecause the stored content never shrinks. The only recovery today is manually callingPOST .../mental-models/{id}/clear.The embedding call chain (
LiteLLMSDKEmbeddings.encode()→litellm.embedding()) had zero length check or truncation.Fix
Add an opt-in
HINDSIGHT_API_EMBEDDINGS_LITELLM_SDK_MAX_INPUT_TOKENScap. When set,encode()truncates each input to that manycl100k_basetokens before callinglitellm.embedding()— mirroring the existing rerankerHINDSIGHT_API_RERANKER_LITELLM_MAX_TOKENS_PER_DOC/_truncate_to_tokenspattern the issue references.log.warningnaming the model and the largest original token count, so it's never silent.8192) with a little headroom (tiktoken only approximates the provider's tokenizer).Scope
This addresses the embedding call overflow (gap #2 in the issue) — the direct cause of the permanent failure — per the "just truncate the vector indexing for now" steer. The deeper root cause (gap #1: unbounded delta-document growth in
apply_operations()) is not addressed here and is worth a follow-up if we want to cap persisted document size.Tests
tests/test_litellm_sdk_embeddings.py:All 29 tests pass; ruff + ty + lint.sh clean.