feat(litellm): add flatten content option for auto_router compatibility - #11133
feat(litellm): add flatten content option for auto_router compatibility#11133ghost wants to merge 1 commit into
Conversation
LiteLLM's auto_router feature passes user content to embedding models for semantic routing. However, embedding models expect plain strings, not arrays of content blocks. This causes errors when Roo Code sends messages with array content format. This commit adds a new `litellmFlattenContent` option (enabled by default) that flattens array content to simple string format before sending to LiteLLM. Changes: - Add litellmFlattenContent option to provider-settings.ts schema - Add flattenMessageContent method to LiteLLMHandler - Apply flattening in createMessage when option is enabled (default: true) - Add UI checkbox in LiteLLM settings component - Add translations for all supported languages - Add comprehensive tests for the flattening logic Users who need multimodal content (images) can disable this option. Fixes #11132
ghost
mentioned this pull request
Reviewed the implementation of the
Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues. |
| // Flatten array content to string for compatibility with LiteLLM's auto_router | ||
| // This is enabled by default (when litellmFlattenContent is undefined or true) | ||
| if (this.options.litellmFlattenContent !== false) { | ||
| openAiMessages = this.flattenMessageContent(openAiMessages) | ||
| } |
There was a problem hiding this comment.
When both litellmFlattenContent (enabled by default) and litellmUsePromptCache are enabled with a model that supports prompt caching, the flattening is undone. The flattening converts array content to strings here, but lines 194-207 below wrap string content back into arrays to add cache_control. This means auto_router users who also enable prompt caching will still encounter the embedding model failure this PR intends to fix. Consider either moving the flattening after the prompt caching logic, or having the prompt caching logic skip re-wrapping when flattening is enabled.
Fix it with Roo Code or mention @roomote and request a fix.
1 participant
Summary
This PR attempts to address Issue #11132.
Problem
LiteLLM's
auto_routerfeature passes user content to embedding models for semantic routing. However, embedding models expect plain strings, not arrays of content blocks. When Roo Code sends messages with array content like:{"role": "user", "content": [{"type": "text", "text": "..."}]}The auto_router fails because it passes this array directly to embeddings, which is incompatible.
Solution
This PR adds a new
litellmFlattenContentoption (enabled by default) that flattens array content to simple string format before sending to LiteLLM:{"role": "user", "content": "..."}Changes
litellmFlattenContentoption to the LiteLLM schemaflattenMessageContent()method and apply it increateMessage()when the option is enabled (default: true)Notes
Feedback and guidance are welcome!
Important
Adds
litellmFlattenContentoption to flatten message content for LiteLLM'sauto_router, with schema, UI, and test updates.litellmFlattenContentoption to flatten array content to strings for LiteLLM'sauto_routercompatibility.litellmFlattenContentis set to false.litellmFlattenContenttoprovider-settings.tsschema.flattenMessageContent()inlite-llm.tsand applies it increateMessage().LiteLLM.tsx.lite-llm.spec.tsto verify flattening logic under different configurations.settings.jsonin 17 locales to include translations for the new option.This description was created by
for 6fe9a9e. You can customize this summary. It will automatically update as commits are pushed.