diff --git a/docs/providers/litellm.md b/docs/providers/litellm.md new file mode 100644 index 00000000..e7242768 --- /dev/null +++ b/docs/providers/litellm.md @@ -0,0 +1,83 @@ +--- +sidebar_label: LiteLLM +--- + +# Using LiteLLM With Roo Code + +LiteLLM is a versatile tool that provides a unified interface to over 100 Large Language Models (LLMs) by offering an OpenAI-compatible API. This allows you to run a local server that can proxy requests to various model providers or serve local models, all accessible through a consistent API endpoint. + +**Website:** [https://litellm.ai/](https://litellm.ai/) (Main project) & [https://docs.litellm.ai/](https://docs.litellm.ai/) (Documentation) + +## Key Benefits + +* **Unified API:** Access a wide range of LLMs (from OpenAI, Anthropic, Cohere, HuggingFace, etc.) through a single, OpenAI-compatible API. +* **Local Deployment:** Run your own LiteLLM server locally, giving you more control over model access and potentially reducing latency. +* **Simplified Configuration:** Manage credentials and model configurations in one place (your LiteLLM server) and let Roo Code connect to it. +* **Cost Management:** LiteLLM offers features for tracking costs across different models and providers. + +## Setting Up Your LiteLLM Server + +To use LiteLLM with Roo Code, you first need to set up and run a LiteLLM server. + +1. **Installation:** Follow the official [LiteLLM installation guide](https://docs.litellm.ai/docs/proxy_server) to install LiteLLM and its dependencies. +2. **Configuration:** Configure your LiteLLM server with the models you want to use. This typically involves setting API keys for the underlying providers (e.g., OpenAI, Anthropic) in your LiteLLM server's configuration. +3. **Start the Server:** Run your LiteLLM server. By default, it usually starts on `http://localhost:4000`. + * You can also configure an API key for your LiteLLM server itself for added security. + +Refer to the [LiteLLM documentation](https://docs.litellm.ai/docs/) for detailed instructions on server setup, model configuration, and advanced features. + +## Configuration in Roo Code + +Once your LiteLLM server is running: + +1. **Open Roo Code Settings:** Click the gear icon () in the Roo Code panel. +2. **Select Provider:** Choose "LiteLLM" from the "API Provider" dropdown. +3. **Enter Base URL:** + * Input the URL of your LiteLLM server. + * Defaults to `http://localhost:4000` if left blank. +4. **Enter API Key (Optional):** + * If you've configured an API key for your LiteLLM server, enter it here. + * If your LiteLLM server doesn't require an API key, Roo Code will use a default dummy key (`"dummy-key"`), which should work fine. +5. **Select Model:** + * Roo Code will attempt to fetch the list of available models from your LiteLLM server by querying the `${baseUrl}/v1/model/info` endpoint. + * The models displayed in the dropdown are sourced from this endpoint. + * If no model is selected, Roo Code defaults to `anthropic/claude-3-7-sonnet-20250219` (this is `litellmDefaultModelId`). Ensure this model (or your desired default) is configured and available on your LiteLLM server. + +Roo Code LiteLLM Provider Settings + +## How Roo Code Fetches and Interprets Model Information + +When you configure the LiteLLM provider, Roo Code interacts with your LiteLLM server to get details about the available models: + +* **Model Discovery:** Roo Code makes a GET request to `${baseUrl}/v1/model/info` on your LiteLLM server. If an API key is provided in Roo Code's settings, it's included in the `Authorization: Bearer ${apiKey}` header. +* **Model Properties:** For each model reported by your LiteLLM server, Roo Code extracts and interprets the following: + * `model_name`: The identifier for the model. + * `maxTokens`: Maximum output tokens. Defaults to `8192` if not specified by LiteLLM. + * `contextWindow`: Maximum context tokens. Defaults to `200000` if not specified by LiteLLM. + * `supportsImages`: Determined from `model_info.supports_vision` provided by LiteLLM. + * `supportsPromptCache`: Determined from `model_info.supports_prompt_caching` provided by LiteLLM. + * `inputPrice` / `outputPrice`: Calculated from `model_info.input_cost_per_token` and `model_info.output_cost_per_token` from LiteLLM. + * `supportsComputerUse`: This flag is set to `true` if the underlying model identifier (from `litellm_params.model`, e.g., `openrouter/anthropic/claude-3.5-sonnet`) matches one of the Anthropic models predefined in Roo Code as suitable for "computer use" (see `COMPUTER_USE_MODELS` in technical details). + +Roo Code uses default values for some of these properties if they are not explicitly provided by your LiteLLM server's `/model/info` endpoint for a given model. The defaults are: +* `maxTokens`: 8192 +* `contextWindow`: 200,000 +* `supportsImages`: `true` +* `supportsComputerUse`: `true` (for the default model ID) +* `supportsPromptCache`: `true` +* `inputPrice`: 3.0 (µUSD per 1k tokens) +* `outputPrice`: 15.0 (µUSD per 1k tokens) + +## Tips and Notes + +* **LiteLLM Server is Key:** The primary configuration for models, API keys for downstream providers (like OpenAI, Anthropic), and other advanced features are managed on your LiteLLM server. Roo Code acts as a client to this server. +* **Model Availability:** The models available in Roo Code's "Model" dropdown depend entirely on what your LiteLLM server exposes through its `/v1/model/info` endpoint. +* **Network Accessibility:** Ensure your LiteLLM server is running and accessible from the machine where VS Code and Roo Code are running (e.g., check firewall rules if not on `localhost`). +* **Troubleshooting:** If models aren't appearing or requests fail: + * Verify your LiteLLM server is running and configured correctly. + * Check the LiteLLM server logs for errors. + * Ensure the Base URL in Roo Code settings matches your LiteLLM server's address. + * Confirm any API key required by your LiteLLM server is correctly entered in Roo Code. +* **Computer Use Models:** The `supportsComputerUse` flag in Roo Code is primarily relevant for certain Anthropic models known to perform well with tool-use and function-calling tasks. If you are routing other models through LiteLLM, this flag might not be automatically set unless the underlying model ID matches the specific Anthropic ones Roo Code recognizes. + +By leveraging LiteLLM, you can significantly expand the range of models accessible to Roo Code while centralizing their management. \ No newline at end of file diff --git a/sidebars.ts b/sidebars.ts index c43e7b38..7d701230 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -111,6 +111,7 @@ const sidebars: SidebarsConfig = { 'providers/groq', 'providers/human-relay', 'providers/lmstudio', + 'providers/litellm', 'providers/mistral', 'providers/ollama', 'providers/openai', diff --git a/static/img/litellm/litellm.png b/static/img/litellm/litellm.png new file mode 100644 index 00000000..3605db1f Binary files /dev/null and b/static/img/litellm/litellm.png differ