Skip to content

feat: add LiteLLM as AI gateway provider#130

Open
RheagalFire wants to merge 1 commit into
jina-ai:mainfrom
RheagalFire:feat/add-litellm-provider
Open

feat: add LiteLLM as AI gateway provider#130
RheagalFire wants to merge 1 commit into
jina-ai:mainfrom
RheagalFire:feat/add-litellm-provider

Conversation

@RheagalFire
Copy link
Copy Markdown

@RheagalFire RheagalFire commented May 25, 2026

Summary

  • Adds LiteLLM as a new LLM provider, enabling node-DeepResearch to route AI requests through 100+ LLM providers (OpenAI, Anthropic, Azure, Bedrock, Vertex AI, Groq, Cohere, Mistral, etc.) via a single LiteLLM proxy.
  • Follows the existing provider pattern (same structure as OpenAI/Gemini providers).

Motivation

LiteLLM acts as a unified AI gateway. Instead of adding individual provider SDKs, users can set LLM_PROVIDER=litellm and point at a LiteLLM proxy that handles provider routing, load balancing, cost tracking, and fallbacks across 100+ providers. Useful for teams running centralized LLM infrastructure, or users who need providers not yet directly supported (e.g., AWS Bedrock, Cohere, Mistral).

Changes

  • src/config.ts - added 'litellm' to LLMProvider type, added litellm case in getModel() using createOpenAI with configurable LITELLM_BASE_URL and LITELLM_API_KEY env vars, added env var exports
  • config.json - added litellm provider entry and model configuration (mirrors openai config with openai/gpt-4o-mini as default model)

Tests

Live E2E - createOpenAI with LiteLLM proxy routing to Claude Sonnet 4.6:

Model: claude-sonnet-4-6
Content: OK
Finish: stop

This uses the exact same createOpenAI({ baseURL }) call path that getModel() uses when LLM_PROVIDER=litellm.

Risk / Compatibility

  • Additive only. Existing openai/gemini/vertex providers untouched.
  • No new dependencies - uses the existing @ai-sdk/openai package since LiteLLM proxy serves an OpenAI-compatible API.
  • 2 files changed.

Example usage

# Set provider and proxy URL
export LLM_PROVIDER=litellm
export LITELLM_BASE_URL=http://localhost:4000/v1
export LITELLM_API_KEY=sk-your-key    # optional

# Start LiteLLM proxy
litellm --model anthropic/claude-3-5-sonnet --port 4000

# Run deep research as usual
npm start

Any LiteLLM model ID works in config.json:

"litellm": {
  "default": {
    "model": "openai/gpt-4o-mini"
  }
}

Programmatic usage (same API as existing providers):

import { getModel } from './src/config';

// Set LLM_PROVIDER=litellm in env, then:
const model = getModel('agent');  // resolves litellm config from config.json
// model is a standard Vercel AI SDK LanguageModel instance

// Or use directly:
import { createOpenAI } from '@ai-sdk/openai';
import { generateText } from 'ai';

const litellm = createOpenAI({
    apiKey: process.env.LITELLM_API_KEY,
    baseURL: process.env.LITELLM_BASE_URL || 'http://localhost:4000/v1',
});

const result = await generateText({
    model: litellm('anthropic/claude-3-5-sonnet'),
    prompt: 'What is deep research?',
});
console.log(result.text);

@RheagalFire
Copy link
Copy Markdown
Author

cc @hanxiao

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant