Skip to content

feat: add LiteLLM as LLM provider#64

Open
RheagalFire wants to merge 1 commit into
melih-unsal:mainfrom
RheagalFire:feat/litellm-provider
Open

feat: add LiteLLM as LLM provider#64
RheagalFire wants to merge 1 commit into
melih-unsal:mainfrom
RheagalFire:feat/litellm-provider

Conversation

@RheagalFire
Copy link
Copy Markdown

Summary

  • Add LiteLLM as a new LLM provider, giving DemoGPT users access to 100+ LLM providers (Anthropic, AWS Bedrock, Vertex AI, Azure, Groq, Cohere, Mistral, Together, Fireworks, etc.) through the LiteLLM Python SDK
  • Works in both the agenthub layer (LiteLLMChatModel for agents/RAG) and the chains layer (via create_llm() factory)
  • Activated by passing provider="litellm" to DemoGPT() or using LiteLLMChatModel directly

Changes

  • demogpt_agenthub/llms/litellm.py -- new LiteLLMChatModel extending both BaseChatModel (LangChain runnable) and BaseLLM, calls litellm.completion() with drop_params=True
  • demogpt_agenthub/llms/__init__.py -- export LiteLLMChatModel (lazy import, no breakage if litellm not installed)
  • demogpt/chains/_llm_factory.py -- shared factory that creates LiteLLMChatModel when provider="litellm", ChatOpenAI otherwise
  • demogpt/chains/chains.py -- Chains.setLlm() and Chains.getModel() use factory, accept provider param
  • demogpt/chains/task_chains.py -- TaskChains.setLlm() uses factory, accepts provider param
  • demogpt/chains/task_chains_seperate.py -- same
  • demogpt/model.py -- DemoGPT.__init__() accepts provider param, passes through to all chain classes
  • pyproject.toml -- add litellm>=1.80.0,<1.87.0 as optional dependency (pip install demogpt[litellm])

Tests

1. Live E2E: LiteLLMChatModel.run() (Azure Foundry -> Claude Sonnet 4.6)

from demogpt_agenthub.llms.litellm import LiteLLMChatModel

model = LiteLLMChatModel(model='azure_ai/claude-sonnet-4-6')
result = model.run('What is 2+2? Reply with just the number.')
Response: 4
Type: str

2. Live E2E: LangChain pipe compatibility (used by agents)

chain = ChatPromptTemplate.from_template('Reply with just: {word}') | model | StrOutputParser()
pipe_result = chain.invoke({'word': 'pong'})
Pipe result: pong

3. Live E2E: Chains factory path

from demogpt.chains._llm_factory import create_llm

llm = create_llm(model='azure_ai/claude-sonnet-4-6', provider='litellm')
result = llm.invoke('Reply with just: pong')
Factory result: pong
Type: LiteLLMChatModel

Example usage

Agenthub (agents, RAG):

from demogpt_agenthub.llms.litellm import LiteLLMChatModel
from demogpt_agenthub.agents import ToolCallingAgent

# Provider API keys read from env: export ANTHROPIC_API_KEY=sk-ant-...
llm = LiteLLMChatModel(model="anthropic/claude-sonnet-4-20250514")
agent = ToolCallingAgent(tools=[...], llm=llm)
result = agent.run("What is the weather in Paris?")

Main DemoGPT app:

from demogpt.model import DemoGPT

# Provider API keys read from env: export ANTHROPIC_API_KEY=sk-ant-...
demo = DemoGPT(
    model_name="anthropic/claude-sonnet-4-20250514",
    provider="litellm",
)

Supported model formats (LiteLLM convention):

  • anthropic/claude-sonnet-4-20250514 -- Anthropic
  • bedrock/anthropic.claude-sonnet-4-20250514-v1:0 -- AWS Bedrock
  • vertex_ai/gemini-2.5-flash -- Google Vertex
  • azure_ai/claude-sonnet-4-6 -- Azure AI Foundry
  • groq/llama-3.3-70b-versatile -- Groq
  • ollama/llama3 -- Local Ollama

Risk / Compatibility

  • Additive only -- no existing providers modified
  • LiteLLMChatModel follows the exact same dual-inheritance pattern as OpenAIChatModel(ChatOpenAI, BaseLLM), just swapping the LangChain base for a custom BaseChatModel that calls litellm SDK
  • provider parameter defaults to None, preserving existing OpenAI behavior
  • litellm is an optional dependency -- existing users without it are unaffected (lazy import with try/except)
  • drop_params=True ensures unsupported params are silently dropped per provider

@RheagalFire
Copy link
Copy Markdown
Author

cc @melih-unsal

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant