Skip to content

Feat/add litellm provider#137

Merged
JDOxygen merged 5 commits into
jd-opensource:mainfrom
RheagalFire:feat/add-litellm-provider
Jun 5, 2026
Merged

Feat/add litellm provider#137
JDOxygen merged 5 commits into
jd-opensource:mainfrom
RheagalFire:feat/add-litellm-provider

Conversation

@RheagalFire

@RheagalFire RheagalFire commented May 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds LiteLLM as a new LLM provider (oxy.LiteLLM), enabling access to 100+ LLM backends (OpenAI, Anthropic, Google, Azure, AWS Bedrock, Ollama, etc.) through a single unified interface.

Changes

  • Feature: oxy.LiteLLM provider added

    New files:

    • oxygent/oxy/llms/litellm_llm.py -- LiteLLM(BaseLLM) class using litellm.acompletion()
    • tests/unittest/test_litellm_llm.py -- 8 unit tests
    • examples/llms/demo_litellm.py -- usage example

    Modified files:

    • oxygent/oxy/llms/__init__.py -- export LiteLLM
    • oxygent/oxy/__init__.py -- export LiteLLM
    • oxygent/oxy_factory.py -- register in OxyFactory._creators
    • docs/docs_en/introduction/agents/select-llm.md -- added LiteLLM to provider table + usage section

    Design decisions:

    • Extends BaseLLM directly (not RemoteLLM) because RemoteLLM's validator requires non-empty base_url, but LiteLLM routes by model prefix and doesn't always need one
    • drop_params=True by default -- silently drops provider-unsupported kwargs (e.g. strict:true on Anthropic, seed on Gemini)
    • api_key is optional -- when unset, LiteLLM falls back to provider-specific env vars (ANTHROPIC_API_KEY, OPENAI_API_KEY, etc.)
    • Lazy-imports litellm inside _execute() so the package stays optional
    • Streaming + reasoning_content (<think> tags) handled identically to OpenAILLM

Tests

Unit tests (8 passing):

tests/unittest/test_litellm_llm.py::test_streaming_success PASSED
tests/unittest/test_litellm_llm.py::test_non_streaming PASSED
tests/unittest/test_litellm_llm.py::test_api_key_forwarded PASSED
tests/unittest/test_litellm_llm.py::test_api_key_omitted_when_unset PASSED
tests/unittest/test_litellm_llm.py::test_base_url_forwarded PASSED
tests/unittest/test_litellm_llm.py::test_drop_params_default_true PASSED
tests/unittest/test_litellm_llm.py::test_llm_params_merged PASSED
tests/unittest/test_litellm_llm.py::test_factory_registration PASSED
8 passed in 0.85s

Existing tests -- no regressions (38 passed):

test_litellm_llm (8) + test_remote_llm (5) + test_http_llm (2) + test_base_llm (1) +
test_mock_llm (3) + test_oxy_factory (12) = 38 passed in 1.88s

Live e2e against Anthropic via LiteLLM:

# Non-streaming
State: OxyState.COMPLETED
Output: 4
Usage: input_tokens=20 output_tokens=5 model_name='anthropic/claude-sonnet-4-20250514' estimation_method='exact'

# Streaming
State: OxyState.COMPLETED
Output: Hello there, friend!
Usage: input_tokens=15 output_tokens=8 model_name='anthropic/claude-sonnet-4-20250514' estimation_method='exact'

Format check: ruff format clean on all new/modified files.

Example usage

import asyncio
from oxygent import MAS, oxy

oxy_space = [
    oxy.LiteLLM(
        name="default_llm",
        model_name="anthropic/claude-sonnet-4-20250514",
        # api_key="sk-...",  # or set ANTHROPIC_API_KEY env var
    ),
    oxy.ReActAgent(
        name="agent",
        llm_model="default_llm",
        system_prompt="You are a helpful assistant.",
    ),
]

async def main():
    async with MAS(oxy_space=oxy_space) as mas:
        result = await mas.call(
            callee="agent",
            arguments={"messages": [{"role": "user", "content": "What is 2 + 2?"}]},
        )
        print("Result:", result.output)

asyncio.run(main())

Checklist

  • been self-reviewed.
    • Code is formatted
    • Tests added/updated
    • All CI checks passed
    • Documentation updated
  • added documentation for new or modified features or behaviors.
  • added new features, such as new agents, new flows, etc.
  • added or updated version, license, or notice information.
  • added or updated demo, integration tests, unit tests, or others.
  • added or updated ui, or had changes in frontend.

@RheagalFire

Copy link
Copy Markdown
Contributor Author

cc @JDOxygen

@RheagalFire

Copy link
Copy Markdown
Contributor Author

@JDOxygen do you have any update on this PR?

@JDOxygen
JDOxygen merged commit 718ce99 into jd-opensource:main Jun 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants