|
1 | 1 | --- |
2 | | -title: Introduction |
3 | | -description: Introducing llms.py - Lightweight OpenAI compatible CLI and server gateway |
| 2 | +title: Installation |
| 3 | +description: How to install llms.py |
4 | 4 | --- |
5 | 5 |
|
6 | | -# Introducing llms.py 🚀 |
| 6 | +## Using pip |
7 | 7 |
|
8 | | -We're excited to announce **[llms.py](https://github.com/ServiceStack/llms)** - a super lightweight CLI tool and OpenAI-compatible server that acts as a **configurable gateway** over multiple configurable Large Language Model (LLM) providers. |
| 8 | +The easiest way to install llms.py is using pip: |
9 | 9 |
|
10 | | -## 🎯 OpenRouter but Local |
| 10 | +<ShellCommand>pip install llms-py</ShellCommand> |
11 | 11 |
|
12 | | -**llms.py** is designed as a **unified gateway** that seamlessly connects you to multiple LLM providers through a single, consistent interface. Whether using cloud APIs or local models, `llms` provides intelligent routing and automatic failover to ensure your AI workflows connect to your chosen providers in your preferred priority - whether optimizing for cost, performance, or availability. |
| 12 | +## Using Docker |
13 | 13 |
|
14 | | -### ⚡ Ultra-Lightweight Architecture |
| 14 | +### Pre-built Docker Images |
15 | 15 |
|
| 16 | +Pull and run the latest image: |
16 | 17 |
|
17 | | -1. **Simplicity**: Just one [llms.py](https://github.com/ServiceStack/llms/blob/main/llms/main.py) file (easily customizable) |
18 | | -2. **Single Dependency**: Only requires `aiohttp` (Pillow optional) |
19 | | - - Zero Dependencies for ComfyUI - Ideal for use in Custom Nodes |
20 | | -3. **Flexibility**: Works with any OpenAI-compatible client or framework |
21 | | -4. **Reliability**: Automatic failover ensures your workflows never break |
22 | | -5. **Economy**: Intelligent routing minimizes API costs |
23 | | -6. **Privacy**: Mix local and cloud models based on your data sensitivity |
24 | | -7. **Future-Proof**: Easily add new providers as they emerge |
25 | | -8. **No Setup**: Just download and use, configure preferred LLMs in [llms.json](https://github.com/ServiceStack/llms/blob/main/llms/llms.json) |
| 18 | +<ShellCommand>docker pull ghcr.io/servicestack/llms:latest</ShellCommand> |
26 | 19 |
|
27 | | -**llms.py** transforms the complexity of managing multiple LLM providers into a simple, unified experience. |
28 | | -Whether you're researching capabilities of new models, building the next breakthrough AI application, |
29 | | -or just want reliable access to the best models available, llms.py has you covered. |
| 20 | +<ShellCommand>docker run -p 8000:8000 -e GROQ_API_KEY=$GROQ_API_KEY ghcr.io/servicestack/llms:latest</ShellCommand> |
30 | 21 |
|
31 | | -Get started today and avoid expensive cloud lock-ins with the freedom of provider-agnostic AI development! 🎉 |
| 22 | +### Using docker compose (Recommended) |
32 | 23 |
|
33 | | -## 🌐 Configurable Multi-Provider Gateway |
| 24 | +1. Download the docker-compose.yml file: |
34 | 25 |
|
35 | | -Acts as an intelligent gateway that can route requests for 160+ models across: |
| 26 | +<ShellCommand>curl -O https://raw.githubusercontent.com/ServiceStack/llms/refs/heads/main/docker-compose.yml</ShellCommand> |
36 | 27 |
|
37 | | -### Cloud Providers with Free Tiers |
| 28 | +2. Create a `.env` file with your API keys: |
38 | 29 |
|
39 | | -- OpenRouter |
40 | | -- Groq |
41 | | -- Codestral |
42 | | -- Google |
| 30 | +```bash |
| 31 | +OPENROUTER_API_KEY=sk-or-... |
| 32 | +GROQ_API_KEY=gsk_... |
| 33 | +GOOGLE_FREE_API_KEY=AIza... |
| 34 | +``` |
43 | 35 |
|
44 | | -### Premium Cloud Providers |
| 36 | +3. Start the service: |
45 | 37 |
|
46 | | -- OpenAI |
47 | | -- Anthropic |
48 | | -- Google |
49 | | -- Grok |
50 | | -- Qwen |
51 | | -- Mistral |
| 38 | +<ShellCommand>docker compose up -d</ShellCommand> |
52 | 39 |
|
53 | | -### Local Providers |
| 40 | +### Build from Source |
54 | 41 |
|
55 | | -- Ollama |
56 | | - - Restrict access to custom models |
57 | | - - Or auto-discovery of installed models |
| 42 | +If you've cloned the repository: |
58 | 43 |
|
59 | | -### Custom Providers |
| 44 | +<ShellCommand>git clone https://github.com/ServiceStack/llms</ShellCommand> |
60 | 45 |
|
61 | | -Use JSON config to add any OpenAI-compatible API endpoints and models |
| 46 | +<ShellCommand>cd llms</ShellCommand> |
62 | 47 |
|
63 | | -## 🔄 Intelligent Request Routing |
| 48 | +<ShellCommand>docker compose -f docker-compose.local.yml up -d --build</ShellCommand> |
64 | 49 |
|
65 | | -- **Automatic Failover**: If one provider fails, automatically retry with the next available provider |
66 | | -- **Cost Optimization**: Define free/cheap/local providers first to minimize costs |
67 | | -- **Model Mapping**: Use unified model names that map to different provider-specific names |
| 50 | +## Setting Up API Keys |
68 | 51 |
|
69 | | -## 🚀 Key Features |
| 52 | +Set environment variables for the providers you want to use: |
70 | 53 |
|
71 | | -### 🎨 ChatGPT-like Web UI |
| 54 | +```bash |
| 55 | +export OPENROUTER_API_KEY="sk-or-..." |
| 56 | +export GROQ_API_KEY="gsk_..." |
| 57 | +export GOOGLE_FREE_API_KEY="AIza..." |
| 58 | +export ANTHROPIC_API_KEY="sk-ant-..." |
| 59 | +export OPENAI_API_KEY="sk-..." |
| 60 | +export GROK_API_KEY="xai-..." |
| 61 | +export DASHSCOPE_API_KEY="sk-..." # Qwen |
| 62 | +export ZAI_API_KEY="sk-..." |
| 63 | +export MISTRAL_API_KEY="..." |
| 64 | +``` |
72 | 65 |
|
73 | | -A modern, fast, and privacy-focused web interface for interacting with all your LLMs. |
| 66 | +### API Key Reference |
74 | 67 |
|
75 | | -- **Offline & Private**: All data stored locally in SQLite |
76 | | -- **Dark Mode**: Automatic or manual dark mode toggle |
77 | | -- **Rich Markdown**: Full markdown support with syntax highlighting |
78 | | -- **Search**: Quickly find past conversations |
79 | | -- **Export/Import**: Backup and transfer chat histories |
| 68 | +| Provider | Environment Variable | Description | |
| 69 | +|----------|---------------------|-------------| |
| 70 | +| openrouter_free | `OPENROUTER_API_KEY` | OpenRouter FREE models | |
| 71 | +| groq | `GROQ_API_KEY` | Groq API key | |
| 72 | +| google_free | `GOOGLE_FREE_API_KEY` | Google FREE API key | |
| 73 | +| codestral | `CODESTRAL_API_KEY` | Codestral API key | |
| 74 | +| ollama | N/A | No API key required | |
| 75 | +| openrouter | `OPENROUTER_API_KEY` | OpenRouter API key | |
| 76 | +| google | `GOOGLE_API_KEY` | Google API key | |
| 77 | +| anthropic | `ANTHROPIC_API_KEY` | Anthropic API key | |
| 78 | +| openai | `OPENAI_API_KEY` | OpenAI API key | |
| 79 | +| grok | `GROK_API_KEY` | Grok (X.AI) API key | |
| 80 | +| qwen | `DASHSCOPE_API_KEY` | Qwen (Alibaba) key | |
| 81 | +| z.ai | `ZAI_API_KEY` | Z.ai API key | |
| 82 | +| mistral | `MISTRAL_API_KEY` | Mistral API key | |
80 | 83 |
|
81 | | -[Learn more about the Web UI →](/docs/features/web-ui) |
| 84 | +## Verification |
82 | 85 |
|
83 | | -### 🖥️ CLI Interface |
| 86 | +After installation, verify it's working: |
84 | 87 |
|
85 | | -- Interactive command-line tool |
86 | | -- Support for all modalities (text, image, audio, files) |
87 | | -- Custom system prompts |
88 | | -- Raw JSON output mode |
| 88 | +<ShellCommand>llms ls</ShellCommand> |
89 | 89 |
|
90 | | -[Learn more about the CLI →](/docs/features/cli) |
| 90 | +## Updating |
91 | 91 |
|
92 | | -### 🎯 Multimodal Support |
| 92 | +To update to the latest version: |
93 | 93 |
|
94 | | -Process text, images, audio, and documents with capable models. |
| 94 | +<ShellCommand>pip install llms-py --upgrade</ShellCommand> |
95 | 95 |
|
96 | | -- **Text Generation**: Chat completions with any supported model |
97 | | -- **Vision Models**: Process images through vision-capable models (GPT-4V, Gemini Vision, etc.) |
98 | | -- **Audio Processing**: Handle audio inputs through audio-capable models |
99 | | -- **Document Processing**: Analyze PDFs and documents with capable models |
100 | | -- **Drag & Drop**: Easy file attachments in the UI |
| 96 | +For Docker: |
101 | 97 |
|
102 | | -[Learn more about Multimodal Support →](/docs/features/multimodal) |
103 | | - |
104 | | -### Flexible Deployment Options |
105 | | -- **CLI Tool**: Interactive command-line interface for quick queries |
106 | | -- **HTTP Server**: OpenAI-compatible server at `http://localhost:{PORT}/v1/chat/completions` |
107 | | -- **Python Module**: Import and use programmatically in your applications |
108 | | -- **ComfyUI Node**: Embed directly in ComfyUI workflows |
109 | | - |
110 | | -### Simple and Customizable |
111 | | -- **Environment Variables**: Secure API key management |
112 | | -- **Provider Management**: Easy enable/disable of providers |
113 | | -- **Custom Models**: Define your own model aliases and mappings |
114 | | -- **Unified Configuration**: Single [llms.json](https://github.com/ServiceStack/llms/blob/main/llms/llms.json) to configure all providers and models |
115 | | - - Custom model aliases and mappings |
116 | | - - Flexible chat templates |
117 | | - - Environment variable support |
118 | | - |
119 | | -[Learn more about Configuration →](docs/configuration) |
120 | | - |
121 | | -### 🌐 OpenAI-Compatible API |
122 | | - |
123 | | -- Drop-in replacement for OpenAI API |
124 | | -- Works with any OpenAI-compatible client |
125 | | -- Streaming support |
126 | | -- Custom parameters |
127 | | - |
128 | | -### 📊 Analytics & Monitoring |
129 | | - |
130 | | -Track costs, usage, and performance across all providers. |
131 | | - |
132 | | -- **Cost Tracking**: Per-message, per-thread, and monthly cost analytics |
133 | | -- **Token Metrics**: Input/output token tracking |
134 | | -- **Activity Logs**: Detailed request history |
135 | | -- **Provider Stats**: Response times and reliability metrics |
136 | | - |
137 | | -[Learn more about Analytics →](/docs/features/analytics) |
138 | | - |
139 | | -### 🔄 Intelligent Provider Routing |
140 | | - |
141 | | -Automatic failover and cost optimization across providers. |
142 | | - |
143 | | -- **Multi-Provider**: Support for 10+ provider types with 160+ models |
144 | | -- **Auto-Failover**: If one provider fails, automatically try the next |
145 | | -- **Cost Optimization**: Route to free/cheap providers first |
146 | | -- **Model Mapping**: Unified model names across providers |
147 | | - |
148 | | -[Learn more about Providers →](/docs/features/providers) |
149 | | - |
150 | | -### 🔒 Security |
151 | | - |
152 | | -- Optional GitHub OAuth authentication |
153 | | -- User access restrictions |
154 | | - |
155 | | -[Learn more about GitHub OAuth →](/docs/deployment/github-oauth) |
156 | | - |
157 | | ---- |
158 | | - |
159 | | -## Links |
160 | | - |
161 | | -- 📚 [GitHub Repository](https://github.com/ServiceStack/llms) |
162 | | -- 📦 [PyPI Package](https://pypi.org/project/llms-py/) |
163 | | -- 🔧 [Source Code](https://github.com/ServiceStack/llms) |
| 98 | +<ShellCommand>docker pull ghcr.io/servicestack/llms:latest</ShellCommand> |
0 commit comments