Skip to content

README.md

Latest commit

 

History

History
354 lines (262 loc) · 11.5 KB

README.md

File metadata and controls

354 lines (262 loc) · 11.5 KB
Raw

LiteLLM-Stack

Production-ready Docker stack for self-hosted AI infrastructure

License: MIT Docker LiteLLM Open WebUI PostgreSQL Nginx PRs Welcome

Features • Installation • Usage • Documentation • Contributing

📖 About

LiteLLM-Stack is a complete, production-ready Docker infrastructure for self-hosting AI services. It combines LiteLLM proxy, Open WebUI interface, PostgreSQL database, and SSL-secured Nginx reverse proxy into a single, easy-to-deploy stack.

Perfect for developers, researchers, and organizations who want full control over their AI infrastructure without vendor lock-in.

✨ Key Features

🚀 Unified AI Gateway

  • LiteLLM Proxy - Connect to 100+ LLM providers (OpenAI, Anthropic, Google, Azure, local models, and more) through a single unified API
  • Load Balancing - Distribute requests across multiple models and providers
  • Cost Tracking - Monitor usage and costs across all providers
  • Caching - Reduce costs with intelligent response caching

🎨 Modern Web Interface

  • Open WebUI - Beautiful, ChatGPT-like interface for interacting with AI models
  • Multi-Model Support - Switch between different models seamlessly
  • Conversation History - Save and manage your AI conversations
  • Team Collaboration - User authentication and role management

🔒 Enterprise-Grade Security

  • SSL/TLS Encryption - Automatic HTTPS with self-signed or custom certificates
  • Nginx Reverse Proxy - Professional-grade reverse proxy with security headers
  • Authentication - Built-in user authentication and API key management
  • Network Isolation - Docker network separation for enhanced security

📦 Easy Deployment

  • One-Command Setup - Get started in minutes with Docker Compose
  • Persistent Storage - PostgreSQL database for configuration and history
  • Health Checks - Automatic monitoring and recovery
  • Zero Configuration - Sensible defaults that work out of the box

🔧 Developer Friendly

  • OpenAI-Compatible API - Drop-in replacement for OpenAI API
  • Detailed Logging - Debug mode for troubleshooting
  • Extensible Configuration - YAML-based configuration for easy customization
  • Docker Native - Container-first architecture

📋 Prerequisites

  • Docker 20.10 or higher
  • Docker Compose 2.0 or higher
  • OpenSSL (for SSL certificate generation)
  • 2GB RAM minimum (4GB recommended)
  • 5GB disk space minimum

🚀 Installation

Quick Start (5 minutes)

# Clone the repository
git clone https://github.com/sieekuu/litellm-stack.git
cd litellm-stack

# Generate SSL certificates
cd scripts
chmod +x generate-ssl.sh
./generate-ssl.sh
cd ..

# Create environment file (optional)
cp .env.example config/litellm/.env
# Edit config/litellm/.env and add your API keys

# Start the stack
docker compose up -d

# Check status
docker compose ps

Accessing Services

After deployment, access your services at:

First-Time Setup

  1. Access Open WebUI at https://localhost:8443
  2. Accept the SSL warning (for self-signed certificates)
  3. Create an admin account (first user becomes admin)
  4. Configure API Keys in Settings → Connections
  5. Start chatting with your AI models!

💡 Usage

Adding API Keys

Method 1: Through Open WebUI (Recommended)

  1. Navigate to Settings → Connections
  2. Add your provider credentials (OpenAI, Anthropic, etc.)
  3. Test connection and start using models

Method 2: Via Configuration File

Edit config/litellm/config.yaml:

model_list:
  - model_name: gpt-4
    litellm_params:
      model: gpt-4
      api_key: os.environ/OPENAI_API_KEY

  - model_name: claude-3-sonnet
    litellm_params:
      model: claude-3-sonnet-20240229
      api_key: os.environ/ANTHROPIC_API_KEY

Add keys to config/litellm/.env:

OPENAI_API_KEY=sk-your-key-here
ANTHROPIC_API_KEY=sk-ant-your-key-here

Restart services:

docker compose restart litellm

Using the API

The stack exposes an OpenAI-compatible API:

# Example: Chat completion
curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-1234" \
  -d '{
    "model": "gpt-4",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Managing Services

# View logs
docker compose logs -f

# Stop services
docker compose stop

# Start services
docker compose start

# Restart services
docker compose restart

# Remove everything (including volumes)
docker compose down -v

🏗️ Architecture

┌─────────────────────────────────────────────────┐
│                    Client                        │
│              (Browser / API Client)              │
└────────────────────┬────────────────────────────┘
                     │ HTTPS (8443) / HTTP (8080)
                     ▼
┌─────────────────────────────────────────────────┐
│                Nginx Reverse Proxy              │
│          (SSL Termination, Security)            │
└────────────────────┬────────────────────────────┘
                     │
          ┌──────────┴──────────┐
          ▼                     ▼
┌──────────────────┐  ┌──────────────────┐
│   Open WebUI     │  │   LiteLLM Proxy  │
│  (Port 8080)     │  │   (Port 4000)    │
└────────┬─────────┘  └────────┬─────────┘
         │                     │
         └──────────┬──────────┘
                    ▼
         ┌────────────────────┐
         │   PostgreSQL DB    │
         │   (Port 5432)      │
         └────────────────────┘

📚 Documentation

🔧 Configuration

Environment Variables

Key environment variables (in docker-compose.yml):

Variable Description Default
LITELLM_MASTER_KEY Master API key for LiteLLM sk-1z4bxV6OoK
DATABASE_URL PostgreSQL connection string Auto-configured
POSTGRES_PASSWORD PostgreSQL password bmFj5xF07m
WEBUI_AUTH Enable Open WebUI authentication true

Ports

Service Internal Port External Port Protocol
Nginx (HTTPS) 443 8443 HTTPS
Nginx (HTTP) 80 8080 HTTP
LiteLLM API 4000 4000 HTTP
PostgreSQL 5432 - Internal
Open WebUI 8080 - Internal

🔐 Security Considerations

For Development/Testing

  • Default SSL uses self-signed certificates
  • Default passwords should be changed
  • Suitable for local development and testing

For Production

  • Replace self-signed certificates with valid SSL certificates (Let's Encrypt)
  • Change all default passwords and API keys
  • Configure firewall rules
  • Enable additional security headers
  • Set up regular backups
  • Implement rate limiting
  • Use Docker secrets for sensitive data

🐛 Troubleshooting

Services won't start 
# Check Docker daemon is running
docker ps

# Check logs for errors
docker compose logs

# Ensure ports are not in use
netstat -tulpn | grep -E '8080|8443|4000'
SSL certificate warnings

This is normal for self-signed certificates. See SSL Setup Guide for:

  • How to accept the warning safely
  • How to install the certificate in your browser
  • How to use Let's Encrypt for production
Can't connect to models
  1. Verify API keys are correctly set in config/litellm/.env
  2. Check LiteLLM logs: docker compose logs litellm
  3. Test API key validity with the provider directly
  4. Ensure model names are correct in configuration
Database connection errors 
# Wait for PostgreSQL to be ready
docker compose logs postgres

# Check health status
docker compose ps

# Restart database
docker compose restart postgres

🤝 Contributing

We welcome contributions! Please see our Contributing Guidelines for details.

Ways to Contribute

  • 🐛 Report bugs
  • 💡 Suggest new features
  • 📝 Improve documentation
  • 🔧 Submit pull requests

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

Copyright © 2026 Krzysztof Siek

🙏 Acknowledgments

This project builds upon excellent open-source software:

📞 Support

⭐ Star History

If you find this project useful, please consider giving it a star! It helps others discover the project.

⬆ Back to Top

Made with ❤️ by Krzysztof Siek