feat: Add production inference service, API keys, A/B testing, and scheduled retraining

.gitignore

@@ -16,6 +16,7 @@ __pycache__/
 *.pyo
 .python-version
 .venv/
+venv/
 .coverage
 **/coverage.xml
 .ruff_cache/

CLAUDE.md

@@ -1,16 +1,23 @@
 # Recotem
 
-Recommender system builder — Docker-first web app for tuning, training, and previewing recommendation models via UI.
+Recommender system builder — Docker-first web app for tuning, training, deploying, and monitoring recommendation models via UI and REST API.
 
 ## Architecture
 
-**5 Docker services** (see `compose.yaml`):
+**7 Docker services** (see `compose.yaml`):
 - `db` — PostgreSQL 17
-- `redis` — Redis 7 (broker db0, channels db1, cache db2)
+- `redis` — Redis 7 (broker db0, channels db1, cache db2, model events db3)
 - `backend` — Django 5.1 + DRF + Channels, served by Daphne (ASGI)
 - `worker` — Celery (same Docker image as backend)
+- `beat` — Celery Beat for scheduled retraining (same Docker image as backend)
+- `inference` — FastAPI service for real-time recommendations (separate image)
 - `proxy` — Nginx + Vue 3 SPA (built in `proxy.dockerfile`)
 
+**Inference-only deployment** (see `compose-inference.yaml`):
+- `db` — PostgreSQL 17
+- `inference` — FastAPI service
+- `proxy` — Nginx (inference routes only, via `nginx-inference.conf`)
+
 **ML stack**: irspack 0.4.0 + Optuna for hyperparameter tuning.
 
 ## Directory Layout
@@ -26,29 +33,44 @@ backend/
       urls.py             # Root URL conf: /api/v1/, /admin/, /ws/
       asgi.py / celery.py
       api/
-        models/           # Django models (Project, TrainingData, TrainedModel, etc.)
+        authentication.py # API key auth (X-API-Key header)
+        models/           # Django models (Project, TrainingData, TrainedModel, ApiKey, etc.)
         views/            # DRF ViewSets + mixins (OwnedResourceMixin, CreatedByResourceMixin)
-        serializers/      # DRF serializers
-        services/         # Business logic (model_service, training_service, tuning_service, pickle_signing)
-        tasks.py          # Celery tasks (start_tuning_job, task_train_recommender, run_search)
+        serializers/      # DRF serializers (api_key, retraining, deployment, ab_test, events)
+        services/         # Business logic (model, training, tuning, pickle_signing, scheduling, ab_testing)
+        tasks.py          # Celery tasks (tuning, training, scheduled retraining)
         consumers.py      # WebSocket consumers (Django Channels)
         urls.py           # API router registration
     tests/                # pytest + pytest-django
       conftest.py         # MovieLens100K fixtures
+inference/
+  Dockerfile              # Multi-stage FastAPI build
+  pyproject.toml
+  inference/
+    main.py               # FastAPI app with lifespan + rate limiting
+    config.py             # Pydantic Settings
+    auth.py               # API key verification (Django PBKDF2 compatible)
+    model_loader.py       # Thread-safe LRU model cache
+    hot_swap.py           # Redis Pub/Sub listener for model updates
+    models.py             # SQLAlchemy read-only models
+    routes/               # predict, project (A/B routing), health
 frontend/
   package.json            # Vue 3.5 + Vite 6 + PrimeVue 4 + Tailwind CSS 4 + Pinia + TanStack Query
   src/
-    pages/                # Page components (Login, Dashboard, Data*, Tuning*, Model*)
+    pages/                # Page components (Login, Dashboard, Data*, Tuning*, Model*, ApiKey*, ABTest*, etc.)
     layouts/              # MainLayout, AuthLayout, ProjectLayout
     stores/               # Pinia stores (auth, project, notification)
     composables/          # useWebSocket, useJobStatus, useAbortOnUnmount, useNotification
-    api/                  # API client (ofetch)
-    types/                # TypeScript types
+    api/                  # API client (ofetch) + production.ts
+    types/                # TypeScript types + production.ts
     utils/                # format.ts (formatDate, formatFileSize, formatScore)
     router/index.ts       # Vue Router with auth guards
-helm/recotem/             # Helm chart (ServiceAccount, PDB, NetworkPolicy)
+helm/recotem/             # Helm chart (ServiceAccount, PDB, NetworkPolicy, HPA)
 envs/                     # Environment files (dev.env, production.env)
-nginx.conf                # Proxy config: SPA + /api/ + /ws/ + /admin/ + /static/
+nginx.conf                # Proxy config: SPA + /api/ + /ws/ + /admin/ + /inference/ + /static/
+docs/
+  guides/                 # Feature guides (inference-api, api-keys, retraining, ab-testing)
+  deployment/             # Deployment guides (docker-compose, kubernetes, aws, gcp, env vars)
 ```
 
 ## Development Setup
@@ -69,6 +91,10 @@ uv run daphne recotem.asgi:application -b 0.0.0.0 -p 8000
 cd backend/recotem
 uv run celery -A recotem worker --loglevel=INFO
 
+# Celery beat (separate terminal, for scheduled retraining)
+cd backend/recotem
+uv run celery -A recotem beat --loglevel=INFO --scheduler django_celery_beat.schedulers:DatabaseScheduler
+
 # Frontend
 cd frontend
 npm install
@@ -102,6 +128,8 @@ docker compose up --build
 
 ## API Structure
 
+### Management API
+
 Base path: `/api/v1/` (backward compat at `/api/`)
 
 | Endpoint | ViewSet |
@@ -115,15 +143,35 @@ Base path: `/api/v1/` (backward compat at `/api/`)
 | `model_configuration/` | ModelConfigurationViewset |
 | `trained_model/` | TrainedModelViewset |
 | `task_log/` | TaskLogViewSet |
+| `api_keys/` | ApiKeyViewSet |
+| `retraining_schedule/` | RetrainingScheduleViewSet |
+| `retraining_run/` | RetrainingRunViewSet |
+| `deployment_slot/` | DeploymentSlotViewSet |
+| `ab_test/` | ABTestViewSet |
+| `conversion_event/` | ConversionEventViewSet |
 | `ping/` | PingView |
 | `project_summary/<id>/` | ProjectSummaryView |
 | `auth/login/` | dj-rest-auth LoginView |
 | `schema/` | drf-spectacular |
 
-Auth: JWT via dj-rest-auth + simplejwt. Tokens stored in localStorage.
+Auth: JWT via dj-rest-auth + simplejwt, or API key via `X-API-Key` header.
 
 WebSocket: `/ws/job/{id}/` for job status updates.
 
+### Inference API
+
+Base path: `/inference/` (proxied to FastAPI service on port 8081)
+
+| Endpoint | Description |
+|---|---|
+| `POST /inference/predict/{model_id}` | Single-user recommendations |
+| `POST /inference/predict/{model_id}/batch` | Multi-user batch recommendations |
+| `POST /inference/predict/project/{project_id}` | Project-level with A/B slot routing |
+| `GET /inference/health` | Health check |
+| `GET /inference/models` | List loaded models |
+
+Auth: API key with `predict` scope via `X-API-Key` header.
+
 ## Data Flow
 
 1. Create Project (user/item/time column definitions)
@@ -132,15 +180,18 @@ WebSocket: `/ws/job/{id}/` for job status updates.
 4. Create ParameterTuningJob → Celery task runs Optuna + irspack
 5. Best ModelConfiguration saved automatically
 6. Train model (auto or manual) → TrainedModel with HMAC-signed serialized file
-7. Preview recommendations from trained model
+7. Create API key with `predict` scope
+8. Create DeploymentSlot(s) pointing to trained model(s)
+9. Call inference API → weighted slot selection → real-time recommendations
+10. Record ConversionEvents → analyze A/B test results
 
 ## Conventions
 
 - **Python**: 3.12, uv for dependency management, Ruff for linting/formatting (line-length 88, rules: E/F/I/W/UP/B/SIM)
 - **Frontend**: TypeScript strict, Vue 3 Composition API, PrimeVue components, Tailwind CSS 4
 - **ViewSets**: Use `OwnedResourceMixin` / `CreatedByResourceMixin` from `views/mixins.py` for ownership filtering
 - **Services**: Business logic in `api/services/`, not in views
-- **Model security**: HMAC-SHA256 signing via `pickle_signing.py` — models signed on save, verified on load
+- **Model security**: HMAC-SHA256 signing via `pickle_signing_core.py` (Django-independent) — models signed on save, verified on load
 - **Uniqueness**: Project.name is per-owner, ModelConfiguration.name is per-training-data
 - **Backend user**: runs as `appuser:1000` in Docker
 - **Proxy**: nginx on port 8000 (non-root)
@@ -153,13 +204,18 @@ Core (see `envs/.env.example`):
 - `DATABASE_URL` — PostgreSQL connection string
 - `CELERY_BROKER_URL` — Redis URL for Celery (db 0)
 - `CACHE_REDIS_URL` — Redis URL for cache (db 2)
+- `MODEL_EVENTS_REDIS_URL` — Redis URL for model event Pub/Sub (db 3)
 - `SECRET_KEY` — Django secret (must change for production)
 - `DEBUG` — true/false
 - `DEFAULT_ADMIN_PASSWORD` — Initial admin password
 - `REDIS_PASSWORD` — Optional Redis auth
 - `ACCESS_TOKEN_LIFETIME` — JWT access token TTL in seconds (default 300)
 - `RECOTEM_STORAGE_TYPE` — Empty for local, "S3" for S3 storage
 - `CELERY_TASK_TIME_LIMIT` — Task timeout in seconds (default 3600)
+- `CELERY_BEAT_SCHEDULER` — `django_celery_beat.schedulers:DatabaseScheduler`
+- `INFERENCE_MAX_LOADED_MODELS` — Max models in inference LRU cache (default 10)
+- `INFERENCE_RATE_LIMIT` — Inference rate limit per API key (default 100/minute)
+- `INFERENCE_PRELOAD_MODEL_IDS` — Comma-separated model IDs to pre-load on startup
 - `LOG_LEVEL`, `DJANGO_LOG_LEVEL`, `CELERY_LOG_LEVEL` — Logging levels
 
 ## CI/CD

CONTRIBUTING.md

@@ -11,39 +11,45 @@ Recotem is a recommendation system platform built with:
 │                     Clients                          │
 │              (Browser / API consumers)               │
 └────────────────────────┬─────────────────────────────┘
-                         │
+                         │ X-API-Key or JWT
                    ┌─────▼─────┐
                    │   nginx   │ :8000
-                   │  (proxy)  │ Static SPA + reverse proxy
+                   │  (proxy)  │ SPA + reverse proxy
                    └──┬────┬───┘
                       │    │
-          ┌───────────▼┐  ┌▼───────────┐
-          │  Frontend  │  │  Backend   │
-          │  Vue 3 SPA │  │  Django 5  │
-          │  (static)  │  │  (daphne)  │
-          └────────────┘  └──┬────┬────┘
-                             │    │
-                    ┌────────▼┐  ┌▼────────┐
-                    │ Celery  │  │PostgreSQL│
-                    │ Worker  │  │   :5432  │
-                    └────┬────┘  └──────────┘
-                         │
-                    ┌────▼────┐
-                    │  Redis  │
-                    │ :6379   │
-                    │ db0=broker, db1=channels, db2=cache
-                    └─────────┘
+          ┌───────────▼┐  ┌▼───────────┐  ┌───────────┐
+          │  Frontend  │  │  Backend   │  │ Inference │
+          │  Vue 3 SPA │  │  Django 5  │  │ FastAPI   │
+          │  (static)  │  │  (daphne)  │  │ :8081     │
+          └────────────┘  └──┬──┬──┬───┘  └──┬────┬───┘
+                             │  │  │         │    │ read-only
+                    ┌────────▼┐ │ ┌▼─────────▼┐  │
+                    │ Celery  │ │ │ PostgreSQL │  │
+                    │ Worker  │ │ │   :5432    │  │
+                    └────┬────┘ │ └────────────┘  │
+                         │     │                  │
+                    ┌────▼─────▼──────────────────▼┐
+                    │          Redis :6379          │
+                    │ db0=broker  db1=channels      │
+                    │ db2=cache   db3=model events  │
+                    └──────────┬───────────────────┘
+                          ┌────▼────┐
+                          │ Celery  │
+                          │  Beat   │
+                          └─────────┘
 ```
 
 ### Services (Docker Compose)
 
 | Service | Image/Build | Role |
 |---------|------------|------|
 | `db` | postgres:17-alpine | Primary data store |
-| `redis` | redis:7-alpine | Celery broker (db 0), Channels (db 1), Cache (db 2) |
+| `redis` | redis:7-alpine | Broker (db 0), Channels (db 1), Cache (db 2), Model events (db 3) |
 | `backend` | `backend/Dockerfile` | Django ASGI server (daphne), REST API + WebSocket |
 | `worker` | `backend/Dockerfile` | Celery worker for async tuning/training tasks |
-| `proxy` | `proxy.dockerfile` | nginx serving Vue SPA + reverse proxy to backend |
+| `beat` | `backend/Dockerfile` | Celery Beat for scheduled retraining |
+| `inference` | `inference/Dockerfile` | FastAPI inference service for real-time recommendations |
+| `proxy` | `proxy.dockerfile` | nginx serving Vue SPA + reverse proxy to backend + inference |
 
 ### Data Flow
 
@@ -122,18 +128,29 @@ recotem/
     recotem/              # Django project
       recotem/
         api/
-          consumers.py    # WebSocket consumers
-          models/         # Django models
-          serializers.py  # DRF serializers
-          services/       # Business logic (training, model serving)
-          tasks.py        # Celery tasks (tuning, training)
-          views/          # DRF viewsets
+          authentication.py  # API key authentication
+          consumers.py       # WebSocket consumers
+          models/            # Django models
+          serializers/       # DRF serializers (api_key, retraining, ab_test, etc.)
+          services/          # Business logic (training, model, scheduling, A/B testing)
+          tasks.py           # Celery tasks (tuning, training, retraining)
+          views/             # DRF viewsets
         settings.py
       manage.py
       tests/              # pytest tests
     pyproject.toml        # Dependencies + tool config (uv)
     uv.lock               # Locked dependency versions
-    Dockerfile            # Multi-stage (backend + worker)
+    Dockerfile            # Multi-stage (backend + worker + beat)
+  inference/              # Standalone FastAPI inference service
+    inference/
+      routes/             # Prediction, project-level, health endpoints
+      auth.py             # API key verification (Django-compatible)
+      config.py           # Pydantic Settings
+      model_loader.py     # LRU model cache with hot-swap
+      hot_swap.py         # Redis Pub/Sub listener
+      models.py           # SQLAlchemy models (read-only)
+    Dockerfile
+    pyproject.toml
   frontend/
     src/
       api/                # API client (ofetch), generated types
@@ -146,12 +163,14 @@ recotem/
       stores/             # Pinia stores (auth, project)
       types/              # TypeScript type definitions + WebSocket types
     e2e/                  # Playwright E2E tests
-  compose.yaml            # Production Docker Compose (5 services)
+  compose.yaml            # Production Docker Compose (7 services)
   compose-dev.yaml        # Development Docker Compose (db + redis)
   proxy.dockerfile        # nginx + frontend SPA build
-  nginx.conf              # Unified SPA + API + WS + Admin proxy
+  nginx.conf              # SPA + API + WS + Admin + Inference proxy
   helm/recotem/           # Helm chart for Kubernetes deployment
-  docs/deployment/        # Deployment documentation
+  docs/
+    guides/               # Feature guides (inference API, API keys, etc.)
+    deployment/           # Deployment documentation
 ```
 
 ## API Development Guide