feat: Add production inference service, API keys, A/B testing, and scheduled retraining

[marevol]

Summary

Add a complete production deployment stack enabling real-time recommendation serving, API key authentication, A/B testing, and automated model retraining.

Changes Made

Inference Service (inference/)

• New FastAPI service for real-time recommendation predictions
• Thread-safe LRU model cache with configurable capacity
• HMAC-SHA256 signature verification for model integrity
• Redis Pub/Sub listener for hot-swapping deployed models
• Rate limiting per API key
• Standalone inference-only deployment via compose-inference.yaml

API Key Authentication (backend/.../authentication.py)

• PBKDF2-based API key auth compatible between Django and FastAPI
• Scoped keys (predict, manage) with expiration support
• X-API-Key header authentication for both management and inference APIs
• Management command create_api_key for CLI key provisioning

Deployment Slots & A/B Testing

• Deployment slots with weighted routing for gradual model rollouts
• A/B testing framework with conversion event tracking
• Statistical analysis endpoints (chi-squared test) for experiment evaluation
• Project-level inference routing through weighted slot selection

Scheduled Retraining

• Celery Beat integration with django-celery-beat for cron-based scheduling
• Retraining schedule model with configurable parameters
• Automatic model deployment after successful retraining
• Run history tracking with status and metrics

Frontend

• ApiKeyPage: Create, view, revoke API keys with scope/expiration management
• DeploymentSlotPage: Manage model deployment slots with weight configuration
• ABTestListPage / ABTestDetailPage: Create experiments, monitor results, view statistical significance
• RetrainingSchedulePage: Configure and monitor automated retraining schedules
• Updated Dashboard with production feature quick-access cards
• Navigation updates in MainLayout

Infrastructure

• compose.yaml: Added beat and inference services
• compose-inference.yaml: Standalone inference-only deployment
• nginx.conf / nginx-inference.conf: Proxy routing for inference endpoints
• Helm chart: inference Deployment, HPA, Service, Beat scheduler, NetworkPolicy updates

Documentation

• docs/guides/inference-api.md: Inference API usage guide
• docs/guides/api-keys.md: API key management guide
• docs/guides/ab-testing.md: A/B testing guide
• docs/guides/retraining.md: Scheduled retraining guide
• docs/guides/standalone-inference.md: Inference-only deployment guide
• Updated deployment docs (docker-compose, kubernetes, aws, gcp, env vars)

Testing

• Backend unit tests pass (uv run pytest recotem/tests/ -v)
• Frontend type-check passes (npm run type-check)
• Docker Compose full stack starts successfully (docker compose up --build)
• Inference-only deployment works (docker compose -f compose-inference.yaml up)
• API key creation and authentication flow works end-to-end
• Model deployment and inference prediction returns correct results
• A/B test creation and conversion tracking works
• Retraining schedule triggers and completes successfully

Breaking Changes

• New environment variables required for production: MODEL_EVENTS_REDIS_URL, CELERY_BEAT_SCHEDULER
• New Django migration 0003_production_features must be applied
• Inference service requires its own INFERENCE_DATABASE_URL and INFERENCE_SECRET_KEY

Additional Notes

• The pickle_signing_core.py module extracts Django-independent signing logic for reuse in the FastAPI inference service
• API key hashing uses Django's PBKDF2 implementation; the inference service includes a compatible verifier
• The inference service is stateless and horizontally scalable (HPA configured in Helm)
• Model hot-swap uses Redis Pub/Sub on channel model_events to notify inference pods of deployments