Add RAG MCP Server with document search and management UI

[crab182]

Summary

This PR introduces a complete self-hosted RAG (Retrieval Augmented Generation) system with MCP (Model Context Protocol) server support, designed for deployment on Unraid. The system enables cloud-based LLMs to search local documents via semantic similarity without exposing documents to external services.

Key Changes

Backend (FastAPI + ChromaDB)

• RAG Engine (backend/app/services/rag_engine.py): Semantic document search using sentence-transformers and ChromaDB with configurable chunking and embedding
• Document Management (backend/app/routers/documents.py): Upload, query, and manage documents across multiple collections with support for PDF, DOCX, XLSX, and text formats
• SMB Browser (backend/app/routers/smb.py, backend/app/services/smb_browser.py): Browse and ingest documents from LAN SMB shares with recursive directory support
• Authentication (backend/app/services/auth.py): API key generation, hashing, and validation for secure access control
• Document Parser (backend/app/services/document_parser.py): Multi-format file parsing (PDF, DOCX, XLSX, code, config files, etc.)

MCP Server (SSE Transport)

• MCP Protocol Implementation (mcp_server/server.py): Server-Sent Events (SSE) based MCP server providing tools for:
  • search_documents: Semantic search with relevance scoring
  • list_collections: Browse available document collections
  • list_documents: List documents in a collection
  • get_server_status: Server health and statistics
• API key authentication via Authorization header
• Session management for SSE connections

Frontend (React + Vite)

• Dashboard (frontend/src/pages/Dashboard.jsx): System overview with document counts and collection statistics
• Documents (frontend/src/pages/Documents.jsx): Upload files, manage collections, reindex documents
• Search (frontend/src/pages/Search.jsx): Semantic search interface with result display
• SMB Browser (frontend/src/pages/SMBBrowser.jsx): Browse SMB shares and ingest documents with path navigation
• API Keys (frontend/src/pages/APIKeys.jsx): Create, revoke, and manage API keys for MCP access
• MCP Config (frontend/src/pages/MCPConfig.jsx): Enable/disable MCP server and view connection details
• Styling (frontend/src/index.css): Dark theme with comprehensive component styles (cards, forms, tables, badges, etc.)

Infrastructure

• Docker Compose (docker-compose.yml): Three-service orchestration (backend, MCP server, frontend)
• Deployment Script (deploy.sh): Automated setup with health checks and service validation
• Configuration (backend/app/config.py): Environment-based settings with persistent JSON config storage
• Nginx (frontend/nginx.conf): Reverse proxy for API requests and SPA routing

Notable Implementation Details

• Semantic Search: Uses all-MiniLM-L6-v2 embedding model by default with configurable alternatives
• Document Chunking: Configurable chunk size (512 tokens) with overlap (64 tokens) for context preservation
• Multi-Collection Support: Organize documents into separate collections with independent indexing
• SMB Integration: Native SMB/CIFS support for accessing Windows shares on LAN without copying files
• API Key Security: Keys are hashed with SHA-256 and stored in persistent config; raw keys shown only once
• SSE Transport: MCP server uses Server-Sent Events for streaming responses, compatible with cloud LLM platforms
• Persistent Storage: ChromaDB vector store and document metadata persisted to /app/data/ volumes

Deployment

The system is containerized and ready for Unraid deployment:

• Backend API: Port 8900
• MCP Server: Port 8901
• Web UI: Port 8902

All services include health checks and automatic restart policies.

https://claude.ai/code/session_01ByAJeYptU8ZosBaBVw1rbd