Merge pull request #41 from devtron-labs/doc-rag

misc: Doc rag

Author: prakash100198

.dockerignore

@@ -0,0 +1,118 @@
+# Git
+.git
+.gitignore
+.gitattributes
+
+# Documentation
+*.md
+!README.md
+docs/
+mcp-docs-server/
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Build artifacts
+*.o
+*.a
+*.so
+*.exe
+*.test
+*.out
+vendor/
+
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+env/
+venv/
+ENV/
+.venv
+pip-log.txt
+pip-delete-this-directory.txt
+.pytest_cache/
+.coverage
+htmlcov/
+*.egg-info/
+dist/
+build/
+
+# Data directories (will be mounted as volumes)
+/data/
+devtron-docs/
+chroma_db/
+
+# Logs
+*.log
+logs/
+
+# Test files
+*_test.go
+test/
+tests/
+
+# CI/CD
+.github/
+.gitlab-ci.yml
+.travis.yml
+
+# Docker
+docker-compose*.yml
+Dockerfile.dev
+.dockerignore
+
+# Temporary files
+tmp/
+temp/
+*.tmp
+*.bak
+*.backup
+
+# Scripts (not needed in image)
+scripts/dev/
+scripts/test/
+start-integrated.sh
+
+# Documentation files (exclude all .md except README)
+STARTUP_FIX.md
+INDEXING_API_GUIDE.md
+INDEXING_CHANGES_SUMMARY.md
+CHANGES_COMPLETE.md
+DATABASE_CONNECTION_LOGS.md
+DOCKERFILE_OPTIMIZATION_GUIDE.md
+DOCKER_OPTIMIZATION_COMPLETE.md
+OPTIMIZATION_SUMMARY.md
+QUICK_START.md
+
+# Node modules (if any)
+node_modules/
+package-lock.json
+yarn.lock
+
+# Large binary files
+*.tar
+*.tar.gz
+*.zip
+*.rar
+
+# Database files
+*.db
+*.sqlite
+*.sqlite3
+
+# Cache directories
+.cache/
+.npm/
+.yarn/
+

.vscode/settings.json

@@ -0,0 +1,2 @@
+{
+}

GET_STARTED.md

@@ -0,0 +1,273 @@
+# 🚀 Get Started - Your Next Steps
+
+Welcome! This guide will help you get started with the Devtron Documentation MCP Server.
+
+## ✅ What You Have
+
+A complete, production-ready MCP server that provides semantic search over Devtron documentation:
+
+- ✅ **16 files** created and configured
+- ✅ **~2,570 lines** of code and documentation
+- ✅ **4 MCP tools** ready to use
+- ✅ **Free tier** AWS Bedrock Titan embeddings
+- ✅ **Comprehensive documentation** for all use cases
+
+## 📋 Quick Checklist
+
+### Step 1: Understand the Project (5 minutes)
+
+Read these files in order:
+
+1. **[README.md](README.md)** - Project overview
+2. **[PROJECT_OVERVIEW.md](PROJECT_OVERVIEW.md)** - Central API details
+3. **[mcp-docs-server/SOLUTION_SUMMARY.md](mcp-docs-server/SOLUTION_SUMMARY.md)** - MCP server architecture
+
+### Step 2: Set Up MCP Server (5 minutes)
+
+```bash
+# Navigate to MCP server directory
+cd mcp-docs-server
+
+# Run automated setup
+./setup.sh
+
+# This will:
+# ✅ Check Python version
+# ✅ Create virtual environment
+# ✅ Install dependencies
+# ✅ Create .env file
+# ✅ Create directories
+```
+
+### Step 3: Configure AWS (2 minutes)
+
+**Option A: Use AWS CLI** (Recommended)
+```bash
+aws configure
+# Enter your AWS credentials when prompted
+```
+
+**Option B: Edit .env file**
+```bash
+nano .env
+# Add:
+# AWS_ACCESS_KEY_ID=your_key
+# AWS_SECRET_ACCESS_KEY=your_secret
+# AWS_REGION=us-east-1
+```
+
+**Enable Bedrock Titan** (One-time, 30 seconds):
+1. Go to: https://console.aws.amazon.com/bedrock/
+2. Click "Model access" → "Manage model access"
+3. Check "Titan Embeddings G1 - Text"
+4. Click "Request model access"
+5. Wait for approval (usually instant)
+
+### Step 4: Test Everything (2 minutes)
+
+```bash
+# Activate virtual environment
+source venv/bin/activate
+
+# Run test suite
+python test_server.py
+```
+
+Expected output:
+```
+✅ AWS Bedrock test passed
+✅ Document processor test passed
+✅ Vector store test passed
+✅ All tests completed!
+```
+
+### Step 5: Run the Server (1 minute)
+
+```bash
+python server.py
+```
+
+You should see:
+```
+INFO - Initializing Devtron Documentation MCP Server...
+INFO - Cloning repository...
+INFO - Indexing documentation...
+INFO - Server initialization complete
+```
+
+### Step 6: Integrate with Your Chatbot (10 minutes)
+
+Follow the integration guide:
+
+**[mcp-docs-server/INTEGRATION_GUIDE.md](mcp-docs-server/INTEGRATION_GUIDE.md)**
+
+Quick example:
+```python
+from mcp import ClientSession
+from mcp.client.stdio import stdio_client
+
+async def search_docs(query):
+    async with stdio_client("python", ["server.py"]) as (read, write):
+        async with ClientSession(read, write) as session:
+            await session.initialize()
+            result = await session.call_tool(
+                "search_docs",
+                {"query": query, "max_results": 3}
+            )
+            return result[0].text
+```
+
+## 📚 Documentation Map
+
+### For Quick Start
+- **[mcp-docs-server/QUICKSTART.md](mcp-docs-server/QUICKSTART.md)** - 5-minute setup guide
+
+### For Understanding
+- **[mcp-docs-server/SOLUTION_SUMMARY.md](mcp-docs-server/SOLUTION_SUMMARY.md)** - Architecture and design
+- **[mcp-docs-server/ALTERNATIVES_COMPARISON.md](mcp-docs-server/ALTERNATIVES_COMPARISON.md)** - Why this solution?
+
+### For Integration
+- **[mcp-docs-server/INTEGRATION_GUIDE.md](mcp-docs-server/INTEGRATION_GUIDE.md)** - Chatbot integration
+- **[mcp-docs-server/README.md](mcp-docs-server/README.md)** - Complete user guide
+
+### For Reference
+- **[mcp-docs-server/FILES_OVERVIEW.md](mcp-docs-server/FILES_OVERVIEW.md)** - File structure
+- **[IMPLEMENTATION_COMPLETE.md](IMPLEMENTATION_COMPLETE.md)** - Implementation summary
+
+## 🎯 Common Use Cases
+
+### Use Case 1: Answer User Questions
+```python
+# User asks: "How do I deploy an application?"
+context = await search_docs("deploy application")
+# Returns relevant documentation chunks
+# Use in your chatbot prompt
+```
+
+### Use Case 2: Get Specific Documentation
+```python
+# Get a specific doc file
+result = await session.call_tool(
+    "get_doc_by_path",
+    {"path": "docs/user-guide/deploying-application.md"}
+)
+```
+
+### Use Case 3: Keep Docs Updated
+```python
+# Manually sync documentation
+result = await session.call_tool("sync_docs", {})
+# Or set up a cron job to run periodically
+```
+
+### Use Case 4: Browse Available Docs
+```python
+# List all documentation sections
+result = await session.call_tool(
+    "list_doc_sections",
+    {"filter": "user-guide"}
+)
+```
+
+## 🔧 Troubleshooting
+
+### Problem: AWS credentials not found
+**Solution**: Run `aws configure` or edit `.env` file
+
+### Problem: Bedrock access denied
+**Solution**: Enable Titan Embeddings in AWS Console (see Step 3)
+
+### Problem: Git clone fails
+**Solution**: Check internet connection, verify GitHub URL
+
+### Problem: ChromaDB error
+**Solution**: Delete `chroma_db/` directory and restart
+
+### Problem: Slow initial startup
+**Solution**: Normal! First run indexes all docs (~2-5 minutes)
+
+## 📊 What Happens Next?
+
+### First Run (2-5 minutes)
+1. Clones Devtron docs from GitHub
+2. Parses all markdown files
+3. Chunks content by headers
+4. Generates embeddings (AWS Bedrock)
+5. Stores in ChromaDB
+6. Ready to serve queries!
+
+### Subsequent Runs (<10 seconds)
+1. Loads existing ChromaDB index
+2. Ready to serve queries immediately!
+
+### When Docs Update
+1. Run `sync_docs` tool
+2. Git pulls latest changes
+3. Only re-indexes changed files
+4. Updates ChromaDB incrementally
+
+## 💡 Pro Tips
+
+1. **Cache Frequent Queries**: Implement caching in your chatbot
+2. **Limit Results**: Use `max_results=3` for faster responses
+3. **Schedule Syncs**: Set up cron job for `sync_docs`
+4. **Monitor Logs**: Check for errors and performance
+5. **Use Docker**: For production deployment
+
+## 🎓 Learning Path
+
+### Day 1: Setup & Test
+- ✅ Run setup script
+- ✅ Configure AWS
+- ✅ Run tests
+- ✅ Start server
+
+### Day 2: Integration
+- ✅ Read integration guide
+- ✅ Implement basic search
+- ✅ Test with sample queries
+
+### Day 3: Production
+- ✅ Set up Docker
+- ✅ Configure monitoring
+- ✅ Schedule doc syncs
+- ✅ Deploy to production
+
+## 📞 Need Help?
+
+1. **Check Documentation**: See files listed above
+2. **Run Tests**: `python test_server.py`
+3. **Check Logs**: Review error messages
+4. **Verify AWS**: Ensure credentials and Bedrock access
+
+## 🎉 Success Criteria
+
+You'll know it's working when:
+- ✅ Tests pass without errors
+- ✅ Server starts and indexes docs
+- ✅ Search returns relevant results
+- ✅ Chatbot gets accurate context
+- ✅ Users get better answers!
+
+## 🚀 Ready to Start?
+
+```bash
+cd mcp-docs-server
+./setup.sh
+```
+
+Then follow the prompts!
+
+---
+
+**Next Steps**:
+1. ✅ Run setup: `./setup.sh`
+2. ✅ Configure AWS credentials
+3. ✅ Run tests: `python test_server.py`
+4. ✅ Start server: `python server.py`
+5. ✅ Integrate with chatbot
+
+**Questions?** Check the documentation files listed above.
+
+**Status**: ✅ Ready to use!
+