Add configuration for MCP servers, updated documentation, created manual testing support

Author: blublinsky

.gitignore

@@ -180,6 +180,9 @@ openai_api_key.txt
 # Dev stuff
 dev/
 
+# Mock server certificates (auto-generated)
+dev-tools/mcp-mock-server/.certs/
+
 # VSCode
 .vscode/
 

README.md

@@ -294,9 +294,20 @@ user_data_collection:
 
 **Note**: The `run.yaml` configuration is currently an implementation detail. In the future, all configuration will be available directly from the lightspeed-core config.
 
+**Important**: Only MCP servers defined in the `lightspeed-stack.yaml` configuration are available to the agents. Tools configured in the llama-stack `run.yaml` are not accessible to lightspeed-core agents.
+
 #### Configuring MCP Servers
 
-MCP (Model Context Protocol) servers provide tools and capabilities to the AI agents. These are configured in the `mcp_servers` section of your `lightspeed-stack.yaml`:
+MCP (Model Context Protocol) servers provide tools and capabilities to the AI agents. These are configured in the `mcp_servers` section of your `lightspeed-stack.yaml`.
+
+**Basic Configuration Structure:**
+
+Each MCP server requires three fields:
+- `name`: Unique identifier for the MCP server
+- `provider_id`: MCP provider identification (typically `"model-context-protocol"`)
+- `url`: The endpoint where the MCP server is running
+
+**Minimal Example:**
 
 ```yaml
 mcp_servers:
@@ -306,24 +317,126 @@ mcp_servers:
   - name: "git-tools"
     provider_id: "model-context-protocol"
     url: "http://localhost:3001"
-  - name: "database-tools"
+```
+
+In addition to the basic configuration above, you can configure authentication headers for your MCP servers to securely communicate with services that require credentials.
+
+#### Configuring MCP Server Authentication
+
+Lightspeed Core Stack supports three methods for authenticating with MCP servers, each suited for different use cases:
+
+##### 1. Static Tokens from Files (Recommended for Service Credentials)
+
+Store authentication tokens in secret files and reference them in your configuration. This is ideal for API keys, service tokens, or any credentials that don't change per-user:
+
+```yaml
+mcp_servers:
+  - name: "api-service"
     provider_id: "model-context-protocol"
-    url: "http://localhost:3002"
+    url: "http://api-service:8080"
+    authorization_headers:
+      Authorization: "/var/secrets/api-token"      # Path to file containing token
+      X-API-Key: "/var/secrets/api-key"            # Multiple headers supported
 ```
 
-**Important**: Only MCP servers defined in the `lightspeed-stack.yaml` configuration are available to the agents. Tools configured in the llama-stack `run.yaml` are not accessible to lightspeed-core agents.
+The secret files should contain only the header value (tokens are automatically stripped of whitespace):
 
-#### Configuring MCP Headers
+```bash
+# /var/secrets/api-token
+Bearer sk-abc123def456...
 
-MCP headers allow you to pass authentication tokens, API keys, or other metadata to MCP servers. These are configured **per request** via the `MCP-HEADERS` HTTP header:
+# /var/secrets/api-key
+my-api-key-value
+```
+
+##### 2. Kubernetes Service Account Tokens (For K8s Deployments)
+
+Use the special `"kubernetes"` keyword to automatically use the authenticated user's Kubernetes token. This is perfect for MCP servers running in the same Kubernetes cluster:
+
+```yaml
+mcp_servers:
+  - name: "k8s-internal-service"
+    provider_id: "model-context-protocol"
+    url: "http://internal-mcp.default.svc.cluster.local:8080"
+    authorization_headers:
+      Authorization: "kubernetes"    # Uses user's k8s token from request auth
+```
+
+The user's Kubernetes token is extracted from the incoming request's `Authorization` header and forwarded to the MCP server.
+
+##### 3. Client-Provided Tokens (For Per-User Authentication)
+
+Use the special `"client"` keyword to allow clients to provide custom tokens per-request. This enables user-specific authentication:
+
+```yaml
+mcp_servers:
+  - name: "user-specific-service"
+    provider_id: "model-context-protocol"
+    url: "http://user-service:8080"
+    authorization_headers:
+      Authorization: "client"         # Token provided via MCP-HEADERS
+      X-User-Token: "client"          # Multiple client headers supported
+```
+
+Clients then provide tokens via the `MCP-HEADERS` HTTP header:
 
 ```bash
 curl -X POST "http://localhost:8080/v1/query" \
   -H "Content-Type: application/json" \
-  -H "MCP-HEADERS: {\"filesystem-tools\": {\"Authorization\": \"Bearer token123\"}}" \
-  -d '{"query": "List files in /tmp"}'
+  -H "MCP-HEADERS: {\"user-specific-service\": {\"Authorization\": \"Bearer user-token-123\", \"X-User-Token\": \"custom-value\"}}" \
+  -d '{"query": "Get my data"}'
 ```
 
+**Note**: The `MCP-HEADERS` dictionary is keyed by **server name** (not URL), matching the `name` field in your MCP server configuration.
+
+##### Combining Authentication Methods
+
+You can mix and match authentication methods across different MCP servers, and even combine multiple methods for a single server:
+
+```yaml
+mcp_servers:
+  # Static credentials for public API
+  - name: "weather-api"
+    provider_id: "model-context-protocol"
+    url: "http://weather-api:8080"
+    authorization_headers:
+      X-API-Key: "/var/secrets/weather-api-key"
+  
+  # Kubernetes auth for internal services
+  - name: "internal-db"
+    provider_id: "model-context-protocol"
+    url: "http://db-mcp.cluster.local:8080"
+    authorization_headers:
+      Authorization: "kubernetes"
+  
+  # Mixed: static API key + per-user token
+  - name: "multi-tenant-service"
+    provider_id: "model-context-protocol"
+    url: "http://multi-tenant:8080"
+    authorization_headers:
+      X-Service-Key: "/var/secrets/service-key"    # Static service credential
+      Authorization: "client"                       # User-specific token
+```
+
+##### Authentication Method Comparison
+
+| Method | Use Case | Configuration | Token Scope | Example |
+|--------|----------|---------------|-------------|---------|
+| **Static File** | Service tokens, API keys | File path in config | Global (all users) | `"/var/secrets/token"` |
+| **Kubernetes** | K8s service accounts | `"kubernetes"` keyword | Per-user (from auth) | `"kubernetes"` |
+| **Client** | User-specific tokens | `"client"` keyword + HTTP header | Per-request | `"client"` |
+
+##### Important: Automatic Server Skipping
+
+**If an MCP server has `authorization_headers` configured but the required tokens cannot be resolved at runtime, the server will be automatically skipped for that request.** This prevents failed authentication attempts to MCP servers.
+
+**Examples:**
+- A server with `Authorization: "kubernetes"` will be skipped if the user's request doesn't include a Kubernetes token
+- A server with `Authorization: "client"` will be skipped if no `MCP-HEADERS` are provided in the request
+- A server with multiple headers will be skipped if **any** required header cannot be resolved
+
+Skipped servers are logged as warnings. Check Lightspeed Core logs to see which servers were skipped and why.
+
 
 ### Llama Stack project and configuration
 
@@ -988,6 +1101,36 @@ The version X.Y.Z indicates:
 * Y is the minor version (backward-compatible), and
 * Z is the patch version (backward-compatible bug fix).
 
+# Development Tools
+
+Lightspeed Core Stack includes development utilities to help with local testing and debugging. These tools are located in the `dev-tools/` directory.
+
+## MCP Mock Server
+
+A lightweight mock MCP server for testing MCP integrations locally without requiring real MCP infrastructure.
+
+**Quick Start:**
+```bash
+# Start the mock server
+python dev-tools/mcp-mock-server/server.py
+
+# Configure Lightspeed Core Stack to use it
+# Add to lightspeed-stack.yaml:
+mcp_servers:
+  - name: "mock-test"
+    url: "http://localhost:3000"
+    authorization_headers:
+      Authorization: "/tmp/test-token"
+```
+
+**Features:**
+- Test authorization header configuration
+- Debug MCP connectivity issues
+- Inspect captured headers via debug endpoints
+- No external dependencies (pure Python stdlib)
+
+For detailed usage instructions, see [`dev-tools/mcp-mock-server/README.md`](dev-tools/mcp-mock-server/README.md).
+
 # Konflux
 
 The official image of Lightspeed Core Stack is built on [Konflux](https://konflux-ui.apps.kflux-prd-rh02.0fk9.p1.openshiftapps.com/ns/lightspeed-core-tenant/applications/lightspeed-stack).

dev-tools/MANUAL_TESTING.md

@@ -0,0 +1,163 @@
+# Manual Testing Guide for MCP Authorization
+
+This guide walks through testing the MCP server authorization feature using the mock MCP server.
+
+## Prerequisites
+
+### 1. Create Test Secret File
+
+```bash
+echo "Bearer test-secret-token-12345" > /tmp/lightspeed-mcp-test-token
+```
+
+### 2. Start Mock MCP Server
+
+```bash
+# Terminal 1: Start mock MCP server on HTTP port 9000
+python3 dev-tools/mcp-mock-server/server.py 9000
+```
+
+Verify the server starts and shows the HTTP endpoint.
+
+### 3. Install Library Mode Dependencies
+
+The test configuration uses Llama Stack in library mode, which requires additional dependencies:
+
+```bash
+uv pip install emoji langdetect aiosqlite pythainlp asyncpg nltk 'mcp>=1.23.0' matplotlib 'sqlalchemy[asyncio]' chardet scikit-learn faiss-cpu pillow 'datasets>=4.0.0' psycopg2-binary pandas pypdf pymongo redis tree_sitter requests
+```
+
+**Note:** These dependencies are needed for Llama Stack's inline providers (agents, vector_io, eval, etc.) to work in library mode. This is a one-time installation.
+
+### 4. Start Lightspeed Core
+
+```bash
+# Terminal 2: Start Lightspeed Core with test config (Llama Stack runs as library)
+# Make sure OPENAI_API_KEY is set first!
+export OPENAI_API_KEY="your-api-key-here"
+uv run src/lightspeed_stack.py --config dev-tools/test-configs/mcp-mock-test.yaml
+```
+
+Wait for Lightspeed Core to start (you should see "Application startup complete").
+
+**Note:** The test configuration uses Llama Stack in library mode with a dedicated test config (`dev-tools/test-configs/llama-stack-mcp-test.yaml`), so you don't need to start it separately!
+
+---
+
+## Test: All Three Authorization Types in One Request
+
+The test configuration defines **3 MCP servers**, which means **every query will contact all 3 servers** to discover their tools. This allows us to test all three authorization types in a single request.
+
+### Step 1: Make a Query Request
+
+```bash
+# Terminal 3: Make test query with all required headers
+curl -X POST http://localhost:8080/v1/streaming_query \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer my-k8s-token" \
+  -H 'MCP-HEADERS: {"mock-client-auth": {"Authorization": "Bearer my-client-token"}}' \
+  -d '{"query": "Test all MCP auth types"}'
+```
+
+**What This Tests:**
+- **`mock-file-auth`**: Uses static token from `/tmp/lightspeed-mcp-test-token`
+- **`mock-k8s-auth`**: Forwards the Kubernetes token from your `Authorization` header
+- **`mock-client-auth`**: Uses the client-provided token from `MCP-HEADERS`
+
+### Step 2: Verify All Three Auth Types Worked
+
+Check the mock server terminal output. You should see **6 requests total** (2 per server: initialize + list_tools):
+
+1. **First pair (mock-file-auth)**:
+   - `Authorization: Bearer test-secret-token-12345`
+   
+2. **Second pair (mock-k8s-auth)**:
+   - `Authorization: Bearer my-k8s-token`
+   
+3. **Third pair (mock-client-auth)**:
+   - `Authorization: Bearer my-client-token`
+
+Or check the debug endpoint:
+
+```bash
+curl http://localhost:9000/debug/requests | jq
+```
+
+### Expected Result
+
+The mock server should return unique tool names for each auth type:
+- `mock_tool_file` - from `mock-file-auth`
+- `mock_tool_k8s` - from `mock-k8s-auth`  
+- `mock_tool_client` - from `mock-client-auth`
+
+Check the Lightspeed Core logs, you should see:
+```
+DEBUG    Configured 3 MCP tools: ['mock-file-auth', 'mock-k8s-auth', 'mock-client-auth']
+```
+
+### Step 3: View Request History
+
+```bash
+curl http://localhost:9000/debug/requests | jq
+```
+
+This will show all 6 requests with their respective Authorization headers.
+
+---
+
+## Additional Test Scenarios
+
+### Test Without Client Headers
+
+Test what happens when you don't provide `MCP-HEADERS`:
+
+```bash
+curl -X POST http://localhost:8080/v1/streaming_query \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer my-k8s-token" \
+  -d '{"query": "Test without client headers"}'
+```
+
+**Expected Result:**
+- `mock-file-auth`: ✅ Works (static token)
+- `mock-k8s-auth`: ✅ Works (uses your k8s token)
+- `mock-client-auth`: ⚠️ **Skipped** (required auth header not available - see warning in logs)
+
+## Cleanup
+
+Stop all services (Ctrl+C in each terminal):
+1. Terminal 1: Mock MCP server
+2. Terminal 2: Lightspeed Core (includes Llama Stack)
+
+Remove test secret:
+```bash
+rm /tmp/lightspeed-mcp-test-token
+```
+
+## Troubleshooting
+
+### Missing OPENAI_API_KEY
+- Error: "API key not found" or authentication errors
+- Solution: Set the environment variable before starting Lightspeed Core
+  ```bash
+  export OPENAI_API_KEY="your-api-key-here"
+  ```
+
+### Lightspeed Core startup errors
+- Check that `dev-tools/test-configs/llama-stack-mcp-test.yaml` exists and is valid
+- Verify OPENAI_API_KEY is set
+- Check the logs for specific error messages
+
+### Mock server not receiving requests
+- Verify mock server is running: `curl http://localhost:9000/debug/headers`
+- Check Lightspeed Core logs for errors
+- Ensure config file path is correct
+
+### Headers not captured
+- Check mock server output for incoming requests
+- Verify the secret file exists and is readable
+- Check that the MCP server name matches in config and MCP-HEADERS
+
+### Connection refused
+- Ensure all services are running on expected ports (9000, 8080)
+- Check firewall settings

dev-tools/README.md

@@ -0,0 +1,32 @@
+# Development Tools
+
+This directory contains utilities and tools for local development and testing of Lightspeed Core Stack.
+
+## Available Tools
+
+### MCP Mock Server
+
+A lightweight mock Model Context Protocol (MCP) server for testing MCP integrations and authorization headers locally.
+
+**Location:** `dev-tools/mcp-mock-server/`
+
+**Use Cases:**
+- Test MCP server authentication headers without real MCP infrastructure
+- Debug authorization header configuration
+- Local development of MCP-related features
+- Validate MCP server connectivity
+
+See [`mcp-mock-server/README.md`](mcp-mock-server/README.md) for usage instructions.
+
+## Testing MCP Integration with Lightspeed Core
+
+For comprehensive step-by-step instructions on manually testing MCP authorization, see [`MANUAL_TESTING.md`](MANUAL_TESTING.md).
+
+## Adding New Tools
+
+When adding new development tools to this directory:
+1. Create a subdirectory for the tool
+2. Include a README.md explaining what it does and how to use it
+3. Update this file to list the new tool
+4. Keep tools self-contained with their own dependencies (if any)
+