feat: implement comprehensive MCP authentication and proxy integration

.docs/summary-terraphim_persistence-src-mcp.rs.md

@@ -0,0 +1,244 @@
+# Summary: crates/terraphim_persistence/src/mcp.rs
+
+## Purpose
+Implements persistence layer for MCP (Model Context Protocol) data using OpenDAL for multi-backend storage (Memory, File, S3, etc.). Provides async CRUD operations for namespaces, endpoints, API keys, tools, tool caching, and audit trails.
+
+## Core Data Models
+
+### McpNamespaceRecord
+- **uuid**: Unique identifier
+- **name**: Namespace display name
+- **description**: Optional description
+- **user_id**: Owner identifier
+- **config_json**: MCP configuration as JSON string
+- **created_at**: Creation timestamp
+- **enabled**: Active/inactive status
+- **visibility**: Public or Private (default: Private)
+
+### McpEndpointRecord
+- **uuid**: Unique identifier
+- **name**: Endpoint display name
+- **namespace_uuid**: Parent namespace reference
+- **auth_type**: Authentication method (e.g., "bearer")
+- **user_id**: Owner identifier
+- **created_at**: Creation timestamp
+- **enabled**: Active/inactive status
+
+### McpApiKeyRecord
+- **uuid**: Unique identifier
+- **key_hash**: SHA256 hash of actual API key (never stores plaintext)
+- **endpoint_uuid**: Associated endpoint
+- **user_id**: Owner identifier
+- **created_at**: Creation timestamp
+- **expires_at**: Optional expiration timestamp
+- **enabled**: Active/inactive status (can be disabled without deletion)
+
+### McpToolRecord
+- **uuid**: Unique identifier
+- **namespace_uuid**: Parent namespace
+- **server_name**: MCP server name (e.g., "filesystem")
+- **tool_name**: Namespaced tool name (e.g., "filesystem__read_file")
+- **original_name**: Original tool name from server
+- **status**: Active or Inactive
+- **override_name**: Optional custom display name
+- **override_description**: Optional custom description
+- **created_at / updated_at**: Timestamps
+
+### ToolDiscoveryCache
+- **namespace_uuid**: Namespace identifier
+- **tools_json**: Cached tool list as JSON string
+- **cached_at**: Cache creation time
+- **expires_at**: Cache expiration time
+- Purpose: Reduce repeated tool discovery API calls
+
+### McpAuditRecord
+- **uuid**: Unique identifier
+- **user_id**: User who executed the action
+- **endpoint_uuid**: Endpoint used
+- **namespace_uuid**: Namespace context
+- **tool_name**: Tool that was called
+- **arguments**: Optional serialized arguments JSON
+- **response**: Optional serialized response JSON
+- **is_error**: Boolean flag for error responses
+- **latency_ms**: Execution time in milliseconds
+- **created_at**: Audit timestamp
+
+## McpPersistence Trait
+
+### Namespace Operations
+- `save_namespace(&self, record) -> Result<()>`
+- `get_namespace(&self, uuid) -> Result<Option<McpNamespaceRecord>>`
+- `list_namespaces(&self, user_id) -> Result<Vec<McpNamespaceRecord>>`
+- `list_namespaces_with_visibility(&self, user_id, include_public) -> Result<Vec<McpNamespaceRecord>>`
+- `delete_namespace(&self, uuid) -> Result<()>`
+
+### Endpoint Operations
+- `save_endpoint(&self, record) -> Result<()>`
+- `get_endpoint(&self, uuid) -> Result<Option<McpEndpointRecord>>`
+- `list_endpoints(&self, user_id) -> Result<Vec<McpEndpointRecord>>`
+- `delete_endpoint(&self, uuid) -> Result<()>`
+
+### API Key Operations
+- `save_api_key(&self, record) -> Result<()>`
+- `get_api_key(&self, uuid) -> Result<Option<McpApiKeyRecord>>`
+- **`verify_api_key(&self, key_hash) -> Result<Option<McpApiKeyRecord>>`** - Critical for authentication
+- `list_api_keys(&self, user_id) -> Result<Vec<McpApiKeyRecord>>`
+- `delete_api_key(&self, uuid) -> Result<()>`
+
+### Tool Operations
+- `save_tool(&self, record) -> Result<()>`
+- `get_tool(&self, uuid) -> Result<Option<McpToolRecord>>`
+- `list_tools(&self, namespace_uuid) -> Result<Vec<McpToolRecord>>`
+- `update_tool_status(&self, uuid, status) -> Result<()>`
+- `delete_tool(&self, uuid) -> Result<()>`
+
+### Tool Cache Operations
+- `save_tool_cache(&self, cache) -> Result<()>`
+- `get_tool_cache(&self, namespace_uuid) -> Result<Option<ToolDiscoveryCache>>`
+- `delete_tool_cache(&self, namespace_uuid) -> Result<()>`
+
+### Audit Operations
+- `save_audit(&self, record) -> Result<()>`
+- `get_audit(&self, uuid) -> Result<Option<McpAuditRecord>>`
+- `list_audits(&self, user_id, endpoint_uuid, limit) -> Result<Vec<McpAuditRecord>>`
+- `delete_audit(&self, uuid) -> Result<()>`
+
+## McpPersistenceImpl
+
+### Storage Architecture
+- **Backend**: OpenDAL Operator wrapped in `Arc<RwLock<Operator>>`
+- **Path Structure**: Hierarchical JSON file organization
+  - `mcp/namespaces/{uuid}.json`
+  - `mcp/endpoints/{uuid}.json`
+  - `mcp/api_keys/{uuid}.json`
+  - `mcp/tools/{uuid}.json`
+  - `mcp/tool_cache/{namespace_uuid}.json`
+  - `mcp/audit/{uuid}.json`
+
+### Key Implementation Details
+
+#### verify_api_key (Authentication Critical)
+```rust
+async fn verify_api_key(&self, key_hash: &str) -> Result<Option<McpApiKeyRecord>> {
+    // Lists all API keys and searches for matching hash
+    // Validates enabled=true
+    // Checks expiration if expires_at is set
+    // Returns None if key not found, disabled, or expired
+}
+```
+- **Performance Note**: Linear search through all keys (acceptable for small datasets)
+- **Security**: Returns None for expired/disabled keys (no error indication)
+
+#### list_audits with Sorting
+```rust
+async fn list_audits(...) -> Result<Vec<McpAuditRecord>> {
+    // Collects matching audits
+    // Sorts by created_at descending (newest first)
+    // Respects limit parameter
+}
+```
+
+#### Namespace Visibility Filtering
+```rust
+async fn list_namespaces_with_visibility(&self, user_id, include_public) -> Result<...> {
+    // If include_public=true: returns user's namespaces + all public namespaces
+    // If include_public=false: returns only user's namespaces
+    // Enables multi-tenant public/private namespace sharing
+}
+```
+
+### Thread Safety
+- `Arc<RwLock<Operator>>` allows concurrent reads, exclusive writes
+- All trait methods are `async` for non-blocking I/O
+- Trait requires `Send + Sync` for multi-threaded usage
+
+## Test Coverage (7 comprehensive tests)
+
+1. **test_namespace_persistence** - CRUD operations for namespaces
+2. **test_api_key_verification** - Authentication verification logic
+3. **test_tool_management** - Tool CRUD and status updates
+4. **test_tool_cache** - Cache save/retrieve/delete
+5. **test_audit_trail** - Audit record management and filtering
+6. **test_namespace_visibility** - Public/private namespace filtering
+
+All tests use Memory backend for fast, isolated testing.
+
+## Critical Design Decisions
+
+### Why Arc<RwLock<Operator>>?
+- **Arc**: Enables cloning for shared ownership across threads
+- **RwLock**: Allows multiple concurrent readers, single writer
+- **Operator**: OpenDAL's backend abstraction
+
+### Why JSON Files?
+- **Flexibility**: Works with any OpenDAL backend (Memory, FS, S3, Azure, etc.)
+- **Human-Readable**: Easy debugging and inspection
+- **Schema Evolution**: JSON allows adding fields without migration
+- **Limitation**: No relational queries, linear search for some operations
+
+### Why verify_api_key Scans All Keys?
+- **Simplicity**: No secondary index needed
+- **Performance Trade-off**: Acceptable for <1000 keys
+- **Future Improvement**: Could add in-memory cache or use database backend
+
+## Known Limitations
+
+1. **No Pagination**: List operations return all matching records
+2. **Linear Search**: API key verification O(n) complexity
+3. **No Transactions**: Multi-step operations not atomic
+4. **No Indexing**: Cannot efficiently query by fields other than UUID
+5. **SQLite Incompatibility**: Hierarchical paths don't map to SQLite blob storage
+
+## Backend Compatibility
+
+### Tested Backends
+- **Memory**: In-memory HashMap (tests, development)
+- **File System**: Local directory storage (production)
+
+### Theoretically Supported (via OpenDAL)
+- AWS S3
+- Azure Blob Storage
+- Google Cloud Storage
+- HTTP/WebDAV
+
+### Known Issues
+- **SQLite**: Blob storage model incompatible with hierarchical paths
+- **RocksDB**: Previously caused locking issues (deprecated)
+
+## File Location
+`crates/terraphim_persistence/src/mcp.rs` (744 lines)
+
+## Dependencies
+- `async_trait`: Async trait definitions
+- `chrono`: Timestamp handling
+- `futures`: Stream utilities
+- `serde`: JSON serialization
+- `tokio::sync::RwLock`: Async read-write lock
+- `opendal`: Multi-backend storage abstraction
+
+## Related Files
+- `terraphim_server/src/api_mcp.rs`: HTTP API handlers using this persistence
+- `terraphim_server/src/mcp_auth.rs`: Authentication middleware calling verify_api_key
+- `terraphim_server/src/lib.rs`: AppState with shared McpPersistenceImpl instance
+- `terraphim_server/tests/mcp_auth_tests.rs`: Integration tests using this persistence
+
+## Security Considerations
+- **API Key Storage**: Only hashes stored, never plaintext
+- **Expiration Enforcement**: Expired keys automatically rejected by verify_api_key
+- **Enabled Status**: Allows disabling keys without deletion
+- **Audit Trail**: Comprehensive logging for security monitoring
+- **User Isolation**: Optional user_id filtering for multi-tenancy
+
+## Performance Characteristics
+- **Read Operations**: O(1) for get by UUID, O(n) for list/search
+- **Write Operations**: O(1) for save/delete
+- **verify_api_key**: O(n) linear search (could be optimized with cache)
+- **list_audits**: O(n log n) due to sorting
+
+## Future Enhancements
+1. Add in-memory cache for API key verification
+2. Implement pagination for list operations
+3. Add secondary indexes for common queries
+4. Support for database backends (PostgreSQL, MySQL)
+5. Batch operations for bulk imports
+6. Metrics and monitoring integration

.docs/summary-terraphim_server-src-api_mcp.rs.md

@@ -0,0 +1,149 @@
+# Summary: terraphim_server/src/api_mcp.rs
+
+## Purpose
+Implements HTTP API handlers for MCP (Model Context Protocol) management, providing CRUD operations for namespaces, endpoints, API keys, tools, and audit trails.
+
+## API Structure
+
+### Response Types
+All responses follow consistent pattern with status, data, and error fields:
+- `McpNamespaceResponse` / `McpNamespaceListResponse`
+- `McpEndpointResponse` / `McpEndpointListResponse`
+- `McpApiKeyResponse` - Includes plaintext `key_value` on creation (one-time only)
+- `McpAuditListResponse` - Paginated audit records
+- `McpHealthResponse` - System health with namespace/endpoint counts
+
+### Request Types
+- `CreateNamespaceRequest` - name, description, user_id, config_json, enabled, visibility
+- `CreateEndpointRequest` - name, namespace_uuid, auth_type, user_id, enabled
+- `CreateApiKeyRequest` - endpoint_uuid, user_id, expires_at, enabled
+
+## API Endpoints
+
+### Namespace Management
+1. **list_namespaces** - `GET /metamcp/namespaces`
+   - Lists all namespaces, optionally filtered by user_id
+   - Returns: `McpNamespaceListResponse`
+
+2. **get_namespace** - `GET /metamcp/namespaces/{uuid}`
+   - Retrieves single namespace by UUID
+   - Returns: `McpNamespaceResponse` with 404 handling
+
+3. **create_namespace** - `POST /metamcp/namespaces`
+   - Creates new namespace with auto-generated UUID
+   - Sets `created_at` timestamp automatically
+   - Returns: Created namespace or error
+
+4. **delete_namespace** - `DELETE /metamcp/namespaces/{uuid}`
+   - Removes namespace permanently
+   - Returns: `204 NO_CONTENT` or `500 INTERNAL_SERVER_ERROR`
+
+### Endpoint Management
+5. **list_endpoints** - `GET /metamcp/endpoints`
+   - Lists all endpoints, optionally filtered by user_id
+   - Returns: `McpEndpointListResponse`
+
+6. **get_endpoint** - `GET /metamcp/endpoints/{uuid}`
+   - Retrieves single endpoint by UUID
+   - Returns: `McpEndpointResponse` with 404 handling
+
+7. **create_endpoint** - `POST /metamcp/endpoints`
+   - Creates new MCP endpoint with auto-generated UUID
+   - Sets `created_at` timestamp automatically
+   - Returns: Created endpoint or error
+
+8. **delete_endpoint** - `DELETE /metamcp/endpoints/{uuid}`
+   - Removes endpoint permanently
+   - Returns: `204 NO_CONTENT` or `500 INTERNAL_SERVER_ERROR`
+
+### API Key Management
+9. **create_api_key** - `POST /metamcp/api_keys`
+   - Generates API key with format: `tpai_{uuid_without_hyphens}`
+   - Hashes key using SHA256 before storage (only hash persisted)
+   - Returns: API key record AND plaintext key_value (one-time retrieval)
+   - Security: Key value never stored, only hash
+
+### Audit Trail
+10. **list_audits** - `GET /metamcp/audits`
+    - Lists up to 100 most recent audit records
+    - Optionally filtered by user_id or endpoint_uuid
+    - Sorted by created_at descending
+    - Returns: Audit records with total count
+
+### Health Check
+11. **get_mcp_health** - `GET /metamcp/health`
+    - Returns system version, timestamp, namespace/endpoint counts
+    - Non-blocking health check for monitoring
+    - Returns: `McpHealthResponse`
+
+## Key Design Patterns
+
+### Shared State Access
+All handlers use `State(app_state): State<AppState>` extraction:
+```rust
+let persistence = app_state.mcp_persistence.clone();
+```
+- Uses `Arc<McpPersistenceImpl>` for thread-safe shared access
+- Critical for authentication middleware compatibility
+
+### UUID Generation
+Auto-generates UUIDs for all resources:
+```rust
+uuid: uuid::Uuid::new_v4().to_string()
+```
+
+### Timestamp Management
+Automatically sets `created_at` using `chrono::Utc::now()`:
+```rust
+created_at: chrono::Utc::now()
+```
+
+### Error Handling Strategy
+- **Graceful Degradation**: Most endpoints return `Status::Error` in response body
+- **HTTP Status Codes**: DELETE operations use proper status codes (204, 500)
+- **Logging**: Errors logged before returning to client
+- **User-Friendly Messages**: Generic error messages to avoid leaking internals
+
+### API Key Security
+Private `hash_api_key()` function in this module:
+```rust
+fn hash_api_key(key: &str) -> String {
+    use sha2::{Digest, Sha256};
+    let mut hasher = Sha256::new();
+    hasher.update(key.as_bytes());
+    format!("{:x}", hasher.finalize())
+}
+```
+Note: Duplicates logic from `mcp_auth.rs` (could be refactored to shared module)
+
+## Authentication Integration
+All endpoints (except health) protected by `validate_api_key` middleware via:
+```rust
+.route_layer(middleware::from_fn_with_state(app_state, mcp_auth::validate_api_key))
+```
+
+## File Location
+`terraphim_server/src/api_mcp.rs` (372 lines)
+
+## Dependencies
+- `axum`: HTTP framework, routing, JSON extraction/responses
+- `chrono`: Timestamp management
+- `serde`: Serialization/deserialization
+- `sha2`: API key hashing
+- `uuid`: UUID generation
+- `terraphim_persistence::mcp`: Persistence trait and record types
+
+## Related Files
+- `terraphim_server/src/mcp_auth.rs`: Authentication middleware
+- `terraphim_server/src/api_mcp_tools.rs`: Tool execution endpoints
+- `terraphim_server/src/api_mcp_openapi.rs`: OpenAPI spec generation
+- `terraphim_persistence/src/mcp.rs`: Persistence layer implementation
+- `terraphim_server/src/lib.rs`: Router configuration
+
+## Potential Improvements
+1. **Deduplicate hash_api_key**: Share implementation with `mcp_auth.rs`
+2. **Pagination**: Add pagination for list endpoints (currently unbounded)
+3. **Batch Operations**: Support bulk create/delete for efficiency
+4. **Rate Limiting**: Add per-user rate limits for creation endpoints
+5. **Validation**: Add input validation (e.g., name length, config JSON schema)
+6. **CORS**: Configure CORS headers for browser clients