Merge pull request #235 from davidruzicka/feat/upstream-mcp-proxy-session-and-tool-discovery

feat: upstream MCP proxy - session foundation and tool discovery (Phase 1 + Phase 2)

Author: davidruzicka

.claude/skills/auto-update-skills/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: auto-update-skills
-description: Propose creating a new skill or update existing after a correction reveals reusable existing knowledge, tools, policies, or preferred communication style. Trigger immediately for critical issues and on repetition for trivial patterns.
+description: After correction/feedback: propose new skill or update existing to capture reusable pattern. Covers knowledge, tools, policies, preferred style. Trigger immediately for critical issues, on repetition for trivial patterns.
 ---
 
 ## Goal

.cursorrules

@@ -0,0 +1,14 @@
+# lean-ctx — Context Engineering Layer
+
+PREFER lean-ctx MCP tools over native equivalents for token savings:
+
+| PREFER | OVER | Why |
+|--------|------|-----|
+| `ctx_read(path)` | `Read` | Cached, 8 compression modes |
+| `ctx_shell(command)` | `Shell` | Pattern compression |
+| `ctx_search(pattern, path)` | `Grep` | Compact results |
+| `ctx_tree(path, depth)` | `ls` / `find` | Directory maps |
+| `ctx_edit(path, old_string, new_string)` | `Edit` (when Read unavailable) | Search-and-replace without native Read |
+
+Edit files: use native Edit/StrReplace if available. If Edit requires Read and Read is unavailable, use ctx_edit.
+Write, Delete, Glob — use normally. NEVER loop on Edit failures — switch to ctx_edit immediately.

.gitignore

@@ -7,3 +7,4 @@ dist/
 coverage/
 junit.xml
 tmp/
+.claude/settings.local.json

.planning/PROJECT.md

@@ -0,0 +1,127 @@
+# mcp4openapi - Enterprise MCP Gateway
+
+## What This Is
+
+A centralized, enterprise-grade MCP gateway that acts as the single front door for all AI clients
+inside the company to reach upstream MCP servers. It authenticates clients via SSO/OIDC or API
+keys, enforces team-level tool access policies, and forwards tool calls to upstream remote HTTP MCP
+servers (internal services and third-party SaaS) using credentials supplied by the client at session
+initialization - the gateway itself stores no upstream secrets.
+
+Built on top of the existing mcp4openapi server, extending it from an OpenAPI-to-MCP adapter into a
+full MCP proxy/gate.
+
+## Core Value
+
+A security boundary between internal AI clients and all upstream MCP servers: one place to
+authenticate, authorize, audit, and proxy every tool call in the company.
+
+## Requirements
+
+### Validated
+
+- Existing capabilities already shipped and working:
+- ✓ MCP server over HTTP (SSE, sessions, MCP spec 2025-03-26) - existing
+- ✓ Profile-driven configuration with Zod-validated schemas - existing
+- ✓ OpenAPI-backed tool generation from REST APIs - existing
+- ✓ OAuth 2.0 provider (PKCE, DCR, token exchange) - existing
+- ✓ Multi-auth support (bearer, query, custom header, OAuth) - existing
+- ✓ Multi-tenant HTTP transport with session isolation - existing
+- ✓ Rate limiting, SSRF protection, token redaction - existing
+- ✓ Prometheus metrics emission (prom-client) - existing
+- ✓ Upstream MCP provider config schema (UpstreamMcpProvider type, Zod schemas) - existing (PR #219)
+
+### Validated
+
+- ✓ Upstream session lifecycle (Phase 01) - per-session `UpstreamConnectionManager` with lazy connect, concurrent-safe `getOrConnect`, heartbeat pings, and session-scoped cleanup wired into HTTP transport destruction lifecycle
+- ✓ Pass-through credential forwarding (Phase 01) - client-supplied Bearer token forwarded directly to upstream; profile-per-upstream model; no credential storage on gateway; `validateCredentials` with SSRF-protected `validation_endpoint` for early auth validation
+- ✓ Auth redaction hardening (Phase 01) - `sanitizeAuthErrorMessage` preserves last-4 Bearer suffix for debuggability; `redactString` fully redacts; token never appears in logs or error responses
+
+### Active
+- [ ] Upstream tool discovery and proxy - tools/list and tools/call forwarded to correct upstream
+  provider; upstream tools appear in tools/list alongside (or instead of) OpenAPI-backed tools
+- [ ] Tool namespacing - upstream tool names prefixed/namespaced to prevent collisions across
+  providers (#215)
+- [ ] Team-level allow/deny policy - each client identity (team/API key/SSO principal) maps to a
+  policy that allows or denies specific upstream servers and/or tool names (#216)
+- [ ] Client authentication gate - SSO/OIDC (Entra ID / Okta / Keycloak) for interactive clients;
+  API keys for M2M; identity resolved before any tool call is processed
+- [ ] Upstream notification forwarding - tools/list_changed and other server-initiated upstream
+  notifications forwarded to downstream SSE clients with replay on reconnect (#214)
+- [ ] Audit log - structured persistent log of every tool call: client identity, team, tool name,
+  upstream server, outcome, timestamp
+- [ ] Request tracing - OpenTelemetry trace context propagated through gateway and forwarded to
+  upstream where possible
+- [ ] Third-party SaaS MCP proxy - remote HTTP MCP endpoints for services like GitHub, Slack, etc.
+  supported through the same upstream provider config model
+- [ ] End-to-end documentation and test coverage for proxy mode (#218)
+
+### Out of Scope
+
+- Stdio upstream MCP processes - execution boundary undefined, risk of process isolation issues;
+  deferred to a later phase behind an explicit feature gate (#217)
+- Server-side upstream credential storage - pass-through model replaces the need; vault integration
+  adds complexity without benefit given the chosen auth model
+- Attribute-based access control (ABAC) - team-level allow/deny covers v1 needs; ABAC adds
+  authoring overhead before any team has adopted the gateway
+- Public internet exposure - on-prem/private cloud deployment only; no multi-cloud SaaS distribution
+  in scope
+
+## Context
+
+- **Existing codebase:** mcp4openapi is a TypeScript/Node.js MCP server (Express, MCP SDK 1.26.0,
+  jose for JWT, Zod for schema validation). The HTTP transport already handles SSE sessions,
+  multi-tenancy, OAuth provider, and interceptor chains (auth -> rate-limit -> retry -> fetch).
+- **Tracking issue:** davidruzicka/mcp4openapi#211 groups the full MCP proxy roadmap. Issues
+  #213-#218 map directly to the active requirements above. #212 (upstream config schema) is done.
+- **Deployment target:** On-prem / private cloud. No public internet exposure. Docker/Kubernetes
+  packaging assumed.
+- **Client auth model:** SSO/OIDC tokens from the company IdP (Entra ID, Okta, Keycloak) for
+  interactive users; API keys for machine-to-machine. Both paths must resolve to a team identity
+  before policy is checked.
+- **Upstream auth model:** Pass-through. Clients supply their own upstream credentials at HTTP
+  session initialization. The gateway extracts and stores them in the session context, then forwards
+  them on each upstream call. No credential storage or rotation responsibility on the gateway.
+- **Security posture:** SSRF protection already in place. Token redaction in logs. Trust boundaries:
+  inbound client auth and upstream auth are fully separate layers.
+
+## Constraints
+
+- **Tech stack:** TypeScript 5 / Node.js 22 / ESM - no runtime changes; extend, don't replace
+- **MCP protocol:** MCP spec 2025-03-26 compliance must be preserved end-to-end (client <-> gateway
+  <-> upstream)
+- **Security:** Inbound client identity must be verified before any upstream connection is
+  established; upstream credentials must never leak into logs or error responses
+- **Compatibility:** Existing OpenAPI-backed tool generation must continue working unchanged;
+  proxy mode is additive, not a replacement
+
+## Key Decisions
+
+| Decision | Rationale | Outcome |
+|----------|-----------|---------|
+| Pass-through upstream credentials | Gateway stores no secrets - client owns their own upstream tokens; simpler security model, no vault dependency | Validated in Phase 01 - profile-per-upstream model, `token: string \| undefined` passed directly |
+| Profile-per-upstream (not session-level credential aggregation) | Simpler than per-session credential bag; one profile = one upstream = one token env var | Validated in Phase 01 - dead X-Upstream-Authorization extractor removed |
+| Remote HTTP upstream first, stdio deferred | Stdio adds process isolation complexity; HTTP upstream covers the primary enterprise use case first | - Pending |
+| Build on mcp4openapi transport stack | Existing SSE session management, OAuth provider, multi-tenant HTTP transport are production-grade; extend rather than rewrite | - Pending |
+| Team-level allow/deny (not RBAC/ABAC) | Explicit allow/deny per team is auditable and predictable; ABAC adds authoring overhead before adoption | - Pending |
+| Tool namespacing by upstream provider | Prevents tool name collisions across providers; makes audit logs and policy rules unambiguous | - Pending |
+
+---
+*Last updated: 2026-03-30 after Phase 01 completion*
+
+## Evolution
+
+This document evolves at phase transitions and milestone boundaries.
+
+**After each phase transition** (via `/gsd:transition`):
+1. Requirements invalidated? -> Move to Out of Scope with reason
+2. Requirements validated? -> Move to Validated with phase reference
+3. New requirements emerged? -> Add to Active
+4. Decisions to log? -> Add to Key Decisions
+5. "What This Is" still accurate? -> Update if drifted
+
+**After each milestone** (via `/gsd:complete-milestone`):
+1. Full review of all sections
+2. Core Value check - still the right priority?
+3. Audit Out of Scope - reasons still valid?
+4. Update Context with current state

.planning/REQUIREMENTS.md

@@ -0,0 +1,130 @@
+# Requirements - Enterprise MCP Gateway
+
+Generated: 2026-03-27
+Project: mcp4openapi enterprise MCP proxy/gate
+Milestone: v1 - Proxy foundation + security gate
+
+---
+
+## v1 Requirements
+
+### Proxy Core
+
+- [x] **PROXY-01**: A downstream client session connecting to a profile backed by an upstream MCP server
+  creates a per-session upstream HTTP connection on first tool use (lazy, not at session init)
+- [x] **PROXY-02**: Client-supplied upstream credentials (Bearer token, custom header, OAuth token)
+  provided at session initialization are stored in the session context and forwarded to the upstream
+  MCP server for all requests in that session; the gateway stores no credentials server-side
+- [x] **PROXY-03**: A tools/list request from a downstream client returns the tool list fetched from
+  the upstream MCP server defined in the active profile (same profile-per-upstream model as OpenAPI
+  profiles; no aggregation or namespacing across providers)
+- [x] **PROXY-04**: A tools/call request is routed to the upstream MCP server defined in the active
+  profile and the upstream response is returned to the downstream client with typed error mapping for
+  upstream failure cases
+
+### Client Authentication
+
+- [ ] **AUTH-01**: Inbound client presenting a JWT is validated against the JWKS endpoint of the
+  configured identity provider (Entra ID, Okta, or Keycloak); session is rejected if validation
+  fails before any upstream connection is made
+- [ ] **AUTH-02**: Inbound M2M client presenting an API key is validated against a configured API
+  key store; a valid key resolves to a client identity before session is established
+- [ ] **AUTH-03**: Client identity (resolved from SSO JWT or API key) is attached to the session
+  context and included in every audit log entry for that session
+
+### Security
+
+- [x] **SEC-01**: Tool definitions received from an upstream MCP server are sanitized before being
+  forwarded to downstream clients; tool names and descriptions are validated against a safe-string
+  allowlist to prevent tool poisoning and prompt injection via upstream tool metadata
+- [x] **SEC-02**: Upstream credential values are redacted from all logs, error responses, and
+  diagnostic output; existing token-redaction infrastructure is extended to cover the new
+  upstream-credential session fields
+
+### Observability
+
+- [ ] **OBS-01**: Every tools/call request produces a structured audit log entry containing: session
+  ID, resolved client identity, tool name, upstream server URL (host only, no credentials),
+  invocation outcome (success/error code), and wall-clock duration
+- [ ] **OBS-02**: Prometheus metrics expose per-upstream and per-client-identity counters and
+  latency histograms for tools/list and tools/call requests; existing prom-client registry is
+  extended (no second registry)
+- [ ] **OBS-03**: GET /health returns 200 when the server is running; GET /ready returns 200 when
+  at least one profile is loaded and the server can accept sessions; both endpoints are unauthenticated
+
+### Reliability
+
+- [x] **REL-01**: Application-level heartbeat pings are sent on upstream SSE connections at a
+  configurable interval (default 30s) to detect silent disconnects before a tool call fails
+- [x] **REL-02**: A session reaper runs on a configurable interval (default 60s) and closes
+  upstream connections for sessions that have been inactive beyond the session timeout; no upstream
+  connections are leaked when downstream clients disconnect without explicit close
+- [x] **REL-03**: Upstream failure cases (connection timeout, auth failure, server unavailable,
+  malformed response) produce typed error responses to the downstream client with correlation IDs;
+  no raw stack traces or upstream credential fragments in error payloads
+- [x] **REL-04**: Upstream notifications/tools/list_changed events received on a live upstream
+  session are forwarded to the connected downstream SSE client; if no stream is attached,
+  notifications are queued and replayed on reconnect using existing SSE replay infrastructure
+
+---
+
+## v2 Requirements (Deferred)
+
+### Policy
+
+- Team-level allow/deny policy: client identity maps to a policy that allows or denies specific
+  upstream MCP servers or tool name patterns - deferred until v1 adoption demonstrates which
+  granularity teams need
+- Policy dry-run mode: evaluate policy without enforcing, surface what would be denied
+
+### Observability
+
+- OpenTelemetry request tracing with trace context propagated to upstream MCP servers - deferred
+  until core pipeline is stable; audit log + Prometheus covers operational needs for v1
+- Per-tool budget and rate limiting by team identity
+
+### Upstream Sources
+
+- Third-party SaaS MCP endpoints (GitHub, Slack, etc.) - same model as internal HTTP upstreams,
+  unblocked by v1; explicit phase for auth/trust configuration differences
+- Stdio upstream MCP processes - deferred until process isolation boundary is defined
+
+### Advanced Proxy
+
+- Tool definition pinning: administrator can pin upstream tool schemas to detect upstream rug-pull
+  changes between deployments
+
+---
+
+## Out of Scope
+
+- Server-side upstream credential storage - pass-through model replaces the need; no vault
+  integration in scope
+- Tool namespacing/aggregation across multiple upstream providers in a single profile - profile-
+  per-upstream model is the architecture; aggregation is a separate product decision
+- Attribute-based access control (ABAC) - team allow/deny covers v1; ABAC adds authoring overhead
+- Admin UI - CLI and profile config files are the management interface
+- Public internet / multi-cloud SaaS distribution - on-prem/private cloud deployment only
+
+---
+
+## Traceability
+
+| REQ-ID | Phase | Status |
+|--------|-------|--------|
+| PROXY-01 | Phase 1 | Complete |
+| PROXY-02 | Phase 1 | Complete |
+| PROXY-03 | Phase 2 | Complete |
+| PROXY-04 | Phase 2 | Complete |
+| AUTH-01 | Phase 4 | Pending |
+| AUTH-02 | Phase 3 | Pending |
+| AUTH-03 | Phase 3 (partial), Phase 4 (complete) | Pending |
+| SEC-01 | Phase 2 | Complete |
+| SEC-02 | Phase 1 | Complete |
+| OBS-01 | Phase 5 | Pending |
+| OBS-02 | Phase 5 | Pending |
+| OBS-03 | Phase 5 | Pending |
+| REL-01 | Phase 1 | Complete |
+| REL-02 | Phase 1 | Complete |
+| REL-03 | Phase 1 | Complete |
+| REL-04 | Phase 2 | Complete |