| title | Architecture | |||||
|---|---|---|---|---|---|---|
| description | System architecture and component overview | |||||
| icon | sitemap | |||||
| seoTitle | Architecture: MCP Server with LangGraph System Design | |||||
| seoDescription | Understand the architecture of MCP Server with LangGraph: multi-LLM routing, OpenFGA authorization, Redis sessions, observability stack, and deployment patterns. | |||||
| keywords |
|
|||||
| contentType | explanation |
MCP Server with LangGraph is a production-ready, multi-layered architecture designed for enterprise AI applications. It combines stateful AI agents, enterprise authentication, fine-grained authorization, and comprehensive observability.
flowchart TB
subgraph Client_Layer["Client Layer"]
WebApp[Web Application]
CLI[CLI Tools]
API[API Clients]
end
subgraph Gateway_Layer["Gateway Layer"]
Ingress[Ingress/Load Balancer]
Kong[Kong API Gateway]
end
subgraph Application_Layer["Application Layer"]
MCP[MCP Server]
Agent[LangGraph Agent]
Auth[Auth Middleware]
end
subgraph AI_LLM_Layer["AI/LLM Layer"]
LiteLLM[LiteLLM Router]
Anthropic[Anthropic Claude]
Google[Google Gemini]
OpenAI[OpenAI GPT]
Ollama[Ollama Local]
end
subgraph Authentication_Layer["Authentication Layer"]
Keycloak[Keycloak SSO]
JWT[JWT Validation]
Sessions[Session Manager]
end
subgraph Authorization_Layer["Authorization Layer"]
OpenFGA[OpenFGA]
RoleMapper[Role Mapper]
end
subgraph Data_Layer["Data Layer"]
Redis[Redis Sessions]
Postgres[(PostgreSQL)]
end
subgraph Observability_Layer["Observability Layer"]
OTel[OpenTelemetry]
Jaeger[Jaeger Tracing]
Prometheus[Prometheus Metrics]
LangSmith[LangSmith]
end
subgraph Secrets_Layer["Secrets Layer"]
Infisical[Infisical]
K8sSecrets[Kubernetes Secrets]
end
WebApp --> Ingress
CLI --> Ingress
API --> Ingress
Ingress --> Kong
Kong --> MCP
MCP --> Agent
MCP --> Auth
Auth --> JWT
Auth --> Sessions
Auth --> Keycloak
Auth --> OpenFGA
Sessions --> Redis
OpenFGA --> RoleMapper
RoleMapper --> Keycloak
OpenFGA --> Postgres
Keycloak --> Postgres
Agent --> LiteLLM
LiteLLM --> Anthropic
LiteLLM --> Google
LiteLLM --> OpenAI
LiteLLM --> Ollama
Agent --> OTel
OTel --> Jaeger
OTel --> Prometheus
Agent --> LangSmith
Agent --> Infisical
Agent --> K8sSecrets
%% ColorBrewer2 Set3 (12-color) palette - optimized for node type distinction
%% Each layer uses distinct colors to avoid visual confusion
classDef clientStyle fill:#8dd3c7,stroke:#2a9d8f,stroke-width:2px,color:#333
classDef ingressStyle fill:#fb8072,stroke:#e74c3c,stroke-width:2px,color:#333
classDef kongStyle fill:#fdb462,stroke:#e67e22,stroke-width:2px,color:#333
classDef mcpStyle fill:#ffed6f,stroke:#f39c12,stroke-width:2px,color:#333
classDef agentStyle fill:#b3de69,stroke:#7cb342,stroke-width:2px,color:#333
classDef authStyle fill:#bebada,stroke:#8e7cc3,stroke-width:2px,color:#333
classDef llmRouterStyle fill:#fdb462,stroke:#e67e22,stroke-width:2px,color:#333
classDef llmProviderStyle fill:#fccde5,stroke:#ec7ab8,stroke-width:2px,color:#333
classDef authSvcStyle fill:#bebada,stroke:#8e7cc3,stroke-width:2px,color:#333
classDef authzStyle fill:#d9d9d9,stroke:#95a5a6,stroke-width:2px,color:#333
classDef redisStyle fill:#80b1d3,stroke:#3498db,stroke-width:2px,color:#333
classDef secretsStyle fill:#ccebc5,stroke:#82c99a,stroke-width:2px,color:#333
class WebApp,CLI,API clientStyle
class Ingress ingressStyle
class Kong,LiteLLM kongStyle
class MCP mcpStyle
class Agent agentStyle
class Auth authStyle
class Anthropic,Google,OpenAI,Ollama llmProviderStyle
class Keycloak,JWT,Sessions authSvcStyle
class OpenFGA,RoleMapper,Postgres authzStyle
class Redis redisStyle
class OTel,Jaeger,Prometheus,LangSmith obsStyle
class Infisical,K8sSecrets secretsStyle
%% Additional styling for observability layer distinction
%% ColorBrewer2 Set3 palette - each component type uniquely colored
classDef obsStyle fill:#ffffb3,stroke:#f1c40f,stroke-width:2px,color:#333
classDef langsmithStyle fill:#bc80bd,stroke:#8e44ad,stroke-width:2px,color:#333
class OTel obsStyle
class Jaeger,Prometheus obsStyle
class LangSmith langsmithStyle
Model Context Protocol server - Exposes AI agents as standard tools.
Standard input/output for CLI integration - Direct agent interaction - Shell scripting support - Development and testing HTTP API with streaming support - REST endpoints - Server-Sent Events (SSE) - Production deploymentsKey Features:
- Protocol-compliant MCP implementation
- Tool registration and discovery
- Streaming responses
- Error handling and retries
Code Location: src/mcp_server_langgraph/server.py
Stateful AI agent with functional API and conditional routing.
## Agent graph structure
START → authenticate → authorize → execute_tool → END
↓ ↓ ↓
unauthorized forbidden tool_error
↓ ↓ ↓
END END retry/ENDComponents:
**AgentState** - Pydantic model for type-safe state```python
class AgentState(BaseModel):
messages: List[Message]
user_id: str
authorized: bool
context: Dict[str, Any]
tool_results: List[Any]
```
**Features**:
- Immutable state transitions
- Type validation
- Checkpointing support
```python
tools = {
"chat": chat_tool,
"search": search_tool,
# Add custom tools
}
```
**Authorization**:
- Per-tool permission checks
- OpenFGA relationship validation
- Fallback admin access
- In-memory checkpoints (development)
- Redis checkpoints (production)
- Resume interrupted conversations
- Audit trail
Code Location: src/mcp_server_langgraph/agent.py
Pluggable authentication with multiple provider support.
%%{init: {'theme':'base', 'themeVariables': {'actorBkg':'#8dd3c7','actorBorder':'#2a9d8f','actorTextColor':'#333','actorLineColor':'#2a9d8f','signalColor':'#7cb342','signalTextColor':'#333','labelBoxBkgColor':'#ffffb3','labelBoxBorderColor':'#f1c40f','labelTextColor':'#333','noteBorderColor':'#e67e22','noteBkgColor':'#fdb462','noteTextColor':'#333','activationBorderColor':'#7cb342','activationBkgColor':'#b3de69','sequenceNumberColor':'#fff'}}}%%
sequenceDiagram
participant Client
participant AuthMiddleware
participant UserProvider
participant Keycloak
participant SessionStore
Client->>AuthMiddleware: Request + credentials
AuthMiddleware->>UserProvider: authenticate()
UserProvider->>Keycloak: OIDC login
Keycloak-->>UserProvider: Access + Refresh tokens
UserProvider-->>AuthMiddleware: AuthResponse
AuthMiddleware->>SessionStore: create_session()
SessionStore-->>AuthMiddleware: Session ID
AuthMiddleware-->>Client: Set-Cookie: session_id
Providers:
**Development and testing**- Pre-defined users (alice, bob, admin)
- No external dependencies
- Fast iteration
- Zero configuration
```python
users = {
"alice": User(username="alice", roles=["admin"]),
"bob": User(username="bob", roles=["user"])
}
```
- OpenID Connect / OAuth2
- JWKS token verification
- Refresh token rotation
- Role/group synchronization
```python
provider = KeycloakUserProvider(
server_url="https://sso.yourdomain.com",
realm="mcp-server-langgraph",
client_id="langgraph-client"
)
```
Implement `UserProvider` interface:
```python
class CustomProvider(UserProvider):
async def authenticate(self, username, password):
# Custom auth logic
return AuthResponse(...)
async def verify_token(self, token):
# Custom verification
return TokenVerification(...)
```
Code Location: src/mcp_server_langgraph/auth/
Fine-grained, relationship-based access control with OpenFGA.
%% ColorBrewer2 Set3 palette - each component type uniquely colored
flowchart LR
User[User: alice] -->|admin| Org[Org: acme]
Org -->|member| Tool1[Tool: chat]
Org -->|member| Tool2[Tool: search]
User2[User: bob] -->|member| Org
Authorization Model:
type user
type organization
relations
define admin: [user]
define member: [user] or admin
type tool
relations
define owner: [user]
define executor: [user, organization#member]
define viewer: [user, organization#member]Features:
- Relationship-based permissions
- Hierarchical roles (admin → member → viewer)
- Multi-tenancy support
- Audit logging
- Keycloak role synchronization
Code Location: src/mcp_server_langgraph/auth/openfga.py
Flexible session storage with Redis or in-memory backends.
```mermaid flowchart LR App[Application] --> SM[SessionManager] SM --> Memory[InMemory Store] SM --> Redis[Redis Store] Redis --> Master[Redis Master] Redis --> Replica[Redis Replica] %% ColorBrewer2 Set3 palette styling
classDef appStyle fill:#b3de69,stroke:#7cb342,stroke-width:2px,color:#333
classDef managerStyle fill:#fdb462,stroke:#e67e22,stroke-width:2px,color:#333
classDef storeStyle fill:#bebada,stroke:#8e7cc3,stroke-width:2px,color:#333
classDef redisStyle fill:#fb8072,stroke:#e74c3c,stroke-width:2px,color:#333
class App appStyle
class SM managerStyle
class Memory storeStyle
class Redis redisStyle
class Master,Replica redisStyle
```
**Advanced**:
- Concurrent session limits
- IP/User agent tracking
- TTL management
- Automatic cleanup
# In-memory backend
SESSION_BACKEND=memory
```
Code Location: src/mcp_server_langgraph/auth/session.py
Multi-LLM routing via LiteLLM with automatic fallback.
flowchart TB
Agent[LangGraph Agent] --> LLMFactory[LLM Factory]
LLMFactory --> LiteLLM[LiteLLM Router]
LiteLLM --> Primary[Primary Model]
LiteLLM --> Fallback1[Fallback 1]
LiteLLM --> Fallback2[Fallback 2]
Primary -.->|Error| Fallback1
Fallback1 -.->|Error| Fallback2
Primary --> Claude[Claude 3.5 Sonnet]
Fallback1 --> Gemini[Gemini 2.5 Flash]
Fallback2 --> GPT[GPT-5]
%% ColorBrewer2 Set3 palette styling
classDef agentStyle fill:#b3de69,stroke:#7cb342,stroke-width:2px,color:#333
classDef routerStyle fill:#fdb462,stroke:#e67e22,stroke-width:2px,color:#333
classDef primaryStyle fill:#8dd3c7,stroke:#2a9d8f,stroke-width:2px,color:#333
classDef fallbackStyle fill:#bebada,stroke:#8e7cc3,stroke-width:2px,color:#333
classDef llmStyle fill:#fccde5,stroke:#ec7ab8,stroke-width:2px,color:#333
class Agent agentStyle
class LLMFactory,LiteLLM routerStyle
class Primary,Claude primaryStyle
class Fallback1,Gemini,Fallback2,GPT fallbackStyle
Supported Providers (100+):
- Cloud: Anthropic, OpenAI, Google, Azure, AWS Bedrock
- Open Source: Ollama (Llama, Mistral, Qwen, DeepSeek)
- Custom: Bring your own endpoints
Features:
- Automatic fallback on errors
- Load balancing
- Rate limiting
- Cost tracking
- Response caching
Configuration:
LLM_PROVIDER=anthropic
MODEL_NAME=claude-sonnet-4-5-20250929
ENABLE_FALLBACK=true
FALLBACK_MODELS=["gemini-2.5-flash-002", "gpt-5.1"]Code Location: src/mcp_server_langgraph/llm_factory.py
Dual observability stack - OpenTelemetry + LangSmith.
**Distributed tracing and metrics**```mermaid
flowchart LR
App[Application] --> Exporter[OTLP Exporter]
Exporter --> Collector[OTel Collector]
Collector --> Jaeger[Jaeger UI]
Collector --> Prometheus[Prometheus]
Prometheus --> Grafana[Grafana]
%% ColorBrewer2 Set3 palette styling
classDef appStyle fill:#b3de69,stroke:#7cb342,stroke-width:2px,color:#333
classDef exporterStyle fill:#fdb462,stroke:#e67e22,stroke-width:2px,color:#333
classDef collectorStyle fill:#bebada,stroke:#8e7cc3,stroke-width:2px,color:#333
classDef backendStyle fill:#ffffb3,stroke:#f1c40f,stroke-width:2px,color:#333
class App appStyle
class Exporter exporterStyle
class Collector collectorStyle
class Jaeger,Prometheus,Grafana backendStyle
```
**Traces**:
- End-to-end request flow
- LLM call timing
- Authorization decisions
- Tool executions
**Metrics** (30+):
- Request rate, latency, errors
- Authentication success/failure
- Authorization decisions
- LLM token usage
- Session lifecycle
- Prompt tracking
- LLM response analysis
- Chain visualization
- Cost attribution
- A/B testing
- Evaluations
```python
LANGSMITH_TRACING=true
LANGSMITH_API_KEY=your-key
LANGSMITH_PROJECT=mcp-server-langgraph
```
```json
{
"timestamp": "2025-10-12T10:30:00Z",
"level": "INFO",
"service": "mcp-server-langgraph",
"trace_id": "abc123...",
"span_id": "def456...",
"user_id": "alice",
"event": "authorization_check",
"resource": "tool:chat",
"authorized": true,
"duration_ms": 15
}
```
Code Location: src/mcp_server_langgraph/observability/
Secure secret storage with Infisical or cloud-native solutions.
**Centralized secret management**- End-to-end encryption
- Secret versioning
- Access controls
- Audit logging
- Secret rotation
```python
INFISICAL_CLIENT_ID=your-client-id
INFISICAL_CLIENT_SECRET=your-secret
INFISICAL_PROJECT_ID=project-id
```
- **GCP**: Secret Manager
- **AWS**: Secrets Manager
- **Azure**: Key Vault
- **Kubernetes**: Secrets + RBAC
Automatic injection via workload identity.
```bash
# .env
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_API_KEY=AIza...
JWT_SECRET_KEY=$(openssl rand -base64 32)
```
<Warning>
Never commit `.env` files to Git!
</Warning>
Code Location: src/mcp_server_langgraph/config.py
%%{init: {'theme':'base', 'themeVariables': {'actorBkg':'#8dd3c7','actorBorder':'#2a9d8f','actorTextColor':'#333','actorLineColor':'#2a9d8f','signalColor':'#7cb342','signalTextColor':'#333','labelBoxBkgColor':'#ffffb3','labelBoxBorderColor':'#f1c40f','labelTextColor':'#333','noteBorderColor':'#e67e22','noteBkgColor':'#fdb462','noteTextColor':'#333','activationBorderColor':'#7cb342','activationBkgColor':'#b3de69','sequenceNumberColor':'#fff'}}}%%
sequenceDiagram
participant Client
participant MCP as MCP Server
participant Auth as Auth Middleware
participant Authz as OpenFGA
participant Agent as LangGraph Agent
participant LLM
participant OTel as Observability
Client->>MCP: POST /message (+ session_id)
MCP->>OTel: Start trace
MCP->>Auth: Verify session
Auth->>Auth: Get session from Redis
Auth->>Auth: Validate session TTL
Auth-->>MCP: User: alice
MCP->>Authz: Check permission (alice, executor, tool:chat)
Authz-->>MCP: Authorized: true
MCP->>Agent: Execute with context
Agent->>LLM: Generate response
LLM-->>Agent: Response + token usage
Agent->>OTel: Record metrics
Agent-->>MCP: AgentResponse
MCP->>Auth: Refresh session (if sliding)
MCP->>OTel: End trace
MCP-->>Client: Response + trace_id
%%{init: {'theme':'base', 'themeVariables': {'actorBkg':'#8dd3c7','actorBorder':'#2a9d8f','actorTextColor':'#333','actorLineColor':'#2a9d8f','signalColor':'#7cb342','signalTextColor':'#333','labelBoxBkgColor':'#ffffb3','labelBoxBorderColor':'#f1c40f','labelTextColor':'#333','noteBorderColor':'#e67e22','noteBkgColor':'#fdb462','noteTextColor':'#333','activationBorderColor':'#7cb342','activationBkgColor':'#b3de69','sequenceNumberColor':'#fff'}}}%%
sequenceDiagram
participant User
participant App as MCP Server
participant KC as Keycloak
participant Redis
participant FGA as OpenFGA
User->>App: Login request
App->>KC: Redirect to login
User->>KC: Enter credentials
KC->>KC: Authenticate
KC-->>App: Authorization code
App->>KC: Exchange code for tokens
KC-->>App: Access + Refresh tokens
App->>KC: Get user info
KC-->>App: User profile + roles
App->>FGA: Sync roles to OpenFGA
App->>Redis: Create session
App-->>User: Set session cookie
flowchart TB
subgraph Docker_Host["Docker Host"]
Agent[Agent Container]
KC[Keycloak Container]
Redis[Redis Container]
FGA[OpenFGA Container]
PG[(PostgreSQL)]
Jaeger[Jaeger Container]
Prom[Prometheus]
end
Agent --> KC
Agent --> Redis
Agent --> FGA
Agent --> Jaeger
KC --> PG
FGA --> PG
Prom --> Agent
%% ColorBrewer2 Set3 palette - each service type gets unique color
classDef agentStyle fill:#b3de69,stroke:#7cb342,stroke-width:2px,color:#333
classDef authStyle fill:#bebada,stroke:#8e7cc3,stroke-width:2px,color:#333
classDef cacheStyle fill:#fb8072,stroke:#e74c3c,stroke-width:2px,color:#333
classDef authzStyle fill:#d9d9d9,stroke:#95a5a6,stroke-width:2px,color:#333
classDef dataStyle fill:#80b1d3,stroke:#3498db,stroke-width:2px,color:#333
classDef tracingStyle fill:#ffffb3,stroke:#f1c40f,stroke-width:2px,color:#333
classDef metricsStyle fill:#fdb462,stroke:#e67e22,stroke-width:2px,color:#333
class Agent agentStyle
class KC authStyle
class Redis cacheStyle
class FGA authzStyle
class PG dataStyle
class Jaeger tracingStyle
class Prom metricsStyle
Command: docker compose up
flowchart TB
Internet[Internet]
subgraph Kubernetes_Cluster["Kubernetes Cluster"]
subgraph Ingress_Layer["Ingress"]
LB[Load Balancer]
IngressCtrl[Ingress Controller]
end
subgraph Application_Pods["Application Pods"]
A1[Agent Pod 1]
A2[Agent Pod 2]
A3[Agent Pod 3]
end
subgraph Auth_Services["Auth Services"]
KC1[Keycloak Pod 1]
KC2[Keycloak Pod 2]
end
subgraph Session_Store["Session Store"]
RM[Redis Master]
RR[Redis Replica]
end
subgraph Authorization_Layer["Authorization"]
FGA1[OpenFGA Pod 1]
FGA2[OpenFGA Pod 2]
end
subgraph Data_Layer["Data"]
PG[(PostgreSQL Primary)]
PGR[(PostgreSQL Replica)]
end
end
Internet --> LB
LB --> IngressCtrl
IngressCtrl --> A1
IngressCtrl --> A2
IngressCtrl --> A3
A1 --> KC1
A2 --> KC2
A3 --> KC1
A1 --> RM
A2 --> RM
A3 --> RM
RM -.-> RR
A1 --> FGA1
A2 --> FGA2
A3 --> FGA1
KC1 --> PG
KC2 --> PG
FGA1 --> PG
FGA2 --> PG
PG -.-> PGR
%% ColorBrewer2 Set3 palette - each component type uniquely colored
classDef externalStyle fill:#8dd3c7,stroke:#2a9d8f,stroke-width:2px,color:#333
classDef ingressStyle fill:#fb8072,stroke:#e74c3c,stroke-width:2px,color:#333
classDef appStyle fill:#b3de69,stroke:#7cb342,stroke-width:2px,color:#333
classDef authStyle fill:#bebada,stroke:#8e44ad,stroke-width:2px,color:#333
classDef sessionStyle fill:#fdb462,stroke:#e67e22,stroke-width:2px,color:#333
classDef authzStyle fill:#d9d9d9,stroke:#95a5a6,stroke-width:2px,color:#333
classDef dataStyle fill:#80b1d3,stroke:#3498db,stroke-width:2px,color:#333
class Internet externalStyle
class LB,IngressCtrl ingressStyle
class A1,A2,A3 appStyle
class KC1,KC2 authStyle
class RM,RR sessionStyle
class FGA1,FGA2 authzStyle
class PG,PGR dataStyle
Deployment: See Kubernetes Guide or Helm Guide
- Pluggable authentication providers - Swappable session stores - Custom authorization models - Extensible tool framework - Pydantic models for all data - Type hints throughout - Validation at boundaries - Property-based testing - Trace every request - Structured logging - Comprehensive metrics - Correlation IDs - Authentication required - Authorization on all tools - Secrets in secure stores - Encrypted communications - Health checks - Graceful shutdown - Error recovery - Rate limiting - Circuit breakers
- **Python 3.10+** - **LangGraph**: Stateful agent framework - **LiteLLM**: Multi-LLM router - **Pydantic**: Data validation - **FastAPI**: HTTP server (StreamableHTTP) - **Keycloak**: SSO and OIDC - **PyJWT**: Token validation - **Redis**: Session storage - **OpenFGA**: Relationship-based authz - **PostgreSQL**: Authorization data - **OpenTelemetry**: Tracing & metrics - **Jaeger**: Trace visualization - **Prometheus**: Metrics storage - **Grafana**: Dashboards - **LangSmith**: LLM observability - **Docker**: Containerization - **Kubernetes**: Orchestration - **Helm**: Package management - **Infisical**: Secrets management
Get started in 5 minutes Configure authentication Setup fine-grained permissions Enable tracing and metrics
**Enterprise-Grade Architecture**: Production-ready components designed for scale and reliability!