Skip to content

Latest commit

 

History

History
896 lines (736 loc) · 24.6 KB

File metadata and controls

896 lines (736 loc) · 24.6 KB
title Architecture
description System architecture and component overview
icon sitemap
seoTitle Architecture: MCP Server with LangGraph System Design
seoDescription Understand the architecture of MCP Server with LangGraph: multi-LLM routing, OpenFGA authorization, Redis sessions, observability stack, and deployment patterns.
keywords
MCP architecture
LangGraph design
microservices
AI agent architecture
system design
contentType explanation

Overview

MCP Server with LangGraph is a production-ready, multi-layered architecture designed for enterprise AI applications. It combines stateful AI agents, enterprise authentication, fine-grained authorization, and comprehensive observability.

High-Level Architecture

flowchart TB
    subgraph Client_Layer["Client Layer"]
        WebApp[Web Application]
        CLI[CLI Tools]
        API[API Clients]
    end

    subgraph Gateway_Layer["Gateway Layer"]
        Ingress[Ingress/Load Balancer]
        Kong[Kong API Gateway]
    end

    subgraph Application_Layer["Application Layer"]
        MCP[MCP Server]
        Agent[LangGraph Agent]
        Auth[Auth Middleware]
    end

    subgraph AI_LLM_Layer["AI/LLM Layer"]
        LiteLLM[LiteLLM Router]
        Anthropic[Anthropic Claude]
        Google[Google Gemini]
        OpenAI[OpenAI GPT]
        Ollama[Ollama Local]
    end

    subgraph Authentication_Layer["Authentication Layer"]
        Keycloak[Keycloak SSO]
        JWT[JWT Validation]
        Sessions[Session Manager]
    end

    subgraph Authorization_Layer["Authorization Layer"]
        OpenFGA[OpenFGA]
        RoleMapper[Role Mapper]
    end

    subgraph Data_Layer["Data Layer"]
        Redis[Redis Sessions]
        Postgres[(PostgreSQL)]
    end

    subgraph Observability_Layer["Observability Layer"]
        OTel[OpenTelemetry]
        Jaeger[Jaeger Tracing]
        Prometheus[Prometheus Metrics]
        LangSmith[LangSmith]
    end

    subgraph Secrets_Layer["Secrets Layer"]
        Infisical[Infisical]
        K8sSecrets[Kubernetes Secrets]
    end

    WebApp --> Ingress
    CLI --> Ingress
    API --> Ingress
    Ingress --> Kong
    Kong --> MCP
    MCP --> Agent
    MCP --> Auth
    Auth --> JWT
    Auth --> Sessions
    Auth --> Keycloak
    Auth --> OpenFGA
    Sessions --> Redis
    OpenFGA --> RoleMapper
    RoleMapper --> Keycloak
    OpenFGA --> Postgres
    Keycloak --> Postgres
    Agent --> LiteLLM
    LiteLLM --> Anthropic
    LiteLLM --> Google
    LiteLLM --> OpenAI
    LiteLLM --> Ollama
    Agent --> OTel
    OTel --> Jaeger
    OTel --> Prometheus
    Agent --> LangSmith
    Agent --> Infisical
    Agent --> K8sSecrets

    %% ColorBrewer2 Set3 (12-color) palette - optimized for node type distinction
    %% Each layer uses distinct colors to avoid visual confusion
    classDef clientStyle fill:#8dd3c7,stroke:#2a9d8f,stroke-width:2px,color:#333
    classDef ingressStyle fill:#fb8072,stroke:#e74c3c,stroke-width:2px,color:#333
    classDef kongStyle fill:#fdb462,stroke:#e67e22,stroke-width:2px,color:#333
    classDef mcpStyle fill:#ffed6f,stroke:#f39c12,stroke-width:2px,color:#333
    classDef agentStyle fill:#b3de69,stroke:#7cb342,stroke-width:2px,color:#333
    classDef authStyle fill:#bebada,stroke:#8e7cc3,stroke-width:2px,color:#333
    classDef llmRouterStyle fill:#fdb462,stroke:#e67e22,stroke-width:2px,color:#333
    classDef llmProviderStyle fill:#fccde5,stroke:#ec7ab8,stroke-width:2px,color:#333
    classDef authSvcStyle fill:#bebada,stroke:#8e7cc3,stroke-width:2px,color:#333
    classDef authzStyle fill:#d9d9d9,stroke:#95a5a6,stroke-width:2px,color:#333
    classDef redisStyle fill:#80b1d3,stroke:#3498db,stroke-width:2px,color:#333
    classDef secretsStyle fill:#ccebc5,stroke:#82c99a,stroke-width:2px,color:#333

    class WebApp,CLI,API clientStyle
    class Ingress ingressStyle
    class Kong,LiteLLM kongStyle
    class MCP mcpStyle
    class Agent agentStyle
    class Auth authStyle
    class Anthropic,Google,OpenAI,Ollama llmProviderStyle
    class Keycloak,JWT,Sessions authSvcStyle
    class OpenFGA,RoleMapper,Postgres authzStyle
    class Redis redisStyle
    class OTel,Jaeger,Prometheus,LangSmith obsStyle
    class Infisical,K8sSecrets secretsStyle

    %% Additional styling for observability layer distinction
    %% ColorBrewer2 Set3 palette - each component type uniquely colored
    classDef obsStyle fill:#ffffb3,stroke:#f1c40f,stroke-width:2px,color:#333
    classDef langsmithStyle fill:#bc80bd,stroke:#8e44ad,stroke-width:2px,color:#333

    class OTel obsStyle
    class Jaeger,Prometheus obsStyle
    class LangSmith langsmithStyle
Loading

Core Components

MCP Server

Model Context Protocol server - Exposes AI agents as standard tools.

Standard input/output for CLI integration - Direct agent interaction - Shell scripting support - Development and testing HTTP API with streaming support - REST endpoints - Server-Sent Events (SSE) - Production deployments

Key Features:

  • Protocol-compliant MCP implementation
  • Tool registration and discovery
  • Streaming responses
  • Error handling and retries

Code Location: src/mcp_server_langgraph/server.py


LangGraph Agent

Stateful AI agent with functional API and conditional routing.

## Agent graph structure
START → authenticate → authorize → execute_tool → END
                ↓           ↓            ↓
          unauthorized  forbidden   tool_error
                ↓           ↓            ↓
               END         END       retry/END

Components:

**AgentState** - Pydantic model for type-safe state
```python
class AgentState(BaseModel):
    messages: List[Message]
    user_id: str
    authorized: bool
    context: Dict[str, Any]
    tool_results: List[Any]
```
**Features**:
- Immutable state transitions
- Type validation
- Checkpointing support
**Dynamic tool routing** based on LLM decisions
```python
tools = {
    "chat": chat_tool,
    "search": search_tool,
    # Add custom tools
}
```
**Authorization**:
- Per-tool permission checks
- OpenFGA relationship validation
- Fallback admin access
**State persistence** for conversation history
- In-memory checkpoints (development)
- Redis checkpoints (production)
- Resume interrupted conversations
- Audit trail

Code Location: src/mcp_server_langgraph/agent.py


Authentication Layer

Pluggable authentication with multiple provider support.

%%{init: {'theme':'base', 'themeVariables': {'actorBkg':'#8dd3c7','actorBorder':'#2a9d8f','actorTextColor':'#333','actorLineColor':'#2a9d8f','signalColor':'#7cb342','signalTextColor':'#333','labelBoxBkgColor':'#ffffb3','labelBoxBorderColor':'#f1c40f','labelTextColor':'#333','noteBorderColor':'#e67e22','noteBkgColor':'#fdb462','noteTextColor':'#333','activationBorderColor':'#7cb342','activationBkgColor':'#b3de69','sequenceNumberColor':'#fff'}}}%%
sequenceDiagram
    participant Client
    participant AuthMiddleware
    participant UserProvider
    participant Keycloak
    participant SessionStore

    Client->>AuthMiddleware: Request + credentials
    AuthMiddleware->>UserProvider: authenticate()
    UserProvider->>Keycloak: OIDC login
    Keycloak-->>UserProvider: Access + Refresh tokens
    UserProvider-->>AuthMiddleware: AuthResponse
    AuthMiddleware->>SessionStore: create_session()
    SessionStore-->>AuthMiddleware: Session ID
    AuthMiddleware-->>Client: Set-Cookie: session_id
Loading

Providers:

**Development and testing**
- Pre-defined users (alice, bob, admin)
- No external dependencies
- Fast iteration
- Zero configuration

```python
users = {
    "alice": User(username="alice", roles=["admin"]),
    "bob": User(username="bob", roles=["user"])
}
```
**Production SSO**
- OpenID Connect / OAuth2
- JWKS token verification
- Refresh token rotation
- Role/group synchronization

```python
provider = KeycloakUserProvider(
    server_url="https://sso.yourdomain.com",
    realm="mcp-server-langgraph",
    client_id="langgraph-client"
)
```
**Extensible architecture**
Implement `UserProvider` interface:

```python
class CustomProvider(UserProvider):
    async def authenticate(self, username, password):
        # Custom auth logic
        return AuthResponse(...)

    async def verify_token(self, token):
        # Custom verification
        return TokenVerification(...)
```

Code Location: src/mcp_server_langgraph/auth/


Authorization Layer

Fine-grained, relationship-based access control with OpenFGA.

%% ColorBrewer2 Set3 palette - each component type uniquely colored
flowchart LR
    User[User: alice] -->|admin| Org[Org: acme]
    Org -->|member| Tool1[Tool: chat]
    Org -->|member| Tool2[Tool: search]
    User2[User: bob] -->|member| Org

Loading

Authorization Model:

type user

type organization
  relations
    define admin: [user]
    define member: [user] or admin

type tool
  relations
    define owner: [user]
    define executor: [user, organization#member]
    define viewer: [user, organization#member]

Features:

  • Relationship-based permissions
  • Hierarchical roles (admin → member → viewer)
  • Multi-tenancy support
  • Audit logging
  • Keycloak role synchronization

Code Location: src/mcp_server_langgraph/auth/openfga.py


Session Management

Flexible session storage with Redis or in-memory backends.

```mermaid flowchart LR App[Application] --> SM[SessionManager] SM --> Memory[InMemory Store] SM --> Redis[Redis Store] Redis --> Master[Redis Master] Redis --> Replica[Redis Replica]
    %% ColorBrewer2 Set3 palette styling
    classDef appStyle fill:#b3de69,stroke:#7cb342,stroke-width:2px,color:#333
    classDef managerStyle fill:#fdb462,stroke:#e67e22,stroke-width:2px,color:#333
    classDef storeStyle fill:#bebada,stroke:#8e7cc3,stroke-width:2px,color:#333
    classDef redisStyle fill:#fb8072,stroke:#e74c3c,stroke-width:2px,color:#333

    class App appStyle
    class SM managerStyle
    class Memory storeStyle
    class Redis redisStyle
    class Master,Replica redisStyle
```
**Session Lifecycle**: - Creation with metadata - Retrieval and validation - Refresh with sliding window - Revocation (single or bulk)
**Advanced**:
- Concurrent session limits
- IP/User agent tracking
- TTL management
- Automatic cleanup
```python # Redis backend SESSION_BACKEND=redis REDIS_URL=redis://redis:6379/0 SESSION_TTL_SECONDS=86400 SESSION_SLIDING_WINDOW=true SESSION_MAX_CONCURRENT=5
# In-memory backend
SESSION_BACKEND=memory
```

Code Location: src/mcp_server_langgraph/auth/session.py


LLM Integration

Multi-LLM routing via LiteLLM with automatic fallback.

flowchart TB
    Agent[LangGraph Agent] --> LLMFactory[LLM Factory]
    LLMFactory --> LiteLLM[LiteLLM Router]

    LiteLLM --> Primary[Primary Model]
    LiteLLM --> Fallback1[Fallback 1]
    LiteLLM --> Fallback2[Fallback 2]

    Primary -.->|Error| Fallback1
    Fallback1 -.->|Error| Fallback2

    Primary --> Claude[Claude 3.5 Sonnet]
    Fallback1 --> Gemini[Gemini 2.5 Flash]
    Fallback2 --> GPT[GPT-5]

    %% ColorBrewer2 Set3 palette styling
    classDef agentStyle fill:#b3de69,stroke:#7cb342,stroke-width:2px,color:#333
    classDef routerStyle fill:#fdb462,stroke:#e67e22,stroke-width:2px,color:#333
    classDef primaryStyle fill:#8dd3c7,stroke:#2a9d8f,stroke-width:2px,color:#333
    classDef fallbackStyle fill:#bebada,stroke:#8e7cc3,stroke-width:2px,color:#333
    classDef llmStyle fill:#fccde5,stroke:#ec7ab8,stroke-width:2px,color:#333

    class Agent agentStyle
    class LLMFactory,LiteLLM routerStyle
    class Primary,Claude primaryStyle
    class Fallback1,Gemini,Fallback2,GPT fallbackStyle
Loading

Supported Providers (100+):

  • Cloud: Anthropic, OpenAI, Google, Azure, AWS Bedrock
  • Open Source: Ollama (Llama, Mistral, Qwen, DeepSeek)
  • Custom: Bring your own endpoints

Features:

  • Automatic fallback on errors
  • Load balancing
  • Rate limiting
  • Cost tracking
  • Response caching

Configuration:

LLM_PROVIDER=anthropic
MODEL_NAME=claude-sonnet-4-5-20250929
ENABLE_FALLBACK=true
FALLBACK_MODELS=["gemini-2.5-flash-002", "gpt-5.1"]

Code Location: src/mcp_server_langgraph/llm_factory.py


Observability

Dual observability stack - OpenTelemetry + LangSmith.

**Distributed tracing and metrics**
```mermaid
flowchart LR
    App[Application] --> Exporter[OTLP Exporter]
    Exporter --> Collector[OTel Collector]
    Collector --> Jaeger[Jaeger UI]
    Collector --> Prometheus[Prometheus]
    Prometheus --> Grafana[Grafana]

    %% ColorBrewer2 Set3 palette styling
    classDef appStyle fill:#b3de69,stroke:#7cb342,stroke-width:2px,color:#333
    classDef exporterStyle fill:#fdb462,stroke:#e67e22,stroke-width:2px,color:#333
    classDef collectorStyle fill:#bebada,stroke:#8e7cc3,stroke-width:2px,color:#333
    classDef backendStyle fill:#ffffb3,stroke:#f1c40f,stroke-width:2px,color:#333

    class App appStyle
    class Exporter exporterStyle
    class Collector collectorStyle
    class Jaeger,Prometheus,Grafana backendStyle
```
**Traces**:
- End-to-end request flow
- LLM call timing
- Authorization decisions
- Tool executions

**Metrics** (30+):
- Request rate, latency, errors
- Authentication success/failure
- Authorization decisions
- LLM token usage
- Session lifecycle
**LLM-specific observability**
- Prompt tracking
- LLM response analysis
- Chain visualization
- Cost attribution
- A/B testing
- Evaluations

```python
LANGSMITH_TRACING=true
LANGSMITH_API_KEY=your-key
LANGSMITH_PROJECT=mcp-server-langgraph
```
**JSON logging with context**
```json
{
  "timestamp": "2025-10-12T10:30:00Z",
  "level": "INFO",
  "service": "mcp-server-langgraph",
  "trace_id": "abc123...",
  "span_id": "def456...",
  "user_id": "alice",
  "event": "authorization_check",
  "resource": "tool:chat",
  "authorized": true,
  "duration_ms": 15
}
```

Code Location: src/mcp_server_langgraph/observability/


Secrets Management

Secure secret storage with Infisical or cloud-native solutions.

**Centralized secret management**
- End-to-end encryption
- Secret versioning
- Access controls
- Audit logging
- Secret rotation

```python
INFISICAL_CLIENT_ID=your-client-id
INFISICAL_CLIENT_SECRET=your-secret
INFISICAL_PROJECT_ID=project-id
```
**Native secret managers**
- **GCP**: Secret Manager
- **AWS**: Secrets Manager
- **Azure**: Key Vault
- **Kubernetes**: Secrets + RBAC

Automatic injection via workload identity.
**.env files with validation**
```bash
# .env
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_API_KEY=AIza...
JWT_SECRET_KEY=$(openssl rand -base64 32)
```
<Warning>
Never commit `.env` files to Git!
</Warning>

Code Location: src/mcp_server_langgraph/config.py


Data Flow

Request Flow (Authenticated Request)

%%{init: {'theme':'base', 'themeVariables': {'actorBkg':'#8dd3c7','actorBorder':'#2a9d8f','actorTextColor':'#333','actorLineColor':'#2a9d8f','signalColor':'#7cb342','signalTextColor':'#333','labelBoxBkgColor':'#ffffb3','labelBoxBorderColor':'#f1c40f','labelTextColor':'#333','noteBorderColor':'#e67e22','noteBkgColor':'#fdb462','noteTextColor':'#333','activationBorderColor':'#7cb342','activationBkgColor':'#b3de69','sequenceNumberColor':'#fff'}}}%%
sequenceDiagram
    participant Client
    participant MCP as MCP Server
    participant Auth as Auth Middleware
    participant Authz as OpenFGA
    participant Agent as LangGraph Agent
    participant LLM
    participant OTel as Observability

    Client->>MCP: POST /message (+ session_id)
    MCP->>OTel: Start trace
    MCP->>Auth: Verify session
    Auth->>Auth: Get session from Redis
    Auth->>Auth: Validate session TTL
    Auth-->>MCP: User: alice

    MCP->>Authz: Check permission (alice, executor, tool:chat)
    Authz-->>MCP: Authorized: true

    MCP->>Agent: Execute with context
    Agent->>LLM: Generate response
    LLM-->>Agent: Response + token usage
    Agent->>OTel: Record metrics
    Agent-->>MCP: AgentResponse

    MCP->>Auth: Refresh session (if sliding)
    MCP->>OTel: End trace
    MCP-->>Client: Response + trace_id
Loading

Authentication Flow (Keycloak SSO)

%%{init: {'theme':'base', 'themeVariables': {'actorBkg':'#8dd3c7','actorBorder':'#2a9d8f','actorTextColor':'#333','actorLineColor':'#2a9d8f','signalColor':'#7cb342','signalTextColor':'#333','labelBoxBkgColor':'#ffffb3','labelBoxBorderColor':'#f1c40f','labelTextColor':'#333','noteBorderColor':'#e67e22','noteBkgColor':'#fdb462','noteTextColor':'#333','activationBorderColor':'#7cb342','activationBkgColor':'#b3de69','sequenceNumberColor':'#fff'}}}%%
sequenceDiagram
    participant User
    participant App as MCP Server
    participant KC as Keycloak
    participant Redis
    participant FGA as OpenFGA

    User->>App: Login request
    App->>KC: Redirect to login
    User->>KC: Enter credentials
    KC->>KC: Authenticate
    KC-->>App: Authorization code
    App->>KC: Exchange code for tokens
    KC-->>App: Access + Refresh tokens
    App->>KC: Get user info
    KC-->>App: User profile + roles
    App->>FGA: Sync roles to OpenFGA
    App->>Redis: Create session
    App-->>User: Set session cookie
Loading

Deployment Architectures

Development (Docker Compose)

flowchart TB
    subgraph Docker_Host["Docker Host"]
        Agent[Agent Container]
        KC[Keycloak Container]
        Redis[Redis Container]
        FGA[OpenFGA Container]
        PG[(PostgreSQL)]
        Jaeger[Jaeger Container]
        Prom[Prometheus]
    end

    Agent --> KC
    Agent --> Redis
    Agent --> FGA
    Agent --> Jaeger
    KC --> PG
    FGA --> PG
    Prom --> Agent

    %% ColorBrewer2 Set3 palette - each service type gets unique color
    classDef agentStyle fill:#b3de69,stroke:#7cb342,stroke-width:2px,color:#333
    classDef authStyle fill:#bebada,stroke:#8e7cc3,stroke-width:2px,color:#333
    classDef cacheStyle fill:#fb8072,stroke:#e74c3c,stroke-width:2px,color:#333
    classDef authzStyle fill:#d9d9d9,stroke:#95a5a6,stroke-width:2px,color:#333
    classDef dataStyle fill:#80b1d3,stroke:#3498db,stroke-width:2px,color:#333
    classDef tracingStyle fill:#ffffb3,stroke:#f1c40f,stroke-width:2px,color:#333
    classDef metricsStyle fill:#fdb462,stroke:#e67e22,stroke-width:2px,color:#333

    class Agent agentStyle
    class KC authStyle
    class Redis cacheStyle
    class FGA authzStyle
    class PG dataStyle
    class Jaeger tracingStyle
    class Prom metricsStyle
Loading

Command: docker compose up


Production (Kubernetes)

flowchart TB
    Internet[Internet]

    subgraph Kubernetes_Cluster["Kubernetes Cluster"]
        subgraph Ingress_Layer["Ingress"]
            LB[Load Balancer]
            IngressCtrl[Ingress Controller]
        end

        subgraph Application_Pods["Application Pods"]
            A1[Agent Pod 1]
            A2[Agent Pod 2]
            A3[Agent Pod 3]
        end

        subgraph Auth_Services["Auth Services"]
            KC1[Keycloak Pod 1]
            KC2[Keycloak Pod 2]
        end

        subgraph Session_Store["Session Store"]
            RM[Redis Master]
            RR[Redis Replica]
        end

        subgraph Authorization_Layer["Authorization"]
            FGA1[OpenFGA Pod 1]
            FGA2[OpenFGA Pod 2]
        end

        subgraph Data_Layer["Data"]
            PG[(PostgreSQL Primary)]
            PGR[(PostgreSQL Replica)]
        end
    end

    Internet --> LB
    LB --> IngressCtrl
    IngressCtrl --> A1
    IngressCtrl --> A2
    IngressCtrl --> A3
    A1 --> KC1
    A2 --> KC2
    A3 --> KC1
    A1 --> RM
    A2 --> RM
    A3 --> RM
    RM -.-> RR
    A1 --> FGA1
    A2 --> FGA2
    A3 --> FGA1
    KC1 --> PG
    KC2 --> PG
    FGA1 --> PG
    FGA2 --> PG
    PG -.-> PGR

    %% ColorBrewer2 Set3 palette - each component type uniquely colored
    classDef externalStyle fill:#8dd3c7,stroke:#2a9d8f,stroke-width:2px,color:#333
    classDef ingressStyle fill:#fb8072,stroke:#e74c3c,stroke-width:2px,color:#333
    classDef appStyle fill:#b3de69,stroke:#7cb342,stroke-width:2px,color:#333
    classDef authStyle fill:#bebada,stroke:#8e44ad,stroke-width:2px,color:#333
    classDef sessionStyle fill:#fdb462,stroke:#e67e22,stroke-width:2px,color:#333
    classDef authzStyle fill:#d9d9d9,stroke:#95a5a6,stroke-width:2px,color:#333
    classDef dataStyle fill:#80b1d3,stroke:#3498db,stroke-width:2px,color:#333

    class Internet externalStyle
    class LB,IngressCtrl ingressStyle
    class A1,A2,A3 appStyle
    class KC1,KC2 authStyle
    class RM,RR sessionStyle
    class FGA1,FGA2 authzStyle
    class PG,PGR dataStyle
Loading

Deployment: See Kubernetes Guide or Helm Guide


Design Principles

- Pluggable authentication providers - Swappable session stores - Custom authorization models - Extensible tool framework - Pydantic models for all data - Type hints throughout - Validation at boundaries - Property-based testing - Trace every request - Structured logging - Comprehensive metrics - Correlation IDs - Authentication required - Authorization on all tools - Secrets in secure stores - Encrypted communications - Health checks - Graceful shutdown - Error recovery - Rate limiting - Circuit breakers

Technology Stack

- **Python 3.10+** - **LangGraph**: Stateful agent framework - **LiteLLM**: Multi-LLM router - **Pydantic**: Data validation - **FastAPI**: HTTP server (StreamableHTTP) - **Keycloak**: SSO and OIDC - **PyJWT**: Token validation - **Redis**: Session storage - **OpenFGA**: Relationship-based authz - **PostgreSQL**: Authorization data - **OpenTelemetry**: Tracing & metrics - **Jaeger**: Trace visualization - **Prometheus**: Metrics storage - **Grafana**: Dashboards - **LangSmith**: LLM observability - **Docker**: Containerization - **Kubernetes**: Orchestration - **Helm**: Package management - **Infisical**: Secrets management

Next Steps

Get started in 5 minutes Configure authentication Setup fine-grained permissions Enable tracing and metrics
**Enterprise-Grade Architecture**: Production-ready components designed for scale and reliability!