Skip to content

Commit ef9e49b

Browse files
vishu-bhjonpspri
vishu-bh
and
jonpspri
authored
feat: Add OpenTelemetry W3C Baggage support for distributed tracing (#4008)
* feat: add OTEL root and client spans for MCP flows Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com> * feat: extend OTEL MCP client trace lifecycle Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com> * feat: implement OpenTelemetry baggage header extraction (GitHub #3976) Implements HTTP header-to-baggage conversion for distributed tracing with strict security controls: **Core Features:** - Dual trust model: UNTRUSTED headers (strict allowlist) vs TRUSTED baggage (permissive) - Size limits (32 items, 8KB) apply ONLY to header-derived baggage - Default behavior: capture as span attributes only - Downstream propagation requires explicit opt-in via config flag **Implementation:** - mcpgateway/baggage.py: Core validation, parsing, and conversion logic - mcpgateway/middleware/baggage_middleware.py: ASGI middleware for extraction - mcpgateway/config.py: 7 new configuration fields with Pydantic validation - mcpgateway/main.py: Middleware registration after OpenTelemetryRequestMiddleware - mcpgateway/observability.py: Auto-inject baggage as span attributes, conditional propagation **Security:** - Reuses existing sanitize_header_value() and ALLOWED_HEADERS_REGEX - Strict allowlist (only configured headers processed) - CRLF injection prevention - Size limits enforced (DoS prevention) - Audit logging for rejected headers - Fail-closed on errors **Configuration:** - OTEL_BAGGAGE_ENABLED: Enable/disable feature (default: false) - OTEL_BAGGAGE_HEADER_MAPPINGS: JSON array of header-to-baggage mappings - OTEL_BAGGAGE_PROPAGATE_TO_EXTERNAL: Enable downstream propagation (default: false) - OTEL_BAGGAGE_MAX_ITEMS: Max items from headers (default: 32) - OTEL_BAGGAGE_MAX_SIZE_BYTES: Max total size (default: 8192) - OTEL_BAGGAGE_LOG_REJECTED: Log rejected headers (default: true) - OTEL_BAGGAGE_LOG_SANITIZATION: Log sanitization events (default: true) **Testing:** - 44 unit tests (baggage.py logic) - 13 integration tests (middleware flow) - 6 E2E tests (full tracing flow) - 18 security tests (deny-path scenarios) Closes #3976 Signed-off-by: Bob Shell <bob@contextforge.ai> Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com> * fix: correct middleware order and inject baggage into request spans - Swap BaggageMiddleware and OpenTelemetryRequestMiddleware registration order (ASGI executes in reverse, so baggage must be set before span creation) - Add baggage injection to OpenTelemetryRequestMiddleware request spans - Update FastTimeUser load test to include baggage test headers - Add baggage configuration documentation to .env.example This ensures baggage attributes appear in all spans, including the root request span created by OpenTelemetryRequestMiddleware. Signed-off-by: Bob Shell <bob@example.com> Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com> * fix: finalize otel baggage header extraction rebase Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com> * chore: update test artifacts and documentation - Update .secrets.baseline - Refine baggage implementation and tests - Update observability documentation Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com> * feat: enhance baggage configuration with comprehensive examples and security documentation - Add structured section header with clear delineation - Document common use cases (multi-tenant, user context, request correlation, feature flags, security) - Provide 4 comprehensive configuration examples with multiple header mappings - Add detailed security implications for OTEL_BAGGAGE_PROPAGATE_TO_EXTERNAL - Include guidance on when to enable external propagation and alternatives - Enhance descriptions for OTEL_BAGGAGE_MAX_ITEMS and OTEL_BAGGAGE_MAX_SIZE_BYTES with W3C spec references Addresses PR feedback for comprehensive baggage configuration documentation. Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com> * fix: correct mock patch location in baggage tracing E2E tests - Patch get_settings in mcpgateway.observability module where it's used - Previously patched mcpgateway.config.get_settings which didn't affect the imported reference - Fixes test_baggage_propagated_to_downstream and test_baggage_sanitized_before_propagation Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com> * test: add comprehensive coverage for baggage middleware main flow - Add test_baggage_middleware_main_flow.py with 7 tests covering lines 210-237 - Add test_baggage_middleware_comprehensive.py with 5 additional middleware tests - Add test_baggage_middleware_coverage.py with 7 edge case tests - Add test_baggage_middleware_final.py with 3 integration-style tests - Add test_baggage_parsing.py with 2 supplementary parsing edge case tests - Add test_observability_baggage_exceptions.py for baggage exception handling - Add test_auth_coverage.py, test_main_baggage_coverage.py, test_main_module_init.py, test_observability_coverage.py for additional coverage - Update test_baggage.py with improved formatting and structure - Update mcpgateway/observability.py with formatting improvements - Achieve 94% coverage for baggage_middleware.py (up from 71%) - Achieve 99% coverage for baggage.py - All 26 new/updated tests passing Remaining coverage gaps (6% in baggage_middleware.py, 1% in baggage.py) are defensive exception paths that are difficult to trigger in unit tests without mocking internal library behavior. Closes coverage gaps identified in diff-cover report for baggage-related files. Signed-off-by: Jonathan Springer <jps@s390x.com> --------- Signed-off-by: Vishu Bhatnagar <vishu.bhatnagar@ibm.com> Signed-off-by: Bob Shell <bob@contextforge.ai> Signed-off-by: Bob Shell <bob@example.com> Signed-off-by: Jonathan Springer <jps@s390x.com> Co-authored-by: Jonathan Springer <jps@s390x.com>
1 parent 65dda0e commit ef9e49b

25 files changed

Lines changed: 5213 additions & 33 deletions

‎.env.example‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2367,6 +2367,120 @@ OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
23672367
# OTEL_BSP_MAX_EXPORT_BATCH_SIZE=512
23682368
# OTEL_BSP_SCHEDULE_DELAY=5000
23692369

2370+
# =============================================================================
2371+
# W3C Baggage Configuration
2372+
# =============================================================================
2373+
# OpenTelemetry baggage allows propagating context across service boundaries.
2374+
# Use this to extract HTTP headers into baggage for distributed tracing.
2375+
#
2376+
# Common use cases:
2377+
# - Multi-tenant request tracking (tenant IDs, organization IDs)
2378+
# - User context propagation (user IDs, session IDs)
2379+
# - Request correlation (trace IDs, request IDs)
2380+
# - Feature flags and A/B testing metadata
2381+
# - Security context (authentication realm, authorization scope)
2382+
2383+
# Enable extraction of HTTP headers into OpenTelemetry baggage
2384+
# Options: true, false (default)
2385+
# When enabled, headers matching OTEL_BAGGAGE_HEADER_MAPPINGS are extracted
2386+
# and attached to the current span's baggage for downstream propagation
2387+
# OTEL_BAGGAGE_ENABLED=false
2388+
2389+
# JSON array mapping HTTP headers to baggage keys
2390+
# Format: [{"header_name": "X-Header", "baggage_key": "baggage.key"}]
2391+
#
2392+
# IMPORTANT: Baggage keys should use dot notation for namespacing
2393+
# Recommended prefixes: tenant., user., request., feature., security.
2394+
#
2395+
# Example 1: Basic multi-tenant tracking
2396+
# OTEL_BAGGAGE_HEADER_MAPPINGS='[
2397+
# {"header_name": "X-Tenant-ID", "baggage_key": "tenant.id"},
2398+
# {"header_name": "X-Organization-ID", "baggage_key": "tenant.org_id"}
2399+
# ]'
2400+
#
2401+
# Example 2: User context and request correlation
2402+
# OTEL_BAGGAGE_HEADER_MAPPINGS='[
2403+
# {"header_name": "X-User-ID", "baggage_key": "user.id"},
2404+
# {"header_name": "X-User-Email", "baggage_key": "user.email"},
2405+
# {"header_name": "X-Session-ID", "baggage_key": "user.session_id"},
2406+
# {"header_name": "X-Request-ID", "baggage_key": "request.id"},
2407+
# {"header_name": "X-Correlation-ID", "baggage_key": "request.correlation_id"}
2408+
# ]'
2409+
#
2410+
# Example 3: Comprehensive tracking (multi-tenant + user + features)
2411+
# OTEL_BAGGAGE_HEADER_MAPPINGS='[
2412+
# {"header_name": "X-Tenant-ID", "baggage_key": "tenant.id"},
2413+
# {"header_name": "X-Tenant-Name", "baggage_key": "tenant.name"},
2414+
# {"header_name": "X-User-ID", "baggage_key": "user.id"},
2415+
# {"header_name": "X-User-Role", "baggage_key": "user.role"},
2416+
# {"header_name": "X-Request-ID", "baggage_key": "request.id"},
2417+
# {"header_name": "X-Feature-Flags", "baggage_key": "feature.flags"},
2418+
# {"header_name": "X-AB-Test-Variant", "baggage_key": "feature.ab_variant"}
2419+
# ]'
2420+
#
2421+
# Example 4: Security and compliance tracking
2422+
# OTEL_BAGGAGE_HEADER_MAPPINGS='[
2423+
# {"header_name": "X-Auth-Realm", "baggage_key": "security.realm"},
2424+
# {"header_name": "X-Auth-Scope", "baggage_key": "security.scope"},
2425+
# {"header_name": "X-Client-IP", "baggage_key": "security.client_ip"},
2426+
# {"header_name": "X-Forwarded-For", "baggage_key": "security.forwarded_for"},
2427+
# {"header_name": "X-Compliance-Level", "baggage_key": "security.compliance_level"}
2428+
# ]'
2429+
2430+
# Propagate baggage to external services
2431+
# Options: true, false (default: false)
2432+
#
2433+
# ⚠️ SECURITY WARNING: DISABLED BY DEFAULT FOR GOOD REASON ⚠️
2434+
#
2435+
# When enabled, baggage is propagated to ALL external HTTP requests made by
2436+
# the gateway, including:
2437+
# - Upstream MCP servers
2438+
# - External APIs and webhooks
2439+
# - Third-party services
2440+
# - Plugin endpoints
2441+
#
2442+
# SECURITY IMPLICATIONS:
2443+
# 1. DATA LEAKAGE: Baggage may contain sensitive tenant/user identifiers that
2444+
# external services should NOT receive. This can leak:
2445+
# - Internal tenant IDs and organizational structure
2446+
# - User identifiers and session tokens
2447+
# - Internal request correlation IDs
2448+
# - Feature flags revealing your product roadmap
2449+
#
2450+
# 2. COMPLIANCE VIOLATIONS: Propagating user data to third parties without
2451+
# consent may violate GDPR, CCPA, HIPAA, or other regulations.
2452+
#
2453+
# 3. ATTACK SURFACE: Malicious external services could harvest baggage data
2454+
# to map your internal architecture or user base.
2455+
#
2456+
# 4. TRUST BOUNDARY: External services are outside your security perimeter.
2457+
# Baggage should only cross trust boundaries when explicitly required.
2458+
#
2459+
# WHEN TO ENABLE:
2460+
# - Only enable if you control ALL external services the gateway calls
2461+
# - Use allowlists to restrict which external hosts receive baggage
2462+
# - Audit external service access to baggage data
2463+
# - Document which external services receive what baggage keys
2464+
# - Consider using separate baggage keys for internal vs external propagation
2465+
#
2466+
# ALTERNATIVES TO CONSIDER:
2467+
# - Use service-specific headers instead of baggage for external calls
2468+
# - Implement a baggage filtering proxy for external requests
2469+
# - Use separate observability backends for internal vs external traces
2470+
#
2471+
# OTEL_BAGGAGE_PROPAGATE_TO_EXTERNAL=false
2472+
2473+
# Maximum number of baggage items (default: 32)
2474+
# Prevents unbounded baggage growth and DoS attacks
2475+
# W3C Baggage spec recommends keeping this low for performance
2476+
# OTEL_BAGGAGE_MAX_ITEMS=32
2477+
2478+
# Maximum total baggage size in bytes (default: 8192)
2479+
# Total size of all baggage key-value pairs combined
2480+
# Prevents header size attacks and network overhead
2481+
# W3C Baggage spec recommends 8KB limit for HTTP header compatibility
2482+
# OTEL_BAGGAGE_MAX_SIZE_BYTES=8192
2483+
23702484
# Prometheus Metrics Configuration
23712485
# Enable Prometheus-compatible metrics endpoint at /metrics/prometheus
23722486
# Options: true, false (default)

‎.secrets.baseline‎

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
"files": "^.secrets.baseline$",
44
"lines": null
55
},
6-
"generated_at": "2026-04-07T19:15:04Z",
6+
"generated_at": "2026-04-07T19:59:03Z",
77
"plugins_used": [
88
{
99
"name": "AWSKeyDetector"
@@ -9166,7 +9166,7 @@
91669166
"hashed_secret": "ff37a98a9963d347e9749a5c1b3936a4a245a6ff",
91679167
"is_secret": false,
91689168
"is_verified": false,
9169-
"line_number": 2179,
9169+
"line_number": 2221,
91709170
"type": "Secret Keyword",
91719171
"verified_result": null
91729172
}
@@ -18920,31 +18920,31 @@
1892018920
"hashed_secret": "d033e22ae348aeb5660fc2140aec35850c4da997",
1892118921
"is_secret": false,
1892218922
"is_verified": false,
18923-
"line_number": 3784,
18923+
"line_number": 3792,
1892418924
"type": "Secret Keyword",
1892518925
"verified_result": null
1892618926
},
1892718927
{
1892818928
"hashed_secret": "d73bccb432c5c7b1b166cfa603b3b99af63fd580",
1892918929
"is_secret": false,
1893018930
"is_verified": false,
18931-
"line_number": 6627,
18931+
"line_number": 6635,
1893218932
"type": "Secret Keyword",
1893318933
"verified_result": null
1893418934
},
1893518935
{
1893618936
"hashed_secret": "20d22126242b5c7e253d84dd6ff0ed7a6d2662a2",
1893718937
"is_secret": false,
1893818938
"is_verified": false,
18939-
"line_number": 6983,
18939+
"line_number": 6991,
1894018940
"type": "Secret Keyword",
1894118941
"verified_result": null
1894218942
},
1894318943
{
1894418944
"hashed_secret": "fa9beb99e4029ad5a6615399e7bbae21356086b3",
1894518945
"is_secret": false,
1894618946
"is_verified": false,
18947-
"line_number": 7007,
18947+
"line_number": 7015,
1894818948
"type": "Secret Keyword",
1894918949
"verified_result": null
1895018950
}
@@ -21436,55 +21436,55 @@
2143621436
"hashed_secret": "9fb7fe1217aed442b04c0f5e43b5d5a7d3287097",
2143721437
"is_secret": false,
2143821438
"is_verified": false,
21439-
"line_number": 2870,
21439+
"line_number": 2871,
2144021440
"type": "Secret Keyword",
2144121441
"verified_result": null
2144221442
},
2144321443
{
2144421444
"hashed_secret": "72cb70dbbafe97e5ea13ad88acd65d08389439b0",
2144521445
"is_secret": false,
2144621446
"is_verified": false,
21447-
"line_number": 3498,
21447+
"line_number": 3499,
2144821448
"type": "Secret Keyword",
2144921449
"verified_result": null
2145021450
},
2145121451
{
2145221452
"hashed_secret": "ee977806d7286510da8b9a7492ba58e2484c0ecc",
2145321453
"is_secret": false,
2145421454
"is_verified": false,
21455-
"line_number": 5777,
21455+
"line_number": 5778,
2145621456
"type": "Secret Keyword",
2145721457
"verified_result": null
2145821458
},
2145921459
{
2146021460
"hashed_secret": "f2e7745f43b0ef0e2c2faf61d6c6a28be2965750",
2146121461
"is_secret": false,
2146221462
"is_verified": false,
21463-
"line_number": 6269,
21463+
"line_number": 6270,
2146421464
"type": "Secret Keyword",
2146521465
"verified_result": null
2146621466
},
2146721467
{
2146821468
"hashed_secret": "4a249743d4d2241bd2ae085b4fe654d089488295",
2146921469
"is_secret": false,
2147021470
"is_verified": false,
21471-
"line_number": 7514,
21471+
"line_number": 7515,
2147221472
"type": "Secret Keyword",
2147321473
"verified_result": null
2147421474
},
2147521475
{
2147621476
"hashed_secret": "0c8d051d3c7eada5d31b53d9936fce6bcc232ae2",
2147721477
"is_secret": false,
2147821478
"is_verified": false,
21479-
"line_number": 7652,
21479+
"line_number": 7653,
2148021480
"type": "Secret Keyword",
2148121481
"verified_result": null
2148221482
},
2148321483
{
2148421484
"hashed_secret": "f2b14f68eb995facb3a1c35287b778d5bd785511",
2148521485
"is_secret": false,
2148621486
"is_verified": false,
21487-
"line_number": 8029,
21487+
"line_number": 8030,
2148821488
"type": "Secret Keyword",
2148921489
"verified_result": null
2149021490
}

0 commit comments

Comments
 (0)