Support MCP observability for Envoy AI Gateway (#13791)

**MAL rules** (new files):
- `gateway-mcp-service.yaml` — 13 MCP service-level metrics (request CPM/latency/percentile, method CPM, error CPM, initialization latency, capabilities, per-backend breakdown)
- `gateway-mcp-instance.yaml` — 13 MCP instance-level metrics

**LAL rules** (modified `envoy-ai-gateway.yaml`):
- Split into two rules: `envoy-ai-gateway-llm-access-log` and `envoy-ai-gateway-mcp-access-log`
- LLM logs: persist error responses (>= 400) and upstream failures only
- MCP logs: persist error responses (>= 400) only
- Both rules tag `ai_route_type` (`llm` or `mcp`) for searchable filtering

**Dashboard** (modified service + instance JSON):
- Added **MCP** tab with 9 widgets (service) / 6 widgets (instance): request CPM, latency avg/percentile, error CPM, method CPM, initialization latency, backend breakdown

**E2E test** (modified):
- Added `mcp-server` service (`tzolov/mcp-everything-server:v3` — MCP reference server with StreamableHttp)
- Added MCP request steps (initialize + tools/list + tools/call)
- Added MCP metric verification cases
- Log query uses `ai_route_type=llm` tag filter

**Config**:
- Added `ai_route_type` to `searchableLogsTags` in `application.yml`
- Fixed aigw healthcheck binary path (`/app` instead of `aigw`)

Author: wu-sheng

docs/en/changes/changes.md

@@ -11,6 +11,7 @@
 * Fix missing parentheses around OR conditions in `JDBCZipkinQueryDAO.getTraces()`, which caused the table filter to be bypassed for all but the first trace ID. Replaced with a proper `IN` clause.
 * Fix missing `and` keyword in `JDBCEBPFProfilingTaskDAO.getTaskRecord()` SQL query, which caused a syntax error on every invocation.
 * Fix duplicate `TABLE_COLUMN` condition in `JDBCMetadataQueryDAO.findEndpoint()`, which was binding the same parameter twice due to a copy-paste error.
+* Support MCP (Model Context Protocol) observability for Envoy AI Gateway: MCP metrics (request CPM/latency, method breakdown, backend breakdown, initialization latency, capabilities), MCP access log sampling (errors only), `ai_route_type` searchable log tag, and MCP dashboard tabs.
 
 #### UI
 

docs/en/setup/backend/backend-envoy-ai-gateway-monitoring.md

@@ -4,8 +4,9 @@
 
 [Envoy AI Gateway](https://aigateway.envoyproxy.io/) is a gateway/proxy for AI/LLM API traffic
 (OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, Google Gemini, etc.) built on top of Envoy Proxy.
-It natively emits GenAI metrics and access logs via OTLP, following
-[OpenTelemetry GenAI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/).
+It natively emits GenAI metrics following
+[OpenTelemetry GenAI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/),
+and also emits MCP (Model Context Protocol) metrics and access logs via OTLP.
 
 SkyWalking receives OTLP metrics and logs directly on its gRPC port (11800) — no OpenTelemetry
 Collector is needed between the AI Gateway and SkyWalking OAP.
@@ -15,7 +16,7 @@ Collector is needed between the AI Gateway and SkyWalking OAP.
   [Envoy AI Gateway getting started](https://aigateway.envoyproxy.io/docs/getting-started/) for installation.
 
 ### Data flow
-1. Envoy AI Gateway processes LLM API requests and records GenAI metrics (token usage, latency, TTFT, TPOT).
+1. Envoy AI Gateway processes LLM API requests and MCP requests, recording GenAI metrics and MCP metrics.
 2. The AI Gateway pushes metrics and access logs via OTLP gRPC to SkyWalking OAP.
 3. SkyWalking OAP parses metrics with [MAL](../../concepts-and-designs/mal.md) rules and access logs
    with [LAL](../../concepts-and-designs/lal.md) rules.
@@ -27,14 +28,14 @@ in SkyWalking OAP. No OAP-side configuration is needed.
 
 Configure the AI Gateway to push OTLP to SkyWalking by setting these environment variables:
 
-| Env Var | Value | Purpose |
-|---------|-------|---------|
-| `OTEL_SERVICE_NAME` | Per-deployment gateway name (e.g., `my-ai-gateway`) | SkyWalking service name |
-| `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://skywalking-oap:11800` | SkyWalking OAP gRPC receiver |
-| `OTEL_EXPORTER_OTLP_PROTOCOL` | `grpc` | OTLP transport |
-| `OTEL_METRICS_EXPORTER` | `otlp` | Enable OTLP metrics push |
-| `OTEL_LOGS_EXPORTER` | `otlp` | Enable OTLP access log push |
-| `OTEL_RESOURCE_ATTRIBUTES` | See below | Routing + instance + layer |
+| Env Var                       | Value                                               | Purpose                      |
+|-------------------------------|-----------------------------------------------------|------------------------------|
+| `OTEL_SERVICE_NAME`           | Per-deployment gateway name (e.g., `my-ai-gateway`) | SkyWalking service name      |
+| `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://skywalking-oap:11800`                       | SkyWalking OAP gRPC receiver |
+| `OTEL_EXPORTER_OTLP_PROTOCOL` | `grpc`                                              | OTLP transport               |
+| `OTEL_METRICS_EXPORTER`       | `otlp`                                              | Enable OTLP metrics push     |
+| `OTEL_LOGS_EXPORTER`          | `otlp`                                              | Enable OTLP access log push  |
+| `OTEL_RESOURCE_ATTRIBUTES`    | See below                                           | Routing + instance + layer   |
 
 **Required resource attributes** (in `OTEL_RESOURCE_ATTRIBUTES`):
 - `job_name=envoy-ai-gateway` — Fixed routing tag for MAL/LAL rules. Same for all AI Gateway deployments.
@@ -58,47 +59,86 @@ is a service, each pod is an instance. Metrics include per-provider and per-mode
 
 #### Service Metrics
 
-| Monitoring Panel | Unit | Metric Name | Description |
-|---|---|---|---|
-| Request CPM | calls/min | meter_envoy_ai_gw_request_cpm | Requests per minute |
-| Request Latency Avg | ms | meter_envoy_ai_gw_request_latency_avg | Average request duration |
-| Request Latency Percentile | ms | meter_envoy_ai_gw_request_latency_percentile | P50/P75/P90/P95/P99 |
-| Input Token Rate | tokens/min | meter_envoy_ai_gw_input_token_rate | Input (prompt) tokens per minute |
-| Output Token Rate | tokens/min | meter_envoy_ai_gw_output_token_rate | Output (completion) tokens per minute |
-| TTFT Avg | ms | meter_envoy_ai_gw_ttft_avg | Time to First Token (streaming only) |
-| TTFT Percentile | ms | meter_envoy_ai_gw_ttft_percentile | P50/P75/P90/P95/P99 TTFT |
-| TPOT Avg | ms | meter_envoy_ai_gw_tpot_avg | Time Per Output Token (streaming only) |
-| TPOT Percentile | ms | meter_envoy_ai_gw_tpot_percentile | P50/P75/P90/P95/P99 TPOT |
+| Monitoring Panel           | Unit       | Metric Name                                  | Description                            |
+|----------------------------|------------|----------------------------------------------|----------------------------------------|
+| Request CPM                | calls/min  | meter_envoy_ai_gw_request_cpm                | Requests per minute                    |
+| Request Latency Avg        | ms         | meter_envoy_ai_gw_request_latency_avg        | Average request duration               |
+| Request Latency Percentile | ms         | meter_envoy_ai_gw_request_latency_percentile | P50/P75/P90/P95/P99                    |
+| Input Token Rate           | tokens/min | meter_envoy_ai_gw_input_token_rate           | Input (prompt) tokens per minute       |
+| Output Token Rate          | tokens/min | meter_envoy_ai_gw_output_token_rate          | Output (completion) tokens per minute  |
+| TTFT Avg                   | ms         | meter_envoy_ai_gw_ttft_avg                   | Time to First Token (streaming only)   |
+| TTFT Percentile            | ms         | meter_envoy_ai_gw_ttft_percentile            | P50/P75/P90/P95/P99 TTFT               |
+| TPOT Avg                   | ms         | meter_envoy_ai_gw_tpot_avg                   | Time Per Output Token (streaming only) |
+| TPOT Percentile            | ms         | meter_envoy_ai_gw_tpot_percentile            | P50/P75/P90/P95/P99 TPOT               |
 
 #### Provider Breakdown Metrics
 
-| Monitoring Panel | Unit | Metric Name | Description |
-|---|---|---|---|
-| Provider Request CPM | calls/min | meter_envoy_ai_gw_provider_request_cpm | Requests by provider |
-| Provider Token Rate | tokens/min | meter_envoy_ai_gw_provider_token_rate | Token rate by provider |
-| Provider Latency Avg | ms | meter_envoy_ai_gw_provider_latency_avg | Latency by provider |
+| Monitoring Panel     | Unit       | Metric Name                            | Description            |
+|----------------------|------------|----------------------------------------|------------------------|
+| Provider Request CPM | calls/min  | meter_envoy_ai_gw_provider_request_cpm | Requests by provider   |
+| Provider Token Rate  | tokens/min | meter_envoy_ai_gw_provider_token_rate  | Token rate by provider |
+| Provider Latency Avg | ms         | meter_envoy_ai_gw_provider_latency_avg | Latency by provider    |
 
 #### Model Breakdown Metrics
 
-| Monitoring Panel | Unit | Metric Name | Description |
-|---|---|---|---|
-| Model Request CPM | calls/min | meter_envoy_ai_gw_model_request_cpm | Requests by model |
-| Model Token Rate | tokens/min | meter_envoy_ai_gw_model_token_rate | Token rate by model |
-| Model Latency Avg | ms | meter_envoy_ai_gw_model_latency_avg | Latency by model |
-| Model TTFT Avg | ms | meter_envoy_ai_gw_model_ttft_avg | TTFT by model |
-| Model TPOT Avg | ms | meter_envoy_ai_gw_model_tpot_avg | TPOT by model |
+| Monitoring Panel  | Unit       | Metric Name                         | Description         |
+|-------------------|------------|-------------------------------------|---------------------|
+| Model Request CPM | calls/min  | meter_envoy_ai_gw_model_request_cpm | Requests by model   |
+| Model Token Rate  | tokens/min | meter_envoy_ai_gw_model_token_rate  | Token rate by model |
+| Model Latency Avg | ms         | meter_envoy_ai_gw_model_latency_avg | Latency by model    |
+| Model TTFT Avg    | ms         | meter_envoy_ai_gw_model_ttft_avg    | TTFT by model       |
+| Model TPOT Avg    | ms         | meter_envoy_ai_gw_model_tpot_avg    | TPOT by model       |
 
 #### Instance Metrics
 
 All service-level metrics are also available per instance (pod) with `meter_envoy_ai_gw_instance_` prefix,
 including per-provider and per-model breakdowns.
 
+### MCP Metrics
+
+When the AI Gateway is configured with MCP (Model Context Protocol) routes, SkyWalking collects
+MCP-specific metrics. These appear in the **MCP** tab on the service and instance dashboards.
+
+#### MCP Service Metrics
+
+| Monitoring Panel                      | Unit      | Metric Name                                             | Description                                                       |
+|---------------------------------------|-----------|---------------------------------------------------------|-------------------------------------------------------------------|
+| MCP Request CPM                       | calls/min | meter_envoy_ai_gw_mcp_request_cpm                       | MCP requests per minute                                           |
+| MCP Request Latency Avg               | ms        | meter_envoy_ai_gw_mcp_request_latency_avg               | Average MCP request duration                                      |
+| MCP Request Latency Percentile        | ms        | meter_envoy_ai_gw_mcp_request_latency_percentile        | P50/P75/P90/P95/P99                                               |
+| MCP Method CPM                        | calls/min | meter_envoy_ai_gw_mcp_method_cpm                        | Requests by MCP method (initialize, tools/list, tools/call, etc.) |
+| MCP Error CPM                         | calls/min | meter_envoy_ai_gw_mcp_error_cpm                         | MCP error requests per minute                                     |
+| MCP Initialization Latency Avg        | ms        | meter_envoy_ai_gw_mcp_initialization_latency_avg        | Average MCP session initialization time                           |
+| MCP Initialization Latency Percentile | ms        | meter_envoy_ai_gw_mcp_initialization_latency_percentile | P50/P75/P90/P95/P99                                               |
+| MCP Capabilities CPM                  | calls/min | meter_envoy_ai_gw_mcp_capabilities_cpm                  | Capabilities negotiated by type                                   |
+
+#### MCP Backend Breakdown Metrics
+
+| Monitoring Panel         | Unit      | Metric Name                                              | Description                    |
+|--------------------------|-----------|----------------------------------------------------------|--------------------------------|
+| Backend Request CPM      | calls/min | meter_envoy_ai_gw_mcp_backend_request_cpm                | Requests by MCP backend        |
+| Backend Latency Avg      | ms        | meter_envoy_ai_gw_mcp_backend_request_latency_avg        | Latency by MCP backend         |
+| Backend Method CPM       | calls/min | meter_envoy_ai_gw_mcp_backend_method_cpm                 | Requests by backend and method |
+| Backend Error CPM        | calls/min | meter_envoy_ai_gw_mcp_backend_error_cpm                  | Errors by MCP backend          |
+| Backend Init Latency Avg | ms        | meter_envoy_ai_gw_mcp_backend_initialization_latency_avg | Init latency by backend        |
+
+#### MCP Instance Metrics
+
+All MCP service-level metrics are also available per instance with `meter_envoy_ai_gw_mcp_instance_` prefix.
+
 ### Access Log Sampling
 
-The LAL rules apply a sampling policy to reduce storage:
+Access logs are tagged with `ai_route_type` (`llm` or `mcp`) for filtering in the log query UI.
+The `ai_route_type` tag is searchable by default.
+
+**LLM route logs:**
 - **Error responses** (HTTP status >= 400) — always persisted.
 - **Upstream failures** — always persisted.
 - **High token cost** (>= 10,000 total tokens) — persisted for cost anomaly detection.
 - Normal successful responses with low token counts are dropped.
 
-The token threshold can be adjusted in `lal/envoy-ai-gateway.yaml`.
+**MCP route logs:**
+- **Error responses** (HTTP status >= 400) — always persisted.
+- Normal MCP requests are dropped (MCP observability is covered by metrics).
+
+The sampling policy can be adjusted in `lal/envoy-ai-gateway.yaml`.

oap-server/server-starter/src/main/resources/application.yml

@@ -118,7 +118,7 @@ core:
     searchableTracesTags: ${SW_SEARCHABLE_TAG_KEYS:http.method,http.status_code,rpc.status_code,db.type,db.instance,mq.queue,mq.topic,mq.broker}
     # Define the set of log tag keys, which should be searchable through the GraphQL.
     # The max length of key=value should be less than 256 or will be dropped.
-    searchableLogsTags: ${SW_SEARCHABLE_LOGS_TAG_KEYS:level,http.status_code}
+    searchableLogsTags: ${SW_SEARCHABLE_LOGS_TAG_KEYS:level,http.status_code,ai_route_type}
     # Define the set of alarm tag keys, which should be searchable through the GraphQL.
     # The max length of key=value should be less than 256 or will be dropped.
     searchableAlarmTags: ${SW_SEARCHABLE_ALARM_TAG_KEYS:level}

oap-server/server-starter/src/main/resources/lal/envoy-ai-gateway.yaml

@@ -15,29 +15,52 @@
 
 # Envoy AI Gateway access log processing via OTLP.
 #
-# Sampling policy: only persist abnormal or expensive requests.
-# Normal 200 responses with low token count and no upstream failure are dropped.
+# Two rules: one for LLM route logs, one for MCP route logs.
+# LLM sampling: persist error responses (>= 400), upstream failures, or high-token requests (>= 10000).
+# MCP sampling: only persist error responses (>= 400).
+# Both tag ai_route_type for searchable filtering in the UI.
 
 rules:
-  - name: envoy-ai-gateway-access-log
+  - name: envoy-ai-gateway-llm-access-log
     layer: ENVOY_AI_GATEWAY
     dsl: |
       filter {
-        // Drop normal logs: response < 400, no upstream failure, low token count
+        // Only process LLM route logs (gen_ai.request.model is always set for LLM routes, even on errors)
+        if (tag("gen_ai.request.model") == "" || tag("gen_ai.request.model") == "-") {
+          abort {}
+        }
+
+        // Keep: error responses (>= 400), upstream failures, or high-token requests (>= 10000 total tokens)
+        // Abort logs without response_code unless there is an upstream failure
+        if (tag("response_code") == "" || tag("response_code") == "-") {
+          if (tag("upstream_transport_failure_reason") == "" || tag("upstream_transport_failure_reason") == "-") {
+            abort {}
+          }
+        }
+        // For normal responses (< 400), check upstream failure and token cost
         if (tag("response_code") != "" && tag("response_code") != "-") {
           if (tag("response_code") as Integer < 400) {
+            if (tag("upstream_transport_failure_reason") != "" && tag("upstream_transport_failure_reason") != "-") {
+              // upstream failure — keep
+            }
             if (tag("upstream_transport_failure_reason") == "" || tag("upstream_transport_failure_reason") == "-") {
+              // no upstream failure — check token cost
               if (tag("gen_ai.usage.input_tokens") != "" && tag("gen_ai.usage.input_tokens") != "-"
                   && tag("gen_ai.usage.output_tokens") != "" && tag("gen_ai.usage.output_tokens") != "-") {
                 if ((tag("gen_ai.usage.input_tokens") as Integer) + (tag("gen_ai.usage.output_tokens") as Integer) < 10000) {
                   abort {}
                 }
               }
+              if (tag("gen_ai.usage.input_tokens") == "" || tag("gen_ai.usage.input_tokens") == "-"
+                  || tag("gen_ai.usage.output_tokens") == "" || tag("gen_ai.usage.output_tokens") == "-") {
+                abort {}
+              }
             }
           }
         }
 
         extractor {
+          tag 'ai_route_type': "llm"
           tag 'gen_ai.request.model': tag("gen_ai.request.model")
           tag 'gen_ai.response.model': tag("gen_ai.response.model")
           tag 'gen_ai.provider.name': tag("gen_ai.provider.name")
@@ -50,3 +73,33 @@ rules:
         sink {
         }
       }
+
+  - name: envoy-ai-gateway-mcp-access-log
+    layer: ENVOY_AI_GATEWAY
+    dsl: |
+      filter {
+        // Only process MCP route logs
+        if (tag("mcp.method.name") == "" || tag("mcp.method.name") == "-") {
+          abort {}
+        }
+
+        // Only persist error responses (>= 400)
+        if (tag("response_code") == "" || tag("response_code") == "-") {
+          abort {}
+        }
+        if (tag("response_code") as Integer < 400) {
+          abort {}
+        }
+
+        extractor {
+          tag 'ai_route_type': "mcp"
+          tag 'mcp.method.name': tag("mcp.method.name")
+          tag 'mcp.provider.name': tag("mcp.provider.name")
+          tag 'mcp.session.id': tag("mcp.session.id")
+          tag 'response_code': tag("response_code")
+          tag 'duration': tag("duration")
+        }
+
+        sink {
+        }
+      }