Document o11y for vMCP

Signed-off-by: Jeremy Drouillard <jeremy@stacklok.com>

Author: jerm-dro

cmd/vmcp/README.md

@@ -14,6 +14,7 @@ The Virtual MCP Server (vmcp) is a standalone binary that aggregates multiple MC
 - ✅ **Session Management**: MCP protocol session tracking with TTL-based cleanup
 - ✅ **Health Endpoints**: `/health` and `/ping` for service monitoring
 - ✅ **Configuration Validation**: `vmcp validate` command for config verification
+- ✅ **Observability**: OpenTelemetry metrics and traces for backend operations and workflow executions
 
 ### In Progress
 - 🚧 **Incoming Authentication** (Issue #165): OIDC, local, anonymous authentication
@@ -121,6 +122,7 @@ vmcp uses a YAML configuration file to define:
 3. **Outgoing Authentication**: Virtual MCP → Backend API token exchange
 4. **Tool Aggregation**: Conflict resolution and filtering strategies
 5. **Operational Settings**: Timeouts, health checks, circuit breakers
+6. **Telemetry**: OpenTelemetry metrics/tracing and Prometheus endpoint
 
 See [examples/vmcp-config.yaml](../../examples/vmcp-config.yaml) for a complete example.
 

docs/observability.md

@@ -84,3 +84,10 @@ The telemetry middleware:
 
 This provides end-to-end visibility across the entire request lifecycle while
 maintaining the modular architecture of ToolHive's middleware system.
+
+## Virtual MCP Server Telemetry
+
+For observability in the Virtual MCP Server (vMCP), including backend request
+metrics, workflow execution telemetry, and distributed tracing, see the
+dedicated [Virtual MCP Server Observability](./operator/virtualmcpserver-observability.md)
+documentation.

docs/operator/virtualmcpserver-api.md

@@ -277,6 +277,40 @@ spec:
               cpu: "1000m"
 ```
 
+### `.spec.telemetry` (optional)
+
+Configures OpenTelemetry-based observability for the Virtual MCP server, including distributed tracing, OTLP metrics export, and Prometheus metrics endpoint.
+
+**Type**: `telemetry.Config`
+
+**Fields**:
+- `endpoint` (string): OTLP collector endpoint (host:port format)
+- `serviceName` (string): Service name for telemetry
+- `serviceVersion` (string): Service version for telemetry
+- `tracingEnabled` (boolean): Enable distributed tracing
+- `metricsEnabled` (boolean): Enable OTLP metrics export
+- `samplingRate` (float64): Trace sampling rate (0.0-1.0)
+- `headers` (map[string]string): Authentication headers for OTLP endpoint
+- `insecure` (boolean): Use HTTP instead of HTTPS
+- `enablePrometheusMetricsPath` (boolean): Expose Prometheus /metrics endpoint
+- `environmentVariables` ([]string): Environment variable names to include as span attributes
+- `customAttributes` (map[string]string): Custom resource attributes for all telemetry signals
+
+**Example**:
+```yaml
+spec:
+  telemetry:
+    endpoint: "otel-collector:4317"
+    serviceName: "my-vmcp"
+    tracingEnabled: true
+    metricsEnabled: true
+    samplingRate: 0.1
+    insecure: true
+    enablePrometheusMetricsPath: true
+```
+
+For details on what metrics and traces are emitted, see the [Virtual MCP Server Observability](./virtualmcpserver-observability.md) documentation.
+
 ## Status Fields
 
 ### `.status.conditions`
@@ -437,6 +471,14 @@ spec:
         failureThreshold: 5
         timeout: 60s
 
+  # Observability
+  telemetry:
+    endpoint: "otel-collector:4317"
+    tracingEnabled: true
+    metricsEnabled: true
+    samplingRate: 0.1
+    enablePrometheusMetricsPath: true
+
 status:
   phase: Ready
   message: "Virtual MCP serving 3 backends with 15 tools"
@@ -502,4 +544,5 @@ The VirtualMCPServer CRD includes comprehensive validation:
 - [MCPServer](./mcpserver-api.md): Individual MCP server instances
 - [MCPExternalAuthConfig](./mcpexternalauthconfig-api.md): External authentication configuration
 - [MCPToolConfig](./toolconfig-api.md): Tool filtering and renaming configuration
+- [Virtual MCP Server Observability](./virtualmcpserver-observability.md): Telemetry and metrics documentation
 - [Virtual MCP Proposal](../proposals/THV-2106-virtual-mcp-server.md): Complete design proposal

docs/operator/virtualmcpserver-observability.md

@@ -0,0 +1,74 @@
+# Virtual MCP Server Observability
+
+This document describes the observability for the Virtual MCP
+Server (vMCP), which aggregates multiple backend MCP servers into a unified
+interface. The vMCP provides OpenTelemetry-based instrumentation for monitoring
+backend operations and composite tool workflow executions.
+
+For general ToolHive observability concepts and proxy runner telemetry, see the
+main [Observability and Telemetry](../observability.md) documentation.
+
+## Overview
+
+The vMCP telemetry provides visibility into:
+
+1. **Backend operations**: Track requests to individual backend MCP servers
+   including tool calls, resource reads, prompt retrieval, and capability listing
+2. **Workflow executions**: Monitor composite tool workflow performance and errors
+3. **Distributed tracing**: Correlate requests across the vMCP and its backends
+
+The vMCP uses a decorator pattern to wrap backend clients and workflow executors
+with telemetry instrumentation. This approach provides consistent metrics and
+tracing without modifying the core business logic.
+
+The implementation of both metrics and traces can be found in `pkg/vmcp/server/telemetry.go`.
+
+## Metrics
+
+The vMCP emits metrics for backend operations and workflow executions. All
+metrics use the `toolhive_vmcp_` prefix.
+
+**Backend metrics** track requests to individual backend MCP servers, including
+request counts, error counts, and request duration histograms. These metrics
+include attributes identifying the target backend (workload ID, name, URL,
+transport type) and the action being performed (tool call, resource read, etc.).
+
+**Workflow metrics** track composite tool workflow executions, including
+execution counts, error counts, and duration histograms. These metrics include
+the workflow name as an attribute.
+
+## Distributed Tracing
+
+The vMCP creates spans for each individual backend operation as well as workflow executions, enabling the attribution of workflow exection errors or latency to specific tool calls.
+
+
+## Configuration
+
+Configure telemetry in the `VirtualMCPServer` resource using the `spec.telemetry`
+field:
+
+```yaml
+apiVersion: toolhive.stacklok.dev/v1alpha1
+kind: VirtualMCPServer
+metadata:
+  name: my-vmcp
+spec:
+  groupRef:
+    name: my-group
+  telemetry:
+    endpoint: "otel-collector:4317"
+    serviceName: "my-vmcp"
+    tracingEnabled: true
+    metricsEnabled: true
+    samplingRate: 0.1
+    insecure: true
+    enablePrometheusMetricsPath: true
+```
+
+See the [VirtualMCPServer API reference](./virtualmcpserver-api.md) for complete
+CRD documentation.
+
+## Related Documentation
+
+- [Observability and Telemetry](../observability.md) - Main ToolHive observability documentation
+- [VirtualMCPServer API Reference](./virtualmcpserver-api.md) - Complete CRD specification

examples/vmcp-config.yaml

@@ -187,3 +187,14 @@ operational:
 #           environment: "{{.steps.confirm_deploy.content.environment}}"
 #         depends_on: ["confirm_deploy"]
 #         condition: "{{.steps.confirm_deploy.action == 'accept'}}"
+
+# ===== OBSERVABILITY =====
+# OpenTelemetry-based metrics and tracing for backend operations and workflows
+telemetry:
+  endpoint: "localhost:4317"  # OTLP collector endpoint
+  serviceName: "engineering-vmcp"
+  tracingEnabled: true
+  metricsEnabled: true
+  samplingRate: 0.1  # 10% sampling
+  insecure: true  # Use HTTP instead of HTTPS
+  enablePrometheusMetricsPath: true  # Expose /metrics endpoint