Skip to content

Commit 194f52f

Browse files
JakeSCahillJakeSCahill
committed
Add real trace examples showing tool execution and error events
1 parent c1f9e53 commit 194f52f

1 file changed

Lines changed: 105 additions & 39 deletions

File tree

‎modules/ai-agents/pages/mcp/remote/monitor-activity.adoc‎

Lines changed: 105 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -79,15 +79,112 @@ Each trace message contains an OpenTelemetry span with the following structure:
7979
}
8080
----
8181

82-
Key fields:
82+
* `traceId`: Unique identifier for the entire trace.
83+
* `spanId`: Unique identifier for this specific span.
84+
* `parentSpanId`: Links child spans to parent operations.
85+
* `name`: Descriptive name of the span.
86+
* `startTimeUnixNano` / `endTimeUnixNano`: Timing information in nanoseconds.
87+
* `attributes`: Key-value pairs with contextual information.
88+
* `status`: Success or error status.
8389

84-
* `traceId`: Unique identifier for the entire trace
85-
* `spanId`: Unique identifier for this specific span
86-
* `parentSpanId`: Links child spans to parent operations
87-
* `name`: Operation name (e.g., `tool_invocation`, `http_request`)
88-
* `startTimeUnixNano` / `endTimeUnixNano`: Timing information in nanoseconds
89-
* `attributes`: Key-value pairs with contextual information
90-
* `status`: Success or error status
90+
=== Real trace examples
91+
92+
Here are actual traces from MCP tool executions showing the parent-child span relationships and detailed attributes.
93+
94+
**Successful tool execution:**
95+
96+
This trace shows an `http_processor` tool that fetched weather data for London. The parent span includes child spans for mutations, HTTP requests, and processing:
97+
98+
[source,json]
99+
----
100+
{
101+
"attributes": [
102+
{
103+
"key": "city_name",
104+
"value": {"stringValue": "london"}
105+
},
106+
{
107+
"key": "result_prefix",
108+
"value": {"stringValue": "{\"city\":\"london\",\"description\":\"Partly cloudy\",\"feels_like\":12,\"humidity\":88,\"metadata\":{\"fetched_at\":\"2025-12-08T12:53:44.660Z\""}
109+
},
110+
{
111+
"key": "result_length",
112+
"value": {"intValue": "198"}
113+
}
114+
],
115+
"endTimeUnixNano": "1765198424660663434",
116+
"flags": 1,
117+
"instrumentationScope": {"name": "rpcn-mcp"},
118+
"kind": 1,
119+
"name": "http_processor",
120+
"spanId": "43ad6bc31a826afd",
121+
"startTimeUnixNano": "1765198415253280028",
122+
"status": {"code": 0, "message": ""},
123+
"traceId": "71cad555b35602fbb35f035d6114db54"
124+
}
125+
----
126+
127+
The parent span (`http_processor`) has child spans showing the processing pipeline:
128+
129+
[,json]
130+
----
131+
{
132+
"attributes": [],
133+
"endTimeUnixNano": "1765198424659619511",
134+
"flags": 1,
135+
"instrumentationScope": {"name": "benthos"},
136+
"kind": 1,
137+
"name": "http",
138+
"parentSpanId": "43ad6bc31a826afd",
139+
"spanId": "ed45544a7d7b08d4",
140+
"startTimeUnixNano": "1765198415253319496",
141+
"status": {"code": 0, "message": ""},
142+
"traceId": "71cad555b35602fbb35f035d6114db54"
143+
}
144+
----
145+
146+
* The parent span (`http_processor`) includes custom attributes like `city_name` and `result_length` that help debug and monitor tool behavior.
147+
* Child spans use the same `traceId` but different `spanId` values.
148+
* The `parentSpanId` in child spans matches the parent's `spanId`, creating the hierarchy.
149+
* Duration can be calculated: `(1765198424660663434 - 1765198415253280028) / 1000000 = 9407ms`.
150+
151+
**Trace with error event:**
152+
153+
This trace shows an HTTP request that encountered an error during execution:
154+
155+
[,json]
156+
----
157+
{
158+
"attributes": [],
159+
"endTimeUnixNano": "1765198423080903265",
160+
"events": [
161+
{
162+
"attributes": [
163+
{
164+
"key": "error",
165+
"value": {"stringValue": "type"}
166+
}
167+
],
168+
"name": "event",
169+
"timeUnixNano": "1765198420254169629"
170+
}
171+
],
172+
"flags": 1,
173+
"instrumentationScope": {"name": "benthos"},
174+
"kind": 1,
175+
"name": "http_request",
176+
"parentSpanId": "43ad6bc31a826afd",
177+
"spanId": "ba332199f3af6d7f",
178+
"startTimeUnixNano": "1765198415253325782",
179+
"status": {"code": 0, "message": ""},
180+
"traceId": "71cad555b35602fbb35f035d6114db54"
181+
}
182+
----
183+
184+
* The span still shows `status.code: 0` (success) overall.
185+
* However, the `events` array contains error information with `error: type`.
186+
* This indicates a recoverable error or warning during execution.
187+
* The event's `timeUnixNano` shows exactly when the error occurred.
91188

92189
== Common monitoring scenarios
93190

@@ -131,37 +228,6 @@ Link MCP server activity to downstream effects:
131228

132229
The `redpanda.otel_traces` topic uses standard OpenTelemetry format, making it compatible with popular observability platforms.
133230

134-
=== Grafana and Tempo
135-
136-
Export traces to Grafana Tempo for visualization:
137-
138-
. Create a Redpanda Connect pipeline that consumes from `redpanda.otel_traces`.
139-
. Transform the JSON spans into OTLP format.
140-
. Send to Tempo's OTLP HTTP endpoint.
141-
. Query and visualize in Grafana.
142-
143-
Example pipeline configuration:
144-
145-
[,yaml]
146-
----
147-
input:
148-
kafka:
149-
addresses: [ "${REDPANDA_BROKERS}" ]
150-
topics: [ "redpanda.otel_traces" ]
151-
consumer_group: "tempo-exporter"
152-
153-
pipeline:
154-
processors:
155-
- mapping: |
156-
root = this
157-
158-
output:
159-
http_client:
160-
url: "http://tempo:4318/v1/traces"
161-
verb: POST
162-
headers:
163-
Content-Type: "application/json"
164-
----
165231

166232
=== Grafana Cloud
167233

0 commit comments

Comments
 (0)