Fix relay Axiom HTTP span view fields

Co-authored-by: codex <codex@users.noreply.github.com>

Author: juliusmarminge

docs/relay-observability.md

@@ -1,68 +1,39 @@
 # Relay observability
 
-The relay Alchemy stack owns Axiom resources for post-hoc diagnostics:
+The relay Alchemy stack owns a focused Axiom trace setup:
 
-- `t3-code-relay-events` for Effect logs and spans
-- `t3-code-relay-metrics` for Effect metrics
-- `t3-code-relay-otel-ingest` for Worker OTLP ingest
-- `t3-code-relay-readonly-query` for human/agent log lookup
-- `T3 Code Relay Operations` dashboard
-- starter views for recent logs and recent failures
-- monitors for warning/error logs, APNS failures, managed tunnel provisioning failures, and quiet log ingestion
+- `t3-code-relay-traces`, an OpenTelemetry trace dataset for Worker requests
+- `t3-code-relay-otel-ingest`, a dataset-scoped ingest token bound to the Worker
+- `t3-code-relay-readonly-query`, a dataset-scoped token for scripted diagnostics
+- `t3-code-relay-recent-spans`, a view of recent request and endpoint spans
 
 Deploy from `infra/relay` with the normal Alchemy workflow:
 
 ```sh
 bun run deploy
 ```
 
-Alchemy resolves Axiom credentials through the Axiom provider. Use either environment credentials or `alchemy login --configure` before deploy.
+Alchemy resolves Axiom deployment credentials through its provider. At runtime, the Worker
+receives only the scoped ingest token; it does not receive the diagnostics query token.
 
-Useful APL queries:
+The Worker emits Effect's built-in HTTP server spans plus endpoint and database child spans.
+Effect's OpenTelemetry exporter stores semantic HTTP attributes below the `attributes.` prefix.
+For example:
 
 ```apl
-['t3-code-relay-events']
+['t3-code-relay-traces']
+| where name startswith 'http.server'
+| project _time, name, trace_id, duration,
+    ['attributes.http.request.method'],
+    ['attributes.url.path'],
+    ['attributes.http.response.status_code']
 | order by _time desc
 | limit 200
 ```
 
-```apl
-['t3-code-relay-events']
-| extend logSeverity = column_ifexists('severityText', '')
-| extend logBody = column_ifexists('body', '')
-| where logSeverity in ("WARN", "WARNING", "ERROR", "FATAL")
-  or logBody contains "failed"
-  or logBody contains "error"
-| order by _time desc
-| limit 200
-```
-
-Metrics intentionally capture product and state signals that are not just trace counts:
-
-- `relay_managed_tunnel_provisions_total`: managed tunnel provisioning outcomes, split by `created` versus `reused`
-- `relay_environment_links_total`: link and unlink lifecycle operations
-- `relay_managed_tunnels_active`: current active managed-tunnel links
-- `relay_environment_links_active`: current active environment links
-- `relay_mobile_devices_registered`: current registered mobile devices
-- `relay_live_activity_targets_active`: current active Live Activity targets
-- `relay_agent_activities_active`: current active agent activity rows
-- `relay_agent_activity_publishes_total`: agent activity publish/replay lifecycle events
-- `relay_apns_deliveries_total`: APNS enqueue/send outcomes for Live Activities and push notifications
-
-The `*_active` and `*_registered` values are gauges refreshed from the relay database, which is the source of truth for current state. Lifecycle counters are updated from the mutation path after successful writes or delivery outcomes.
-
-Useful metrics queries:
-
-```mpl
-`t3-code-relay-metrics`:`relay_managed_tunnels_active`
-| group using sum
-```
-
-```mpl
-`t3-code-relay-metrics`:`relay_managed_tunnel_provisions_total`
-| map increase
-| align to 5m using sum
-| group by outcome, tunnelProvisionKind using sum
-```
+Endpoint failure annotations and other relay-specific attributes are also emitted under
+`attributes.relay.*` when present on a span.
 
-Agents should prefer Axiom views or APL queries for completed incidents instead of tailing the Cloudflare Worker. Use the read-only query token when scripted access is needed; keep the ingest token reserved for the Worker.
+Agents should prefer the provisioned view or APL queries for completed incidents instead of
+tailing the Cloudflare Worker. Use the read-only query token when scripted access is needed;
+keep the ingest token reserved for the Worker.

infra/relay/src/infra/RelayObservability.test.ts

@@ -4,6 +4,7 @@ import {
   RELAY_AXIOM_TRACE_DATASET,
   relayAxiomIngestDatasetCapabilities,
   relayAxiomQueryDatasetCapabilities,
+  relayRecentSpansQuery,
   relayTraceQuery,
 } from "./RelayObservability.ts";
 
@@ -25,4 +26,15 @@ describe("RelayObservability", () => {
       "['relay-traces-test']\n| where name == 'GET /health'",
     );
   });
+
+  it("projects Effect HTTP span attributes through their OTLP field names", () => {
+    const query = relayRecentSpansQuery("relay-traces-test");
+
+    expect(query).toContain("['relay-traces-test']");
+    expect(query).toContain("attributes.http.request.method");
+    expect(query).toContain("attributes.http.response.status_code");
+    expect(query).toContain("attributes.url.path");
+    expect(query).toContain("attributes.relay.endpoint");
+    expect(query).not.toContain("['http.request.method']");
+  });
 });

infra/relay/src/infra/RelayObservability.ts

@@ -8,6 +8,12 @@ export const RELAY_AXIOM_TRACE_DATASET = "t3-code-relay-traces";
 export const relayTraceQuery = (query: string, dataset: string = RELAY_AXIOM_TRACE_DATASET) =>
   `['${dataset}']\n${query}`;
 
+export const relayRecentSpansQuery = (dataset: string = RELAY_AXIOM_TRACE_DATASET) =>
+  relayTraceQuery(
+    "| where isnotnull(span_id) or isnotnull(trace_id)\n| extend requestMethod = column_ifexists('attributes.http.request.method', ''), path = column_ifexists('attributes.url.path', ''), statusCode = column_ifexists('attributes.http.response.status_code', 0), endpoint = column_ifexists('attributes.relay.endpoint', '')\n| project _time, name, trace_id, span_id, duration, requestMethod, path, statusCode, endpoint\n| order by _time desc\n| limit 200",
+    dataset,
+  );
+
 export const relayAxiomIngestDatasetCapabilities = (
   dataset: string = RELAY_AXIOM_TRACE_DATASET,
 ) => ({
@@ -44,9 +50,7 @@ export const provisionRelayObservability = Effect.gen(function* () {
     name: "t3-code-relay-recent-spans",
     description: "Recent relay HTTP request spans.",
     datasets: [RELAY_AXIOM_TRACE_DATASET],
-    aplQuery: relayTraceQuery(
-      "| where isnotnull(span_id) or isnotnull(trace_id)\n| project _time, name, trace_id, span_id, duration, ['http.request.method'], ['url.path'], ['http.response.status_code'], ['relay.endpoint']\n| order by _time desc\n| limit 200",
-    ),
+    aplQuery: relayRecentSpansQuery(),
   });
 
   return { traces, ingestToken, queryToken } as const;