Skip to content

Commit 65af803

Browse files
authored
feat(agent): export cloud run telemetry over OTLP (#3478)
1 parent c9605b5 commit 65af803

32 files changed

Lines changed: 2970 additions & 298 deletions

packages/agent/README.md

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,16 @@ Required environment variables (validated by zod in `src/server/bin.ts`):
171171
- `POSTHOG_PERSONAL_API_KEY` — API key for PostHog requests
172172
- `POSTHOG_PROJECT_ID` — numeric project ID
173173

174+
Optional run telemetry (the logs pair must both be set, otherwise telemetry stays off):
175+
176+
- `POSTHOG_AGENT_OTEL_LOGS_URL` — full OTLP logs URL for run metadata, e.g. `https://us.i.posthog.com/i/v1/logs`
177+
- `POSTHOG_AGENT_OTEL_LOGS_TOKEN` — project API key of the telemetry project
178+
- `POSTHOG_AGENT_OTEL_TRACES_URL` — full OTLP traces URL, e.g. `https://us.i.posthog.com/i/v1/traces`; additionally enables one APM trace per run
179+
180+
When set, `AgentServer` ships an allowlisted metadata subset of the session log (run/turn/tool lifecycle, usage, error provenance — never message content, tool arguments, or raw error text; see `src/otel-telemetry.ts`) to PostHog Logs, tagged with `service.name=posthog-code-agent` and `run_id`/`task_id`/`team_id`/`user_id`/`distinct_id` resource attributes so cloud runs are filterable per user. With the traces URL set, each run also produces an APM trace (`task_run` root span, a `turn` span per prompt, a `tool_call:<kind>` span per tool call; see `src/otel-trace-builder.ts`), and log records carry the matching trace/span ids so Logs and APM cross-link.
181+
182+
The `task_run` root span is ended and exported at the run's in-process terminal point — a background run's prompt settling, a terminal failure, or session cleanup (`close`). It cannot wait for teardown: agent-server is an exec'd process inside the sandbox, so `docker stop` / Modal terminate kill it without SIGTERM ever arriving, and a span still open at that moment is lost. Interactive sessions that end via hard teardown (e.g. inactivity timeout) therefore lose the root span; turn/tool spans and logs still assemble under the same trace id.
183+
174184
## Agent SDK
175185

176186
The `Agent` class (`src/agent.ts`) is the entrypoint for local/programmatic usage. It handles LLM gateway configuration, log writer setup, and model filtering — then delegates to `createAcpConnection()`.
@@ -209,12 +219,9 @@ Logs serve two purposes: real-time observability and session resume. Every ACP m
209219

210220
### Writing logs
211221

212-
`SessionLogWriter` (`src/session-log-writer.ts`) is a per-session multiplexer that buffers raw ndJson lines. On flush (auto-scheduled 500ms after writes, or explicit), it dispatches to whichever backend is configured:
213-
214-
- **OTEL** (`src/otel-log-writer.ts`) — preferred path. Creates an OpenTelemetry `LoggerProvider` per session with resource attributes (`task_id`, `run_id`, `device_type`) set once and indexed via `resource_fingerprint`. Each ndJson line is emitted as an OTEL log record with an `event_type` attribute (the ACP method name) and exported via OTLP HTTP to PostHog's `/i/v1/agent-logs` endpoint. Batch flush interval defaults to 500ms.
215-
- **Legacy S3** — falls back to `PostHogAPIClient.appendTaskRunLog()`, which POSTs batched `StoredNotification` entries to the Django API. The API stores them as the task run's `log_url`.
222+
`SessionLogWriter` (`src/session-log-writer.ts`) is a per-session multiplexer that buffers raw ndJson lines. On flush (auto-scheduled 500ms after writes, or explicit), it POSTs batched `StoredNotification` entries to the Django API via `PostHogAPIClient.appendTaskRunLog()`; the API stores them in S3 as the task run's `log_url`. This S3 log is the source of truth for the full transcript and for session resume.
216223

217-
Both backends can be active simultaneously — OTEL for fast indexed queries, S3 for full log download.
224+
Independently of that flush cycle, the writer tees every parsed non-chunk entry to optional `SessionLogSink`s at append time. In cloud, `OtelRunTelemetry` (`src/otel-telemetry.ts`) is wired as a sink and exports an allowlisted metadata subset (run/turn/tool lifecycle, usage, error provenance — never message content, tool arguments, or raw error text) to PostHog Logs, plus an APM trace per run when configured. See "Optional run telemetry" above for the env vars and behavior. Sink failures can never break S3 log persistence.
218225

219226
### Resuming from logs
220227

packages/agent/package.json

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -163,10 +163,13 @@
163163
"@earendil-works/pi-coding-agent": "catalog:",
164164
"@hono/node-server": "^1.19.9",
165165
"@openai/codex": "0.144.0",
166+
"@opentelemetry/api": "^1.9.1",
166167
"@opentelemetry/api-logs": "^0.208.0",
167168
"@opentelemetry/exporter-logs-otlp-http": "^0.208.0",
169+
"@opentelemetry/exporter-trace-otlp-http": "^0.208.0",
168170
"@opentelemetry/resources": "^2.0.0",
169171
"@opentelemetry/sdk-logs": "^0.208.0",
172+
"@opentelemetry/sdk-trace-base": "^2.8.0",
170173
"@opentelemetry/semantic-conventions": "^1.28.0",
171174
"@types/jsonwebtoken": "^9.0.10",
172175
"commander": "^14.0.2",
Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
const MAX_BODY_CHARS = 2000;
2+
// PostHog Logs only facets attribute key/value pairs shorter than 256 chars,
3+
// so free-text attribute values are capped well below that.
4+
const MAX_ATTR_CHARS = 200;
5+
// The SDK default export timeout is 30s; a hanging endpoint must not hold up
6+
// session cleanup (the sandbox can be torn down right after), so keep it short.
7+
const EXPORT_TIMEOUT_MS = 5000;
8+
9+
// Batch flush cadence shared by the log and span processors.
10+
const DEFAULT_FLUSH_INTERVAL_MS = 2000;
11+
12+
export type AttributeValue = string | number | boolean;
13+
export type Attributes = Record<string, AttributeValue>;
14+
15+
export {
16+
DEFAULT_FLUSH_INTERVAL_MS,
17+
EXPORT_TIMEOUT_MS,
18+
MAX_ATTR_CHARS,
19+
MAX_BODY_CHARS,
20+
};
21+
22+
/**
23+
* extNotification() can double-prefix custom methods (see matchesExt in
24+
* acp-extensions.ts); normalize so both spellings map identically.
25+
*/
26+
export function normalizeMethod(method: string): string {
27+
return method.startsWith("__posthog/") ? method.slice(1) : method;
28+
}
29+
30+
export function asString(value: unknown): string | undefined {
31+
return typeof value === "string" ? value : undefined;
32+
}
33+
34+
export function truncate(value: string, max: number): string {
35+
return value.length <= max ? value : `${value.slice(0, max)}…`;
36+
}
37+
38+
export function asRecord(value: unknown): Record<string, unknown> | undefined {
39+
return typeof value === "object" && value !== null && !Array.isArray(value)
40+
? (value as Record<string, unknown>)
41+
: undefined;
42+
}
43+
44+
export function strAttr(
45+
attrs: Attributes,
46+
key: string,
47+
value: unknown,
48+
max = MAX_ATTR_CHARS,
49+
): string | undefined {
50+
if (typeof value !== "string" || value.length === 0) return undefined;
51+
const truncated = truncate(value, max);
52+
attrs[key] = truncated;
53+
return truncated;
54+
}
55+
56+
export function numAttr(attrs: Attributes, key: string, value: unknown): void {
57+
if (typeof value === "number" && Number.isFinite(value)) {
58+
attrs[key] = value;
59+
}
60+
}
61+
62+
export function usageAttributes(params: Record<string, unknown>): Attributes {
63+
const attrs: Attributes = {};
64+
const used = asRecord(params.used);
65+
if (used) {
66+
numAttr(attrs, "tokens_input", used.inputTokens);
67+
numAttr(attrs, "tokens_output", used.outputTokens);
68+
numAttr(attrs, "tokens_cached_read", used.cachedReadTokens);
69+
numAttr(attrs, "tokens_cached_write", used.cachedWriteTokens);
70+
}
71+
// Claude sends a plain number; other shapes carry { amount }.
72+
const cost =
73+
typeof params.cost === "number"
74+
? params.cost
75+
: asRecord(params.cost)?.amount;
76+
numAttr(attrs, "cost_usd", cost);
77+
return attrs;
78+
}
79+
80+
/** Timestamp of a stored entry as a Date, falling back to now when invalid. */
81+
export function entryTime(timestamp: string): Date {
82+
const parsed = new Date(timestamp);
83+
return Number.isNaN(parsed.getTime()) ? new Date() : parsed;
84+
}

packages/agent/src/otel-log-writer.test.ts

Lines changed: 0 additions & 105 deletions
This file was deleted.

packages/agent/src/otel-log-writer.ts

Lines changed: 0 additions & 94 deletions
This file was deleted.

0 commit comments

Comments
 (0)