|
| 1 | +# Agent run telemetry: PostHog Code cloud tasks → PostHog Logs + APM |
| 2 | + |
| 3 | +This branch is one half of a two-repo change; the companion branch is `posthog-code/agent-run-otel-telemetry` in `PostHog/posthog`. |
| 4 | +This repo (`PostHog/code`) carries the telemetry emitter inside the agent; `PostHog/posthog` carries the configuration/injection side. |
| 5 | + |
| 6 | +## Goal |
| 7 | + |
| 8 | +Cloud task runs executed by PostHog Code should be observable in PostHog itself: |
| 9 | +every run's lifecycle metadata delivered as OTel **logs** into the Logs product, and one OTel **trace** per run (root span, per-turn spans, per-tool-call spans) into APM, with logs and spans cross-linked via `trace_id`/`span_id`. |
| 10 | + |
| 11 | +Hard requirements: |
| 12 | + |
| 13 | +- **Filterable per user**: the Logs UI must answer "show me everything cloud runs did for this user" directly, so `user_id`/`distinct_id` (plus `team_id`, `task_id`, `run_id`) are OTel resource attributes, which PostHog Logs facets via `resource_fingerprint`. |
| 14 | +- **Cloud tasks only** for now; desktop local runs do not export session telemetry. |
| 15 | +- **Metadata only**: the S3 session log remains the source of truth for full transcripts. Telemetry never carries prompts, agent message/thought text, tool arguments, or tool output. |
| 16 | + |
| 17 | +## Background: what existed before |
| 18 | + |
| 19 | +- Agent session logs flow from the sandbox `agent-server` through `SessionLogWriter` to the Django endpoint `POST /api/projects/{team}/tasks/{task}/runs/{run}/append_log/`, which appends NDJSON to S3 (`TaskRun.append_log`, 30-day TTL). Two more delivery legs exist: an NDJSON event-ingest stream and SSE to connected clients. |
| 20 | +- A February attempt at OTel logs export (`OtelLogWriter`, commits `6abadc79` → `99a3aea8` → `8876c8fb` in `PostHog/code`) was unwired, and its default endpoint `/i/v1/agent-logs` does not exist in the ingest service; it would 404 today. |
| 21 | +- The correct ingest is the `capture-logs` Rust service (`rust/capture-logs/`): `POST /i/v1/logs` and `POST /i/v1/traces`, OTLP http/protobuf or http/JSON, auth `Authorization: Bearer <project API key>`, 2 MB request cap, billed by uncompressed bytes, severity normalized to lowercase, prod host `https://us.i.posthog.com`. |
| 22 | +- Working dogfood precedents followed here: the desktop Electron transport (`posthog-code-desktop` → `/i/v1/logs`), the engineering-analytics CI log emitter, the streamlit sandbox proxy (env-injected OTLP config), and the plugin-server metrics exporter (telemetry off unless URL + token are both set). |
| 23 | + |
| 24 | +## Architecture |
| 25 | + |
| 26 | +``` |
| 27 | +sandbox (agentsh) |
| 28 | +└── agent-server (PostHog/code, packages/agent) |
| 29 | + ├── ACP streams (tapped) ──► SessionLogWriter ──► Django append_log ──► S3 (product log, unchanged) |
| 30 | + │ │ |
| 31 | + │ └─ sink ──► OtelRunTelemetry |
| 32 | + │ ├─ log records ──► POST {POSTHOG_AGENT_OTEL_LOGS_URL} |
| 33 | + │ └─ RunTraceBuilder spans ──► POST {POSTHOG_AGENT_OTEL_TRACES_URL} |
| 34 | + └── terminal error events ──────────────────────► mirrored into OtelRunTelemetry directly |
| 35 | +
|
| 36 | +capture-logs (Rust) ──► Kafka ──► ClickHouse (logs / trace_spans) ──► Logs + APM UI |
| 37 | +``` |
| 38 | + |
| 39 | +Telemetry is emitted **from inside the sandbox** by the agent-server process, deliberately independent of the Django/S3 product path: if `append_log` is degraded, telemetry still flows, and a broken product log pipeline is exactly the failure telemetry must capture. |
| 40 | +Egress works because `*.posthog.com` is in the agentsh `INFRASTRUCTURE_DOMAINS` allowlist (prod) and the new `SANDBOX_AGENT_OTEL_*_URL` hosts join the DEBUG-only firewall list (local dev). |
| 41 | + |
| 42 | +## What ships per run |
| 43 | + |
| 44 | +### Log records (service.name=posthog-code-agent) |
| 45 | + |
| 46 | +Resource attributes on every record: `service.name`, `service.version` (agent version), `run_id`, `task_id`, `team_id`, `user_id`, `distinct_id`, `device_type` (`cloud`), `adapter` (`claude`/`codex`), `run_mode` (`interactive`/`background`). |
| 47 | + |
| 48 | +Exported events (allowlist, everything else is dropped): |
| 49 | + |
| 50 | +| Event | Severity | Notable attributes | |
| 51 | +| --- | --- | --- | |
| 52 | +| `_posthog/run_started` | info | `agent_version`, `session_id` | |
| 53 | +| `_posthog/sdk_session` | info | `adapter`, `session_id` | |
| 54 | +| `_posthog/usage_update` (and the `session/update` variant) | info | `tokens_input/output/cached_read/cached_write`, `cost_usd` | |
| 55 | +| `_posthog/turn_complete` | info | `stop_reason` | |
| 56 | +| `_posthog/task_complete` | info | `stop_reason` | |
| 57 | +| `_posthog/error` | **error** | `error_source`, `stop_reason`, message in body (capped) | |
| 58 | +| `_posthog/console` | mapped from level | agent-server internal logs | |
| 59 | +| `_posthog/progress` | info | `progress_group/step/status` | |
| 60 | +| `_posthog/git_checkpoint`, `_posthog/branch_created` | info | `branch` | |
| 61 | +| `_posthog/mode_change`, `_posthog/compact_boundary` | info | | |
| 62 | +| `_posthog/permission_request/response/resolved` | info | `request_id`, `tool_call_id` (identifiers only; tool content excluded) | |
| 63 | +| `session/update: tool_call` | info | `tool_call_id`, `tool_kind`, `tool_status` (no title, no rawInput) | |
| 64 | +| `session/update: tool_call_update` (terminal only) | info / warn on `failed` | `tool_call_id`, `tool_status` | |
| 65 | + |
| 66 | +Deliberately dropped: `agent_message`, `agent_message_chunk`, `agent_thought_chunk`, `user_message`, `session/prompt` bodies, in-progress `tool_call_update` snapshots (they re-send the growing tool input/output), `available_commands_update`, and any unknown method (fail-closed allowlist). |
| 67 | +Bodies are capped at 2000 chars; free-text attribute values at 200 chars (the `log_attributes` faceting table only indexes key/value pairs under 256 chars). |
| 68 | + |
| 69 | +### APM trace (one per run) |
| 70 | + |
| 71 | +- `task_run` root span (kind SERVER): opened at session init, closed at session cleanup; status OK on `task_complete`, ERROR with the error message on `_posthog/error`. |
| 72 | +- `turn` child spans: opened on each ACP `session/prompt` (used purely as a boundary marker; its content is never read), closed on `_posthog/turn_complete`; attributes `turn_index`, `stop_reason`, plus the turn's token counts and `cost_usd` lifted from usage updates, so APM can rank slow or expensive turns directly. |
| 73 | +- `tool_call:<kind>` grandchild spans (`execute`, `read`, `edit`, ...): opened on `tool_call`, closed on the terminal `tool_call_update`; status ERROR on `failed`; attributes `tool_call_id`, `tool_kind`, `tool_status`. Per-kind span names stay low-cardinality and make APM latency breakdowns by tool kind useful. |
| 74 | +- Robustness: orphaned spans are closed (status unset) and exported at shutdown; a new prompt while a turn is open closes the stale turn; duplicate `tool_call` events are idempotent; an error cascades ERROR status through open tool and turn spans to the root. |
| 75 | +- Every log record is emitted under the OTel context of the span it belongs to (tool logs on the tool span, lifecycle logs on the root), so `trace_id`/`span_id` land in the `logs` table columns and the UI links Logs ⇄ trace waterfall. |
| 76 | + |
| 77 | +### Delivery timing |
| 78 | + |
| 79 | +Telemetry is near-realtime, not end-of-turn: records are created the moment each notification flows through the writer and batched for at most 2 s (`BatchLogRecordProcessor` / `BatchSpanProcessor`, `scheduledDelayMillis` 2000). |
| 80 | +Spans export when they end (tools mid-turn, turns at `turn_complete`, root at cleanup). |
| 81 | +Flush safety nets: an explicit flush after a terminal error in `signalTaskComplete`, a full shutdown-flush in `cleanupSession` (which SIGTERM reaches via `stop()`), so sandbox teardown cannot eat the tail of a run's telemetry. |
| 82 | + |
| 83 | +## Changes in PostHog/code (this branch) |
| 84 | + |
| 85 | +- `packages/agent/src/otel-telemetry.ts` (renamed from `otel-log-writer.ts`): `OtelRunTelemetry`, the single `SessionLogSink` owning the OTLP log exporter and (when a traces URL is configured) the `RunTraceBuilder`. Contains the pure `mapNotificationToLogRecord()` allowlist mapper. Fixes the dead `/i/v1/agent-logs` default. Never throws into the run; ignores entries for other sessions. |
| 86 | +- `packages/agent/src/otel-trace-builder.ts` (new): `RunTraceBuilder`, the span state machine described above; `handle(entry)` returns the context each log record should attach to. |
| 87 | +- `packages/agent/src/otel-attributes.ts` (new): shared pure helpers (`strAttr`, `numAttr`, `usageAttributes`, truncation, caps). |
| 88 | +- `packages/agent/src/session-log-writer.ts`: optional `sinks: SessionLogSink[]`, teed in `appendRawLine` after the entry is built; a throwing sink warns once and can never break product log persistence; message chunks never reach sinks. `SessionContext` moved here from the otel module. |
| 89 | +- `packages/agent/src/server/agent-server.ts`: builds the telemetry per session from config (`createRunTelemetry`), passes it as the writer sink, stores it on the session, shuts it down in `cleanupSession`, flushes after terminal errors, and mirrors `enqueueTaskTerminalEvent` payloads into it directly (terminal `_posthog/error` events bypass `SessionLogWriter`, and a failed run is exactly what telemetry must record). |
| 90 | +- `packages/agent/src/server/bin.ts` + `server/types.ts`: zod-validated env `POSTHOG_AGENT_OTEL_LOGS_URL`, `POSTHOG_AGENT_OTEL_LOGS_TOKEN`, `POSTHOG_AGENT_OTEL_TRACES_URL` → `AgentServerConfig.otelLogsUrl/otelLogsToken/otelTracesUrl`. Telemetry is off unless the logs pair is set; spans additionally require the traces URL (per-signal kill switch). |
| 91 | +- `packages/agent/src/types.ts`: deleted the dead `OtelTransportConfig`/`AgentConfig.otelTransport` left over from the February attempt. |
| 92 | +- Dependencies: `@opentelemetry/api`, `@opentelemetry/sdk-trace-base`, `@opentelemetry/exporter-trace-otlp-http`, version-aligned with the existing logs SDK (0.208.x experimental / 2.x stable line). |
| 93 | +- Tests (71 passing): parameterized log-mapping matrix, a hard privacy test asserting exported payloads never contain tool args/titles/output, per-user resource attributes, session-mismatch guard, never-throws guard, sink isolation in `SessionLogWriter`, and four trace tests (span tree + statuses + attributes, log⇄span id correlation, error cascade, orphan export on shutdown). |
| 94 | +- `packages/agent/README.md`: documents the env vars and behavior. |
| 95 | + |
| 96 | +## Changes in PostHog/posthog (companion branch) |
| 97 | + |
| 98 | +- `posthog/settings/temporal.py`: new optional settings `SANDBOX_AGENT_OTEL_LOGS_URL`, `SANDBOX_AGENT_OTEL_LOGS_TOKEN`, `SANDBOX_AGENT_OTEL_TRACES_URL` (all default unset = telemetry off). |
| 99 | +- `products/tasks/backend/temporal/process_task/utils.py`: `get_sandbox_otel_env_vars()` maps those settings to the sandbox env vars, gated on the logs pair; called from **both** env assembly paths so fresh provisioning and snapshot-resume behave identically: |
| 100 | + - `activities/provision_sandbox.py` `_build_environment_variables` |
| 101 | + - `utils.py` `build_sandbox_environment_variables` (used by `create_sandbox_from_snapshot`) |
| 102 | +- `products/tasks/backend/constants.py`: the three env keys added to `RESERVED_SANDBOX_ENVIRONMENT_VARIABLE_KEYS` so user-supplied SandboxEnvironment vars cannot override them. |
| 103 | +- `products/tasks/backend/logic/services/agentsh.py`: `SANDBOX_AGENT_OTEL_LOGS_URL`/`SANDBOX_AGENT_OTEL_TRACES_URL` added to `_DEBUG_SANDBOX_URL_SETTINGS` so local-dev hosts pass the agentsh syscall firewall (prod egress already covered by `*.posthog.com`). |
| 104 | +- `products/tasks/backend/logic/services/docker_sandbox.py`: `POSTHOG_AGENT_OTEL_LOGS_URL`/`POSTHOG_AGENT_OTEL_TRACES_URL` added to `_DOCKER_URL_ENV_KEYS` so localhost URLs are rewritten to `host.docker.internal` for local Docker sandboxes. |
| 105 | +- Tests: parameterized gating matrix on `_build_environment_variables` (5 rows: full config, logs-only, partial configs, traces-without-logs all correctly gated) and a `SimpleTestCase` wiring guard on the snapshot-resume path. |
| 106 | +- `docs/internal/sandboxes-setup-guide.md`: local-dev setup section for the new settings. |
| 107 | + |
| 108 | +## Key design decisions |
| 109 | + |
| 110 | +1. **Emit from the sandbox, not from Django.** Independence from the product log path (see Architecture), matching the streamlit sandbox precedent. The Django-tee alternative would add an outbound call to a hot API path and go dark precisely when `append_log` breaks. |
| 111 | +2. **`POSTHOG_`-prefixed env vars instead of standard `OTEL_*` names.** The sandbox env is inherited by the user's own processes (their tests, their apps). Standard `OTEL_EXPORTER_OTLP_*` vars would make any OTel SDK in user code silently auto-export the user's telemetry into our internal project. Custom names mean only agent-server reads the config. |
| 112 | +3. **`service.name=posthog-code-agent`, not `posthog-code`.** `service.name` identifies the emitting process, not the product: the desktop app already ships its process logs as `posthog-code-desktop`, and `service_name` is both the primary Logs UI facet and part of the ClickHouse sort key `(team_id, service_name, timestamp)`, so component-level names keep streams separable and queries narrow. House pattern matches (`posthog-django-*`, `node-*`, `github-ci-logs`). |
| 113 | +4. **Fail-closed allowlist for content.** Only known lifecycle events are exported; unknown methods are dropped. This is a privacy boundary (customer prompts/repo content must not reach the telemetry project) and a cost control (logs are billed by bytes; in-progress tool snapshots re-send growing output). |
| 114 | +5. **Generic `SessionLogSink` instead of hardcoding OTel into `SessionLogWriter`.** The February attempt was removed partly because of hard coupling; the sink interface keeps the writer single-purpose, is desktop-neutral (no sinks wired there), and isolates sink failures. |
| 115 | +6. **Terminal-error mirror.** `enqueueTaskTerminalEvent` feeds only the event-ingest stream, bypassing `SessionLogWriter`; without the explicit mirror the most important record (run failed) would be missing from telemetry. |
| 116 | +7. **Per-signal kill switch.** Logs and spans have separate URLs; unsetting the traces URL disables spans without touching logs, and unsetting either of the logs pair disables everything. |
| 117 | +8. **Token exposure is acceptable by design.** The sandbox receives a project API key of the telemetry project: a write-only, public-by-design key class (the same class that ships in client SDKs), far weaker than the `POSTHOG_PERSONAL_API_KEY` already present in the sandbox. Worst case is junk telemetry writes; `capture-logs` has a token drop list as the kill switch. |
| 118 | + |
| 119 | +## Verification |
| 120 | + |
| 121 | +- `PostHog/code`: 71 tests pass in the agent package (including the new telemetry suite), `tsc --noEmit` clean via turbo, biome clean on all touched files (one pre-existing warning untouched). The package's pre-existing test failures in this environment (missing Postgres/git fixtures) were confirmed byte-identical with and without these changes by running the failing files against a stashed tree. |
| 122 | +- `PostHog/posthog`: 21 tests pass across `test_provision_sandbox.py` and the new `TestBuildSandboxEnvironmentVariables`; `ruff check`/`format` clean on all touched files. DB-dependent suites in this sandbox fail identically with and without the change (no Postgres available). |
| 123 | +- Local-dev routing verified: Caddy serves `/i/v1/logs`/`/i/v1/traces` on `localhost:8000` and proxies to `capture-logs`, and the Docker URL rewrite covers the new vars. |
| 124 | + |
| 125 | +## Rollout runbook (what remains) |
| 126 | + |
| 127 | +1. Choose the destination telemetry project and create/locate its project API key. Recommendation: the shared internal project where `posthog-code-desktop` logs and Code analytics already land, so desktop and cloud correlate in one Logs view (separable by `service_name`). |
| 128 | +2. Set in prod US: |
| 129 | + - `SANDBOX_AGENT_OTEL_LOGS_URL=https://us.i.posthog.com/i/v1/logs` |
| 130 | + - `SANDBOX_AGENT_OTEL_LOGS_TOKEN=<project API key>` |
| 131 | + - `SANDBOX_AGENT_OTEL_TRACES_URL=https://us.i.posthog.com/i/v1/traces` |
| 132 | +3. Run one cloud task; verify in the destination project: Logs filtered by `service.name=posthog-code-agent` (facet by `distinct_id`/`user_id`/`run_id`), and the APM trace for the run (`task_run` → `turn` → `tool_call:*` waterfall, logs linked from spans). |
| 133 | +4. Add a saved Logs view and alerts (error severity on the service; volume anomaly), and watch billed bytes for a week; the event allowlist and body caps are the tuning knobs. |
| 134 | +5. Optional follow-ups: unify the desktop transport with the new telemetry module; consider a shared `product` resource attribute across `posthog-code-*` services; sampling if volume warrants. |
| 135 | + |
| 136 | +## Configuration reference |
| 137 | + |
| 138 | +| Where | Name | Meaning | |
| 139 | +| --- | --- | --- | |
| 140 | +| Django settings | `SANDBOX_AGENT_OTEL_LOGS_URL` | Full OTLP logs ingest URL; unset = telemetry off | |
| 141 | +| Django settings | `SANDBOX_AGENT_OTEL_LOGS_TOKEN` | Project API key of the telemetry project; unset = telemetry off | |
| 142 | +| Django settings | `SANDBOX_AGENT_OTEL_TRACES_URL` | Full OTLP traces ingest URL; unset = spans off, logs unaffected | |
| 143 | +| Sandbox env (injected) | `POSTHOG_AGENT_OTEL_LOGS_URL` / `_TOKEN` / `POSTHOG_AGENT_OTEL_TRACES_URL` | Read by `agent-server` (`bin.ts`); reserved keys, not user-overridable | |
0 commit comments