fix(agent): export error provenance only and isolate telemetry flush

Addresses four review findings on the run telemetry export:

- _posthog/error records and the root span no longer carry the raw error
  string: it is free text that can embed prompt or repo content (exception
  paths, provider errors). Exported signal is now the generic "run error"
  body plus error_source/stop_reason attributes; the full message stays in
  the session log and the task run's error_message.
- Telemetry flush/shutdown use Promise.allSettled so logs and traces
  complete independently, and both batch processors cap exports at 5s
  (exportTimeoutMillis) instead of the SDK's 30s default - a rejecting or
  hanging traces endpoint can no longer starve log delivery or delay
  session cleanup.
- A run error now closes still-open tool spans as ERROR with
  tool_status=interrupted, so APM never shows a healthy-looking active
  tool under a failed run.
- OtelTransportConfig/AgentConfig.otelTransport are restored as
  @deprecated ignored stubs: @posthog/agent is published, so removing
  exported types is an API break reserved for a major.

Generated-By: PostHog Code
Task-Id: 0c511836-2180-455a-9b58-45df2a0661ec

Author: tatoalo

REPORT.md

@@ -54,7 +54,7 @@ Exported events (allowlist, everything else is dropped):
 | `_posthog/usage_update` (and the `session/update` variant) | info | `tokens_input/output/cached_read/cached_write`, `cost_usd` |
 | `_posthog/turn_complete` | info | `stop_reason` |
 | `_posthog/task_complete` | info | `stop_reason` |
-| `_posthog/error` | **error** | `error_source`, `stop_reason`, message in body (capped) |
+| `_posthog/error` | **error** | `error_source`, `stop_reason`; body is the generic "run error" — the raw message is free text that can embed prompt/repo content, so it stays in the session log and the run's `error_message` |
 | `_posthog/progress` | info | `progress_group/step/status` |
 | `_posthog/git_checkpoint`, `_posthog/branch_created` | info | `branch` |
 | `_posthog/mode_change`, `_posthog/compact_boundary` | info | |
@@ -67,17 +67,18 @@ Bodies are capped at 2000 chars; free-text attribute values at 200 chars (the `l
 
 ### APM trace (one per run)
 
-- `task_run` root span (kind SERVER): opened at session init, closed at session cleanup. Status is resolved at shutdown from the latest turn outcome (the sandbox never emits `task_complete` for successful runs — the terminal "completed" status is decided by the workflow outside — so the last turn is the in-sandbox success signal): OK when the last turn ended with `end_turn`, ERROR with the error message on `_posthog/error` (an error always wins, later turn completions cannot flip it back) or when the last turn stopped with `error`, unset otherwise (cancelled / refused / timed out / no completed turns). Resolving at shutdown rather than per-turn means an early clean turn cannot leave a stale OK on a run whose final turn was cancelled.
+- `task_run` root span (kind SERVER): opened at session init, closed at session cleanup. Status is resolved at shutdown from the latest turn outcome (the sandbox never emits `task_complete` for successful runs — the terminal "completed" status is decided by the workflow outside — so the last turn is the in-sandbox success signal): OK when the last turn ended with `end_turn`, ERROR on `_posthog/error` (an error always wins, later turn completions cannot flip it back; `error_source` lands as a root-span attribute while the raw message is withheld — see the error row above) or when the last turn stopped with `error`, unset otherwise (cancelled / refused / timed out / no completed turns). Resolving at shutdown rather than per-turn means an early clean turn cannot leave a stale OK on a run whose final turn was cancelled.
 - `turn` child spans: opened on each ACP `session/prompt` (used purely as a boundary marker; its content is never read), closed on `_posthog/turn_complete`; attributes `turn_index`, `stop_reason`, plus the turn's token counts and `cost_usd` lifted from usage updates, so APM can rank slow or expensive turns directly.
 - `tool_call:<kind>` grandchild spans (`execute`, `read`, `edit`, ...): opened on `tool_call`, closed on the terminal `tool_call_update`; status ERROR on `failed`; attributes `tool_call_id`, `tool_kind`, `tool_status`. Per-kind span names stay low-cardinality and make APM latency breakdowns by tool kind useful.
-- Robustness: orphaned spans are closed (status unset) and exported at shutdown; a new prompt while a turn is open closes the stale turn; duplicate `tool_call` events are idempotent; an error cascades ERROR status through open tool and turn spans to the root.
+- Robustness: orphaned spans are closed (status unset) and exported at shutdown; a new prompt while a turn is open closes the stale turn; duplicate `tool_call` events are idempotent; a run error cascades ERROR status to the open turn and the root, and closes still-open tool spans as ERROR with `tool_status=interrupted` so APM never shows a healthy-looking active tool under a failed run.
 - Every log record is emitted under the OTel context of the span it belongs to (tool logs on the tool span, lifecycle logs on the root), so `trace_id`/`span_id` land in the `logs` table columns and the UI links Logs ⇄ trace waterfall.
 
 ### Delivery timing
 
 Telemetry is near-realtime, not end-of-turn: records are created the moment each notification flows through the writer and batched for at most 2 s (`BatchLogRecordProcessor` / `BatchSpanProcessor`, `scheduledDelayMillis` 2000).
 Spans export when they end (tools mid-turn, turns at `turn_complete`, root at cleanup).
 Flush safety nets: an explicit flush after a terminal error in `signalTaskComplete`, a full shutdown-flush in `cleanupSession` (which SIGTERM reaches via `stop()`), so sandbox teardown cannot eat the tail of a run's telemetry.
+Flush and shutdown are best-effort and per-signal independent (`Promise.allSettled`), and every export is capped at 5 s (`exportTimeoutMillis`, down from the SDK's 30 s default), so a rejecting or hanging traces endpoint can neither starve log delivery nor hold up session cleanup.
 
 ## Changes in PostHog/code (this branch)
 
@@ -87,9 +88,9 @@ Flush safety nets: an explicit flush after a terminal error in `signalTaskComple
 - `packages/agent/src/session-log-writer.ts`: optional `sinks: SessionLogSink[]`, teed in `appendRawLine` after the entry is built; a throwing sink warns once and can never break product log persistence; message chunks never reach sinks. `SessionContext` moved here from the otel module.
 - `packages/agent/src/server/agent-server.ts`: builds the telemetry per session from config (`createRunTelemetry`), passes it as the writer sink, stores it on the session, shuts it down in `cleanupSession`, flushes after terminal errors, and mirrors `enqueueTaskTerminalEvent` payloads into it directly (terminal `_posthog/error` events bypass `SessionLogWriter`, and a failed run is exactly what telemetry must record). Fatal crashes (`reportFatalError`, the uncaught-exception/unhandled-rejection path) also mirror an error record (`error_source=agent_server_crash`) and shut telemetry down, so hard process deaths reach the telemetry project instead of vanishing.
 - `packages/agent/src/server/bin.ts` + `server/types.ts`: zod-validated env `POSTHOG_AGENT_OTEL_LOGS_URL`, `POSTHOG_AGENT_OTEL_LOGS_TOKEN`, `POSTHOG_AGENT_OTEL_TRACES_URL` → `AgentServerConfig.otelLogsUrl/otelLogsToken/otelTracesUrl`. Telemetry is off unless the logs pair is set; spans additionally require the traces URL (per-signal kill switch).
-- `packages/agent/src/types.ts`: deleted the dead `OtelTransportConfig`/`AgentConfig.otelTransport` left over from the February attempt.
+- `packages/agent/src/types.ts`: the February `OtelTransportConfig`/`AgentConfig.otelTransport` remain as `@deprecated`, ignored stubs — `@posthog/agent` is a published package, so removing exported types is an API break reserved for a major.
 - Dependencies: `@opentelemetry/api`, `@opentelemetry/sdk-trace-base`, `@opentelemetry/exporter-trace-otlp-http`, version-aligned with the existing logs SDK (0.208.x experimental / 2.x stable line).
-- Tests (71 passing): parameterized log-mapping matrix, a hard privacy test asserting exported payloads never contain tool args/titles/output, per-user resource attributes, session-mismatch guard, never-throws guard, sink isolation in `SessionLogWriter`, and four trace tests (span tree + statuses + attributes, log⇄span id correlation, error cascade, orphan export on shutdown).
+- Tests (74 passing): parameterized log-mapping matrix, hard privacy tests asserting exported payloads never contain tool args/titles/output or raw error messages, per-user resource attributes, session-mismatch guard, never-throws guard, sink isolation in `SessionLogWriter`, and five trace tests (span tree + statuses + attributes, log⇄span id correlation, error cascade incl. interrupted tools, orphan export on shutdown, log shutdown isolated from a failing traces endpoint).
 - `packages/agent/README.md`: documents the env vars and behavior.
 
 ## Changes in PostHog/posthog (companion branch)
@@ -117,7 +118,7 @@ Flush safety nets: an explicit flush after a terminal error in `signalTaskComple
 
 ## Verification
 
-- `PostHog/code`: 71 tests pass in the agent package (including the new telemetry suite), `tsc --noEmit` clean via turbo, biome clean on all touched files (one pre-existing warning untouched). The package's pre-existing test failures in this environment (missing Postgres/git fixtures) were confirmed byte-identical with and without these changes by running the failing files against a stashed tree.
+- `PostHog/code`: 74 tests pass in the agent package (including the new telemetry suite), `tsc --noEmit` clean via turbo, biome clean on all touched files (one pre-existing warning untouched). The package's pre-existing test failures in this environment (missing Postgres/git fixtures) were confirmed byte-identical with and without these changes by running the failing files against a stashed tree.
 - `PostHog/posthog`: 21 tests pass across `test_provision_sandbox.py` and the new `TestBuildSandboxEnvironmentVariables`; `ruff check`/`format` clean on all touched files. DB-dependent suites in this sandbox fail identically with and without the change (no Postgres available).
 - Local-dev routing verified: Caddy serves `/i/v1/logs`/`/i/v1/traces` on `localhost:8000` and proxies to `capture-logs`, and the Docker URL rewrite covers the new vars.
 

packages/agent/README.md

@@ -172,7 +172,7 @@ Optional run telemetry (the logs pair must both be set, otherwise telemetry stay
 - `POSTHOG_AGENT_OTEL_LOGS_TOKEN` — project API key of the telemetry project
 - `POSTHOG_AGENT_OTEL_TRACES_URL` — full OTLP traces URL, e.g. `https://us.i.posthog.com/i/v1/traces`; additionally enables one APM trace per run
 
-When set, `AgentServer` ships an allowlisted metadata subset of the session log (run/turn/tool lifecycle, usage, errors — never message content or tool arguments; see `src/otel-telemetry.ts`) to PostHog Logs, tagged with `service.name=posthog-code-agent` and `run_id`/`task_id`/`team_id`/`user_id`/`distinct_id` resource attributes so cloud runs are filterable per user. With the traces URL set, each run also produces an APM trace (`task_run` root span, a `turn` span per prompt, a `tool_call:<kind>` span per tool call; see `src/otel-trace-builder.ts`), and log records carry the matching trace/span ids so Logs and APM cross-link.
+When set, `AgentServer` ships an allowlisted metadata subset of the session log (run/turn/tool lifecycle, usage, error provenance — never message content, tool arguments, or raw error text; see `src/otel-telemetry.ts`) to PostHog Logs, tagged with `service.name=posthog-code-agent` and `run_id`/`task_id`/`team_id`/`user_id`/`distinct_id` resource attributes so cloud runs are filterable per user. With the traces URL set, each run also produces an APM trace (`task_run` root span, a `turn` span per prompt, a `tool_call:<kind>` span per tool call; see `src/otel-trace-builder.ts`), and log records carry the matching trace/span ids so Logs and APM cross-link.
 
 ## Agent SDK
 
@@ -214,7 +214,7 @@ Logs serve two purposes: real-time observability and session resume. Every ACP m
 
 `SessionLogWriter` (`src/session-log-writer.ts`) is a per-session multiplexer that buffers raw ndJson lines. On flush (auto-scheduled 500ms after writes, or explicit), it POSTs batched `StoredNotification` entries to the Django API via `PostHogAPIClient.appendTaskRunLog()`; the API stores them in S3 as the task run's `log_url`. This S3 log is the source of truth for the full transcript and for session resume.
 
-Independently of that flush cycle, the writer tees every parsed non-chunk entry to optional `SessionLogSink`s at append time. In cloud, `OtelRunTelemetry` (`src/otel-telemetry.ts`) is wired as a sink and exports an allowlisted metadata subset (run/turn/tool lifecycle, usage, errors — never message content or tool arguments) to PostHog Logs, plus an APM trace per run when configured. See "Optional run telemetry" above for the env vars and behavior. Sink failures can never break S3 log persistence.
+Independently of that flush cycle, the writer tees every parsed non-chunk entry to optional `SessionLogSink`s at append time. In cloud, `OtelRunTelemetry` (`src/otel-telemetry.ts`) is wired as a sink and exports an allowlisted metadata subset (run/turn/tool lifecycle, usage, error provenance — never message content, tool arguments, or raw error text) to PostHog Logs, plus an APM trace per run when configured. See "Optional run telemetry" above for the env vars and behavior. Sink failures can never break S3 log persistence.
 
 ### Resuming from logs
 

packages/agent/src/otel-attributes.ts

@@ -2,11 +2,14 @@ const MAX_BODY_CHARS = 2000;
 // PostHog Logs only facets attribute key/value pairs shorter than 256 chars,
 // so free-text attribute values are capped well below that.
 const MAX_ATTR_CHARS = 200;
+// The SDK default export timeout is 30s; a hanging endpoint must not hold up
+// session cleanup (the sandbox can be torn down right after), so keep it short.
+const EXPORT_TIMEOUT_MS = 5000;
 
 export type AttributeValue = string | number | boolean;
 export type Attributes = Record<string, AttributeValue>;
 
-export { MAX_ATTR_CHARS, MAX_BODY_CHARS };
+export { EXPORT_TIMEOUT_MS, MAX_ATTR_CHARS, MAX_BODY_CHARS };
 
 export function truncate(value: string, max: number): string {
   return value.length <= max ? value : `${value.slice(0, max)}…`;

packages/agent/src/otel-telemetry.test.ts

@@ -11,17 +11,20 @@ const mockSpanExport = vi.fn((_spans, callback) => {
   callback({ code: 0 }); // Success
 });
 
+const mockLogShutdown = vi.fn(() => Promise.resolve());
+const mockSpanShutdown = vi.fn(() => Promise.resolve());
+
 vi.mock("@opentelemetry/exporter-logs-otlp-http", () => ({
   OTLPLogExporter: class {
     export = mockLogExport;
-    shutdown = vi.fn().mockResolvedValue(undefined);
+    shutdown = mockLogShutdown;
   },
 }));
 
 vi.mock("@opentelemetry/exporter-trace-otlp-http", () => ({
   OTLPTraceExporter: class {
     export = mockSpanExport;
-    shutdown = vi.fn().mockResolvedValue(undefined);
+    shutdown = mockSpanShutdown;
   },
 }));
 
@@ -88,6 +91,8 @@ describe("OtelRunTelemetry", () => {
   beforeEach(() => {
     mockLogExport.mockClear();
     mockSpanExport.mockClear();
+    mockLogShutdown.mockClear();
+    mockSpanShutdown.mockClear();
   });
 
   describe("mapNotificationToLogRecord", () => {
@@ -164,11 +169,12 @@ describe("OtelRunTelemetry", () => {
         name: "error",
         entry: makeEntry("_posthog/error", {
           source: "agent_server",
+          stopReason: "error",
           error: "boom",
         }),
         severityText: "ERROR",
-        body: "error: boom",
-        attrs: { error_source: "agent_server" },
+        body: "run error",
+        attrs: { error_source: "agent_server", stop_reason: "error" },
       },
       {
         name: "progress",
@@ -324,11 +330,29 @@ describe("OtelRunTelemetry", () => {
       expect(mapNotificationToLogRecord(entry)).toBeNull();
     });
 
-    it("caps body length", () => {
+    // Run errors export provenance only: the raw message is free text that
+    // can embed prompt or repo content (exception paths, provider errors).
+    it("never exports the raw error message", () => {
       const mapped = mapNotificationToLogRecord(
         makeEntry("_posthog/error", {
-          source: "agent_server",
-          error: "x".repeat(5000),
+          source: "agent_server_crash",
+          error: "Agent server crashed: ENOENT open '/repos/acme/SECRET/.env'",
+        }),
+      );
+
+      expect(mapped).not.toBeNull();
+      expect(JSON.stringify([mapped?.body, mapped?.attributes])).not.toContain(
+        "SECRET",
+      );
+    });
+
+    it("caps body length", () => {
+      const mapped = mapNotificationToLogRecord(
+        makeEntry("_posthog/progress", {
+          group: "setup:run-1",
+          step: "agent",
+          status: "completed",
+          label: "x".repeat(5000),
         }),
       );
 
@@ -574,7 +598,7 @@ describe("OtelRunTelemetry", () => {
       }
     });
 
-    it("marks failed tools, errored turns, and the errored run", async () => {
+    it("marks failed tools, interrupted tools, errored turns, and the errored run", async () => {
       telemetry.append(RUN_ID, makeEntry("session/prompt", {}));
       telemetry.append(
         RUN_ID,
@@ -592,11 +616,21 @@ describe("OtelRunTelemetry", () => {
           status: "failed",
         }),
       );
+      // Still open when the run error below lands.
+      telemetry.append(
+        RUN_ID,
+        sessionUpdate({
+          sessionUpdate: "tool_call",
+          toolCallId: "t2",
+          kind: "fetch",
+        }),
+      );
       telemetry.append(
         RUN_ID,
         makeEntry("_posthog/error", {
           source: "agent_server",
-          error: "gateway exploded",
+          stopReason: "error",
+          error: "gateway exploded reading SECRET",
         }),
       );
       // A turn completion arriving after the error must not flip the run
@@ -611,10 +645,37 @@ describe("OtelRunTelemetry", () => {
       expect(spanByName("tool_call:execute").status.code).toBe(
         SpanStatusCode.ERROR,
       );
-      expect(spanByName("turn").status.code).toBe(SpanStatusCode.ERROR);
+      const interrupted = spanByName("tool_call:fetch");
+      expect(interrupted.status.code).toBe(SpanStatusCode.ERROR);
+      expect(interrupted.attributes).toMatchObject({
+        tool_status: "interrupted",
+      });
+      const turn = spanByName("turn");
+      expect(turn.status.code).toBe(SpanStatusCode.ERROR);
+      expect(turn.attributes).toMatchObject({ stop_reason: "error" });
       const root = spanByName("task_run");
       expect(root.status.code).toBe(SpanStatusCode.ERROR);
-      expect(root.status.message).toBe("gateway exploded");
+      expect(root.attributes).toMatchObject({ error_source: "agent_server" });
+      // The raw error message is free text; it must not reach span status
+      // messages or attributes.
+      const surface = JSON.stringify(
+        exportedSpans().map((span) => [
+          span.name,
+          span.attributes,
+          span.status,
+        ]),
+      );
+      expect(surface).not.toContain("SECRET");
+    });
+
+    it("shuts down logs even when the traces endpoint fails", async () => {
+      mockSpanShutdown.mockRejectedValueOnce(new Error("traces endpoint down"));
+      telemetry.append(RUN_ID, makeEntry("_posthog/run_started", {}));
+
+      await expect(telemetry.shutdown()).resolves.toBeUndefined();
+
+      expect(mockLogShutdown).toHaveBeenCalled();
+      expect(exportedLogs().map((log) => log.body)).toContain("run started");
     });
 
     it("exports open spans on shutdown even without terminal events", async () => {