Fix telemetry: unify projects, add missing dimensions, extract keys from source

[willwashburn]

Summary

Audit of telemetry across relay (this repo) and relaycast (hosted side) revealed three classes of problems:

1. Data is fragmented across two PostHog projects with no shared identity, so we can't answer basic usage questions.
2. Missing dimensions mean even within each project we can't segment by harness, can't split CLI vs SDK, and can't count unique humans.
3. PostHog keys are hardcoded in source, so forks pollute our data and we have no dev/staging separation.

This issue tracks the full fix plan, prioritized.

---

What we can answer today

Question	Status	Notes
Total users of agent-relay	⚠️ Partial	Two PostHog projects; client uses hashed machine-ID, server uses workspace-ID. No join.
CLI vs SDK split	❌ No	SDK (packages/sdk/) emits zero telemetry.
Agents spun up	✅ Yes (client)	Broker agent_spawn event in src/telemetry.rs + src/main.rs:~1643.
OS distribution	✅ Captured, ❌ not surfaced	os/os_version/arch on every event (packages/telemetry/src/events.ts:55-76); no dashboard segments on it.
Messages sent	✅ Yes	Client: message_send. Server: usageRecords.messagesSent + messageLogs.
Which harnesses used most	❌ No	Broker tracks the spawned child's CLI (normalize_cli_name, src/main.rs:6666), not the parent harness (Claude Code vs Cursor vs Codex vs Gemini).

---

Gaps

1. No SDK telemetry — packages/sdk/ has no track() calls. Can't measure SDK adoption.
2. No orchestrator/harness identification — broker doesn't record what's driving it.
3. No install / update / first-run events — runInstallCommand in src/cli/lib/core-maintenance.ts is silent. No adoption curve.
4. No broker crash/panic telemetry — Rust panics fire nothing.
5. No unique-user metric on the server — distinct_id = workspace_id (relaycast/packages/server/src/lib/telemetry.ts:18). 1 workspace with 10 humans looks like 10 workspaces with 1 human.
6. No cross-workspace admin dashboard — /console/stats (relaycast/packages/server/src/routes/console.ts) is per-workspace only.
7. No workflow step / swarm worker granularity — workflow_run fires once at completion; no per-step timing or failure data.
8. MCP telemetry is a third silo — src/cli/relaycast-mcp.ts uses @relaycast/mcp telemetry which goes to the relaycast PostHog, not relay's.
9. Dashboard apps have zero client analytics — observer-dashboard, site/ have no PostHog JS.

---

Set up poorly

1. Two PostHog projects with no shared identity — same human shows up as a hashed machine-ID in one, workspace-ID in the other.
2. Workspace ID as distinct_id on the server side — should be authenticated user with workspace as a group/property.
3. origin_client is always '@relaycast/mcp' (relaycast/packages/mcp/src/telemetry.ts:219) — useless for harness segmentation.
4. PostHog is doing double duty as log sink (relaycast/packages/server/src/lib/logger.ts:166) and event store — OTel logs aren't indexed as events and pollute volume.
5. No server-side raw event log — if PostHog is down, events vanish.
6. Event schema drift risk — relay/packages/telemetry/src/events.ts and relaycast/packages/types/src/telemetry.ts define overlapping events independently.

---

Fix plan

P0 — Answer the business questions (this week)

• Harness detection in broker. Walk getppid() / process tree, match claude / cursor / codex / gemini-cli / aider / etc. Add orchestrator_harness to CommonProperties in packages/telemetry/src/events.ts:55. Default "unknown" so we can size the long tail.
• X-Relaycast-Harness header on relay → relaycast calls; read it in relaycast/packages/server/src/lib/serverTelemetry.ts:40 and store on every server event.
• install and update events in runInstallCommand / self-update path. ~5 lines.
• Minimal SDK telemetry — sdk_workflow_run and sdk_method_call events in packages/sdk/ with surface=sdk.

P0.5 — Extract PostHog keys from source

• Default to off. Source constant is empty / None; track() no-ops when empty.
• Inject at build time. GitHub Actions release workflow injects POSTHOG_KEY from a secret:
  • Rust: option_env!(\"POSTHOG_KEY\") in src/telemetry.rs (kill the hardcoded phc_2uDu01GtnLABJpVkWw4ri1OgScLU90aEmXmDjufGdqr)
  • TS: bundler define in packages/telemetry/src/client.ts
  • MCP: kill hardcoded phc_OAqBdey9... default in relaycast/packages/mcp/src/telemetry.ts:33
• Per-environment keys. POSTHOG_KEY_PROD / POSTHOG_KEY_STAGING as separate GitHub secrets.
• Update .github/workflows/release.yml to pass the secret to build steps.

PostHog phc_* keys are write-only and safe to be public — the reasons to extract them are: stopping forks from polluting prod data, enabling staging separation, and quota protection.

P1 — Make the data trustworthy (this sprint)

• Consolidate to one PostHog project. Pick relay's project (more mature schema). Point relaycast's POSTHOG_API_KEY env at it. Add app: property everywhere (broker / cli / sdk / relaycast-server / dashboard) so clean separation comes from properties, not from project boundaries. Export historical data from the old project if needed before sunset.
• Unify distinct_id. Send hashed machine-ID from client + include as property when relaycast receives MCP/HTTP calls. Use PostHog $identify/alias to stitch workspace_id ↔ machine_id ↔ user_id.
• Broker panic capture. std::panic::set_hook in src/main.rs fires a synchronous broker_panic event with error_class and first-frame symbol (no message — PII-safe).
• Move PostHog logs out of the events project. Either separate PostHog project for logs or route to Axiom / BetterStack. Right now log volume distorts event volume.
• Share the event schema. Publish @relay/telemetry-events consumed by both repos, or generate one from the other.

P2 — Queryable without PostHog UI

• Admin dashboard route in relaycast: /admin/analytics with workspaces_active, agent_spawns_by_cli, messages_by_day, harness_mix, os_mix. Backed by HogQL or daily rollup table.
• Instrument the dashboard apps with PostHog JS (observer-dashboard, site/).
• Per-step workflow telemetry — workflow_step_run with step name, duration, success.

---

Repo scope

The repos should stay separate (different release cycles, different deploy targets, different blast radius). The fragmentation problem is in the data, not the code boundary — fixed by unifying the PostHog project + the event schema, not by merging the repos.

---

Files most likely to change

relay:

• src/telemetry.rs — key extraction, harness field, panic hook
• src/main.rs — harness detection, panic hook
• packages/telemetry/src/events.ts — schema additions (orchestrator_harness, app)
• packages/telemetry/src/client.ts — key from build env, no-op when missing
• packages/sdk/ — new telemetry hooks
• src/cli/lib/core-maintenance.ts — install/update events
• .github/workflows/release.yml — POSTHOG_KEY secret injection

relaycast:

• packages/server/src/lib/serverTelemetry.ts — read X-Relaycast-Harness, store on event
• packages/server/src/lib/telemetry.ts — distinct_id strategy
• packages/mcp/src/telemetry.ts — kill hardcoded key default, send harness header
• packages/server/src/routes/console.ts (or new admin.ts) — cross-workspace analytics endpoint
• packages/server/src/lib/logger.ts — split logs off PostHog