telemetry: onboarding funnel instrumentation (install → activation → success → retention)

[Pritom14]

Goal

Instrument the ReverbCode journey (install → first PR raised → reviewed → revised → merged → returns) so drop-off is visible per install. This is a v1 measurement-only pass: no UX changes, no first-run wizard, no user-visible surface. If it emits an event we couldn't already observe, it belongs here; if it changes user behavior, it doesn't.

Locked definitions

• Activation = first PR raised on this install
• Success = first PR merged on this install
• Retention magic number = 4 returns (distinct calendar days after install day)
• Scope = app-first (Electron). CLI joins later on the same events by sharing the install_id file.

12-stage funnel

#	Stage	Event	Emitter
1	acquire	(off-product)	—
2	install	ao.daemon.started (pre-existing)	daemon boot
3	first launch	ao.app.active + is_first_launch=true	renderer
4	prereqs ready	ao.onboarding.prereqs_checked + gated prereqs_ready	daemon boot-goroutine
5	first project	ao.onboarding.first_project_added (pre-existing)	session svc
6	first session	ao.onboarding.first_session_spawned (pre-existing)	session svc
7	first agent output	ao.session.first_agent_output	lifecycle manager
8	first PR raised (ACTIVATION)	ao.session.pr_raised + first_pr_raised	CDC subscriber
9	PR reviewed	ao.session.pr_reviewed + first_pr_reviewed	CDC subscriber
10	changes made	ao.session.pr_revised + first_pr_revised	CDC subscriber
11	PR merged (SUCCESS)	ao.session.pr_merged + first_pr_merged	CDC subscriber
12	returns (RETENTION 4x)	ao.app.returned with return_count, is_retained, days_since_install	renderer

first_* events are the once-per-install onboarding milestones used for clean funnel math in PostHog. Per-session ao.session.* events fire every time the fact occurs (multiple PRs raised, multiple reviews, etc.).

User flows being measured

1. Cold-start install → activation. User installs the app → app supervisor launches daemon on first run → prereqs probe emits prereqs_checked → user adds first project → spawns first session → agent emits first output → agent opens a PR. Every step is a checkpoint; drop-off between any two is now visible.

2. Review-and-revise loop. Reviewer approves or requests changes → pr_reviewed; agent addresses feedback and the review thread is resolved → pr_revised. Same signals fire on every subsequent PR too, so the loop's throughput is measurable, not just its first traversal.

3. Success. PR is merged → pr_merged. First merge per install → first_pr_merged.

4. Retention. App is opened on a new calendar day after install → ao.app.returned with a return_count. When return_count >= 4 on any launch, is_retained=true flips permanently.

Once-per-install gating

The CDC subscriber runs off pr_created / pr_updated / pr_review_thread_resolved broadcaster events, which carry no prior-state history. To avoid re-emitting first_* events after a daemon restart, a durable file-based milestoneStore at <dataDir>/telemetry_milestones.json records the first-claim per key. Dedup keys used:

• first_pr_raised / first_pr_merged / first_pr_reviewed / first_pr_revised — one per install
• pr_merged:<url> — one pr_merged per PR (later pr_updated events on merged PRs are ignored)
• pr_reviewed:<url>:<decision> — one pr_reviewed per (PR, verdict) pair
• pr_revised:<pr>:<thread> — one pr_revised per resolved review thread
• prereqs_ready — one per install

The renderer's return/retention counter uses its own file (<dataDir>/telemetry_app_launches.json) so the daemon never writes it and it is immune to install-id races.

Checkpoint pointers (where the events are emitted)

Backend (Go daemon):

• backend/internal/daemon/onboarding_cdc.go — CDC subscriber: PR-lifecycle events + first_* milestones
• backend/internal/daemon/onboarding_prereqs.go — boot-goroutine probe: git / tmux (POSIX) / claude|codex / gh auth
• backend/internal/lifecycle/manager.go — firstAgentOutputEvent, fires once per spawn from ApplyActivitySignal
• backend/internal/daemon/daemon.go — wires startOnboardingCDC and emitPrereqsTelemetry into Run()

Frontend (Electron):

• frontend/src/shared/telemetry.ts — launch state (installDay, distinctActiveDays), computeLaunchUpdate, RETENTION_MAGIC_NUMBER = 4, extra bootstrap fields (isFirstLaunch, isReturnDay, returnCount, isRetained, daysSinceInstall)
• frontend/src/renderer/lib/telemetry.ts — emits ao.app.active with is_first_launch and ao.app.returned on new-day launches

Allowlist (property filtering for PostHog):

• backend/internal/adapters/telemetry/posthog.go — remotePayloadAllowlist entries for every new event. Events not listed still send with base props but the custom payload is dropped.

Storage

• Shared distinct_id: <dataDir>/telemetry_install_id (daemon writes/reads; renderer reads via bootstrap). Ensures backend and frontend events are one PostHog person.
• Once-per-install milestone store: <dataDir>/telemetry_milestones.json (daemon-owned).
• Return/retention counter: <dataDir>/telemetry_app_launches.json (renderer-owned).

Verification

• Unit tests: manager_test.go (first_agent_output), onboarding_cdc_test.go (all CDC paths + dedup), posthog_test.go (allowlist entries), telemetry.test.ts × 2 (launch-state transitions + renderer bootstrap).
• E2E: onboarding_cdc_e2e_test.go — drives real SQLite store → triggers → CDC poller → broadcaster → subscriber → sink.
• Packaged build: npm run make succeeded; all six funnel strings present in Contents/Resources/daemon/ao.
• Install-time verification via isolated-HOME sandbox: HOME=$(mktemp -d) AO_PORT=<free> AO_TELEMETRY_REMOTE=posthog … daemon — fresh install id, fresh DB, real PostHog project, ao.daemon.started + ao.onboarding.prereqs_checked observed both in local telemetry_event and remote sink with no rejection warnings.
• Not yet verified live: PR-stage events (raised/merged/reviewed/revised, first_agent_output) require real agent + GitHub interaction. Covered by unit + E2E only.

PostHog funnel steps (once configured)

Product Analytics → New Insight → Funnel. Use first_* events for clean per-install math. 14-30d window (merges lag). Retention magic#4 = separate Retention insight, not a funnel step.

1. ao.renderer.loaded
2. ao.onboarding.prereqs_ready
3. ao.onboarding.first_project_added
4. ao.onboarding.first_session_spawned
5. ao.session.first_agent_output
6. ao.onboarding.first_pr_raised (ACTIVATION)
7. ao.onboarding.first_pr_reviewed
8. ao.onboarding.first_pr_revised
9. ao.onboarding.first_pr_merged (SUCCESS)

Non-goals (v1)

• No first-run wizard or UX change.
• No CLI-side emitters (CLI joins later on the same install_id / event names).
• No dashboard build in this PR — configuration in PostHog is a follow-up once dashboard access lands.