Skip to content

feat(telemetry): instrument 12-stage onboarding funnel#433

Open
Pritom14 wants to merge 3 commits into
mainfrom
feat/onboarding-funnel-telemetry
Open

feat(telemetry): instrument 12-stage onboarding funnel#433
Pritom14 wants to merge 3 commits into
mainfrom
feat/onboarding-funnel-telemetry

Conversation

@Pritom14

@Pritom14 Pritom14 commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Closes #432.

Summary

Measurement-only instrumentation of the 12-stage onboarding funnel (install → activation → success → retention). No UX changes, no first-run wizard. Activation = first PR raised, success = first PR merged, retention magic number = 4 returns on distinct calendar days.

Backend emitters:

  • backend/internal/daemon/onboarding_cdc.go — subscribes to the CDC broadcaster; turns pr_created / pr_updated / pr_review_thread_resolved into per-session ao.session.pr_* events plus once-per-install ao.onboarding.first_pr_* milestones. Gated by a durable file-based milestoneStore at <dataDir>/telemetry_milestones.json (needed because broadcaster events carry no prior-state history, and daemon restarts must not re-emit first_*).
  • backend/internal/daemon/onboarding_prereqs.go — boot-goroutine probe: ao.onboarding.prereqs_checked (per-check booleans for git / tmux on POSIX / claude|codex / gh auth) plus one-shot prereqs_ready when all pass. Skips entirely on later boots once readiness is claimed.
  • backend/internal/lifecycle/manager.go — firstAgentOutputEvent fires once per spawn on the first authoritative activity signal.
  • backend/internal/adapters/telemetry/posthog.go — remotePayloadAllowlist entries for every new event.

Frontend emitters:

  • frontend/src/shared/telemetry.ts — LaunchState persisted to <dataDir>/telemetry_app_launches.json (renderer-owned; avoids install-id race). Pure computeLaunchUpdate derives isFirstLaunch / isReturnDay / returnCount / isRetained / daysSinceInstall. RETENTION_MAGIC_NUMBER = 4.
  • frontend/src/renderer/lib/telemetry.ts — ao.app.active now carries is_first_launch; ao.app.returned fires on new-day launches with return_count / is_retained / days_since_install.

Shared distinct_id (<dataDir>/telemetry_install_id) keeps daemon + renderer events on one PostHog person.

See #432 for the full stage-by-stage table, dedup key list, user flows, and PostHog funnel configuration.

Test plan

  • Backend unit: go test ./backend/internal/daemon/... ./backend/internal/lifecycle/... ./backend/internal/adapters/telemetry/...
  • Backend E2E: TestE2E_OnboardingFunnelThroughStore drives real SQLite store → triggers → CDC poller → broadcaster → subscriber → sink
  • Frontend unit: npm test -- telemetry in frontend/
  • Packaged build (npm run make) contains all six funnel event strings in bundled daemon
  • Install-time telemetry verified via isolated-HOME sandbox (HOME=$(mktemp -d) AO_PORT=<free> AO_TELEMETRY_REMOTE=posthog … daemon) — ao.daemon.started + ao.onboarding.prereqs_checked reached PostHog with no rejection
  • Live PR-stage events (raised / merged / reviewed / revised / first_agent_output) — require real agent + GitHub interaction; unit + E2E only in this PR, manual GUI verification is a follow-up
  • PostHog dashboard build — deferred pending project access

🤖 Generated with Claude Code

Pritom14 and others added 2 commits July 1, 2026 13:13
@Pritom14 @claude
feat(telemetry): instrument 12-stage onboarding funnel
2eb5b44 
Adds measurement-only telemetry across the install → activation → success →
retention journey. Activation = first PR raised, success = first PR merged,
retention magic number = 4 returns on distinct calendar days.

Backend:
- daemon/onboarding_cdc.go: CDC subscriber turns pr_created/pr_updated/
  pr_review_thread_resolved into per-session ao.session.pr_* events plus
  once-per-install ao.onboarding.first_pr_* milestones, gated by a durable
  file-based milestoneStore at <dataDir>/telemetry_milestones.json.
- daemon/onboarding_prereqs.go: boot-goroutine probe emits
  ao.onboarding.prereqs_checked (per-check booleans) and a one-shot
  prereqs_ready when git + tmux (POSIX) + claude|codex + gh auth all pass.
- lifecycle/manager.go: ApplyActivitySignal emits ao.session.first_agent_output
  once per spawn on the first authoritative activity signal.
- telemetry/posthog.go: allowlist entries for every new event.

Frontend:
- shared/telemetry.ts: launch-state (installDay, distinctActiveDays) persisted
  to <dataDir>/telemetry_app_launches.json; pure computeLaunchUpdate derives
  isFirstLaunch/isReturnDay/returnCount/isRetained/daysSinceInstall.
- renderer/lib/telemetry.ts: ao.app.active carries is_first_launch;
  ao.app.returned fires on new-day launches with return_count / is_retained /
  days_since_install.

Storage: shared distinct_id from telemetry_install_id keeps daemon + renderer
events on one PostHog person. milestoneStore and launch-state files are
daemon-owned and renderer-owned respectively, avoiding the install-id race.

Refs #432

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@github-actions
chore: format with prettier [skip ci]
2081f2b
@Pritom14

Pritom14 commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator Author

Heads-up on PR size — the +5.5k / -3.7k header is misleading.

Only my commit 2eb5b44 (13 files, +964 / -9) is the funnel work. The rest comes from the Prettier GitHub Actions workflow, which runs npx prettier@3 --write . on every push and auto-commits the diff back to the branch. That auto-commit reformatted frontend/pnpm-lock.yaml (7746 lines — ~95% of the noise) plus a handful of docs that were never Prettier@3-clean on main. Root cause is that frontend/pnpm-lock.yaml isn't in .prettierignore (its npm/yarn cousins are), and main has drifted.

Both are repo-wide baseline concerns, not scope of this PR. Happy to open a follow-up adding the lockfile to .prettierignore if useful. Please review my commit only (2eb5b44).

@Pritom14 @claude
feat(telemetry): instrument renderer failure and CTA events
6410554 
Add PostHog instrumentation for app failures and user CTAs:
- daemon_failure (machine-readable code on DaemonStatus, IPC-forwarded)
- api_error central interceptor (categorized, IDs stripped)
- terminal_attach_failed (pane error + open timeout)
- CTA triads: task_create, session_kill, settings_save,
  orchestrator_spawn (board/restore_dialog), notification open/read

All events sanitized: project_id hashed, enum-only codes, raw error
messages never sent.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@Pritom14

Pritom14 commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator Author

Events captured

Full telemetry surface on this branch. All share one distinct_id (the install_id from <dataDir>/telemetry_install_id), so daemon-side and renderer-side events stitch into one person in PostHog.

Onboarding funnel — daemon-side (ao.onboarding.*)

First-touch, once-per-install milestones (use these for clean funnel math):

  • prereqs_checked, prereqs_ready — git/runtime/harness/github probe on boot
  • first_project_added
  • first_session_spawned
  • first_pr_raised (ACTIVATION)
  • first_pr_reviewed
  • first_pr_revised
  • first_pr_merged (SUCCESS)

Session / PR lifecycle — daemon-side (ao.session.*, ao.projects.*)

Every-occurrence counterparts to the milestones:

  • ao.session.spawned, ao.session.spawn_failed, ao.session.first_agent_output
  • ao.session.pr_raised, pr_reviewed, pr_revised, pr_merged
  • ao.session.waiting_input_entered, waiting_input_exited
  • ao.projects.created

App / daemon / CLI — (ao.daemon.*, ao.app.*, ao.cli.*)

  • ao.daemon.started, ao.daemon.panic
  • ao.app.active, ao.app.returned (retention; magic#4 at return_count=4)
  • ao.cli.invoked, ao.cli.usage_errors

Renderer navigation / funnel entry — (ao.renderer.*, ao.app.*)

  • ao.renderer.loaded, ao.renderer.route_viewed
  • ao.renderer.project_add_requested/succeeded, project_removed
  • ao.renderer.orchestrator_open_requested

NEW in this commit — failure signals (ao.renderer.*)

  • daemon_failure — { daemon_state, code, exit_code, signal } (8 machine-readable codes, set at main.ts + daemon-attach, IPC-forwarded over daemon:status)
  • api_error — { operation, error_category, status } (central interceptor; categories daemon_unavailable / network_error / http_4xx / http_5xx; operation IDs stripped; 30s dedupe)
  • terminal_attach_failed — { reason } (pane_error + first-attempt open_timeout)

NEW in this commit — CTA triads (ao.renderer.*)

Each is *_requested / *_succeeded / *_failed:

  • task_create (New task dialog)
  • session_kill (topbar kill confirm)
  • settings_save (project settings form)
  • orchestrator_spawn — { source: board | restore_dialog }
  • notification_opened — { target: pr | session }
  • notification_marked_read — { scope: single | all }

Sanitization guarantees

All renderer events pass through sanitizeRendererProperties: project_id is hashed, only enum codes are sent (never raw error messages or paths), and any event not on the allowlist sends base props only with its custom payload dropped.

Verification

Whole funnel + retention + failures + all six CTA categories fired end-to-end from a fresh packaged-app install in an isolated sandbox HOME and confirmed live in PostHog (project 475752). Props verified sanitized (hashed project_id, enum-only source/target/scope). orchestrator_spawn fired requested+failed in the sandbox (harness not configured for the throwaway repo → daemon 400); the renderer triad itself is correct.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

telemetry: onboarding funnel instrumentation (install → activation → success → retention)

1 participant

@Pritom14