feat: Add web analytics doctor

transformation-config/skills/web-analytics/config.yaml

@@ -0,0 +1,10 @@
+type: docs-only
+template: description.md
+description: Audit a PostHog web analytics setup for misconfigurations
+tags: [web-analytics, audit]
+shared_docs: []
+variants:
+  - id: doctor
+    display_name: Web Analytics Doctor
+    tags: []
+    docs_urls: []

transformation-config/skills/web-analytics/description.md

@@ -0,0 +1,115 @@
+# PostHog Web Analytics Doctor
+
+This skill audits an existing PostHog project for common web-analytics misconfigurations. It is a **read-only audit**: it never writes code or PostHog config. The output is a markdown report at `posthog-web-analytics-report.md` and a structured JSON file at `posthog-web-analytics-findings.json`, both summarizing findings with severity and remediation links.
+
+## Reference files
+
+{references}
+
+## Guiding tenets
+
+1. **Read-only.** Never modify the user's source code or PostHog project settings. This skill audits; it does not remediate. Remediation is a follow-up the user runs themselves.
+
+2. **Quote real data.** Every finding must cite the HogQL query that produced it and at least one concrete value (host name, event count, ratio). If a query returns nothing actionable, omit the finding — don't speculate.
+
+3. **No fabricated severities.** Use the severity rules in `references/checks.md` exactly. If a check's data doesn't pass the threshold for `warning`, it's `info` or omitted.
+
+4. **Skip checks gracefully.** If a check's prerequisite query fails mid-audit (missing permission, transient error, etc.), skip that check, record it in the `skipped` array of the JSON output, and continue. The JSON file MUST still be written even if some checks didn't run. Do **not** abort unless a hard precondition fails (see Abort statuses).
+
+5. **Bounded query window.** See `references/checks.md` for the 7-day default and 30-day expansion rule. The window is decided once at pre-flight and reused for every check.
+
+## Available MCP tools
+
+- `mcp__posthog__query-run` (HogQL) — primary tool. Use for all event queries.
+- `mcp__posthog__project-get` — fetch project settings, including `app_urls` (the user-configured authorized URLs).
+- `mcp__posthog__feature-flag-get-definition` — only if a check needs flag context.
+- `mcp__posthog__docs-search` — to fetch the latest doc URL for a remediation link.
+
+## Pre-flight
+
+Before running any checks, verify the project has web analytics events and decide the analysis window in a single query:
+
+```sql
+SELECT
+  countIf(timestamp > now() - INTERVAL 7 DAY) AS p7,
+  count() AS p30
+FROM events
+WHERE event = '$pageview'
+  AND timestamp > now() - INTERVAL 30 DAY
+```
+
+- If `p30 = 0`, emit `[ABORT] No web analytics events`. The wizard middleware catches `[ABORT]` and terminates the run cleanly — do not halt yourself.
+- If `p7 >= 100`, use a 7-day window for every check. Otherwise use 30 days and set `expandedFromDefault: true` in the JSON output.
+- If the query returns a permission error, emit `[ABORT] Insufficient permissions`.
+
+Do not re-query to decide the window per check — the decision is made once here and reused.
+
+## How to run the audit
+
+Run ALL checks in `references/checks.md` without pausing for user confirmation. The user will review the final report.
+
+The checks share no state — issue their `query-run` calls in parallel when your tool harness supports concurrent calls. Note that Checks 3 and 4 share a single combined query (see `checks.md`); run that query once and apply both pass/fail rules to the result.
+
+For each check:
+
+1. Read the check's HogQL query and pass/fail rule.
+2. Run the query verbatim via `query-run`. Do not modify the query (other than swapping `INTERVAL 7 DAY` for `INTERVAL 30 DAY` if pre-flight selected the 30-day window).
+3. Apply the rule. If it fires, capture: severity (per the rule), host(s) affected, raw counts, and the remediation doc URL listed in the check.
+4. Report `[STATUS] Running check N: <name>` before each check.
+5. Append the finding (or "passed") to your in-memory findings list.
+
+Each check is independent and required. Do not skip a check based on intermediate findings. Do not invent new checks beyond the ones listed.
+
+## Output
+
+Produce the audit results in **three places**:
+
+1. **Write** `posthog-web-analytics-report.md` to the project root, following the format in `references/report-format.md`. Human-readable, archival.
+2. **Write** `posthog-web-analytics-findings.json` to the project root, following the schema in `references/findings-schema.md`. Machine-readable; the wizard parses this to render the structured findings screen. The file MUST exist — the wizard's report screen depends on it.
+3. **Display** the same report contents in chat as plain markdown (no extra commentary, no fenced code block wrapping the whole thing — just the report). This is what the user reads directly when running the skill outside the wizard.
+
+Both files MUST exist. The JSON and the markdown must describe the same set of findings — they are two views of the same audit.
+
+The report includes:
+
+- A summary section: total findings by severity (critical / warning / info), checks skipped.
+- One section per finding: title, severity, affected host(s), evidence (query + counts), remediation steps, doc link.
+- A "Checks passed" section listing every check that found no issues, so the user knows the audit was thorough.
+- A "Checks skipped" section listing any checks that couldn't run, with the reason.
+
+After displaying the markdown report, output one final line confirming both file paths (e.g. `Report saved to posthog-web-analytics-report.md and posthog-web-analytics-findings.json`). No further commentary, no recap.
+
+## Constraints
+
+- Do NOT modify any source files.
+- Do NOT write to PostHog (no creating dashboards, insights, actions, etc.).
+- Do NOT run queries against more than 30 days of data — performance budget.
+- Do NOT include personally identifiable information in the report (no email addresses, no user IDs, no session IDs, no IP addresses — host names, paths, and counts only).
+- Do NOT fabricate or estimate values. Only report what the queries return.
+- Findings that don't meet a check's threshold are simply omitted, not labeled "OK". (Severity values are defined in `references/checks.md`.)
+
+## Status
+
+Report progress with `[STATUS]` prefixed messages:
+
+- Verifying web analytics events
+- Running check 1: Partial reverse proxy
+- Running check 2: Dark authorized URLs
+- Running check 3: Pageleave coverage per host
+- Running check 4: Web Vitals coverage per host
+- Running check 5: Duplicate canonical URLs across hosts
+- Writing audit report
+- Audit complete
+
+## Abort statuses
+
+Report abort states with `[ABORT]` prefixed messages — wording must match exactly so the wizard renders the right error UI:
+
+- `[ABORT] No web analytics events` — pre-flight finds no `$pageview` events in the last 30 days.
+- `[ABORT] Insufficient permissions` — `query-run` returns a permissions error on the pre-flight query.
+
+Stop all further work after emitting `[ABORT]`.
+
+## Framework guidelines
+
+{commandments}

transformation-config/skills/web-analytics/references/checks.md

@@ -0,0 +1,171 @@
+# Web Analytics Doctor — Audit Checks
+
+Each check is independent. Run them in order; a failure in one does not block the others. Severity values are **fixed** — do not adjust them.
+
+Time window: every query below uses `INTERVAL 7 DAY`. The pre-flight in `description.md` decides once whether to use 7 days or 30 days for the entire run; if it picked 30, swap `INTERVAL 7 DAY` for `INTERVAL 30 DAY` everywhere in this file before running.
+
+---
+
+## Check 1 — Partial reverse proxy
+
+**What it detects:** Some hosts in the project route via a reverse proxy (so events survive ad blockers); others go directly to PostHog (where ad blockers can drop events). This is the support-case pattern: `example.com` proxied, `go.example.com` not.
+
+**Query:**
+
+```sql
+SELECT
+  properties.$host AS host,
+  countIf(
+    coalesce(properties.$lib_custom_api_host, '') = ''
+    OR coalesce(properties.$lib_custom_api_host, '') LIKE '%i.posthog.com%'
+    OR coalesce(properties.$lib_custom_api_host, '') LIKE '%posthog.com%'
+  ) AS direct,
+  countIf(
+    coalesce(properties.$lib_custom_api_host, '') != ''
+    AND coalesce(properties.$lib_custom_api_host, '') NOT LIKE '%posthog.com%'
+  ) AS proxied,
+  count() AS total
+FROM events
+WHERE event = '$pageview'
+  AND timestamp > now() - INTERVAL 7 DAY
+  AND properties.$host IS NOT NULL
+  AND properties.$host != ''
+GROUP BY host
+HAVING total >= 100
+ORDER BY total DESC
+LIMIT 50
+```
+
+> **HogQL NULL gotcha:** in HogQL, `NULL != ''` evaluates to TRUE (not NULL as in standard SQL), so an unwrapped `properties.$lib_custom_api_host != ''` will silently match every row where the property is missing. Always wrap the property in `coalesce(..., '')` before comparing or using `LIKE`/`NOT LIKE`.
+
+**Pass/fail rule:**
+- **Warning** if any host has `proxied >= 1` AND any other host has `proxied = 0 AND direct >= 1`, AND the two hosts share a common registrable parent domain (see "Parent-domain heuristic" below).
+- **Info** if `proxied` and `direct` hosts exist but parent domains differ — cross-environment mixing (e.g. dev vs prod) is plausible and shouldn't trigger a warning.
+- Otherwise: pass.
+
+**Parent-domain heuristic:** strip leading `www.`, then take the rightmost two dot-separated labels (so `go.example.com` and `app.example.com` both yield `example.com`). For these compound public suffixes, take the rightmost three labels instead: `co.uk`, `com.au`, `co.jp`, `co.nz`, `com.br`, `github.io`, `vercel.app`, `netlify.app`, `pages.dev`. This is a heuristic, not a public-suffix list — when in doubt, downgrade to **Info**.
+
+**Evidence to include in finding:** the proxied host(s), the direct host(s), each with their event counts.
+
+**Remediation:** https://posthog.com/docs/advanced/proxy
+
+---
+
+## Check 2 — Dark authorized URLs
+
+**What it detects:** The user configured authorized URLs in PostHog (via project settings), but one or more of those URLs has zero events in the analysis window. This often means an ad blocker is silently eating events for that domain — or the SDK was never installed there.
+
+**Step A — fetch authorized URLs:** call `mcp__posthog__project-get` and read `app_urls` from the response.
+
+**Step B — query event volume per known host:**
+
+```sql
+SELECT
+  properties.$host AS host,
+  count() AS pageviews
+FROM events
+WHERE event = '$pageview'
+  AND timestamp > now() - INTERVAL 7 DAY
+GROUP BY host
+ORDER BY pageviews DESC
+LIMIT 200
+```
+
+**Step C — compare:** for each entry in `app_urls`, parse out the hostname and check it against the query results in memory (do not issue one query per URL).
+
+Define `total_project_pageviews` = sum of `pageviews` across every row returned by Step B (all hosts, not just authorized ones).
+
+**Pass/fail rule:**
+- **Warning** for each authorized URL whose hostname is absent from Step B's results entirely.
+- **Warning** for each authorized URL whose `pageviews / total_project_pageviews < 0.01`, when at least one *other* authorized URL has `pageviews / total_project_pageviews >= 0.10`. (The peer threshold avoids firing on projects where every authorized URL is low-volume.)
+- Skip this check if `app_urls` is empty (note in report).
+
+**Evidence:** the dark host name, total project pageviews for context, and any peer authorized hosts with healthy volume.
+
+**Remediation:** https://posthog.com/docs/web-analytics/faq
+
+---
+
+## Checks 3 & 4 — Pageleave and Web Vitals coverage per host
+
+These two checks share a single combined query — run it once and apply both pass/fail rules to the result.
+
+**What Check 3 detects:** A host emits `$pageview` but few or no `$pageleave` events. Pageleave drives bounce rate and session duration — without it, web analytics dashboards under-report engagement for that domain.
+
+**What Check 4 detects:** A host has pageviews but no `$web_vitals` events, meaning LCP/CLS/INP performance metrics aren't captured.
+
+**Combined query:**
+
+```sql
+SELECT
+  properties.$host AS host,
+  countIf(event = '$pageview') AS pageviews,
+  countIf(event = '$pageleave') AS pageleaves,
+  countIf(event = '$web_vitals') AS web_vitals,
+  round(countIf(event = '$pageleave') / nullif(countIf(event = '$pageview'), 0), 3) AS pageleave_ratio
+FROM events
+WHERE event IN ('$pageview', '$pageleave', '$web_vitals')
+  AND timestamp > now() - INTERVAL 7 DAY
+  AND properties.$host IS NOT NULL
+  AND properties.$host != ''
+GROUP BY host
+HAVING pageviews >= 100
+ORDER BY pageviews DESC
+LIMIT 50
+```
+
+**Check 3 — Pageleave pass/fail rule:**
+- **Info** for any host with `pageleaves = 0` (and `pageviews >= 100`, which the `HAVING` already enforces).
+- **Warning** for any host with `pageleaves > 0 AND pageleave_ratio < 0.5` (less than half as many pageleaves as pageviews). Mutually exclusive with the Info rule above — never emit both for the same host.
+- **Evidence:** host, pageview count, pageleave count, ratio.
+- **Remediation:** https://posthog.com/docs/libraries/js#config — set `capture_pageleave: true`.
+
+**Check 4 — Web Vitals pass/fail rule:**
+- **Info** (not warning — many setups intentionally skip vitals) for any host with `web_vitals = 0`.
+- **Evidence:** host, pageview count.
+- **Remediation:** https://posthog.com/docs/web-analytics/web-vitals
+
+Emit Check 3 and Check 4 as separate findings (with their own `checkId` values per `findings-schema.md`) even though they came from one query.
+
+---
+
+## Check 5 — Duplicate canonical URLs across hosts
+
+**What it detects:** The same path (e.g. `/pricing`) is tracked under multiple `$host` values. This often means you have a marketing site and an app subdomain that both render the same content but get separate analytics — a sign that one is leaking or that proxy/canonical config is inconsistent.
+
+**Query:**
+
+```sql
+SELECT
+  properties.$pathname AS path,
+  groupUniqArray(properties.$host) AS hosts,
+  count() AS pageviews
+FROM events
+WHERE event = '$pageview'
+  AND timestamp > now() - INTERVAL 7 DAY
+  AND properties.$host IS NOT NULL
+  AND properties.$host != ''
+  AND properties.$pathname IS NOT NULL
+  AND properties.$pathname != ''
+GROUP BY path
+HAVING length(hosts) >= 2 AND pageviews >= 100
+ORDER BY pageviews DESC
+LIMIT 25
+```
+
+**Pass/fail rule:**
+- **Info** if any path appears under ≥ 2 hosts with combined `pageviews >= 100`.
+
+**Evidence:** the path, the list of hosts, combined pageview count.
+
+**Remediation:** https://posthog.com/docs/web-analytics/faq — review `$current_url` / `$host` capture, confirm reverse proxy and canonicalization across all domains.
+
+---
+
+## Severity ladder
+
+| Severity | When to use |
+| --- | --- |
+| `critical` | Reserved — none of the current checks emit critical. Future checks may. |
+| `warning` | Active misconfiguration likely causing data loss. Action required. |
+| `info` | Possibly intentional, but worth surfacing. The user decides. |