Skip to content

chore(replay-vision): add quality tab docs#72902

Closed
arnohillen wants to merge 3 commits into
masterfrom
ahillen/replay-vision-quality-docs
Closed

chore(replay-vision): add quality tab docs#72902
arnohillen wants to merge 3 commits into
masterfrom
ahillen/replay-vision-quality-docs

Conversation

@arnohillen

@arnohillen arnohillen commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Note

Superseded by PostHog/posthog.com#18803. The Replay Vision docs section lives in the posthog.com repo (contents/docs/replay-vision/), so the page moved there to sit with its siblings and get a sidebar entry.

Problem

Replay vision's Quality tab (rate scanner results, feedback themes, ratings-over-time, AI configuration recommendations with test-before-apply) has no user-facing documentation yet. With the feature rolling out, users need a page explaining the rating loop and what each panel does.

Changes

Adds a first docs page at docs/published/docs/session-replay/replay-vision/quality.md, which publishes to posthog.com/docs/session-replay/replay-vision/quality on merge.

The page covers:

  • Rating results (thumbs up/down, written feedback, the unrated queue, confidence-first rating, version column)
  • Feedback themes and how they steer recommendations
  • The ratings-over-time chart and per-version accuracy chips
  • Configuration recommendations covering the prompt and the other behavior fields (tag vocabulary, scale, summary length, flags), with diff, rationale, regenerate/staleness, and history
  • Test-before-apply (fixed/still wrong/kept/regressed, quota cost, scorer/summarizer preview mode)
  • Apply/dismiss semantics and the overall improvement loop

Screenshots were taken from a local dev environment with Hedgebox demo data only (no customer data) and are hosted on PostHog/pr-assets. Happy to re-host on Cloudinary if that's preferred for posthog.com docs.

TODO before merge

  • Retake the screenshots with more realistic data: the current Hedgebox demo shots have synthetic session IDs (demo-fresh-*) and uniform confidence values. Use a scanner with real-looking sessions.

Preview of the page's hero image:

quality-tab-overview

How did you test this code?

Docs-only change, no code paths affected. I verified the copy against the implementation (products/replay_vision/frontend/replay_scanners/components/ScannerQualityTab.tsx, scannerQualityLogic.ts, and backend/prompt_suggestions.py) so each described behavior matches what ships: default unrated filter, daily recommendation refresh, per-version accuracy attribution, evaluation outcome labels, and quota charging for test re-runs.

Automatic notifications

  • Publish to changelog?
  • Alert Sales and Marketing teams?

Docs update

This PR is itself the docs update.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

I (well, Claude Code) wrote the page after reading the Quality tab frontend and the prompt-suggestion backend, then captured the screenshots with a Playwright script against a local dev stack seeded with Hedgebox demo data (dev banners hidden, light theme, retina scale), uploaded via hogli pr:upload-image. Decisions: placed the page under docs/published/docs/ (the monorepo path that publishes to posthog.com, following the existing surveys precedent) rather than the posthog.com repo, and used pr-assets image URLs as a first version pending Cloudinary hosting. No repo skills were applicable (docs-only, no migrations/DRF/tests).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@arnohillen arnohillen added skip-inkeep-docs Use this label to skip an Inkeep docs PR in posthog.com stamphog Request AI approval (no full review) labels Jul 22, 2026
@arnohillen
arnohillen requested review from a team, TueHaulund, fasyy612 and ksvat and removed request for a team July 22, 2026 13:52
@github-actions

github-actions Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

🤖 CI report

ℹ️ Docs preview — preview build triggered

Docs from this PR will be published at posthog.com.

Project Preview Updated (UTC)
posthog.com Open preview Jul 22, 2026, 2:07 PM

The preview should be ready in about 10 minutes. Open the preview at /handbook/engineering/.

arnohillen and others added 2 commits July 22, 2026 15:01
…y prompts

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@arnohillen arnohillen closed this Jul 22, 2026
@arnohillen
arnohillen deleted the ahillen/replay-vision-quality-docs branch July 22, 2026 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-inkeep-docs Use this label to skip an Inkeep docs PR in posthog.com stamphog Request AI approval (no full review)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant