docs: add homepage hero slogan A/B test with PostHog tracking.#3161
docs: add homepage hero slogan A/B test with PostHog tracking.#3161clemra wants to merge 4 commits into
Conversation
Rotate the main headline between the existing platform positioning and an agent evals/tracing variant, update the hero subheading, and track exposures in PostHog for conversion analysis. Co-authored-by: Cursor <cursoragent@cursor.com>
dosubot
Bot
added
the
size:L
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
@claude review |
Keep Open Source on its own line and group Agent Evals / and Tracing on the second line with clearer gaps between phrases. Co-authored-by: Cursor <cursoragent@cursor.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| export function HeroSlogan() { | ||
| const posthog = usePostHog(); | ||
| const capture = usePostHogClientCapture(); | ||
| const [variant, setVariant] = useState<HeroSloganVariant>("ai-engineering"); | ||
|
|
||
| useEffect(() => { | ||
| const stored = readStoredVariant(); | ||
| const nextVariant = stored ?? pickVariant(); | ||
| const isNewAssignment = !stored; | ||
|
|
||
| if (!stored) { | ||
| persistVariant(nextVariant); | ||
| } | ||
|
|
||
| setVariant(nextVariant); | ||
|
|
||
| posthog.register({ hero_slogan_variant: nextVariant }); | ||
| capture("hero_slogan_exposure", { | ||
| variant: nextVariant, | ||
| is_new_assignment: isNewAssignment, | ||
| }); |
There was a problem hiding this comment.
agent-evals users
The component hard-codes "ai-engineering" as the initial state, so every server render and first client paint shows the control variant. The useEffect then reads localStorage and may switch to "agent-evals", causing a visible flash of the wrong heading for roughly half of new visitors. For returning users assigned to "agent-evals" the flash happens on every page load.
A lazy state initialiser avoids the re-render on the client side (localStorage access is already guard-wrapped for SSR): useState<HeroSloganVariant>(() => readStoredVariant() ?? pickVariant()). You'd also want to move persistVariant into the same initialiser block so the variant is stored before any render.
Prompt To Fix With AI
This is a comment left during a code review.
Path: components/home/HeroSlogan.tsx
Line: 112-132
Comment:
**Visible heading flash for `agent-evals` users**
The component hard-codes `"ai-engineering"` as the initial state, so every server render and first client paint shows the control variant. The `useEffect` then reads localStorage and may switch to `"agent-evals"`, causing a visible flash of the wrong heading for roughly half of new visitors. For returning users assigned to `"agent-evals"` the flash happens on every page load.
A lazy state initialiser avoids the re-render on the client side (localStorage access is already guard-wrapped for SSR): `useState<HeroSloganVariant>(() => readStoredVariant() ?? pickVariant())`. You'd also want to move `persistVariant` into the same initialiser block so the variant is stored before any render.
How can I resolve this? If you propose a fix, please make it concise.| function pickVariant(): HeroSloganVariant { | ||
| return Math.random() < 0.5 ? "ai-engineering" : "agent-evals"; |
There was a problem hiding this comment.
PostHog's experiments best practices require using getFeatureFlag() or the React equivalent useFeatureFlagVariantKey() for experiment assignment. Without this, PostHog does not record the built-in exposure event tied to an experiment flag, so those users are excluded from PostHog's experiment results and statistical significance calculations. The hero_slogan_exposure custom event captures the exposure for raw analysis, but the stated goal of "conversion analysis" in PostHog's Experiments UI won't work with this approach.
Consider migrating variant assignment to a PostHog feature flag with two variants — this provides cohort consistency, proper exposure tracking, and the Experiments results dashboard out of the box. See PostHog Experiments best practices.
Prompt To Fix With AI
This is a comment left during a code review.
Path: components/home/HeroSlogan.tsx
Line: 14-15
Comment:
**Custom random assignment bypasses PostHog Experiments**
PostHog's experiments best practices require using `getFeatureFlag()` or the React equivalent `useFeatureFlagVariantKey()` for experiment assignment. Without this, PostHog does not record the built-in exposure event tied to an experiment flag, so those users are excluded from PostHog's experiment results and statistical significance calculations. The `hero_slogan_exposure` custom event captures the exposure for raw analysis, but the stated goal of "conversion analysis" in PostHog's Experiments UI won't work with this approach.
Consider migrating variant assignment to a PostHog feature flag with two variants — this provides cohort consistency, proper exposure tracking, and the Experiments results dashboard out of the box. See [PostHog Experiments best practices](https://posthog.com/docs/experiments/best-practices).
How can I resolve this? If you propose a fix, please make it concise.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
| if (!stored) { | ||
| persistVariant(nextVariant); | ||
| } | ||
|
|
There was a problem hiding this comment.
posthog.register may fire before PostHog has finished initialising
usePostHog() returns the PostHog instance synchronously, but the underlying SDK's async bootstrap (feature-flag fetch, session init) may not have completed when this effect runs immediately after mount. Calling posthog.register() at this point can silently succeed but the super-property may not propagate to events captured in the same tick. A safer approach is to gate the registration on posthog.isFeatureEnabled or listen to the onFeatureFlags callback, ensuring the SDK is ready before registering super-properties.
Prompt To Fix With AI
This is a comment left during a code review.
Path: components/home/HeroSlogan.tsx
Line: 125
Comment:
**`posthog.register` may fire before PostHog has finished initialising**
`usePostHog()` returns the PostHog instance synchronously, but the underlying SDK's async bootstrap (feature-flag fetch, session init) may not have completed when this effect runs immediately after mount. Calling `posthog.register()` at this point can silently succeed but the super-property may not propagate to events captured in the same tick. A safer approach is to gate the registration on `posthog.isFeatureEnabled` or listen to the `onFeatureFlags` callback, ensuring the SDK is ready before registering super-properties.
How can I resolve this? If you propose a fix, please make it concise.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Assign the headline variant via middleware cookie and request header so SSR renders the correct slogan without hydration flicker, mirror desktop word spacing on the agent evals line, and defer PostHog registration until the SDK session is ready. Co-authored-by: Cursor <cursoragent@cursor.com>
|
@claude review |
| useEffect(() => { | ||
| const hadLocalStorage = readStoredVariant() !== null; | ||
| persistVariant(initialVariant); | ||
|
|
||
| if (trackedRef.current) { | ||
| return; | ||
| } | ||
|
|
||
| const runTracking = () => { | ||
| if (trackedRef.current) { | ||
| return; | ||
| } | ||
| trackedRef.current = true; | ||
| trackHeroSloganExposure( | ||
| posthog, | ||
| capture, | ||
| initialVariant, | ||
| !hadLocalStorage, | ||
| ); | ||
| }; |
There was a problem hiding this comment.
🔴 🟡 is_new_assignment is derived from localStorage (hadLocalStorage = readStoredVariant() !== null in components/home/HeroSlogan.tsx:130), but after this PR variant assignment authoritatively lives in the server-side cookie set by middleware.ts — localStorage is now just a write-only mirror that persistVariant(initialVariant) overwrites. The two stores diverge in real-world cleanup scenarios (Safari ITP cookie expiry clears cookies but not localStorage; quota eviction / dev tools / extensions clear localStorage but not cookies), producing false positives and false negatives on the very A/B exposure metric this PR exists to capture. Fix by computing the flag at the middleware/SSR boundary (whether the incoming cookie was already a valid variant) and threading it through as a second prop alongside initialVariant.
Extended reasoning...
What the bug is
The PR moves variant assignment from the client to the server: middleware.ts is the source of truth — it reads the langfuse-hero-slogan-variant cookie and, only if missing or invalid, calls pickHeroSloganVariant() (which uses Math.random()) and sets the cookie. app/(home)/page.tsx reads that cookie/header and passes initialVariant to HeroSlogan. The component then unconditionally calls persistVariant(initialVariant) to mirror the value into localStorage.
But is_new_assignment is still computed the old way (HeroSlogan.tsx:129-148):
const hadLocalStorage = readStoredVariant() !== null;
persistVariant(initialVariant);
// ...
trackHeroSloganExposure(posthog, capture, initialVariant, !hadLocalStorage);hadLocalStorage is a signal about the client-side mirror, not about whether the server just minted a fresh assignment. These two stores can and do diverge.
Step-by-step proof — false positive (returning user, localStorage cleared)
- A returning user visits
/. The browser still has thelangfuse-hero-slogan-variantcookie (it has a 1-yearmaxAge). - But they cleared localStorage (Safari ITP can clear it on a different schedule than cookies, devtools, extensions, quota eviction, private mode).
middleware.tsreads the cookie, findsexistingis valid, reuses it — no new assignment, cookie not re-set.page.tsxpasses that same variant asinitialVariant.- In
HeroSlogan's effect:readStoredVariant()returnsnull(empty localStorage) →hadLocalStorage = false→ firesis_new_assignment: true. - Wrong. No new assignment happened server-side.
Step-by-step proof — false negative (returning user, cookies cleared)
- A returning user visits
/with cleared cookies but intact localStorage (localStorage["langfuse-hero-slogan-variant"] === "ai-engineering"). middleware.tssees no cookie → callspickHeroSloganVariant()→ coin-flips to e.g."agent-evals"— a genuinely new server-side assignment that may even flip the bucket.page.tsxpasses"agent-evals"asinitialVariant.- In
HeroSlogan's effect:readStoredVariant()returns"ai-engineering"→hadLocalStorage = true→ firesis_new_assignment: false. - Wrong. Server just made a fresh assignment, and worse:
persistVariant(initialVariant)immediately overwrites localStorage with"agent-evals", silently masking the flip in any future debugging.
Why existing code doesn't prevent it
There is no signal threaded from middleware → page → component about whether the cookie was a hit or a miss. The component only sees the final initialVariant string; from its perspective, every variant looks the same regardless of whether it came from a reused cookie or a fresh Math.random() call.
Impact
This biases hero_slogan_exposure.is_new_assignment in opposite directions for the two failure modes, polluting the exposure metric the PR is specifically shipping for conversion analysis. The variant itself is still correctly assigned and shown to the user — so the visible behavior is fine — but the analytics signal that decides whether the A/B test is significant is contaminated. The practical bias is bounded to users whose cookie/localStorage state has drifted, but ITP and aggressive privacy tooling make that a non-trivial slice of Safari traffic.
Fix
Compute isNewAssignment authoritatively at the layer that knows: middleware.ts already evaluates existing && isHeroSloganVariant(existing) to decide whether to reuse vs. pick. Plumb that boolean through (e.g. via a second header x-hero-slogan-new-assignment, or via the absence/presence of the existing cookie at the page layer) and pass it as a second prop isNewAssignment alongside initialVariant. Then drop the readStoredVariant / hadLocalStorage logic from the effect entirely — localStorage is now redundant with the cookie and can either be removed or kept as a pure mirror that no longer feeds analytics decisions.
| export default async function HomePage() { | ||
| const [cookieStore, headerStore] = await Promise.all([ | ||
| cookies(), | ||
| headers(), | ||
| ]); | ||
| const heroSloganVariant = getHeroSloganVariantFromCookieValue( | ||
| headerStore.get(HERO_SLOGAN_VARIANT_HEADER) ?? | ||
| cookieStore.get(HERO_SLOGAN_VARIANT_KEY)?.value, | ||
| ); | ||
|
|
||
| return <Home heroSloganVariant={heroSloganVariant} />; |
There was a problem hiding this comment.
🔴 pnpm build:static (package.json:16, used for the Cloudflare static-export build path with maintained lib/_headers + scripts/generate-cloudflare-redirects.js infrastructure) breaks after this PR: under STATIC_EXPORT=true → output: "export" (next.config.mjs:43), Next.js does not support middleware.ts, and calling cookies()/headers() in app/(home)/page.tsx unconditionally opts the route into dynamic rendering — which the export build rejects ("headers was called in a route that is statically generated"). The active Vercel/CI path uses plain next build and is unaffected, but the static path is silently broken. Fix by either gating the dynamic code on process.env.STATIC_EXPORT !== "true" (with a deterministic default variant for static builds, plus an analogous matcher/no-op in middleware.ts), or by removing build:static + the supporting Cloudflare scripts if that target is retired.
Extended reasoning...
What the bug is
The repo ships two build paths:
pnpm build— plainnext build, used by Vercel and CI.pnpm build:static—cross-env STATIC_EXPORT=true ... next build && pnpm run postbuild:static(package.json:16), which next.config.mjs:43-48 wires tooutput: "export". The postbuild step runsscripts/generate-cloudflare-redirects.jsandcp lib/_headers out/— these are Cloudflare Pages convention artifacts (lib/_headersliterally contains "Static exports on Cloudflare should not be indexed").next-sitemap.config.jsandnext.config.mjsboth also branch onSTATIC_EXPORT === "true"(sitemap excludes,images.unoptimized), so this is deliberately maintained infrastructure, not dead code.
This PR introduces two features that are explicitly incompatible with output: "export":
-
A new top-level
middleware.ts. Next.js docs explicitly list Middleware as unsupported in static export — there is no Node/Edge runtime to execute it, and the build refuses to bundle it. -
app/(home)/page.tsxnow callscookies()andheaders()unconditionally:const [cookieStore, headerStore] = await Promise.all([cookies(), headers()]);
These are dynamic Server APIs; calling them in a Server Component opts the route into dynamic rendering, which
output: "export"rejects at build time with "headerswas called in a route that is statically generated".
Why existing code doesn't prevent it
There is no STATIC_EXPORT gate in either the page or the middleware. middleware.ts exports a matcher of "/" unconditionally, and the page imports/awaits cookies()/headers() on every render path. Before this PR no file in the repo imported from next/headers and there was no middleware — grep-ing the tree confirms the static build was working.
Step-by-step proof
- Run
pnpm build:static. The script setsSTATIC_EXPORT=trueand invokesnext build. - next.config.mjs:43 reads
process.env.STATIC_EXPORT === "true"and merges{ output: "export", trailingSlash: true, distDir: "out" }into the config. - Next.js begins compiling. It sees
middleware.tsat the project root and emits an error: middleware is not supported withoutput: "export". - Even if middleware were silently dropped, the page render fails:
app/(home)/page.tsxis a Server Component for a route that the export build attempts to statically render. When it hitsawait headers()/await cookies(), Next.js opts the route into dynamic rendering. Withoutput: "export"set, the build aborts with the static-rendering-required error. - Result:
build:staticexits non-zero beforepostbuild:staticever runs, so the Cloudflareout/directory is never produced.
Impact
CI (pnpm build) and the Vercel production deploy use the regular build path and are unaffected — this is why no verifier flagged this as merge-blocking on the active deploy. But the static export path is a real, maintained alternate target (Cloudflare Pages noindex mirror), and this PR silently breaks it. Anyone who runs pnpm build:static locally or in a separate deploy job will get a build failure that has nothing to do with their change. There is also a secondary semantic concern: even if it built, the prerendered HTML would freeze a single variant chosen at build time and never rotate per user — defeating the A/B test for the static deployment.
How to fix
Either:
- Gate the dynamic codepath. In
app/(home)/page.tsx, checkprocess.env.STATIC_EXPORT === "true"and short-circuit to a deterministic default variant (e.g.<Home heroSloganVariant="ai-engineering" />) without callingcookies()/headers(). Inmiddleware.ts, either gate the matcher to be empty underSTATIC_EXPORTor accept that middleware will be dropped (Next.js will still error unless the file is excluded from the build — easiest is to move the middleware behind a build-time check or rename it conditionally via a prebuild script). - Remove the static path. If the Cloudflare mirror is retired, delete
build:static+postbuild:staticfrom package.json, dropSTATIC_EXPORTbranches fromnext.config.mjsandnext-sitemap.config.js, removescripts/generate-cloudflare-redirects.jsandlib/_headers. Either way the codebase becomes coherent.
| pickHeroSloganVariant, | ||
| } from "@/lib/hero-slogan-variant"; | ||
|
|
||
| const HERO_SLOGAN_VARIANT_HEADER = "x-hero-slogan-variant"; |
There was a problem hiding this comment.
🟡 HERO_SLOGAN_VARIANT_HEADER = "x-hero-slogan-variant" is declared locally here, but lib/hero-slogan-variant.ts also exports the same constant (which app/(home)/page.tsx imports for the SSR read). The two literals must stay byte-identical for the middleware → page handshake to work, with no test or type linking them. middleware.ts already imports HERO_SLOGAN_VARIANT_KEY/isHeroSloganVariant/pickHeroSloganVariant from the same lib — add HERO_SLOGAN_VARIANT_HEADER to that import and remove the local const.
Extended reasoning...
What the bug is
lib/hero-slogan-variant.ts:1 exports HERO_SLOGAN_VARIANT_HEADER = "x-hero-slogan-variant", and app/(home)/page.tsx:4 imports it from there to read the SSR header value. But middleware.ts:9 re-declares its own local const HERO_SLOGAN_VARIANT_HEADER = "x-hero-slogan-variant" instead of importing it — even though the same import block already pulls HERO_SLOGAN_VARIANT_KEY, isHeroSloganVariant, and pickHeroSloganVariant from @/lib/hero-slogan-variant. The omission of HERO_SLOGAN_VARIANT_HEADER from that import list is clearly an oversight, not intentional.
Why this is not a live bug today
The two string literals are byte-identical ("x-hero-slogan-variant"), so the middleware → page handshake works correctly: middleware writes the header, page reads it, and the variant flows through. No current incorrectness.
Why it is still worth fixing
This is the textbook "two strings that must stay in sync" footgun, with nothing — no test, no shared type — linking the two literals. If one is ever edited (e.g. namespacing to x-langfuse-hero-slogan-variant) and the other is missed, middleware sets a header that page.tsx never reads. The page would then fall through to cookieStore.get(HERO_SLOGAN_VARIANT_KEY)?.value, which works for returning visitors but fails for first-time requests where the cookie is set only in the response — meaning getHeroSloganVariantFromCookieValue receives undefined and falls through to pickHeroSloganVariant(), producing a fresh Math.random() assignment on every SSR render that may disagree with the cookie middleware just set. The variant the user is bucketed into for tracking and the variant they see on first render would silently drift.
Step-by-step proof of the failure mode if literals drift
Suppose someone later edits lib/hero-slogan-variant.ts:1 to export const HERO_SLOGAN_VARIANT_HEADER = "x-langfuse-hero-slogan-variant" (a reasonable namespacing change) and does not edit middleware.ts:9:
- First-time visitor hits
/. Middleware runs, picks"agent-evals", setsrequestHeaderswith key"x-hero-slogan-variant"(its local literal) and queues aSet-Cookieforlangfuse-hero-slogan-variant=agent-evals. app/(home)/page.tsxruns.headerStore.get(HERO_SLOGAN_VARIANT_HEADER)— whereHERO_SLOGAN_VARIANT_HEADERis the imported"x-langfuse-hero-slogan-variant"— returnsnull(the middleware set it under the old key). The cookie was only just queued in the response and is not visible tocookieStore.get()on this same request, which returnsundefined.getHeroSloganVariantFromCookieValue(undefined)falls through topickHeroSloganVariant()→ freshMath.random()→ say"ai-engineering".- The user is served
"ai-engineering"HTML, but their cookie now persists"agent-evals". Next request they get"agent-evals". Variant flip on second pageview, plus a randomised SSR variant that disagrees with the persisted assignment on every first visit — directly undermining the A/B test.
Fix
Trivial — extend the existing import block at the top of middleware.ts:
import {
HERO_SLOGAN_VARIANT_HEADER,
HERO_SLOGAN_VARIANT_KEY,
isHeroSloganVariant,
pickHeroSloganVariant,
} from "@/lib/hero-slogan-variant";and delete const HERO_SLOGAN_VARIANT_HEADER = "x-hero-slogan-variant"; on line 9. Single source of truth, no behavior change.
1 participant
Rotate the main headline between the existing platform positioning and an agent evals/tracing variant, update the hero subheading, and track exposures in PostHog for conversion analysis.
Greptile Summary
This PR adds an A/B test for the homepage hero heading, rotating between "Open Source AI Engineering Platform" (control) and an "Agent Evals and Tracing" variant, persisting the assignment in
localStorageand firing ahero_slogan_exposureevent to PostHog on each mount.HeroSloganclient component encapsulates variant selection, localStorage persistence, and PostHog tracking, replacing the inline heading JSX inHero.tsx.usePostHogClientCapturehook is extended with typed definitions for the newhero_slogan_exposureevent.Math.random()and localStorage rather than PostHog's native Experiments feature flags, which limits the available analysis tooling in the PostHog UI.Confidence Score: 3/5
The change introduces a visible heading flash for ~50% of first-time visitors and uses a custom random assignment approach that bypasses PostHog's experiment analysis infrastructure, both of which should be addressed before merging.
The hard-coded initial state causes a flash of the wrong heading variant for every user assigned to 'agent-evals', which is half of new visitors on every first visit. Separately, the core objective — conversion analysis in PostHog — won't benefit from PostHog's built-in statistical tooling because variant assignment is done with Math.random() rather than a PostHog feature flag, meaning proper exposure events tied to an experiment won't be recorded.
components/home/HeroSlogan.tsx needs the most attention: the variant initialisation strategy (lazy state vs. effect-driven), and the decision to use custom random assignment vs. PostHog's Experiments feature flags.
Sequence Diagram
%%{init: {'theme': 'neutral'}}%% sequenceDiagram participant Browser participant HeroSlogan participant localStorage participant PostHog Browser->>HeroSlogan: "SSR / first paint (variant = "ai-engineering" default)" Note over HeroSlogan: All users see control variant initially Browser->>HeroSlogan: Hydration complete → useEffect fires HeroSlogan->>localStorage: readStoredVariant() alt Stored variant found localStorage-->>HeroSlogan: "ai-engineering" | "agent-evals" else No stored variant HeroSlogan->>HeroSlogan: "pickVariant() → Math.random() < 0.5" HeroSlogan->>localStorage: persistVariant(nextVariant) end HeroSlogan->>HeroSlogan: setVariant(nextVariant) Note over Browser,HeroSlogan: ⚠️ If nextVariant = "agent-evals",<br/>visible flash occurs here HeroSlogan->>PostHog: "posthog.register({ hero_slogan_variant })" HeroSlogan->>PostHog: "capture("hero_slogan_exposure", { variant, is_new_assignment })" PostHog-->>HeroSlogan: void%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%% sequenceDiagram participant Browser participant HeroSlogan participant localStorage participant PostHog Browser->>HeroSlogan: "SSR / first paint (variant = "ai-engineering" default)" Note over HeroSlogan: All users see control variant initially Browser->>HeroSlogan: Hydration complete → useEffect fires HeroSlogan->>localStorage: readStoredVariant() alt Stored variant found localStorage-->>HeroSlogan: "ai-engineering" | "agent-evals" else No stored variant HeroSlogan->>HeroSlogan: "pickVariant() → Math.random() < 0.5" HeroSlogan->>localStorage: persistVariant(nextVariant) end HeroSlogan->>HeroSlogan: setVariant(nextVariant) Note over Browser,HeroSlogan: ⚠️ If nextVariant = "agent-evals",<br/>visible flash occurs here HeroSlogan->>PostHog: "posthog.register({ hero_slogan_variant })" HeroSlogan->>PostHog: "capture("hero_slogan_exposure", { variant, is_new_assignment })" PostHog-->>HeroSlogan: voidPrompt To Fix All With AI
Reviews (1): Last reviewed commit: "Fix agent evals hero slogan layout and w..." | Re-trigger Greptile