Skip to content

docs: add homepage hero slogan A/B test with PostHog tracking. #3161

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
clemra wants to merge 4 commits into
base: main
Choose a base branch
from
Open

docs: add homepage hero slogan A/B test with PostHog tracking. #3161

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
29df6e7
Add homepage hero slogan A/B test with PostHog tracking.
clemra Jun 19, 2026
3cfca6c
Fix agent evals hero slogan layout and word spacing.
clemra Jun 19, 2026
b758615
Address hero slogan review: SSR variant, spacing, PostHog timing.
clemra Jun 19, 2026
6b3ad0c
Merge branch 'main' into feat/homepage-hero-slogan-experiment
clemra Jun 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 2 additions & 43 deletions components/home/Hero.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,10 @@
import { Button } from "@/components/ui/button";
import { CornerBox } from "@/components/ui/corner-box";
import { Heading } from "@/components/ui/heading";
import { Text } from "@/components/ui/text";
import { TextHighlight } from "@/components/ui";
import { HomeSection } from "@/components/home/HomeSection";
import { EnterpriseLogoGrid } from "@/components/shared/EnterpriseLogoGrid";
import { cn } from "@/lib/utils";
import { HeroStatsStrip } from "@/components/home/HeroStatsStrip";
import { HeroSlogan } from "@/components/home/HeroSlogan";

export function Hero() {
return (
Expand All @@ -15,46 +13,7 @@ export function Hero() {
<HeroStatsStrip />
</CornerBox>
<CornerBox className="flex flex-col gap-4 sm:gap-8 md:gap-10 items-center px-4 py-8 sm:px-8 sm:py-10">
<Heading
as="h1"
size="big"
className={cn(
"flex-col items-center gap-0.5 sm:gap-1 md:gap-1.5 text-center font-medium leading-[105%] max-md:max-w-[500px]",
"[leading-trim:both] [text-edge:cap]",
)}
>
<TextHighlight
highlightClassName="mix-blend-multiply"
className="whitespace-nowrap"
>
Open Source<span className="inline max-[499px]:hidden">&nbsp;</span>
</TextHighlight>
<span className="flex min-[500px]:inline">
<TextHighlight
highlightClassName="mix-blend-multiply"
className="max-[499px]:pr-1.75"
>
AI
</TextHighlight>
<TextHighlight
highlightClassName="mix-blend-multiply"
className="min-[500px]:pr-2"
>
Engineering
</TextHighlight>
</span>
<TextHighlight highlightClassName="mix-blend-multiply">
Platform
</TextHighlight>
</Heading>
<Heading
as="h1"
size="big"
className={cn(
"flex sm:hidden flex-col items-center gap-1.5 text-center font-medium leading-[105%]",
"[leading-trim:both] [text-edge:cap]",
)}
></Heading>
<HeroSlogan />
<div className="flex flex-col gap-6">
<Text className="max-w-xl">
Trace and evaluate AI Agents. Collaborate with your team to
Expand Down
143 changes: 143 additions & 0 deletions components/home/HeroSlogan.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
"use client";

import { useEffect, useState } from "react";
import { usePostHog } from "posthog-js/react";
import { Heading } from "@/components/ui/heading";
import { TextHighlight } from "@/components/ui";
import { cn } from "@/lib/utils";
import { usePostHogClientCapture } from "@/src/usePostHogClientCapture";

const STORAGE_KEY = "langfuse-hero-slogan-variant";

type HeroSloganVariant = "ai-engineering" | "agent-evals";

function pickVariant(): HeroSloganVariant {
return Math.random() < 0.5 ? "ai-engineering" : "agent-evals";

@greptile-apps greptile-apps Bot Jun 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 partner Custom random assignment bypasses PostHog Experiments

PostHog's experiments best practices require using getFeatureFlag() or the React equivalent useFeatureFlagVariantKey() for experiment assignment. Without this, PostHog does not record the built-in exposure event tied to an experiment flag, so those users are excluded from PostHog's experiment results and statistical significance calculations. The hero_slogan_exposure custom event captures the exposure for raw analysis, but the stated goal of "conversion analysis" in PostHog's Experiments UI won't work with this approach.

Consider migrating variant assignment to a PostHog feature flag with two variants — this provides cohort consistency, proper exposure tracking, and the Experiments results dashboard out of the box. See PostHog Experiments best practices.

Prompt To Fix With AI 
This is a comment left during a code review.
Path: components/home/HeroSlogan.tsx
Line: 14-15

Comment:
**Custom random assignment bypasses PostHog Experiments**

PostHog's experiments best practices require using `getFeatureFlag()` or the React equivalent `useFeatureFlagVariantKey()` for experiment assignment. Without this, PostHog does not record the built-in exposure event tied to an experiment flag, so those users are excluded from PostHog's experiment results and statistical significance calculations. The `hero_slogan_exposure` custom event captures the exposure for raw analysis, but the stated goal of "conversion analysis" in PostHog's Experiments UI won't work with this approach.

Consider migrating variant assignment to a PostHog feature flag with two variants — this provides cohort consistency, proper exposure tracking, and the Experiments results dashboard out of the box. See [PostHog Experiments best practices](https://posthog.com/docs/experiments/best-practices).

How can I resolve this? If you propose a fix, please make it concise.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

}

function readStoredVariant(): HeroSloganVariant | null {
try {
const stored = localStorage.getItem(STORAGE_KEY);
if (stored === "ai-engineering" || stored === "agent-evals") {
return stored;
}
} catch {
// localStorage unavailable (e.g. private browsing restrictions)
}
return null;
}

function persistVariant(variant: HeroSloganVariant) {
try {
localStorage.setItem(STORAGE_KEY, variant);
} catch {
// ignore write failures
}
}

const headingClassName = cn(
"flex-col items-center gap-0.5 sm:gap-1 md:gap-1.5 text-center font-medium leading-[105%] max-md:max-w-[500px]",
"[leading-trim:both] [text-edge:cap]",
);

function AiEngineeringSlogan() {
return (
<>
<TextHighlight
highlightClassName="mix-blend-multiply"
className="whitespace-nowrap"
>
Open Source<span className="inline max-[499px]:hidden">&nbsp;</span>
</TextHighlight>
<span className="flex min-[500px]:inline">
<TextHighlight
highlightClassName="mix-blend-multiply"
className="max-[499px]:pr-1.75"
>
AI
</TextHighlight>
<TextHighlight
highlightClassName="mix-blend-multiply"
className="min-[500px]:pr-2"
>
Engineering
</TextHighlight>
</span>
<TextHighlight highlightClassName="mix-blend-multiply">
Platform
</TextHighlight>
</>
);
}

function AgentEvalsSlogan() {
return (
<>
<TextHighlight
highlightClassName="mix-blend-multiply"
className="whitespace-nowrap"
>
Open Source<span className="inline max-[499px]:hidden">&nbsp;</span>
</TextHighlight>
<span className="flex min-[500px]:inline">
<TextHighlight
highlightClassName="mix-blend-multiply"
className="max-[499px]:pr-1.75"
>
Agent
</TextHighlight>
<TextHighlight
highlightClassName="mix-blend-multiply"
className="max-[499px]:pr-1.75"
>
Evals
</TextHighlight>
<TextHighlight
highlightClassName="mix-blend-multiply"
className="max-[499px]:pr-1.75"
>
and
</TextHighlight>
<TextHighlight highlightClassName="mix-blend-multiply">
Tracing
</TextHighlight>
</span>

Check failure on line 104 in components/home/HeroSlogan.tsx

View check run for this annotation

Claude / Claude Code Review

AgentEvals slogan layout broken at both breakpoints 

The new AgentEvalsSlogan variant reuses the inner-span layout (`<span className="flex min-[500px]:inline">`) from the 2-word AI Engineering variant but stuffs 4 words into it, which breaks at both breakpoints: at <500px the `flex` row with default `flex-wrap: nowrap` forces "Agent Evals and Tracing" onto a single line that overflows ~320–440px phone widths at 35–44px and gets clipped silently by `main`'s `overflow-hidden`; at ≥500px three words have only `max-[499px]:pr-1.75` (mobile-only) and "
Comment thread
claude[bot] marked this conversation as resolved.
Outdated
</>
);
}

export function HeroSlogan() {
const posthog = usePostHog();
const capture = usePostHogClientCapture();
const [variant, setVariant] = useState<HeroSloganVariant>("ai-engineering");

useEffect(() => {
const stored = readStoredVariant();
const nextVariant = stored ?? pickVariant();
const isNewAssignment = !stored;

if (!stored) {
persistVariant(nextVariant);
}

@greptile-apps greptile-apps Bot Jun 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 partner posthog.register may fire before PostHog has finished initialising

usePostHog() returns the PostHog instance synchronously, but the underlying SDK's async bootstrap (feature-flag fetch, session init) may not have completed when this effect runs immediately after mount. Calling posthog.register() at this point can silently succeed but the super-property may not propagate to events captured in the same tick. A safer approach is to gate the registration on posthog.isFeatureEnabled or listen to the onFeatureFlags callback, ensuring the SDK is ready before registering super-properties.

Prompt To Fix With AI 
This is a comment left during a code review.
Path: components/home/HeroSlogan.tsx
Line: 125

Comment:
**`posthog.register` may fire before PostHog has finished initialising**

`usePostHog()` returns the PostHog instance synchronously, but the underlying SDK's async bootstrap (feature-flag fetch, session init) may not have completed when this effect runs immediately after mount. Calling `posthog.register()` at this point can silently succeed but the super-property may not propagate to events captured in the same tick. A safer approach is to gate the registration on `posthog.isFeatureEnabled` or listen to the `onFeatureFlags` callback, ensuring the SDK is ready before registering super-properties.

How can I resolve this? If you propose a fix, please make it concise.

setVariant(nextVariant);

posthog.register({ hero_slogan_variant: nextVariant });
capture("hero_slogan_exposure", {
variant: nextVariant,
is_new_assignment: isNewAssignment,
});

@greptile-apps greptile-apps Bot Jun 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Visible heading flash for agent-evals users

The component hard-codes "ai-engineering" as the initial state, so every server render and first client paint shows the control variant. The useEffect then reads localStorage and may switch to "agent-evals", causing a visible flash of the wrong heading for roughly half of new visitors. For returning users assigned to "agent-evals" the flash happens on every page load.

A lazy state initialiser avoids the re-render on the client side (localStorage access is already guard-wrapped for SSR): useState<HeroSloganVariant>(() => readStoredVariant() ?? pickVariant()). You'd also want to move persistVariant into the same initialiser block so the variant is stored before any render.

Prompt To Fix With AI 
This is a comment left during a code review.
Path: components/home/HeroSlogan.tsx
Line: 112-132

Comment:
**Visible heading flash for `agent-evals` users**

The component hard-codes `"ai-engineering"` as the initial state, so every server render and first client paint shows the control variant. The `useEffect` then reads localStorage and may switch to `"agent-evals"`, causing a visible flash of the wrong heading for roughly half of new visitors. For returning users assigned to `"agent-evals"` the flash happens on every page load.

A lazy state initialiser avoids the re-render on the client side (localStorage access is already guard-wrapped for SSR): `useState<HeroSloganVariant>(() => readStoredVariant() ?? pickVariant())`. You'd also want to move `persistVariant` into the same initialiser block so the variant is stored before any render.

How can I resolve this? If you propose a fix, please make it concise.

// Track once when the hero mounts on the homepage.
// eslint-disable-next-line react-hooks/exhaustive-deps
}, []);

Check failure on line 132 in components/home/HeroSlogan.tsx

View check run for this annotation

Claude / Claude Code Review

Hero slogan flashes on hydration, biasing A/B test 

The HeroSlogan client component initializes `useState` to `"ai-engineering"`, so SSR and the first hydration paint always render the AI Engineering headline — even for users who will be bucketed into `agent-evals`. The variant decision runs only inside `useEffect` after mount, so ~50% of new visitors (and every returning visitor whose stored variant is `agent-evals`) see a visible headline swap + CLS, and the `hero_slogan_exposure` event records a variant the user did not actually see first — bi
Comment thread
claude[bot] marked this conversation as resolved.
Outdated

return (
<Heading as="h1" size="big" className={headingClassName}>
{variant === "ai-engineering" ? (
<AiEngineeringSlogan />
) : (
<AgentEvalsSlogan />
)}
</Heading>
);
}
4 changes: 4 additions & 0 deletions src/usePostHogClientCapture.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@ import { usePostHog } from "posthog-js/react";
// This preserves existing PostHog event structure while adding type safety
interface EventDefinitions {
copy_page: { type: "copy" | "chatgpt" | "claude" | "mcp" };
hero_slogan_exposure: {
variant: "ai-engineering" | "agent-evals";
is_new_assignment: boolean;
};
}

type EventName = keyof EventDefinitions;
Expand Down
Loading