feat(web): rename PostHog chat events, add tool_used tracking, and client-side ask events (#1111)

msukkari · claude · web-flow · commit 5305019eba35 · 2026-04-13T18:21:47.000-07:00
* feat(web): rename PostHog chat events, add tool_used tracking, and add client-side ask events Renames server-side chat events to drop the wa_ prefix (since they fire from multiple sources), adds a new tool_used event for unified tool call tracking across the ask agent and MCP server, and introduces client-side wa_ask_* events for accurate web-only usage dashboards. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add CHANGELOG entry for PostHog event changes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(web): add missing properties to client-side ask events Add isAnonymous to wa_ask_thread_created and messageCount to wa_ask_message_sent so they carry the same data as the server-side events. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(web): remove redundant api_code_search_request event This event is fully covered by api_request which fires for every API call and carries path + source. Updated the Any Feature Usage PostHog action to remove the api_code_search_request step. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(web): remove redundant wa_ask_thread_created event Redundant with ask_thread_created which fires server-side for the same thread creation (including from the web UI via createChat server action). Dashboards track ask usage via wa_ask_message_sent, not thread creation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(web): remove wa_ask_message_sent, set source header on chat transport Instead of a separate client-side event, the web client now sets the X-Sourcebot-Client-Source header on DefaultChatTransport. This makes ask_message_sent fire with source='sourcebot-web-client' for web UI calls, enabling web-only filtering directly on the server-side event. Updated Ask Usage and Any Feature Usage PostHog actions to filter ask_message_sent by source=sourcebot-web-client. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: consolidate CHANGELOG entries for #1111 into single entry Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(web): handle unhandled promise rejections in tool_used captureEvent calls Add .catch(() => {}) to all fire-and-forget captureEvent('tool_used') calls in adapters.ts and server.ts, matching the existing pattern in apiHandler.ts. Prevents unhandled promise rejections if telemetry fails. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(web): fix inflated MCP DAU by threading user identity through tool telemetry captureEvent() relies on tryGetPostHogDistinctId() which reads cookies, session, or API key headers. In the MCP server and ask agent contexts, none of these are available, so every tool_used event got a random distinct_id — inflating DAU counts (e.g., 8 tool calls = 8 "users"). Fix: add optional distinctId to ToolContext and captureEvent(). Thread user.id from the auth context through: - MCP route → createMcpServer(userId) → ToolContext.distinctId - chat route / askCodebase → createMessageStream(distinctId) → createAgentStream → createTools Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(web): rename distinctId to userId in ToolContext The field carries a user ID, not a PostHog-specific concept. The captureEvent options parameter remains distinctId since that's the PostHog term, but the application-level interfaces use userId. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(web): use try/finally pattern in registerMcpTool for tool_used event Match the same pattern used in toVercelAITool — single captureEvent call in a finally block with a success boolean, instead of duplicated calls in try and catch. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(web): move captureEvent error handling into captureEvent itself captureEvent now wraps its body in try/catch so callers are guaranteed it won't throw. Removed all .catch() handlers from call sites in adapters.ts, server.ts, and apiHandler.ts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(web): fix tryGetPostHogDistinctId to use getAuthenticatedUser tryGetPostHogDistinctId() was missing OAuth Bearer token resolution, causing all MCP tool calls to get random distinct_ids (inflated DAU). Fix: replace the manual cookie/session/API key checks with a call to getAuthenticatedUser(), which already handles all auth methods (session, OAuth, API keys). This removes the need to thread userId through ToolContext, agent options, and all callers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(web): add error logging in captureEvent catch block Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -13,6 +13,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Changed
 - Deprecated `GOOGLE_VERTEX_THINKING_BUDGET_TOKENS` environment variable in favor of per-model `thinkingBudget` config. [#1110](https://github.com/sourcebot-dev/sourcebot/pull/1110)
 - Removed `GOOGLE_VERTEX_INCLUDE_THOUGHTS` environment variable. Thoughts are now always included. [#1110](https://github.com/sourcebot-dev/sourcebot/pull/1110)
+- Renamed and consolidated PostHog chat events (`wa_chat_thread_created` -> `ask_thread_created`, `wa_chat_message_sent` -> `ask_message_sent`, `wa_chat_tool_used` -> `tool_used`), added unified `tool_used` tracking across the ask agent and MCP server, and removed the redundant `api_code_search_request` event. [#1111](https://github.com/sourcebot-dev/sourcebot/pull/1111)
 
 ## [4.16.8] - 2026-04-09
 
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -57,6 +57,12 @@ if (!value) { return; }
 if (condition) doSomething();
 ```
 
+## PostHog Event Naming
+
+- The `wa_` prefix is reserved for events that can ONLY be fired from the web app (e.g., `wa_login_with_github`, `wa_chat_feedback_submitted`).
+- Events fired from multiple sources (web app, MCP server, API) must NOT use the `wa_` prefix (e.g., `ask_message_sent`, `tool_used`).
+- Multi-source events should include a `source` property to identify the origin (e.g., `'sourcebot-web-client'`, `'sourcebot-mcp-server'`, `'sourcebot-ask-agent'`).
+
 ## Tailwind CSS
 
 Use Tailwind color classes directly instead of CSS variable syntax:
diff --git a/packages/web/src/app/api/(server)/chat/route.ts b/packages/web/src/app/api/(server)/chat/route.ts
@@ -91,12 +91,15 @@ export const POST = apiHandler(async (req: NextRequest) => {
                 return [];
             }))).flat();
 
-            await captureEvent('wa_chat_message_sent', {
+            const source = req.headers.get('X-Sourcebot-Client-Source') ?? undefined;
+
+            await captureEvent('ask_message_sent', {
                 chatId: id,
                 messageCount: messages.length,
                 selectedReposCount: expandedRepos.length,
+                source,
                 ...(env.EXPERIMENT_ASK_GH_ENABLED === 'true' ? { selectedRepos: expandedRepos } : {}),
-            } );
+            });
 
             const stream = await createMessageStream({
                 chatId: id,
diff --git a/packages/web/src/app/api/(server)/search/route.ts b/packages/web/src/app/api/(server)/search/route.ts
@@ -2,7 +2,6 @@
 
 import { search, searchRequestSchema } from "@/features/search";
 import { apiHandler } from "@/lib/apiHandler";
-import { captureEvent } from "@/lib/posthog";
 import { requestBodySchemaValidationError, serviceErrorResponse } from "@/lib/serviceError";
 import { isServiceError } from "@/lib/utils";
 import { NextRequest } from "next/server";
@@ -21,12 +20,6 @@ export const POST = apiHandler(async (request: NextRequest) => {
         ...options
     } = parsed.data;
 
-    const source = request.headers.get('X-Sourcebot-Client-Source') ?? 'unknown';
-    await captureEvent('api_code_search_request', {
-        source,
-        type: 'blocking',
-    });
-
     const response = await search({
         queryType: 'string',
         query,
diff --git a/packages/web/src/app/api/(server)/stream_search/route.ts b/packages/web/src/app/api/(server)/stream_search/route.ts
@@ -2,7 +2,6 @@
 
 import { streamSearch, searchRequestSchema } from '@/features/search';
 import { apiHandler } from '@/lib/apiHandler';
-import { captureEvent } from '@/lib/posthog';
 import { requestBodySchemaValidationError, serviceErrorResponse } from '@/lib/serviceError';
 import { isServiceError } from '@/lib/utils';
 import { NextRequest } from 'next/server';
@@ -20,12 +19,6 @@ export const POST = apiHandler(async (request: NextRequest) => {
         ...options
     } = parsed.data;
 
-    const source = request.headers.get('X-Sourcebot-Client-Source') ?? 'unknown';
-    await captureEvent('api_code_search_request', {
-        source,
-        type: 'streamed',
-    });
-
     const stream = await streamSearch({
         queryType: 'string',
         query,
diff --git a/packages/web/src/features/chat/actions.ts b/packages/web/src/features/chat/actions.ts
@@ -49,9 +49,10 @@ export const createChat = async ({ source }: { source?: string } = {}) => sew(()
             });
         }
 
-        await captureEvent('wa_chat_thread_created', {
+        await captureEvent('ask_thread_created', {
             chatId: chat.id,
             isAnonymous: isGuestUser,
+            source,
         });
 
         return {
diff --git a/packages/web/src/features/chat/agent.ts b/packages/web/src/features/chat/agent.ts
@@ -1,7 +1,6 @@
 import { SBChatMessage, SBChatMessageMetadata } from "@/features/chat/types";
 import { getAnswerPartFromAssistantMessage } from "@/features/chat/utils";
 import { getFileSource } from '@/features/git';
-import { captureEvent } from "@/lib/posthog";
 import { isServiceError } from "@/lib/utils";
 import { LanguageModelV3 as AISDKLanguageModelV3 } from "@ai-sdk/provider";
 import { ProviderOptions } from "@ai-sdk/provider-utils";
@@ -210,13 +209,7 @@ const createAgentStream = async ({
         ],
         toolChoice: "auto",
         onStepFinish: ({ toolResults }) => {
-            toolResults.forEach(({ toolName, output, dynamic }) => {
-                captureEvent('wa_chat_tool_used', {
-                    chatId,
-                    toolName,
-                    success: !isServiceError(output),
-                });
-
+            toolResults.forEach(({ output, dynamic }) => {
                 if (dynamic || isServiceError(output)) {
                     return;
                 }
diff --git a/packages/web/src/features/chat/components/chatThread/chatThread.tsx b/packages/web/src/features/chat/components/chatThread/chatThread.tsx
@@ -102,6 +102,9 @@ export const ChatThread = ({
         messages: initialMessages,
         transport: new DefaultChatTransport({
             api: '/api/chat',
+            headers: {
+                'X-Sourcebot-Client-Source': 'sourcebot-web-client',
+            },
         }),
         onData: (dataPart) => {
             // Keeps sources added by the assistant in sync.
diff --git a/packages/web/src/features/mcp/askCodebase.ts b/packages/web/src/features/mcp/askCodebase.ts
@@ -86,9 +86,10 @@ export const askCodebase = (params: AskCodebaseParams): Promise<AskCodebaseResul
                 },
             });
 
-            await captureEvent('wa_chat_thread_created', {
+            await captureEvent('ask_thread_created', {
                 chatId: chat.id,
                 isAnonymous: !user,
+                source,
             });
 
             if (user) {
@@ -137,10 +138,11 @@ export const askCodebase = (params: AskCodebaseParams): Promise<AskCodebaseResul
 
             let finalMessages: SBChatMessage[] = [];
 
-            await captureEvent('wa_chat_message_sent', {
+            await captureEvent('ask_message_sent', {
                 chatId: chat.id,
                 messageCount: 1,
                 selectedReposCount: selectedRepos.length,
+                source,
                 ...(env.EXPERIMENT_ASK_GH_ENABLED === 'true' ? {
                     selectedRepos: selectedRepos.map(r => r.value)
                 } : {}),
diff --git a/packages/web/src/features/mcp/server.ts b/packages/web/src/features/mcp/server.ts
@@ -2,6 +2,7 @@ import {
     languageModelInfoSchema,
 } from '@/features/chat/types';
 import { askCodebase } from '@/features/mcp/askCodebase';
+import { captureEvent } from '@/lib/posthog';
 import { isServiceError } from '@/lib/utils';
 import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
 import { ChatVisibility } from '@sourcebot/db';
@@ -57,6 +58,11 @@ export async function createMcpServer(): Promise<McpServer> {
         },
         async () => {
             const models = await getConfiguredLanguageModelsInfo();
+            captureEvent('tool_used', {
+                toolName: 'list_language_models',
+                source: 'sourcebot-mcp-server',
+                success: true,
+            });
             return { content: [{ type: "text", text: JSON.stringify(models) }] };
         }
     );
@@ -101,11 +107,22 @@ export async function createMcpServer(): Promise<McpServer> {
                 });
 
                 if (isServiceError(result)) {
+                    captureEvent('tool_used', {
+                        toolName: 'ask_codebase',
+                        source: 'sourcebot-mcp-server',
+                        success: false,
+                    });
                     return {
                         content: [{ type: "text", text: `Failed to ask codebase: ${result.message}` }],
                     };
                 }
 
+                captureEvent('tool_used', {
+                    toolName: 'ask_codebase',
+                    source: 'sourcebot-mcp-server',
+                    success: true,
+                });
+
                 const formattedResponse = dedent`
                 ${result.answer}
 
diff --git a/packages/web/src/features/tools/adapters.ts b/packages/web/src/features/tools/adapters.ts
@@ -1,6 +1,7 @@
 import { tool } from "ai";
 import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
 import { z } from "zod";
+import { captureEvent } from "@/lib/posthog";
 import { ToolContext, ToolDefinition } from "./types";
 
 export function toVercelAITool<TName extends string, TShape extends z.ZodRawShape, TMetadata>(
@@ -11,7 +12,21 @@ export function toVercelAITool<TName extends string, TShape extends z.ZodRawShap
         description: def.description,
         inputSchema: def.inputSchema,
         title: def.title,
-        execute: (input) => def.execute(input, context),
+        execute: async (input) => {
+            let success = true;
+            try {
+                return await def.execute(input, context);
+            } catch (error) {
+                success = false;
+                throw error;
+            } finally {
+                captureEvent('tool_used', {
+                    toolName: def.name,
+                    source: context.source ?? 'unknown',
+                    success,
+                });
+            }
+        },
         toModelOutput: ({ output }) => ({
             type: "content",
             value: [{ type: "text", text: output.output }],
@@ -38,13 +53,21 @@ export function registerMcpTool<TName extends string, TShape extends z.ZodRawSha
             },
         },
         async (input) => {
+            let success = true;
             try {
                 const parsed = def.inputSchema.parse(input);
                 const result = await def.execute(parsed, context);
                 return { content: [{ type: "text" as const, text: result.output }] };
             } catch (error) {
+                success = false;
                 const message = error instanceof Error ? error.message : String(error);
                 return { content: [{ type: "text" as const, text: `Tool "${def.name}" failed: ${message}` }], isError: true };
+            } finally {
+                captureEvent('tool_used', {
+                    toolName: def.name,
+                    source: context.source ?? 'unknown',
+                    success,
+                });
             }
         },
     );
diff --git a/packages/web/src/lib/apiHandler.ts b/packages/web/src/lib/apiHandler.ts
@@ -47,9 +47,7 @@ export function apiHandler<H extends AnyHandler>(
             const source = request.headers.get('X-Sourcebot-Client-Source') ?? 'unknown';
 
             // Fire and forget - don't await to avoid blocking the request
-            captureEvent('api_request', { path, method, source }).catch(() => {
-                // Silently ignore tracking errors
-            });
+            captureEvent('api_request', { path, method, source });
         }
 
         // Call the original handler with all arguments
diff --git a/packages/web/src/lib/posthog.ts b/packages/web/src/lib/posthog.ts
@@ -1,11 +1,12 @@
 import { PostHog } from 'posthog-node'
-import { env, SOURCEBOT_VERSION } from '@sourcebot/shared'
+import { createLogger, env, SOURCEBOT_VERSION } from '@sourcebot/shared'
 import { RequestCookies } from 'next/dist/compiled/@edge-runtime/cookies';
 import * as Sentry from "@sentry/nextjs";
 import { PosthogEvent, PosthogEventMap } from './posthogEvents';
 import { cookies, headers } from 'next/headers';
-import { auth } from '@/auth';
-import { getVerifiedApiObject } from '@/middleware/withAuth';
+import { getAuthenticatedUser } from '@/middleware/withAuth';
+
+const logger = createLogger('posthog');
 
 /**
  * @note: This is a subset of the properties stored in the
@@ -53,28 +54,19 @@ const getPostHogCookie = (cookieStore: Pick<RequestCookies, 'get'>): PostHogCook
  * Attempts to retrieve the distinct id of the current user.
  */
 export const tryGetPostHogDistinctId = async () => {
-    // First, attempt to retrieve the distinct id from the cookie.
+    // First, attempt to retrieve the distinct id from the PostHog cookie
+    // (set by the client-side PostHog SDK). This preserves identity
+    // continuity between client-side and server-side events.
     const cookieStore = await cookies();
     const cookie = getPostHogCookie(cookieStore);
     if (cookie) {
         return cookie.distinct_id;
     }
 
-    // Next, from the session.
-    const session = await auth();
-    if (session) {
-        return session.user.id;
-    }
-
-    // Finally, from the api key.
-    const headersList = await headers();
-    const apiKeyString = headersList.get("X-Sourcebot-Api-Key") ?? undefined;
-    if (!apiKeyString) {
-        return undefined;
-    }
-
-    const apiKey = await getVerifiedApiObject(apiKeyString);
-    return apiKey?.createdById;
+    // Fall back to the authenticated user's ID. This covers all auth
+    // methods: session cookies, OAuth Bearer tokens, and API keys.
+    const authResult = await getAuthenticatedUser();
+    return authResult?.user.id;
 }
 
 export const createPostHogClient = async () => {
@@ -88,24 +80,28 @@ export const createPostHogClient = async () => {
 }
 
 export async function captureEvent<E extends PosthogEvent>(event: E, properties: PosthogEventMap[E]) {
-    if (env.SOURCEBOT_TELEMETRY_DISABLED === 'true') {
-        return;
-    }
+    try {
+        if (env.SOURCEBOT_TELEMETRY_DISABLED === 'true') {
+            return;
+        }
 
-    const distinctId = await tryGetPostHogDistinctId();
-    const posthog = await createPostHogClient();
-
-    const headersList = await headers();
-    const host = headersList.get("host") ?? undefined;
-
-    posthog.capture({
-        event,
-        properties: {
-            ...properties,
-            sourcebot_version: SOURCEBOT_VERSION,
-            install_id: env.SOURCEBOT_INSTALL_ID,
-            $host: host,
-        },
-        distinctId,
-    });
+        const distinctId = await tryGetPostHogDistinctId();
+        const posthog = await createPostHogClient();
+
+        const headersList = await headers();
+        const host = headersList.get("host") ?? undefined;
+
+        posthog.capture({
+            event,
+            properties: {
+                ...properties,
+                sourcebot_version: SOURCEBOT_VERSION,
+                install_id: env.SOURCEBOT_INSTALL_ID,
+                $host: host,
+            },
+            distinctId,
+        });
+    } catch (error) {
+        logger.error('Failed to capture PostHog event:', error);
+    }
 }
diff --git a/packages/web/src/lib/posthogEvents.ts b/packages/web/src/lib/posthogEvents.ts
