Add the search metrics for the quick setup onboarding experiment according to the experiment design#8658
Conversation
…rding to the experiment design
This stack of pull requests is managed by Graphite. Learn more about stacking. |
Move the search-count pixels for the quick-setup onboarding experiment out of feature-toggles-impl into the app module, alongside the OnboardingQuickSetupToggles feature flag. The plugin now injects the toggle directly instead of looking it up by name in the inventory, so removing the flag at experiment end will surface this code as a compile-time usage to clean up. The metrics are now only emitted when the experiment toggle is enabled and the user is enrolled. Adds value=6 to the daily-windowed counts (was missing from the 2..10 sequence). Task/Issue URL: https://app.asana.com/1/137249556945/project/1211724162604201/task/1214753739886022 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
lmac012
left a comment
lmac012
left a comment
There was a problem hiding this comment.
I think we should consider moving search metrics to SearchMetricPixelsPlugin. Yes that would mean we would log them for all experiments, but that won't cause any harm. Otherwise, if we just deduplicate them now across 2 plugins, someone else can add more metrics / conversion windows to SearchMetricPixelsPlugin at any time, and not realize they are creating duplicates until the experiment is shipped and the data is looks off.
| MetricsPixel( | ||
| metric = "search", | ||
| type = MetricType.COUNT_WHEN_IN_WINDOW, | ||
| value = "6", | ||
| toggle = toggle, | ||
| conversionWindow = d0toD7Daily, | ||
| ), |
There was a problem hiding this comment.
This is duplicating metrics already defined in SearchMetricPixelsPlugin. If I understand correctly, both metrics will share the underlying storage for the counter, so we are going to increment the counter 2x for each search event and fire the pixel after just 3 searches instead of 6. cc @marcosholgado
There was a problem hiding this comment.
I removed the duplicate and left only the ones that are not already in SearchMetricPixelsPlugin
SearchMetricPixelsPlugin already emits value=6 with daily 0-14 windows for every active experiment toggle, which overlaps with the quick-setup plugin's value=6 daily 0-7. Both share the same storage key, so the counter incremented twice per ATB refresh and would have fired at 3 searches instead of 6. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
@lmac012 the reason why I created a different plugin for these metrics was because I didn’t want to send the metrics for all of the experiments as they are not needed. This PR adds a lot of metrics (we are measuring the search count for d0, d1,…d7, and we check if there is 1,2…10 searches for each of these windows. That would mean a lot of metrics being sent when they are not needed. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit b5e755a. Configure here.
| value = "7", | ||
| toggle = toggle, | ||
| conversionWindow = d0toD7Daily, | ||
| ), |
There was a problem hiding this comment.
Value "6" missing from sequential search metric range
High Severity
The metrics list emits values "1", "2", "3", "4", "5", "7", "8", "9", "10" — jumping from "5" directly to "7" and omitting "6". This leaves a gap in the intended sequential 1–10 range, meaning the 6th-search milestone will never be recorded for this experiment. The test assertion on the expected set mirrors the same omission, so it passes despite the bug.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit b5e755a. Configure here.
There was a problem hiding this comment.
This is already covered by SearchMetricPixelsPlugin
2 participants
Task/Issue URL: https://app.asana.com/1/137249556945/project/1211724162604201/task/1214753739886022
Description
Adds the search-count metrics for the quick-setup onboarding experiment (
onboardingQuickSetupExperimentMay26).The plugin lives alongside the experiment's feature flag in
com.duckduckgo.app.onboardingquicksetupand is wired as anAtbLifecyclePluginthat fires ononSearchRetentionAtbRefreshed. It injectsOnboardingQuickSetupTogglesdirectly (not via inventory-by-name) so removing the flag at experiment end will surface this code as a compile-time usage to clean up.Metrics emitted per search-refresh ATB:
searchvalue1on a singleD1..D3conversion window (mirrors the experiment's "first search" milestone).searchvalues2..10on dailyD0..D7conversion windows.Both gates are required: the experiment toggle must be
isEnabled()andisEnrolled()— otherwise no metrics are emitted.Steps to test this PR
Quick-setup search metrics
onboardingQuickSetupExperimentMay26toggle is disabled or the user is not enrolled, refresh the search retention ATB and confirm nosearch_*experiment pixels are sent for this toggle.searchpixels are emitted with values1..10and the documented conversion windows.OnboardingQuickSetupSearchMetricsAtbLifecyclePluginTestpass (./gradlew :app:testPlayDebugUnitTest --tests "com.duckduckgo.app.onboardingquicksetup.OnboardingQuickSetupSearchMetricsAtbLifecyclePluginTest").UI changes
Note
Low Risk
Analytics-only plugin with strict toggle enrollment gates; no user-facing or security-sensitive behavior.
Overview
Adds an
AtbLifecyclePluginthat emits experimentsearchmetrics when the search retention ATB refreshes, gated ononboardingQuickSetupExperimentMay26being both enabled and enrolled.On each refresh it sends
COUNT_WHEN_IN_WINDOWpixels for search counts 1 (single D1–D3 window) and 2, 3, 4, 5, 7, 8, 9, 10 (each with daily D0–D7 windows).onAppRetentionAtbRefreshedis left as a no-op. Unit tests cover toggle gating, window shapes, and metric binding to the experiment toggle.Reviewed by Cursor Bugbot for commit b5e755a. Bugbot is set up for automated code reviews on this repo. Configure here.