Skip to content

Consolidate unified-input Maestro tests into one file#8628

Merged
malmstein merged 17 commits into
developfrom
feature/david/consolidate_unified_input_e2e
May 22, 2026
Merged

Consolidate unified-input Maestro tests into one file#8628
malmstein merged 17 commits into
developfrom
feature/david/consolidate_unified_input_e2e

Conversation

@malmstein
Copy link
Copy Markdown
Contributor

@malmstein malmstein commented May 19, 2026

Task/Issue URL: https://app.asana.com/1/137249556945/project/488551667048375/task/1214927873722256?focus=true
Testing scenarios: https://app.asana.com/1/137249556945/project/488551667048375/task/1215056189493585?focus=true

Description

Lands the full unified-input Maestro suite under .maestro/unified_input_screen/. Replaces the six per-combination unified_input_with_favorites_*.yaml files from the seeders work with a single consolidated test, and adds six new top-level tests covering the From-Search / New-Tab scenarios from the Native Input Widget E2E backlog plus a Duck.ai chat → Search navigation test.

Every top-level test pins nativeInputToggle: "true", uses addFavorites: "example.com;duckduckgo.com;privacytest.com", and wraps each iteration in retry: maxRetries: 3. The Duck.ai-side tests pin inputWithAiToggle: "true" and sweep omnibarPosition ∈ {top, bottom, split} (three iterations each). unified_input_with_seeded_favorites keeps the AI on/off sweep (six iterations) because the assertion is AI-independent.

Layout under .maestro/unified_input_screen/:

unified_input_with_seeded_favorites.yaml      # 6 iters (positions × AI)
unified_input_open_chat_move_to_search.yaml   # 3 iters (positions, AI on)
unified_input_search_suggestions.yaml         # 3 iters
unified_input_ask_duckai_suggestion.yaml      # 3 iters
unified_input_duckai_model_reasoning.yaml     # 3 iters
unified_input_duckai_tool_pills.yaml          # 3 iters
unified_input_duckai_web_search_prompt.yaml   # 3 iters
shared/
  launch_and_skip_onboarding.yaml             # local skip helper (see below)
  assert_seeded_favorites_visible.yaml
  open_new_chat.yaml
  flow_search_suggestions.yaml
  flow_ask_duckai_suggestion.yaml
  flow_duckai_model_reasoning.yaml
  flow_duckai_tool_pills.yaml
  flow_duckai_web_search_prompt.yaml

Scenarios covered

Test What it verifies
unified_input_with_seeded_favorites Three seeded favorites visible on the NTP across all omnibar positions and AI states
unified_input_open_chat_move_to_search Open Duck.ai chat, submit query, switch back to Search mode and run a regular search
unified_input_search_suggestions Autocomplete suggestions appear when typing in Search mode; submitting dismisses the unified input and loads the SERP
unified_input_ask_duckai_suggestion "Ask Duck.ai" autocomplete row opens Duck.ai with the typed query submitted
unified_input_duckai_model_reasoning Model picker opens, switching model updates the chip; reasoning picker opens, picking a non-default effort persists on re-open
unified_input_duckai_tool_pills Selecting Web Search / Create Image from the options menu places an active pill; tapping the pill removes the tool
unified_input_duckai_web_search_prompt Web Search applies to the first prompt and clears for the follow-up — the tool is per-prompt, not sticky

Infrastructure notes

  • shared/launch_and_skip_onboarding.yaml — robust local replacement for ../../shared/skip_all_onboarding.yaml. The upstream skip falls back to ../../shared/pre_onboarding.yaml whose asserted welcome copy has drifted from the new onboarding build, and the Skip Onboarding button bounds straddle the status bar height so tap-by-id is intercepted by the system bar. This helper waits for the button, taps by point at the bottom of its bounds, and waits for the unified input to settle.
  • addFavorites values are URL-list strings (semicolon-separated hosts) — matches the landed AddFavoritesSeederPlugin semantics from Add Maestro test seeding for NativeInputWidget E2E tests #8418.
  • All YAMLs target appId: com.duckduckgo.mobile.android (not .debug) to match the binary used in CI / Robin / Maestro Cloud.

CI wiring

.github/workflows/e2e-nightly-non-blockers-suite.yml adds a Unified Input Field step that runs the suite on Maestro Cloud, scoped via the unifiedInputTest tag, and reports failures to Asana.

Steps to test this PR

  • Install: ./gradlew installPlayDebug
  • Run the whole suite: maestro test .maestro/unified_input_screen --include-tags unifiedInputTest
  • All seven top-level tests pass — 24 iterations total (six tests × three positions plus the seeded-favorites test's six AI-and-position combos)

UI changes

Before After
N/A — test-only change N/A — test-only change

@malmstein malmstein force-pushed the feature/david/consolidate_unified_input_e2e branch 2 times, most recently from a8c7aa3 to 394607d Compare May 20, 2026 21:08
Base automatically changed from feature/david/unified_input_e2e_tests to develop May 22, 2026 11:11
David Gonzalez and others added 4 commits May 22, 2026 13:20
Replaces the six unified_input_with_favorites_* files (one per
omnibarPosition x inputWithAiToggle combo) with a single file that runs
the six combos sequentially. Verification of the seeded favorites is
extracted to a shared flow so future journeys (Duck.ai chat to Search)
can reuse the same launch shape.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add shared/open_new_chat.yaml covering chat entry, text input,
clear, newline, and submit-opens-Duck.ai. Wire it into
iteration 1/6.

Align appId in assert_seeded_favorites_visible.yaml with the
parent test. Drop per-iteration maxRetries wrappers.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The test-seeder TD landed on per-parameter mutations (addFavorites: "3")
rather than canned scenario keys. Update the consolidated test and its
shared verify flow to match: replace testScenario: "favorites_3" with
addFavorites: "3" and assert against the generic Favorite N titles
produced by AddFavoritesSeederPlugin.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@malmstein malmstein force-pushed the feature/david/consolidate_unified_input_e2e branch from 394607d to f443a04 Compare May 22, 2026 11:20
David Gonzalez and others added 8 commits May 22, 2026 13:35
Adds the five top-level test files for the From-Search/New-Tab batch.
Each runs three iterations sweeping omnibarPosition (top/bottom/split)
with nativeInputToggle and inputWithAiToggle pinned true. Inner flows
land in subsequent commits.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The merged AddFavoritesSeederPlugin (PR 8418 on develop) takes a
semicolon-separated URL list rather than a count, and uses the host
as each favorite's title. Switch the existing unified-input tests to
"example.com;duckduckgo.com;privacytest.com" and assert against those
host names instead of "Favorite 1/2/3". Add an extendedWaitUntil at
the head of the shared assertion to ride out the seeding latency.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Verifies the unified input shows autocomplete suggestions when the user
types in Search mode, submits the query, and dismisses cleanly into the
SERP.

Adds addFavorites to all five new top-level tests as a workaround for
the seeder/routing race that prevents Skip Onboarding from landing when
no mutation seeders are queued — this matches what PR 8628's existing
unified-input tests already do.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds:
- shared/flow_ask_duckai_suggestion.yaml — types a query, taps the
  Ask Duck.ai autocomplete row, and asserts the Duck.ai chat opens with
  duckAIHeader + the "Ask anything privately…" input placeholder.
- shared/launch_and_skip_onboarding.yaml — a robust local replacement
  for ../../shared/skip_all_onboarding.yaml. The upstream version falls
  back to ../../shared/pre_onboarding.yaml whose asserted welcome copy
  has drifted from the current onboarding build; that fallback throws
  and aborts our flows. This one taps Skip Onboarding directly and
  waits for inputModeWidget with a 30s budget.

All seven unified-input tests (the two existing PR 8628 ones plus the
five new ones) now route through the local skip flow.

Adds an extendedWaitUntil for inputModeWidget at the head of
flow_search_suggestions.yaml so it doesn't race the skip.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Verifies the model and reasoning-effort pickers in Duck.ai mode:
- model picker opens, shows alternatives, swapping to GPT-4o mini updates
  the chip text from "GPT-5" to "4o-mini"
- switching back to GPT-5 mini re-exposes the reasoning picker (4o-mini
  does not support reasoning so the pill is hidden)
- reasoning picker opens with "Fast" and "Reasoning" options, picking
  "Reasoning" dismisses the menu and the selection persists on re-open

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Verifies the Web Search and Create Image tool flows in Duck.ai mode.
Tools are picked from the options menu (sliders icon) which lists
"Web Search" and "Create Image". Selecting one places an active pill
in the bottom row with content-desc "Remove <tool>"; tapping the pill
removes it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Verifies that Web Search applies to the first prompt and clears for the
follow-up:
- selects Web Search from the options menu, asserts the active pill
- types a prompt with Web Search active and submits
- after the response, asserts the unified input re-opens with Web Search
  cleared so the tool is not sticky across prompts
- types a follow-up and confirms the pill remains absent

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Skip Onboarding button bounds straddle the status bar height on
the Pixel 9 — tapping the button by id resolves to its centre, which
the system status bar overlay intercepts before the tap reaches the
app. Switch to a point tap at the bottom of the button (50%, 9%) so
the tap lands inside the button rect but below the system bar.

Without this, the skip step is intermittent: the button visibly
flashes on tap but the welcome card does not dismiss.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@malmstein malmstein marked this pull request as ready for review May 22, 2026 15:05
Comment thread .github/workflows/e2e-nightly-non-blockers-suite.yml Outdated
Copy link
Copy Markdown
Contributor

@joshliebe joshliebe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check the comments from Cursor, otherwise LGTM

David Gonzalez and others added 2 commits May 22, 2026 17:22
Match the rest of the unifiedInputTest suite — every other top-level
test in this directory wraps each iteration in `retry: maxRetries: 3`,
but these two were relying on Maestro's lower default retry count and
were the only outliers. Bring them in line so CI flakiness is bounded
the same way across the whole suite.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The non-debug build is what gets exercised in CI and on Robin/Maestro
Cloud, so target com.duckduckgo.mobile.android directly. Strips the
.debug suffix from every top-level test and shared flow under
.maestro/unified_input_screen/.

No resource-id selectors used the .debug prefix, so the in-flow taps
and asserts are unaffected.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit d6b01c7. Configure here.

text: 'Agree and Continue'
- tapOn:
optional: true
text: "Agree and Continue"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing wait for T&C dialog causes test flakiness

Medium Severity

The T&C handling in open_new_chat.yaml uses assertVisible: optional: true + tapOn: optional: true without any extendedWaitUntil, unlike the other new flows in this PR (flow_ask_duckai_suggestion.yaml, flow_duckai_web_search_prompt.yaml) which correctly use runFlow: ../../shared/accept_duckai_agreement.yaml. That shared flow waits up to 10 seconds for the dialog to appear. Since every iteration launches with clearState: true, the T&C dialog will always appear on first Duck.ai use, but the inline optional check will race the WebView load — if it completes before the dialog renders, the dialog goes undismissed and blocks subsequent steps like tapOn: "Ask anything privately…".

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit d6b01c7. Configure here.

@malmstein malmstein merged commit bcf8d28 into develop May 22, 2026
18 checks passed
@malmstein malmstein deleted the feature/david/consolidate_unified_input_e2e branch May 22, 2026 15:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants