Skip to content

test: guard QuickPick interactions against orphaned pickers#599

Merged
kevineger merged 3 commits into
mainfrom
fix-e2e-picker-flake
May 20, 2026
Merged

test: guard QuickPick interactions against orphaned pickers#599
kevineger merged 3 commits into
mainfrom
fix-e2e-picker-flake

Conversation

@kevineger

Copy link
Copy Markdown
Member

There's been a new batch of flakes pop up (likely due to a change in VS Code or our internal test harness infrastructure). This set of fixes should hopefully fix them.

VS Code uses a single shared <input> element for all QuickPick/InputBox surfaces (command palette, kernel picker, "Remove Server", etc.) and just swaps aria-label/placeholder/title-bar text when the active picker changes. vscode-extension-tester's InputBox.create() returns a handle to whatever picker is on screen with no check that it's the one the caller intended, so any prior step that leaves a picker open silently corrupts every subsequent setText / executeCommand.

This produced two related flake modes seen in our internal probe runs:

  1. Orphan picker: a cleanup selectQuickPickItem('Colab CPU') times out against the Remove Server picker; the try/catch swallows the error but the picker stays open. Every subsequent executeCommand sends Ctrl+Shift+P, which is captured by the still-focused picker, and the >command text is typed into its filter instead of being executed. The next test then fails with Notebook editor did not load in time.

  2. Picker-switch mid-call: while polling for "Python" in the kernel picker, the active picker on screen flips to Remove Server (e.g. from a cascading error flow); setText('Python') lands on the wrong picker's filter, the test eventually times out, and the failure screenshot shows "Remove Server" with "Python" typed into it.

Fixes:

  • pickerIdentity() / pickerIdentityMatches() read the title bar, input aria-label, and placeholder, and check whether the open picker matches a caller-supplied substring.
  • selectQuickPickItem, selectQuickPickItemIfShown, hasQuickPickItem, confirmInputBoxWithDefault, and selectQuickPicksInOrder now take an expectedPickerSubstring and skip iterations where the open picker isn't the expected one. This prevents picker-switch-mid-call (mode 2). Call sites pass the substring inline at each call (per go/tott/643: favor DAMP over DRY in tests so each step is self-documenting), e.g. selectQuickPickItem(driver, 'Remove Server', 'Colab CPU').
  • dismissAnyOpenInput() presses ESCAPE and waits for the input to disappear; called from the catch branch of the cleanup's Colab CPU selection in test-setup.ts, and prepended to afterEach. Prevents orphan-picker poisoning (mode 1).
  • safeExecuteCommand() wraps Workbench.executeCommand with a dismissAnyOpenInput precondition. All e2e callers now use this instead of the raw workbench.executeCommand(...) so "no picker open" is enforced before every command-palette invocation.
  • selectQuickPickItem's timeout message now reports the observed picker identity and the QuickPick items it offered, so the next failure points directly at "wrong picker open" or "expected item missing" instead of looking identical to a generic timeout.

src/test/e2e/connect-execute.e2e.test.ts # modified: src/test/e2e/mount-drive.e2e.test.ts # modified:
src/test/e2e/resource-view.e2e.test.ts # modified: src/test/e2e/test-setup.ts

There's been a new batch of flakes pop up (likely due to a change in VS Code or
our internal test harness infrastructure). This set of fixes should hopefully
fix them.

VS Code uses a single shared `<input>` element for all QuickPick/InputBox
surfaces (command palette, kernel picker, "Remove Server", etc.) and just swaps
`aria-label`/`placeholder`/title-bar text when the active picker changes.
`vscode-extension-tester`'s `InputBox.create()` returns a handle to whatever
picker is on screen with no check that it's the one the caller intended, so any
prior step that leaves a picker open silently corrupts every subsequent
`setText` / `executeCommand`.

This produced two related flake modes seen in our internal probe runs:

  1. Orphan picker: a cleanup `selectQuickPickItem('Colab CPU')` times out
     against the Remove Server picker; the `try/catch` swallows the error but
     the picker stays open. Every subsequent `executeCommand` sends
     Ctrl+Shift+P, which is captured by the still-focused picker, and the
     `>command` text is typed into its filter instead of being executed. The
     next test then fails with `Notebook editor did not load in time`.

  2. Picker-switch mid-call: while polling for "Python" in the kernel picker,
     the active picker on screen flips to Remove Server (e.g. from a cascading
     error flow); `setText('Python')` lands on the wrong picker's filter, the
     test eventually times out, and the failure screenshot shows "Remove Server"
     with "Python" typed into it.

Fixes:

  * `pickerIdentity()` / `pickerIdentityMatches()` read the title bar, input
    aria-label, and placeholder, and check whether the open picker matches a
    caller-supplied substring.
  * `selectQuickPickItem`, `selectQuickPickItemIfShown`, `hasQuickPickItem`,
    `confirmInputBoxWithDefault`, and `selectQuickPicksInOrder` now take an
    `expectedPickerSubstring` and skip iterations where the open picker isn't
    the expected one. This prevents picker-switch-mid-call (mode 2). Call sites
    pass the substring inline at each call (per go/tott/643: favor DAMP over DRY
    in tests so each step is self-documenting), e.g.
    `selectQuickPickItem(driver, 'Remove Server', 'Colab CPU')`.
  * `dismissAnyOpenInput()` presses ESCAPE and waits for the input to disappear;
    called from the `catch` branch of the cleanup's `Colab CPU` selection in
    `test-setup.ts`, and prepended to `afterEach`. Prevents orphan-picker
    poisoning (mode 1).
  * `safeExecuteCommand()` wraps `Workbench.executeCommand` with a
    `dismissAnyOpenInput` precondition. All e2e callers now use this instead of
    the raw `workbench.executeCommand(...)` so "no picker open" is enforced
    before every command-palette invocation.
  * `selectQuickPickItem`'s timeout message now reports the observed picker
    identity and the QuickPick items it offered, so the next failure points
    directly at "wrong picker open" or "expected item missing" instead of
    looking identical to a generic timeout.

src/test/e2e/connect-execute.e2e.test.ts #	modified:
src/test/e2e/mount-drive.e2e.test.ts #	modified:
src/test/e2e/resource-view.e2e.test.ts #	modified:   src/test/e2e/test-setup.ts

@hjjackyang hjjackyang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice improvements!

@kevineger kevineger merged commit e2280f9 into main May 20, 2026
10 checks passed
@kevineger kevineger deleted the fix-e2e-picker-flake branch May 20, 2026 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants