test: guard QuickPick interactions against orphaned pickers#599
Merged
Conversation
There's been a new batch of flakes pop up (likely due to a change in VS Code or
our internal test harness infrastructure). This set of fixes should hopefully
fix them.
VS Code uses a single shared `<input>` element for all QuickPick/InputBox
surfaces (command palette, kernel picker, "Remove Server", etc.) and just swaps
`aria-label`/`placeholder`/title-bar text when the active picker changes.
`vscode-extension-tester`'s `InputBox.create()` returns a handle to whatever
picker is on screen with no check that it's the one the caller intended, so any
prior step that leaves a picker open silently corrupts every subsequent
`setText` / `executeCommand`.
This produced two related flake modes seen in our internal probe runs:
1. Orphan picker: a cleanup `selectQuickPickItem('Colab CPU')` times out
against the Remove Server picker; the `try/catch` swallows the error but
the picker stays open. Every subsequent `executeCommand` sends
Ctrl+Shift+P, which is captured by the still-focused picker, and the
`>command` text is typed into its filter instead of being executed. The
next test then fails with `Notebook editor did not load in time`.
2. Picker-switch mid-call: while polling for "Python" in the kernel picker,
the active picker on screen flips to Remove Server (e.g. from a cascading
error flow); `setText('Python')` lands on the wrong picker's filter, the
test eventually times out, and the failure screenshot shows "Remove Server"
with "Python" typed into it.
Fixes:
* `pickerIdentity()` / `pickerIdentityMatches()` read the title bar, input
aria-label, and placeholder, and check whether the open picker matches a
caller-supplied substring.
* `selectQuickPickItem`, `selectQuickPickItemIfShown`, `hasQuickPickItem`,
`confirmInputBoxWithDefault`, and `selectQuickPicksInOrder` now take an
`expectedPickerSubstring` and skip iterations where the open picker isn't
the expected one. This prevents picker-switch-mid-call (mode 2). Call sites
pass the substring inline at each call (per go/tott/643: favor DAMP over DRY
in tests so each step is self-documenting), e.g.
`selectQuickPickItem(driver, 'Remove Server', 'Colab CPU')`.
* `dismissAnyOpenInput()` presses ESCAPE and waits for the input to disappear;
called from the `catch` branch of the cleanup's `Colab CPU` selection in
`test-setup.ts`, and prepended to `afterEach`. Prevents orphan-picker
poisoning (mode 1).
* `safeExecuteCommand()` wraps `Workbench.executeCommand` with a
`dismissAnyOpenInput` precondition. All e2e callers now use this instead of
the raw `workbench.executeCommand(...)` so "no picker open" is enforced
before every command-palette invocation.
* `selectQuickPickItem`'s timeout message now reports the observed picker
identity and the QuickPick items it offered, so the next failure points
directly at "wrong picker open" or "expected item missing" instead of
looking identical to a generic timeout.
src/test/e2e/connect-execute.e2e.test.ts # modified:
src/test/e2e/mount-drive.e2e.test.ts # modified:
src/test/e2e/resource-view.e2e.test.ts # modified: src/test/e2e/test-setup.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
There's been a new batch of flakes pop up (likely due to a change in VS Code or our internal test harness infrastructure). This set of fixes should hopefully fix them.
VS Code uses a single shared
<input>element for all QuickPick/InputBox surfaces (command palette, kernel picker, "Remove Server", etc.) and just swapsaria-label/placeholder/title-bar text when the active picker changes.vscode-extension-tester'sInputBox.create()returns a handle to whatever picker is on screen with no check that it's the one the caller intended, so any prior step that leaves a picker open silently corrupts every subsequentsetText/executeCommand.This produced two related flake modes seen in our internal probe runs:
Orphan picker: a cleanup
selectQuickPickItem('Colab CPU')times out against the Remove Server picker; thetry/catchswallows the error but the picker stays open. Every subsequentexecuteCommandsends Ctrl+Shift+P, which is captured by the still-focused picker, and the>commandtext is typed into its filter instead of being executed. The next test then fails withNotebook editor did not load in time.Picker-switch mid-call: while polling for "Python" in the kernel picker, the active picker on screen flips to Remove Server (e.g. from a cascading error flow);
setText('Python')lands on the wrong picker's filter, the test eventually times out, and the failure screenshot shows "Remove Server" with "Python" typed into it.Fixes:
pickerIdentity()/pickerIdentityMatches()read the title bar, input aria-label, and placeholder, and check whether the open picker matches a caller-supplied substring.selectQuickPickItem,selectQuickPickItemIfShown,hasQuickPickItem,confirmInputBoxWithDefault, andselectQuickPicksInOrdernow take anexpectedPickerSubstringand skip iterations where the open picker isn't the expected one. This prevents picker-switch-mid-call (mode 2). Call sites pass the substring inline at each call (per go/tott/643: favor DAMP over DRY in tests so each step is self-documenting), e.g.selectQuickPickItem(driver, 'Remove Server', 'Colab CPU').dismissAnyOpenInput()presses ESCAPE and waits for the input to disappear; called from thecatchbranch of the cleanup'sColab CPUselection intest-setup.ts, and prepended toafterEach. Prevents orphan-picker poisoning (mode 1).safeExecuteCommand()wrapsWorkbench.executeCommandwith adismissAnyOpenInputprecondition. All e2e callers now use this instead of the rawworkbench.executeCommand(...)so "no picker open" is enforced before every command-palette invocation.selectQuickPickItem's timeout message now reports the observed picker identity and the QuickPick items it offered, so the next failure points directly at "wrong picker open" or "expected item missing" instead of looking identical to a generic timeout.src/test/e2e/connect-execute.e2e.test.ts # modified: src/test/e2e/mount-drive.e2e.test.ts # modified:
src/test/e2e/resource-view.e2e.test.ts # modified: src/test/e2e/test-setup.ts