-
Notifications
You must be signed in to change notification settings - Fork 100
test(e2e): add Phase C plans — project create, export jar, delete permanently #1029
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+488
−5
Merged
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
64dbb6d
test(e2e): add Phase C plans — project create, export jar, delete per…
wenytang-ms e633cd0
test(e2e): drop flaky LLM verify on expandTreeItem steps
wenytang-ms 2e5553e
test(e2e): drop flaky LLM verify on select-interface
wenytang-ms b06302e
test(e2e): address Copilot review on header comments
wenytang-ms File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,144 @@ | ||
| # Test Plan: Java Dependency — Permanent Delete | ||
| # | ||
| # Covers java.view.package.deleteFilePermanently (the "shift+delete" / | ||
| # non-trash branch of file removal). The companion command | ||
| # java.view.package.moveFileToTrash is already covered by | ||
| # java-dep-file-operations.yaml. | ||
| # | ||
| # Why this is a separate plan | ||
| # ─────────────────────────── | ||
| # The permanent-delete command has no plain-click UI affordance on regular | ||
| # local files: the JAVA PROJECTS context-menu entry (package.json:812-820) | ||
| # and the `delete` keybinding (package.json:485-490) are both gated on | ||
| # `!explorerResourceMoveableToTrash`, so files in `test/maven` only ever | ||
| # see "Delete" (move-to-trash). We therefore invoke the command directly | ||
| # via id and rely on tree selection to supply the target node — the same | ||
| # pattern the classpath plan uses for `removeLibrary`. | ||
| # | ||
| # The confirmation dialog also differs from moveFileToTrash: the prompt | ||
| # says "permanently delete" instead of "delete", and the action button is | ||
| # labelled "Delete" instead of "Move to Recycle Bin" (see | ||
| # src/explorerCommands/delete.ts line 17 + getInformationMessage). | ||
| # | ||
| # Usage: | ||
| # npx autotest run test/e2e-plans/java-dep-delete-permanent.yaml --vsix <path-to-vsix> | ||
|
|
||
| name: "Java Dependency — Permanent Delete" | ||
| description: | | ||
| Tests the java.view.package.deleteFilePermanently command on a regular | ||
| Maven-project file. Invokes the command by id (no UI affordance on local | ||
| files), confirms the "permanently delete" dialog, and verifies the file | ||
| is gone from disk and from the Java Projects tree. | ||
|
|
||
| setup: | ||
| extension: "redhat.java" | ||
| vscodeVersion: "stable" | ||
| workspace: "../maven" | ||
| timeout: 180 | ||
| settings: | ||
| java.configuration.checkProjectSettingsExclusions: false | ||
| workbench.startupEditor: "none" | ||
|
|
||
| steps: | ||
| # ── Setup: wait for LS, free sidebar space, focus Java Projects ── | ||
| - id: "ls-ready" | ||
| action: "waitForLanguageServer" | ||
| # No `verify:` — `waitForLanguageServer` is itself the deterministic | ||
| # readiness check. The AFTER screenshot may transiently show | ||
| # "Java: Building - 0%" which a strict LLM mis-reads as a failure. | ||
| timeout: 180 | ||
|
|
||
| - id: "close-aux-bar" | ||
| action: "executeVSCodeCommand workbench.action.closeAuxiliaryBar" | ||
| verify: "Auxiliary bar (Chat) closed" | ||
|
|
||
| - id: "collapse-outline" | ||
| action: "collapseSidebarSection OUTLINE" | ||
|
|
||
| - id: "collapse-timeline" | ||
| action: "collapseSidebarSection TIMELINE" | ||
|
|
||
| - id: "collapse-workspace-root" | ||
| action: "collapseWorkspaceRoot" | ||
|
|
||
| - id: "focus-java-projects" | ||
| action: "executeVSCodeCommand javaProjectExplorer.focus" | ||
| verify: "Java Projects view is focused" | ||
|
|
||
| - id: "wait-tree-load" | ||
| action: "wait 3 seconds" | ||
|
|
||
| # ── Reveal App1.java via link-with-editor, then select it ── | ||
| # Opening the file makes link-with-editor expand the tree path and | ||
| # reveal+select App1 deterministically — much more reliable than | ||
| # manual expandTreeItem chains on the virtualised tree. | ||
| - id: "open-target-file" | ||
| action: "open file App1.java" | ||
| waitBefore: 2 | ||
|
|
||
| - id: "collapse-workspace-root-2" | ||
| action: "collapseWorkspaceRoot" | ||
|
|
||
| - id: "focus-java-projects-2" | ||
| action: "executeVSCodeCommand javaProjectExplorer.focus" | ||
| waitBefore: 2 | ||
|
|
||
| # Click the tree item to guarantee it's the active selection — getCmdNode | ||
| # (explorerCommands/utility.ts:38) falls back to `selectedNodes[0]` when | ||
| # the command is invoked without a node argument. Linking-with-editor | ||
| # already auto-selected App1 when the file was opened, but clicking | ||
| # re-asserts the selection deterministically after the focus-java-projects | ||
| # round trip. | ||
| # | ||
| # We intentionally do NOT verify the App1 tree row here. On 1024x768 CI | ||
| # displays the row gets virtualised out of view in the brief window | ||
| # between `open file` and the click, and an inView:"Java Projects" | ||
| # verifyTreeItem then times out — even though the underlying selection | ||
| # is set correctly (proven by the subsequent delete succeeding). The | ||
| # post-delete `verify-file-gone` + `verify-tree-item-gone` block is the | ||
| # authoritative ground truth for whether the right node was targeted. | ||
| - id: "select-app1" | ||
| action: "click App1 tree item" | ||
| waitBefore: 1 | ||
|
|
||
| # ── Invoke java.view.package.deleteFilePermanently ── | ||
| # The handler at views/dependencyExplorer.ts:177 takes `node?: DataNode`. | ||
| # With no node arg, getCmdNode uses the current selection (set by the | ||
| # `click App1 tree item` step above). DataNode is a class instance with | ||
| # methods (.getChildren()) so passing a POJO would crash — selection | ||
| # fallback is the only viable path from a smoke-test. | ||
| - id: "invoke-delete-permanently" | ||
| action: "executeVSCodeCommand java.view.package.deleteFilePermanently" | ||
|
|
||
| # `expectConfirmDialog` waits for the dialog and clicks the first | ||
| # recognized confirm button (autotest knows "Delete" is one of them, see | ||
| # dialogOperations.ts:16). It's the strict variant — throws if no dialog | ||
| # appears, surfacing a silently-failed command invocation immediately | ||
| # instead of 15s later when verifyFile times out. | ||
| - id: "confirm-delete" | ||
| action: "expectConfirmDialog" | ||
|
|
||
| - id: "wait-after-delete" | ||
| action: "wait 5 seconds" | ||
|
|
||
| # ── Verify deletion on disk AND in the tree ── | ||
| # The disk check is the strongest signal: useTrash=false routes through | ||
| # workspace.fs.delete with the OS-level unlink, so a passing verifyFile | ||
| # exists:false proves the permanent-delete path actually fired (rather | ||
| # than silently downgrading to a no-op or moving to trash). | ||
| - id: "verify-file-gone" | ||
| action: "wait 1 seconds" | ||
| verifyFile: | ||
| path: "${workspaceFolder}/src/main/java/com/mycompany/app1/App1.java" | ||
| exists: false | ||
| timeout: 15 | ||
|
|
||
| - id: "verify-tree-item-gone" | ||
| action: "wait 1 seconds" | ||
| # No `verify:` — verifyTreeItem is authoritative. | ||
| verifyTreeItem: | ||
| name: "App1" | ||
| exact: true | ||
| visible: false | ||
| inView: "Java Projects" | ||
| timeout: 15 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,162 @@ | ||
| # Test Plan: Java Dependency — Export Jar | ||
| # | ||
| # Covers java.view.package.exportJar — the multi-step wizard that builds a | ||
| # runnable jar from a Java project. The command is contributed both as the | ||
| # title-bar `$(export)` icon on the JAVA PROJECTS workspace-root node | ||
| # (package.json:858-861, group=inline) and as a context-menu entry, but we | ||
| # invoke it directly by id: the inline icon is rendered against the | ||
| # `java:workspace` viewItem (not the Maven project node `my-app`), which | ||
| # is fragile to locate by name, and ResolveJavaProject auto-resolves when | ||
| # there's exactly one project in the workspace — so direct invocation | ||
| # bypasses one wizard step that's not the command's responsibility. | ||
| # | ||
| # Wizard step machine (BuildArtifactTaskProvider.ts:284 `createJarFile`): | ||
| # 1. ResolveJavaProject → auto when single-project; quick-pick otherwise | ||
| # 2. ResolveMainClass → quick-pick of main classes + "<without main class>" | ||
| # 3. GenerateJar | ||
| # a. generateClasspaths → multi-select quick-pick if >1 dependency item | ||
| # b. showSaveDialog → only if outputPath === "" (skipped when | ||
| # java.project.exportJar.targetPath is set | ||
| # to a non-empty value; we set it via | ||
| # workspaceSettings to keep the output | ||
| # filename deterministic) | ||
| # c. Jdtls.exportJar → writes the jar file | ||
| # | ||
| # Verification strategy | ||
| # ───────────────────── | ||
| # The jar is written to a known absolute path inside the workspace | ||
| # (`output.jar`). We assert with `verifyFile exists: true` — the strongest | ||
| # possible signal that the full wizard completed end-to-end, not just that | ||
| # the command was dispatched. The Jdtls export runs in a hidden Pseudoterminal | ||
| # (BuildArtifactTaskProvider line 91: `presentationOptions.reveal = Never`), | ||
| # so there is no terminal text to inspect; the file on disk is the only | ||
| # unambiguous post-condition. | ||
| # | ||
| # Usage: | ||
| # npx autotest run test/e2e-plans/java-dep-export-jar.yaml --vsix <path-to-vsix> | ||
|
|
||
| name: "Java Dependency — Export Jar" | ||
| description: | | ||
| Exercises the multi-step Export Jar wizard end-to-end on the maven | ||
| fixture: triggers the command, picks the main class, accepts the default | ||
| classpath element selection, and verifies that the resulting jar file | ||
| exists at the configured target path. | ||
|
|
||
| setup: | ||
| extension: "redhat.java" | ||
| vscodeVersion: "stable" | ||
| workspace: "../maven" | ||
| # Bumped above the standard 180s — the wizard runs a full workspace build | ||
| # (await buildWorkspace() in executeExportJarTask) before the first | ||
| # quick-pick appears, which on a cold JDT-LS warmup commonly takes 60-90s. | ||
| timeout: 360 | ||
| settings: | ||
| java.configuration.checkProjectSettingsExclusions: false | ||
| workbench.startupEditor: "none" | ||
| # Pinning java.project.exportJar.targetPath to an absolute deterministic | ||
| # path bypasses the showSaveDialog branch in GenerateJarExecutor | ||
| # (BuildArtifactTaskProvider lines 240-244 short-circuit only when | ||
| # outputPath === ""), and lets the verifyFile assertion target a stable | ||
| # location regardless of how the fixture's worktree is named. | ||
| # | ||
| # We use user-level `settings:` (not `workspaceSettings:`) because | ||
| # autotest's workspaceSettings merge uses JSON.parse on the existing | ||
| # `.vscode/settings.json`, and the maven fixture's settings.json | ||
| # contains JSONC `//` comments which fail JSON.parse. User settings | ||
| # are written fresh each run (user-data-dir is wiped on launch) so | ||
| # there is no JSONC merge hazard. The Settings.getExportJarTargetPath | ||
| # config read is unscoped, so user-level setting takes effect identically. | ||
| java.project.exportJar.targetPath: "${workspaceFolder}/output.jar" | ||
|
|
||
| steps: | ||
| # ── Setup: wait for LS, free sidebar space, focus Java Projects ── | ||
| - id: "ls-ready" | ||
| action: "waitForLanguageServer" | ||
| # No `verify:` — `waitForLanguageServer` is itself the deterministic | ||
| # readiness check. The AFTER screenshot may transiently show | ||
| # "Java: Building - 0%" which a strict LLM mis-reads as a failure. | ||
| timeout: 180 | ||
|
|
||
| - id: "close-aux-bar" | ||
| action: "executeVSCodeCommand workbench.action.closeAuxiliaryBar" | ||
| verify: "Auxiliary bar (Chat) closed" | ||
|
|
||
| - id: "collapse-outline" | ||
| action: "collapseSidebarSection OUTLINE" | ||
|
|
||
| - id: "collapse-timeline" | ||
| action: "collapseSidebarSection TIMELINE" | ||
|
|
||
| - id: "collapse-workspace-root" | ||
| action: "collapseWorkspaceRoot" | ||
|
|
||
| - id: "focus-java-projects" | ||
| action: "executeVSCodeCommand javaProjectExplorer.focus" | ||
| verify: "Java Projects view is focused" | ||
|
|
||
| - id: "wait-tree-load" | ||
| action: "wait 3 seconds" | ||
|
|
||
| # ── Trigger the Export Jar wizard ── | ||
| # Direct command invocation avoids the brittle inline-icon-on-workspace- | ||
| # node click path (the `$(export)` icon is contributed against viewItem | ||
| # `java:workspace`, not against the Maven project node `my-app`, and | ||
| # locating workspace-level inline icons in autotest is fragile). | ||
| # ResolveJavaProject auto-resolves for single-project workspaces. | ||
| - id: "invoke-export-jar" | ||
| action: "executeVSCodeCommand java.view.package.exportJar" | ||
|
|
||
| # The wizard first triggers a full workspace build (await buildWorkspace() | ||
| # in executeExportJarTask) before any UI appears. On a cold JDT-LS this | ||
| # typically takes 30-60s; the Jdtls.getMainClasses fetch then needs | ||
| # another 2-5s before the first quick-pick is shown. Wait generously. | ||
| - id: "wait-build-complete" | ||
| action: "wait 60 seconds" | ||
|
|
||
| # ── Step 2: pick the main class ── | ||
| # The maven fixture has a single class with a `public static void main` | ||
| # entry point: com.mycompany.app.App. App1 has no main, so the only | ||
| # quick-pick options are "App" and "<without main class>". We pick "App" | ||
| # explicitly so the resulting jar is runnable. | ||
| - id: "pick-main-class" | ||
| action: "select App option" | ||
| # No `verify:` — selectPaletteOption is the action and verification in | ||
| # one. The quick-pick closes on selection; the AFTER screenshot may | ||
| # already show the next quick-pick (classpath elements), which a | ||
| # strict LLM could mis-read as "App selection was lost". | ||
|
|
||
| - id: "wait-after-main-class" | ||
| action: "wait 5 seconds" | ||
|
|
||
| # ── Step 3a: accept the pre-selected classpath elements ── | ||
| # GenerateJarExecutor.generateClasspaths builds a list of runtime / test | ||
| # output folders + dependency artifacts, pre-selects everything tagged | ||
| # `runtime`, and shows a multi-select QuickPick. For the maven fixture | ||
| # this includes target/classes plus any resolved JUnit jars under test | ||
| # scope. The pre-selection is the sane default — accepting it gives a | ||
| # runnable jar without needing to compute which items to check. | ||
| # | ||
| # `confirmQuickInput` presses Enter on the open quick-pick widget without | ||
| # typing anything, which leaves selections untouched and submits. | ||
| - id: "accept-classpath-elements" | ||
| action: "confirmQuickInput" | ||
|
|
||
| # ── Step 3c: wait for Jdtls to generate the jar ── | ||
| # Jdtls.exportJar writes the jar via a JDT LSP request. The custom | ||
| # pseudoterminal stays hidden (presentationOptions.reveal=Never), so | ||
| # there is no terminal-text signal — we just wait long enough for the | ||
| # filesystem write to settle. | ||
| - id: "wait-jar-generated" | ||
| action: "wait 30 seconds" | ||
|
|
||
| # ── Verification: jar file exists at the configured target path ── | ||
| # This is the strongest possible end-to-end check: the file appearing | ||
| # on disk proves ResolveJavaProject + ResolveMainClass + generateClasspaths | ||
| # + Jdtls.exportJar all completed in order. A failure here pinpoints | ||
| # the wizard breaking; a pass means the full happy path worked. | ||
| - id: "verify-jar-created" | ||
| action: "wait 1 seconds" | ||
| verifyFile: | ||
| path: "${workspaceFolder}/output.jar" | ||
| exists: true | ||
| timeout: 60 | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.