Skip to content

Commit e633cd0

Browse files
wenytang-msCopilot
andcommitted
test(e2e): drop flaky LLM verify on expandTreeItem steps
The LLM screenshot check on expand-existing-pkg / expand-source-root / expand-project is inherently flaky on already-expanded trees — the recent CI run downgraded a mechanically-successful expand from pass to fail, then immediately contradicted itself in the post-mortem analysis ('was already expanded before the action and remains expanded after'). The deterministic verifyTreeItem steps that follow each expand provide authoritative ground truth: if the expand didn't happen, the child item won't be visible. So the LLM verify lines are pure redundancy with no upside and a non-trivial false-negative rate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 64dbb6d commit e633cd0

1 file changed

Lines changed: 7 additions & 4 deletions

File tree

test/e2e-plans/java-dep-refresh-generated-files.yaml

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -55,13 +55,15 @@ steps:
5555
- id: "wait-tree-load"
5656
action: "wait 3 seconds"
5757

58+
# NOTE: no `verify:` on expand steps — the LLM screenshot check is flaky
59+
# ("LLM downgraded pass -> fail" on already-expanded trees). The next
60+
# verifyTreeItem (or the chained expand below) provides authoritative
61+
# ground truth: if the expand didn't happen the child won't be visible.
5862
- id: "expand-project"
5963
action: "expandTreeItem my-app"
60-
verify: "my-app project expanded"
6164

6265
- id: "expand-source-root"
6366
action: "expandTreeItem src/main/java"
64-
verify: "source root src/main/java expanded"
6567

6668
- id: "baseline-existing-pkg"
6769
action: "wait 1 seconds"
@@ -111,10 +113,11 @@ steps:
111113
visible: true
112114
timeout: 15
113115

114-
# Sanity: a new class in an existing package also appears.
116+
# Sanity: a new class in an existing package also appears. No `verify:` here
117+
# for the same reason as the expand steps above — check-existing-pkg-class
118+
# below is the deterministic assertion.
115119
- id: "expand-existing-pkg"
116120
action: "expandTreeItem com.mycompany.app"
117-
verify: "existing package com.mycompany.app expanded"
118121

119122
- id: "check-existing-pkg-class"
120123
action: "wait 1 seconds"

0 commit comments

Comments
 (0)