You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
test(autotest): drop verify: on ls-ready and 2 newly-flaky steps
CI run 25663760786 surfaced 5 NEW LLM-downgrade flakes (different plans
than rounds 1-3):
- java-debugger: verify-breakpoint — LLM missed the yellow execution-line
marker on the screenshot (off-viewport when debug toolbar pushes editor
down). Deterministic ground truth is the next debugStepOver action,
which can only succeed when the debugger is paused.
- java-extension-pack: configure-classpath — Project Settings webview
lazy-loads, command step screenshot caught empty frame. Moved the LLM
check onto the next wait step (5s) which captures the rendered UI.
- java-maven, java-maven-java25, java-single-file: ls-ready —
waitForLanguageServer returns when status reaches 'Java: Ready' but
the LS often re-enters Building/Searching for incremental compilation
right after Maven import, so the AFTER snapshot can catch that
intermediate state.
Fix: drop verify: text on ls-ready across all plans (preventive — 11
other plans were carrying the same brittle text) and on the two
specific flaky steps. The deterministic verifiers
(verifyProblems.errors:0, debugStepOver success, subsequent verify-page
wait) remain as ground truth.
Local: all 5 failing plans now pass with --no-llm.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
0 commit comments