test: convert silent t.Skip to t.Fatal in baseline-scan checks#1578
Merged
omercnet merged 5 commits intoMay 21, 2026
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Updates end-to-end tests to fail loudly when “baseline” vulnerability scans unexpectedly return zero findings, preventing silent skips from masking scanner/test-data regressions.
Changes:
- Replace several
t.Skip(...)baseline “no vulns” branches witht.Fatal/Fatalf(...)in Go and frontend e2e tests. - Improve readability of an intentional integration-test skip by adding a
Skipfreason that includes the local image ref andCOPA_BUILDKIT_ADDR.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| test/e2e/golang/golang_test.go | Convert baseline “no vulnerabilities” skip conditions into hard failures for Go e2e cases. |
| test/e2e/frontend/frontend_test.go | Fail the multiplatform frontend test when no platform reports can be generated. |
| integration/singlearch/patch_test.go | Add a reason string to a legitimate skip for local-only images on non-docker:// BuildKit. |
This was referenced Apr 30, 2026
fde06a9 to
07dc53a
Compare
This was referenced May 6, 2026
Three sites in test/e2e/golang and one in test/e2e/frontend were silently skipping tests when their baseline assertion failed. This masks real regressions: - test/e2e/golang/golang_test.go (TestGoBaselineImages, TestMultiBinaryImages, TestDistrolessImages): each test selects a baseline image specifically because it carries known CVEs. If the baseline scan returns zero vulnerabilities, the test should FAIL loudly — it indicates a scanner regression, a stale Trivy DB, an upstream image rebuild that resolved CVEs, or a network failure during DB download. Silently skipping made the test green when in fact the test was unable to assert anything. - test/e2e/frontend/frontend_test.go (TestMultiplatformFrontend): when *every* platform's report generation fails the test was skipping with a hand-wavy 'possible disk space or network issue' message. Per-platform failures are already surfaced via t.Logf within the loop; if the cumulative result is zero successful reports the failure is real, not flake. Each Fatalf message includes enough context (image ref, mode) for the on-call to triage without re-reading the test. Also adds a reason string to the legitimate skip in integration/singlearch/patch_test.go where local-name images are not testable on non-docker buildkit drivers — this skip is correct, but previously had no message which made gotestsum output unreadable. No production code changes. Signed-off-by: Omer <omer@descope.com>
07dc53a to
d41de20
Compare
ashnamehrotra
approved these changes
May 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Converts 4 silent
t.Skipcalls intot.Fatalin e2e tests where the skip was masking real regressions. Adds a reason string to one legitimate skip inintegration/singlearchfor log readability.No production code changed.
Why
Three sites in
test/e2e/golang/golang_test.go(TestGoBinaryPatchingByCategory,TestMultiBinaryImages,TestDistrolessImages) and one intest/e2e/frontend/frontend_test.go(runFrontendMultiplatformTest) were silently skipping when their baseline assertion failed. Each test selects a baseline image specifically because it carries known CVEs — if the baseline scan returns zero vulnerabilities, the test should fail loudly. A silent skip masks:In the frontend case, the skip fired only when every platform's report generation failed cumulatively — at that point the failure is real, not flake.
Risk
This change could turn currently-green CI red if a baseline image was rebuilt upstream and now legitimately has zero CVEs. If that happens, the right response is to update the baseline test data, not to re-silence the failure. I'd actually consider that a feature of this PR — surfacing exactly the kind of test-data drift that silent skips have been hiding.
Detail
Each
Fatalfcarries enough context (image ref, mode) for an on-call to triage without re-reading the test:The
integration/singlearch/patch_test.goskip (line 80) for local-name images on non-docker buildkit drivers is a legitimate skip — local images can only be tested withdocker://. Previously it hadt.Skip()with no message, makinggotestsumoutput unreadable. Adds an informativeSkipfwith the image ref + currentCOPA_BUILDKIT_ADDR.Test plan
go vetclean on touched files. The next CI run on this branch will surface any baseline currently in the silent-failure state.