Skip to content

test: convert silent t.Skip to t.Fatal in baseline-scan checks#1578

Merged
omercnet merged 5 commits into
project-copacetic:mainfrom
omercnet:coverage/pr6-skip-to-fatal
May 21, 2026
Merged

test: convert silent t.Skip to t.Fatal in baseline-scan checks#1578
omercnet merged 5 commits into
project-copacetic:mainfrom
omercnet:coverage/pr6-skip-to-fatal

Conversation

@omercnet
Copy link
Copy Markdown
Contributor

@omercnet omercnet commented Apr 30, 2026

Part of the coverage-improvement series tracked in #1580.

Summary

Converts 4 silent t.Skip calls into t.Fatal in e2e tests where the skip was masking real regressions. Adds a reason string to one legitimate skip in integration/singlearch for log readability.

No production code changed.

Why

Three sites in test/e2e/golang/golang_test.go (TestGoBinaryPatchingByCategory, TestMultiBinaryImages, TestDistrolessImages) and one in test/e2e/frontend/frontend_test.go (runFrontendMultiplatformTest) were silently skipping when their baseline assertion failed. Each test selects a baseline image specifically because it carries known CVEs — if the baseline scan returns zero vulnerabilities, the test should fail loudly. A silent skip masks:

  • A scanner regression
  • A stale Trivy DB
  • An upstream image rebuild that resolved the CVEs (test data needs updating)
  • A network failure during DB download

In the frontend case, the skip fired only when every platform's report generation failed cumulatively — at that point the failure is real, not flake.

Risk

This change could turn currently-green CI red if a baseline image was rebuilt upstream and now legitimately has zero CVEs. If that happens, the right response is to update the baseline test data, not to re-silence the failure. I'd actually consider that a feature of this PR — surfacing exactly the kind of test-data drift that silent skips have been hiding.

Detail

Each Fatalf carries enough context (image ref, mode) for an on-call to triage without re-reading the test:

t.Fatalf("no vulnerabilities found in baseline scan of %s; this baseline is selected because it has known CVEs — a zero result indicates a scanner regression, stale Trivy DB, or upstream image rebuild", ref)

The integration/singlearch/patch_test.go skip (line 80) for local-name images on non-docker buildkit drivers is a legitimate skip — local images can only be tested with docker://. Previously it had t.Skip() with no message, making gotestsum output unreadable. Adds an informative Skipf with the image ref + current COPA_BUILDKIT_ADDR.

Test plan

go vet clean on touched files. The next CI run on this branch will surface any baseline currently in the silent-failure state.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates end-to-end tests to fail loudly when “baseline” vulnerability scans unexpectedly return zero findings, preventing silent skips from masking scanner/test-data regressions.

Changes:

  • Replace several t.Skip(...) baseline “no vulns” branches with t.Fatal/Fatalf(...) in Go and frontend e2e tests.
  • Improve readability of an intentional integration-test skip by adding a Skipf reason that includes the local image ref and COPA_BUILDKIT_ADDR.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
test/e2e/golang/golang_test.go Convert baseline “no vulnerabilities” skip conditions into hard failures for Go e2e cases.
test/e2e/frontend/frontend_test.go Fail the multiplatform frontend test when no platform reports can be generated.
integration/singlearch/patch_test.go Add a reason string to a legitimate skip for local-only images on non-docker:// BuildKit.

Comment thread test/e2e/golang/golang_test.go
Comment thread integration/singlearch/patch_test.go Outdated
Three sites in test/e2e/golang and one in test/e2e/frontend were silently
skipping tests when their baseline assertion failed. This masks real
regressions:

- test/e2e/golang/golang_test.go (TestGoBaselineImages,
  TestMultiBinaryImages, TestDistrolessImages): each test selects a
  baseline image specifically because it carries known CVEs. If the
  baseline scan returns zero vulnerabilities, the test should FAIL
  loudly — it indicates a scanner regression, a stale Trivy DB, an
  upstream image rebuild that resolved CVEs, or a network failure
  during DB download. Silently skipping made the test green when in
  fact the test was unable to assert anything.

- test/e2e/frontend/frontend_test.go (TestMultiplatformFrontend): when
  *every* platform's report generation fails the test was skipping
  with a hand-wavy 'possible disk space or network issue' message.
  Per-platform failures are already surfaced via t.Logf within the
  loop; if the cumulative result is zero successful reports the
  failure is real, not flake.

Each Fatalf message includes enough context (image ref, mode) for the
on-call to triage without re-reading the test.

Also adds a reason string to the legitimate skip in
integration/singlearch/patch_test.go where local-name images are not
testable on non-docker buildkit drivers — this skip is correct, but
previously had no message which made gotestsum output unreadable.

No production code changes.

Signed-off-by: Omer <omer@descope.com>
@omercnet omercnet force-pushed the coverage/pr6-skip-to-fatal branch from 07dc53a to d41de20 Compare May 8, 2026 17:52
@omercnet omercnet merged commit 3f8d217 into project-copacetic:main May 21, 2026
12 checks passed
@github-project-automation github-project-automation Bot moved this from 🆕 New to ✅ Done in Copacetic Workboard May 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

3 participants