ci: add Trivy vulnerability scan for broadcom and mellanox platform containers and host rootfs#27322
Closed
xq9mend wants to merge 6 commits into
Closed
ci: add Trivy vulnerability scan for broadcom and mellanox platform containers and host rootfs#27322xq9mend wants to merge 6 commits into
xq9mend wants to merge 6 commits into
Conversation
Add comprehensive Trivy vulnerability scanning in a new TrivyScan stage that runs after the Build stage, one job per platform (broadcom, mellanox). Each platform job: 1. Downloads sonic-buildimage.<platform> artifact 2. Scans all docker-*.gz container images (platform-specific containers like docker-syncd-brcm, docker-syncd-mlnx plus all shared images) 3. Extracts host rootfs from sonic-<platform>.bin and runs trivy rootfs New file: .azure-pipelines/trivy-scan-platform.yml (parameterized template) Phase 1: exit-code 1 (fails on HIGH/CRITICAL findings), continueOnError: true (does not block PR merge). Signed-off-by: xq9mend <xq9mend@users.noreply.github.com>
Collaborator
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines successfully started running 1 pipeline(s). |
…atform - Replace `unsquashfs --version` with `unsquashfs -v` (--version is not a valid flag; it prints SYNTAX/help and exits 1) - Add itemPattern to DownloadPipelineArtifact to only fetch docker-*.gz and sonic-<platform>.bin instead of the full 27.9 GB artifact (no itemPattern caused OOM-kill on the broadcom scan agent)
Collaborator
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Collaborator
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Collaborator
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Replace the parallel docker-*.gz scan with a Python script that extracts dockerfs.tar.gz directly from the .bin installer and reconstructs docker-save format tarballs for trivy image scanning. - Add .azure-pipelines/scripts/trivy-scan-bin.py - Update trivy-scan-platform.yml to download only the .bin and invoke the script (removes parallelism parameter) - Fix docker-sonic-mgmt.yml artifact name conflict Tested locally on sonic-mellanox.bin (27 images) and sonic-broadcom.bin (32 images), both scan successfully.
Collaborator
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines successfully started running 1 pipeline(s). |
- trivy-scan-bin.py: print full trivy table per container to stdout - trivy-scan-platform.yml: write rootfs result to results/ subdir, publish results/ only (avoids dockerdata/ permission error)
Collaborator
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines successfully started running 1 pipeline(s). |
7 tasks
Collaborator
|
I have a full SBOM and Vuln scan PR here (waiting on final results of first run) which does a deeper analysis than this PR and covers all build assets: #27455 While this PR is much more simplistic, there's a wide coverage gap in the found vulnerabilities. |
trivy returns exit code 1 when vulnerabilities are found (--exit-code 1). The previous logic only set overall_exit=1 for unexpected errors (rc not in 0,1), silently swallowing trivy's own vuln-found exit code and causing the step to always pass. Fix: move overall_exit update outside the error-only branch so any non-zero trivy exit propagates to sys.exit(overall_exit).
Collaborator
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines successfully started running 1 pipeline(s). |
xq9mend
marked this pull request as ready for review
May 22, 2026 23:03
qiluo-msft
pushed a commit
that referenced
this pull request
May 28, 2026
#27455) Why I did it SONiC currently ships no Software Bill of Materials. There is no inventory of what is in each .bin, no record of the patches and forks SONiC carries on top of upstream sources, no per-artifact vulnerability surface, and no machine-readable provenance. This makes CVE response, license attribution, supply-chain audits (SLSA, SBOM regulations like EO 14028 / CRA), and reproducibility checks expensive or impossible. This PR adds opt-in SBOM generation and SBOM-based vulnerability scanning. When ENABLE_SBOM=y is passed at build time, every artifact (.bin, .img, .swi, plus the standalone test-container .gzs) gets a CycloneDX 1.6 SBOM sidecar covering Debian packages, Python wheels, language-ecosystem lockfile contents (Rust, Go, npm, pnpm, yarn), Docker image layers, vendor blobs, and SONiC-built sources. The SBOMs preserve fork-of-upstream provenance (e.g. the FRR submodule pinned to a specific upstream tag plus SONiC's patch set) via pedigree.ancestors[], so a CVE in upstream FRR can be tied back to the SONiC build that carries the corresponding patch. A separate set of standalone scripts consume the SBOMs to produce vulnerability reports (grype + OpenVEX suppressions) and reproducibility diffs. The default build path is unchanged. With ENABLE_SBOM unset (the default), all SBOM-related work is short-circuited in slave.mk via a make-level $(if) guard, so non-SBOM builds incur zero overhead — no extra subprocesses, no Python startup cost, no apt-cache calls. The 5 added build dependencies (syft, grype, cyclonedx-cli, plus their installer) are only fetched when SBOM is enabled, through the existing sonic-build-hooks wget shim so versions-web tracking still applies. The Azure Pipelines side replaces the recently-merged Trivy scans (PR #27079) with the new SBOM-based scanner so CI vulnerability reports come from the same data that ships in the artifact, not a separate re-scan of the image tarball. Comparison with the trivy-based PRs this would supersede The Azure Pipelines half of this PR replaces / overlaps with two recent trivy PRs: PR ci: add Trivy vulnerability scan for docker-ptf and docker-sonic-mgmt #27079 — informational trivy scan of docker-ptf and docker-sonic-mgmt. Already merged. This PR removes those two trivy_scan jobs. PR ci: add Trivy vulnerability scan for broadcom and mellanox platform containers and host rootfs #27322 — informational trivy scan of all docker-*.gz containers and the host rootfs for broadcom and mellanox platforms. Not yet merged. This PR addresses the same goal more thoroughly. Trivy is a perfectly reasonable image scanner. But trivy alone solves only the vulnerability scanning slice of the supply-chain problem, and it does so by re-discovering what's in the artifact at scan time rather than capturing what the build actually put there. Crucially, the SBOM-based approach finds more CVEs, not the same set — because the input components carry richer provenance, the scanner can match them against CVE databases that trivy can't reach from a plain rootfs scan.
Contributor
Author
|
Closing in favor of #27455 |
securely1g
pushed a commit
to securely1g/sonic-buildimage
that referenced
this pull request
Jun 4, 2026
sonic-net#27455) Why I did it SONiC currently ships no Software Bill of Materials. There is no inventory of what is in each .bin, no record of the patches and forks SONiC carries on top of upstream sources, no per-artifact vulnerability surface, and no machine-readable provenance. This makes CVE response, license attribution, supply-chain audits (SLSA, SBOM regulations like EO 14028 / CRA), and reproducibility checks expensive or impossible. This PR adds opt-in SBOM generation and SBOM-based vulnerability scanning. When ENABLE_SBOM=y is passed at build time, every artifact (.bin, .img, .swi, plus the standalone test-container .gzs) gets a CycloneDX 1.6 SBOM sidecar covering Debian packages, Python wheels, language-ecosystem lockfile contents (Rust, Go, npm, pnpm, yarn), Docker image layers, vendor blobs, and SONiC-built sources. The SBOMs preserve fork-of-upstream provenance (e.g. the FRR submodule pinned to a specific upstream tag plus SONiC's patch set) via pedigree.ancestors[], so a CVE in upstream FRR can be tied back to the SONiC build that carries the corresponding patch. A separate set of standalone scripts consume the SBOMs to produce vulnerability reports (grype + OpenVEX suppressions) and reproducibility diffs. The default build path is unchanged. With ENABLE_SBOM unset (the default), all SBOM-related work is short-circuited in slave.mk via a make-level $(if) guard, so non-SBOM builds incur zero overhead — no extra subprocesses, no Python startup cost, no apt-cache calls. The 5 added build dependencies (syft, grype, cyclonedx-cli, plus their installer) are only fetched when SBOM is enabled, through the existing sonic-build-hooks wget shim so versions-web tracking still applies. The Azure Pipelines side replaces the recently-merged Trivy scans (PR sonic-net#27079) with the new SBOM-based scanner so CI vulnerability reports come from the same data that ships in the artifact, not a separate re-scan of the image tarball. Comparison with the trivy-based PRs this would supersede The Azure Pipelines half of this PR replaces / overlaps with two recent trivy PRs: PR ci: add Trivy vulnerability scan for docker-ptf and docker-sonic-mgmt sonic-net#27079 — informational trivy scan of docker-ptf and docker-sonic-mgmt. Already merged. This PR removes those two trivy_scan jobs. PR ci: add Trivy vulnerability scan for broadcom and mellanox platform containers and host rootfs sonic-net#27322 — informational trivy scan of all docker-*.gz containers and the host rootfs for broadcom and mellanox platforms. Not yet merged. This PR addresses the same goal more thoroughly. Trivy is a perfectly reasonable image scanner. But trivy alone solves only the vulnerability scanning slice of the supply-chain problem, and it does so by re-discovering what's in the artifact at scan time rather than capturing what the build actually put there. Crucially, the SBOM-based approach finds more CVEs, not the same set — because the input components carry richer provenance, the scanner can match them against CVE databases that trivy can't reach from a plain rootfs scan.
purush-nexthop
pushed a commit
to nexthop-ai/sonic-buildimage
that referenced
this pull request
Jun 10, 2026
sonic-net#27455) Why I did it SONiC currently ships no Software Bill of Materials. There is no inventory of what is in each .bin, no record of the patches and forks SONiC carries on top of upstream sources, no per-artifact vulnerability surface, and no machine-readable provenance. This makes CVE response, license attribution, supply-chain audits (SLSA, SBOM regulations like EO 14028 / CRA), and reproducibility checks expensive or impossible. This PR adds opt-in SBOM generation and SBOM-based vulnerability scanning. When ENABLE_SBOM=y is passed at build time, every artifact (.bin, .img, .swi, plus the standalone test-container .gzs) gets a CycloneDX 1.6 SBOM sidecar covering Debian packages, Python wheels, language-ecosystem lockfile contents (Rust, Go, npm, pnpm, yarn), Docker image layers, vendor blobs, and SONiC-built sources. The SBOMs preserve fork-of-upstream provenance (e.g. the FRR submodule pinned to a specific upstream tag plus SONiC's patch set) via pedigree.ancestors[], so a CVE in upstream FRR can be tied back to the SONiC build that carries the corresponding patch. A separate set of standalone scripts consume the SBOMs to produce vulnerability reports (grype + OpenVEX suppressions) and reproducibility diffs. The default build path is unchanged. With ENABLE_SBOM unset (the default), all SBOM-related work is short-circuited in slave.mk via a make-level $(if) guard, so non-SBOM builds incur zero overhead — no extra subprocesses, no Python startup cost, no apt-cache calls. The 5 added build dependencies (syft, grype, cyclonedx-cli, plus their installer) are only fetched when SBOM is enabled, through the existing sonic-build-hooks wget shim so versions-web tracking still applies. The Azure Pipelines side replaces the recently-merged Trivy scans (PR sonic-net#27079) with the new SBOM-based scanner so CI vulnerability reports come from the same data that ships in the artifact, not a separate re-scan of the image tarball. Comparison with the trivy-based PRs this would supersede The Azure Pipelines half of this PR replaces / overlaps with two recent trivy PRs: PR ci: add Trivy vulnerability scan for docker-ptf and docker-sonic-mgmt sonic-net#27079 — informational trivy scan of docker-ptf and docker-sonic-mgmt. Already merged. This PR removes those two trivy_scan jobs. PR ci: add Trivy vulnerability scan for broadcom and mellanox platform containers and host rootfs sonic-net#27322 — informational trivy scan of all docker-*.gz containers and the host rootfs for broadcom and mellanox platforms. Not yet merged. This PR addresses the same goal more thoroughly. Trivy is a perfectly reasonable image scanner. But trivy alone solves only the vulnerability scanning slice of the supply-chain problem, and it does so by re-discovering what's in the artifact at scan time rather than capturing what the build actually put there. Crucially, the SBOM-based approach finds more CVEs, not the same set — because the input components carry richer provenance, the scanner can match them against CVE databases that trivy can't reach from a plain rootfs scan.
roger530-ho
pushed a commit
to roger530-ho/sonic-buildimage
that referenced
this pull request
Jun 23, 2026
sonic-net#27455) Why I did it SONiC currently ships no Software Bill of Materials. There is no inventory of what is in each .bin, no record of the patches and forks SONiC carries on top of upstream sources, no per-artifact vulnerability surface, and no machine-readable provenance. This makes CVE response, license attribution, supply-chain audits (SLSA, SBOM regulations like EO 14028 / CRA), and reproducibility checks expensive or impossible. This PR adds opt-in SBOM generation and SBOM-based vulnerability scanning. When ENABLE_SBOM=y is passed at build time, every artifact (.bin, .img, .swi, plus the standalone test-container .gzs) gets a CycloneDX 1.6 SBOM sidecar covering Debian packages, Python wheels, language-ecosystem lockfile contents (Rust, Go, npm, pnpm, yarn), Docker image layers, vendor blobs, and SONiC-built sources. The SBOMs preserve fork-of-upstream provenance (e.g. the FRR submodule pinned to a specific upstream tag plus SONiC's patch set) via pedigree.ancestors[], so a CVE in upstream FRR can be tied back to the SONiC build that carries the corresponding patch. A separate set of standalone scripts consume the SBOMs to produce vulnerability reports (grype + OpenVEX suppressions) and reproducibility diffs. The default build path is unchanged. With ENABLE_SBOM unset (the default), all SBOM-related work is short-circuited in slave.mk via a make-level $(if) guard, so non-SBOM builds incur zero overhead — no extra subprocesses, no Python startup cost, no apt-cache calls. The 5 added build dependencies (syft, grype, cyclonedx-cli, plus their installer) are only fetched when SBOM is enabled, through the existing sonic-build-hooks wget shim so versions-web tracking still applies. The Azure Pipelines side replaces the recently-merged Trivy scans (PR sonic-net#27079) with the new SBOM-based scanner so CI vulnerability reports come from the same data that ships in the artifact, not a separate re-scan of the image tarball. Comparison with the trivy-based PRs this would supersede The Azure Pipelines half of this PR replaces / overlaps with two recent trivy PRs: PR ci: add Trivy vulnerability scan for docker-ptf and docker-sonic-mgmt sonic-net#27079 — informational trivy scan of docker-ptf and docker-sonic-mgmt. Already merged. This PR removes those two trivy_scan jobs. PR ci: add Trivy vulnerability scan for broadcom and mellanox platform containers and host rootfs sonic-net#27322 — informational trivy scan of all docker-*.gz containers and the host rootfs for broadcom and mellanox platforms. Not yet merged. This PR addresses the same goal more thoroughly. Trivy is a perfectly reasonable image scanner. But trivy alone solves only the vulnerability scanning slice of the supply-chain problem, and it does so by re-discovering what's in the artifact at scan time rather than capturing what the build actually put there. Crucially, the SBOM-based approach finds more CVEs, not the same set — because the input components carry richer provenance, the scanner can match them against CVE databases that trivy can't reach from a plain rootfs scan.
xdqi
pushed a commit
to canonical/sonic-buildimage
that referenced
this pull request
Jul 6, 2026
sonic-net#27455) Why I did it SONiC currently ships no Software Bill of Materials. There is no inventory of what is in each .bin, no record of the patches and forks SONiC carries on top of upstream sources, no per-artifact vulnerability surface, and no machine-readable provenance. This makes CVE response, license attribution, supply-chain audits (SLSA, SBOM regulations like EO 14028 / CRA), and reproducibility checks expensive or impossible. This PR adds opt-in SBOM generation and SBOM-based vulnerability scanning. When ENABLE_SBOM=y is passed at build time, every artifact (.bin, .img, .swi, plus the standalone test-container .gzs) gets a CycloneDX 1.6 SBOM sidecar covering Debian packages, Python wheels, language-ecosystem lockfile contents (Rust, Go, npm, pnpm, yarn), Docker image layers, vendor blobs, and SONiC-built sources. The SBOMs preserve fork-of-upstream provenance (e.g. the FRR submodule pinned to a specific upstream tag plus SONiC's patch set) via pedigree.ancestors[], so a CVE in upstream FRR can be tied back to the SONiC build that carries the corresponding patch. A separate set of standalone scripts consume the SBOMs to produce vulnerability reports (grype + OpenVEX suppressions) and reproducibility diffs. The default build path is unchanged. With ENABLE_SBOM unset (the default), all SBOM-related work is short-circuited in slave.mk via a make-level $(if) guard, so non-SBOM builds incur zero overhead — no extra subprocesses, no Python startup cost, no apt-cache calls. The 5 added build dependencies (syft, grype, cyclonedx-cli, plus their installer) are only fetched when SBOM is enabled, through the existing sonic-build-hooks wget shim so versions-web tracking still applies. The Azure Pipelines side replaces the recently-merged Trivy scans (PR sonic-net#27079) with the new SBOM-based scanner so CI vulnerability reports come from the same data that ships in the artifact, not a separate re-scan of the image tarball. Comparison with the trivy-based PRs this would supersede The Azure Pipelines half of this PR replaces / overlaps with two recent trivy PRs: PR ci: add Trivy vulnerability scan for docker-ptf and docker-sonic-mgmt sonic-net#27079 — informational trivy scan of docker-ptf and docker-sonic-mgmt. Already merged. This PR removes those two trivy_scan jobs. PR ci: add Trivy vulnerability scan for broadcom and mellanox platform containers and host rootfs sonic-net#27322 — informational trivy scan of all docker-*.gz containers and the host rootfs for broadcom and mellanox platforms. Not yet merged. This PR addresses the same goal more thoroughly. Trivy is a perfectly reasonable image scanner. But trivy alone solves only the vulnerability scanning slice of the supply-chain problem, and it does so by re-discovering what's in the artifact at scan time rather than capturing what the build actually put there. Crucially, the SBOM-based approach finds more CVEs, not the same set — because the input components carry richer provenance, the scanner can match them against CVE databases that trivy can't reach from a plain rootfs scan.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why I did it
SONiC CI currently has no automated vulnerability scanning for published platform container images or the host rootfs. This PR adds a parameterized Trivy scan template covering both, starting with broadcom and mellanox platforms. Phase 1 is informational only — scans run after the Build stage and never block a PR merge.
Work item tracking
How I did it
Added a reusable pipeline template
.azure-pipelines/trivy-scan-platform.ymlparameterized byplatformandparallelism(default 4). Each invocation:sonic-buildimage.<platform>)--skip-db-updatedocker-*.gzcontainer images in parallel (bounded byparallelismparameter) — HIGH and CRITICAL severity,--ignore-unfixed.bininstaller (fs.squashfsviainstaller/fs.zip) and scans it withtrivy rootfsBoth jobs use
exit-code 1andcontinueOnError: true— findings are visible but do not block merge.A new
TrivyScanstage inazure-pipelines.ymldepends onBuildand invokes the template for broadcom and mellanox in parallel.How to verify it
TrivyScanstage in the CI run for this PR — two jobs (Trivy scan (broadcom)andTrivy scan (mellanox)) should appear and complete independently.trivy-scan-broadcom/trivy-scan-mellanoxartifacts for full table output.continueOnErrorand the overall pipeline passes.Which release branch to backport (provide reason below if selected)
Tested branch (Please provide the tested image version)
Description for the changelog
Add Trivy vulnerability scanning CI jobs for broadcom and mellanox platform containers and host rootfs (informational, non-blocking)
Link to config_db schema for YANG module changes
N/A — no YANG changes