fix(cosign-scan/oras-scan): Referrers-based classification, Trivy cache reuse, report cleanups#114
Merged
Merged
Conversation
added 17 commits
June 30, 2026 09:55
Trivy's default `fs` cache backend uses BoltDB and holds an exclusive
lock on `<cache-dir>/fanal/cache.db` for the duration of every scan,
not just during DB updates. The previous two configurations both had
real downsides:
* Per-image `--cache-dir` (no DB seeding) -> every parallel worker
redownloaded the entire vuln DB. On 30+ images this was the
dominant wall-clock cost (~7 min wasted per run).
* Shared default cache + `--skip-db-update` -> workers
raced for the BoltDB lock and most timed out with
`unable to initialize fs cache: cache may be in use by another
process`.
This change combines the upsides of both:
* `prepare_trivy_db()` still downloads the vuln DB once into Trivy's
default cache and now records the location in `_TRIVY_DB_SRC`.
* `_get_worker_cache_dir()` is a threading.local helper that, on
first use by each worker thread, mints a fresh tempdir and copies
the immutable vuln-DB files (`db/`, optional `java-db/`) from
`_TRIVY_DB_SRC`. The pool is bounded by `args.parallel`, so peak
extra disk usage is N x ~30-50 MB - trivial on a CI runner.
* `run_trivy_scan()` passes `--cache-dir <worker-dir> --skip-db-update`,
so Trivy reuses the seeded DB without redownloading and without
fighting other workers for a shared lock.
* `atexit` cleans up every per-worker dir on process exit.
Net effect: the ORAS scan keeps its pre-fix wall-clock improvement
(~10 min -> ~2-3 min) while producing correct results for every
image instead of failing 35 of 39 scans.
Two changes to collapse the long tail of skipped images on ``cosign-scan``: 1. ``ImageScanner.detect_attestation_type`` now does the cheap ``oras discover`` (single Referrers API GET, capped at 15 s) BEFORE invoking ``cosign verify``. Aleph-Alpha images attach all signature, SBOM and triage bundles via OCI 1.1 referrers; if the referrers list is empty we can mark the image UNSIGNED immediately and avoid a minutes-long ``cosign verify`` round-trip that was guaranteed to fail anyway. ``cosign verify`` only runs once we know there is a sigstore bundle to validate. 2. ``CosignVerifier`` gains a ``verify_timeout`` ceiling, defaulted to ``timeout`` (no behavior change for standalone callers) but set to 60 s by ``ImageScanner``. Previously verify inherited the full scan timeout (default 600 s = 10 min), so a hung Harbor request could block one worker for the entire window. 60 s is plenty for a keyless verify against a healthy registry while preventing the tail-end stall. Measured on the helm-chart pipeline: tail-end scans of skipped images (``pharia-os-app:1.30.4``, ``pharia-os-manager:0.15.7`` and friends) were each consuming ~2-3 minutes of worker time despite never running trivy.
Previous commit unintentionally marked images UNSIGNED when ``oras discover`` returned an empty list for *any* reason -- including a transient timeout or network glitch -- because the helper conflated "discovery failed" with "no referrers exist". A run on the helm-chart pipeline misclassified ``feature-flags:0.11.11`` (previously a successful ✓ scan) as unsigned. Make the discovery API tristate: * ``None`` -> discovery failed (timeout / parse error / unsupported registry); caller must use the slow path. * ``[]`` -> discovery succeeded, image has no referrers attached; caller can short-circuit. * ``[..]`` -> discovery succeeded, referrers found. ``ImageScanner.detect_attestation_type`` now only short-circuits to UNSIGNED on the definitive empty case and falls back to ``cosign verify`` on any failure, so a flaky Referrers API call no longer corrupts the report. The legacy private ``_discover_referrers`` keeps its old ``List[dict]`` contract (treats ``None`` as ``[]``) so existing call-sites that don't need the distinction are unaffected.
``CosignVerifier.verify()`` is the gate used by ``ImageScanner`` to decide if an image is signed at all. Without ``--new-bundle-format`` (available in cosign >= 2.5 and the default for future versions) cosign only probes the legacy ``<image>:sha256-<digest>.sig`` tag. When an image is signed purely via OCI 1.1 sigstore-bundle referrers (``application/vnd.dev.sigstore.bundle.v0.3+json``) the probe 404s and cosign returns "no signatures found", so scanner-py labels the image ``UNSIGNED`` and skips the scan. Verified against Harbor: the four skipped pharia-os images (``pharia-os-app:1.30.4``, ``pharia-os-manager:0.15.7``, ``phariaos-applications-proxy:0.8.9``, ``feature-flags:0.11.11``) each have a valid ``https://sigstore.dev/cosign/sign/v1`` bundle in their referrers, signed by the expected ``Aleph-Alpha/shared-workflows/build-and-push.yaml`` identity, plus SBOM/triage/SLSA/vuln attestations. They are false-negatives caused solely by the missing discovery flag. The sister ``AttestationExtractor._verify_attestation`` helper has always passed ``--new-bundle-format``; this change aligns signature verification with attestation verification so both halves of ``ImageScanner.detect_attestation_type`` look at the same storage backend.
…images With --new-bundle-format (support commit 53b98da) cosign enumerates every OCI 1.1 referrer on the image to find a signature bundle. Harbor's Referrers API is O(N) in referrer count: refs=36 -> ~8 s (feature-flags: verified) refs=66 -> ~13 s (phariaos-applications-proxy) refs=237 -> ~48 s (pharia-os-manager) refs=403 -> ~95 s (pharia-os-app) The previous 60 s cap was tight enough that all three pharia-os images with >60 SBOM/triage/vuln/SLSA referrers hit "context deadline exceeded", which ``detect_attestation_type`` mapped to ``UNSIGNED`` even though they are perfectly signed. Bumping the verify timeout to 180 s gives cosign enough headroom to page through the Referrers API plus a couple of HTTP round trips to fetch the matching sigstore bundle and complete the Fulcio / Rekor checks. This does not re-introduce the original long-tail problem: the ``oras discover`` pre-check in ``ImageScanner.detect_attestation_type`` still short-circuits truly unsigned images in ~1 RTT without ever entering the ``cosign verify`` slow path.
…ce of truth
``ImageScanner.detect_attestation_type`` was making three round-trips
through Harbor's OCI 1.1 Referrers API per image:
1. ``discover_referrers`` (oras discover, 15 s cap)
2. ``verifier.verify`` (cosign re-enumerates referrers)
3. ``list_attestations`` (oras discover + N×2 manifest fetches)
Harbor's referrers endpoint is O(N) in referrer count -- empirically
~8 s for 36 refs and ~95 s for 403 refs. The 15 s cap on step 1 plus
the 60 / 180 s budget on step 2 meant every heavily-attested image
(pharia-os-app at 403 referrers, pharia-os-manager at 237, ...) hit
"context deadline exceeded" somewhere in this chain and was reported
``UNSIGNED`` even though the image carried a perfectly valid cosign
signature.
Restructure detect_attestation_type around a single trip through
``list_attestations``:
* The signature half of a cosign-signed image is itself a sigstore
bundle with predicateType ``https://sigstore.dev/cosign/sign/v1``.
Its presence in the OCI 1.1 referrer list is sufficient evidence
of signing -- the cert chain is still validated downstream when
``extract_sbom`` / ``extract_triage`` actually pull the bundle.
* Add ``AttestationList.has_signature()`` for that check and
``AttestationList.COSIGN_SIGNATURE_PREDICATE`` for the constant.
* Drop the standalone ``cosign verify`` call in this method (the
verifier is still wired up for callers that need a verified
signature, e.g. signing pipelines).
* Switch ``list_attestations`` to the tristate ``discover_referrers``
so a transient registry failure can no longer be silently
misclassified as "no SBOM".
* Bump the oras-discover ``fast_timeout`` from 15 s to 120 s so a
Harbor referrers call against a 400-referrer image actually
completes; this keeps unsigned images fast (those still finish in
1 RTT once the registry returns an empty list) while letting
signed-but-attested images reach the predicate decoding step.
Net effect: pharia-os-app / pharia-os-manager /
phariaos-applications-proxy / feature-flags / data-eval-load all
classify correctly again, without per-image cosign verify latency.
The remaining 🚫 in scan reports are the genuinely unsigned
inference-worker images that have only a Harbor-generated SBOM and no
sigstore bundles in their referrers.
``oras discover --format json`` already returns each referrer's manifest annotations, which for sigstore bundles includes ``dev.sigstore.bundle.predicateType`` (set by cosign at sign time). Reading the annotation lets ``list_attestations`` derive every predicate type from the single ``oras discover`` call we already make, instead of issuing two extra HTTP calls (manifest fetch + blob fetch + DSSE decode) per bundle. For an image with 400 referrers that drops the per-image cost from ~800 sequential HTTP round-trips (5-worker pool, several minutes wall-clock) to zero. The previous full-bundle decode path is kept as a fallback for the rare bundle that lacks the annotation (older or hand-crafted artifacts) so behaviour for non-cosign-emitted bundles is unchanged. This restores cosign-scan latency to ~7-9 minutes for the pharia-ai image set even after the holistic ``detect_attestation_type`` rewrite that now exclusively relies on ``list_attestations`` for signed/SBOM/triage classification.
…parallelism Harbor / cosign sets the same ``dev.sigstore.bundle.predicateType = https://sigstore.dev/cosign/sign/v1`` annotation on *every* sigstore bundle manifest, regardless of whether the bundle wraps an image signature, an SBOM, a triage statement, etc. The previous commit (7101d3a) read this annotation and -- correctly per its contract but unhelpfully in practice -- classified all bundles as cosign signatures, making every signed image fall into ``COSIGN_NO_SBOM``. The real in-toto predicate type lives inside each bundle's DSSE payload, which means we have to pull manifest + blob per bundle to decode it. That's 2 HTTP round-trips per bundle, and pharia-os-app has 400+ bundles. The original 5-worker pool serialised this into ~4 minutes per image; raise it to 20 workers so the heaviest known image classifies in ~30 s while still being polite to Harbor.
…ication is stable For images with hundreds of OCI 1.1 sigstore-bundle referrers (e.g. pharia-os-app at 400+) decoding every bundle's DSSE payload to read the in-toto predicate type was the cosign-scan critical path -- 800+ HTTP round-trips per image, even with the 20-worker pool from 1a8580c. ``ImageScanner.detect_attestation_type`` only consumes three predicate classes from ``list_attestations``: the cosign signature predicate, an SBOM predicate (CycloneDX or SPDX), and the Aleph-Alpha triage predicate. Once at least one bundle of each kind has been observed, no further bundle can change the cosign-scan classification -- counts may go up but the SIGNED / SBOM / TRIAGE flags are already final. Track the predicate set as bundles complete and, the moment the "decisive" set is filled, cancel pending futures and break out of the ``as_completed`` loop. ``ThreadPoolExecutor.shutdown(wait=True)`` will still join the at-most-20 in-flight workers, so this is safe and deterministic; cancelled futures simply never execute their HTTP work. Empirically this drops classify latency for the heaviest signed image from "doesn't finish in 20 minutes" down to ~5-15 s, restoring the overall cosign-scan job to ~3-5 minutes wall-clock for the full pharia-ai image set. For the small minority of signed images that legitimately ship an SBOM but no triage (or vice versa) the loop falls through to a full enumeration -- but those images have a small number of bundles to begin with, so wall-clock impact is negligible.
…tract_triage The previous early-termination commit (daf7810) made ``list_attestations`` fast for heavily-attested images, but the cosign-scan job remained slow because the *next* step, ``extract_sbom`` / ``extract_triage``, called ``_find_attestation_digests`` which serially re-enumerated every sigstore bundle on the image (no parallelism, no shared work with ``list_attestations``). For pharia-os-app's 400+ bundles that's 800+ sequential HTTP round-trips per extraction call, twice per image. Add a per-instance cache mapping ``(image_name, digest) -> {predicate_type: [bundle_digests]}``. ``list_attestations`` populates it as it decodes each bundle's DSSE payload, and ``_find_attestation_digests`` consults it before falling back to the slow path. The cosign-scan flow now decodes each bundle exactly once (during ``detect_attestation_type``) and reuses the result for both SBOM and triage extraction. Also parallelise the cache-miss fallback in ``_find_attestation_digests`` (20 workers, mirroring ``list_attestations``) so unusual call patterns -- callers that ask for a predicate type ``list_attestations`` did not observe before early-terminating, e.g. SLSA / vuln -- still benefit from the concurrency improvements.
…evel When detect_attestation_type classifies an image as UNSIGNED, the reason was only visible at DEBUG, which is gated by --verbose. In CI this meant any false UNSIGNED (e.g. heavily-attested pharia-os-app under 15x parallel scan load) was indistinguishable from a genuinely unsigned image. Surface three diagnostics at INFO so cosign-scan CI logs capture the failure mode directly: * list_attestations now returns AttestationList.discovery_failed=True when oras discover timed out or when the sigstore bundle set was enumerated but the signature predicate never showed up in the decoded map (typical of bundle fetches silently failing under high load). * list_attestations logs per-image bundle decode outcomes (ok / err / empty counts) whenever any decode failure occurred. * detect_attestation_type logs the predicate map and discovery_failed flag alongside the UNSIGNED classification. Local 15-way parallel repro against Harbor still classifies all 15 test images correctly in 81 s wall-clock, so this change is diagnostic-only; no behavior change for the healthy path. Follow-up can wire a retry or an authoritative fallback once the CI logs confirm which failure mode is actually hitting us.
Previous commit routed the diagnostic messages through ``logger.info``, but the CLI wraps every parallel-scan worker in ``suppress_logging`` (which bumps the root logger level to CRITICAL+1) so the progress bar has exclusive stderr control. That silently swallowed every INFO line in CI, defeating the whole point of the diagnostic. Switch the three UNSIGNED / discovery_failed diagnostics to direct ``print(..., file=sys.stderr, flush=True)`` so they survive ``suppress_logging`` and appear in the cosign-scan job log next to the progress bar output. Still diagnostic-only; no behavioral change on the healthy path.
Previous CI runs misclassified the heaviest signed image (``pharia-os-images/pharia-os-app:1.30.4``, 403 OCI 1.1 referrers) as UNSIGNED. The stderr diagnostic added in the prior commit pinpointed the cause: ``oras discover`` on that image takes ~95 s locally but exceeded the 120 s cap from GitHub-hosted runners (extra network round-trip + TLS cost amplifies the per-referrer Referrers-API latency). On timeout ``discover_referrers`` returned ``None`` without falling through to the tag-based path, so the downstream classifier saw zero referrers and declared the image unsigned. Two fixes: * Raise ``fast_timeout`` from 120 s to 240 s. Still bounded, still governed by the caller-supplied ``self.timeout``, but now comfortably above the worst observed tail (~150 s for pharia-os-app from CI). * Fall through to the tag-based ``oras discover`` retry on *any* digest-based failure, including timeout. Harbor caches tag->digest separately, so a tag-based request frequently succeeds when the digest-based one did not. The previous "bail out on timeout" short circuit turned the tag-based retry into dead code for the exact failure mode that needed it most. Also replaces the ``logger.debug`` calls with ``sys.stderr`` prints in the failure paths so operators can see *which* attempt timed out without rerunning with --verbose (same rationale as the prior diagnostic commit).
The scan reports were historically written for the Kubernetes-namespace use case (`scanner-py cosign-scan --namespace pharia-ai`), but are now also driven off static image-file lists by downstream tooling such as pharia-ai-helm-chart. That mix produced two chronic papercuts: * Per-image tables were emitted in *scan-completion order*, i.e. effectively random across reruns, making the reports useless for diffing and painful to read. * The header row read `| Namespace | <full path to images.txt> |` whenever `--image-file` was used, which is both wrong semantically and visually broken once the path is long. This change keeps full back-compat (defaults behave identically when `--namespace` is still the source of truth) while: * adding a `--source-label` flag to `cosign-scan` and `generate-sbom-report` for callers that want to set an explicit human-readable label; * falling back to `basename(--image-file)` instead of the raw path for the header when `--source-label` isn't set; * renaming the rendered header from "Namespace" to "Image source" in `generate_markdown_summary` (cosign) and `generate_markdown_report` (oras); * sorting the per-image rows alphabetically by full image reference in both markdown and CLI renderers (cosign + oras + generate-report), by basename for the triage report, and by image name for the SBOM detailed report; * preserving the Harbor project segment (e.g. `pharia-ai-images/foo:1.0`) in the displayed image name so duplicate basenames across projects stay distinguishable, only truncating very long refs; * replacing the grammatically broken SBOM header line "Source: Successful SBOM scans from <path> namespace" with "**Image source:** `<label>`". No semantic change to scan behaviour — this is purely presentation.
The image column in the cosign-scan, ORAS, and standalone CVE reports previously rendered images as `<project>/<image>:<tag>` to disambiguate basename collisions across registry projects. In a typical chart-wide scan the project prefix is identical for nearly every row, so it adds noise without signal and pushes long names into the truncation cap. Render only `<image>:<tag>` in the image column of: * cosign-cve-summary.md (k8s_scanner.generate_markdown_summary) * standalone CVE markdown report (generate_report.generate_markdown_report) * oras-result.md (oras_scan.generate_markdown_report) The matching CLI/terminal print summaries (k8s_scanner.print_summary, oras_scan.print_cli_summary) get the same treatment for consistency. Truncation thresholds raised slightly (45->50 markdown, kept at 33/43 for terminal) since basenames are shorter and previously truncated names like `pharia-search-transition-actor:0.30.22` now fit in full. Caller is responsible for ensuring image basenames are unique across the input set; collisions would now render as visually identical rows.
Tables previously sorted by the full image path (`<host>/<project>/<image>:<tag>`).
After the prior commit started rendering only `<image>:<tag>` in the
image column, that left the visible rows looking unsorted: e.g. all
`data-*` images appeared scrambled across the table because they live
in different projects (`pharia-data/data-engine`, `pharia-data/data-etl`,
`assistant-container-images/translation-service`, etc.) but the project
prefix was hidden.
Sort by `image.rsplit("/", 1)[-1].lower()` in all six renderers
(markdown + CLI for cosign-scan, ORAS, and standalone CVE report) so
the row order matches what the reader actually sees.
Trivy's default vulnerability DB order now prefers mirror.gcr.io before ghcr.io. The mirror can lag behind the official upstream publish, which means `trivy image --download-db-only` may fetch a stale DB even after Aquasecurity has already published a newer one. Pin the DB download in both cosign-scan and oras-scan to the official GHCR trivy-db image so scheduled scans prefer freshness over mirror latency.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Performance and correctness fixes for the cosign/oras scanner, plus report-presentation cleanups. These changes have been running pinned as
fix/oras-scan-trivy-cache-reuseinpharia-ai-helm-chart'sscan-docker-images.yaml; this PR rebases them onto currentmainso the pin can eventually return tomain.oras-scan / Trivy
ghcr.io/aquasecurity/trivy-db:2so scans don't silently consume a stalemirror.gcr.iocopy.cosign-scan
cosign verifyround trips (Harbor enumerates the Referrers API again, linear in referrer count — the main long-tail timeout cause).discover_referrers()now returnsNoneon discovery failure vs[]for "genuinely no referrers".--new-bundle-formatso OCI 1.1 signatures are found; verify timeout bumped to 180s for heavily-attested images.extract_sbom/extract_triagereuse parallel decode work; early-terminate predicate enumeration once classification is stable.Reports
Rebase note — conflict resolutions to review
This branch had diverged from
main(17 ahead / 7 behind). Rebasing produced three conflicts inscanner_py/core/attestation.py, all from the same root cause:mainand this branch made conflicting design choices fordiscover_referrers().mainadded a per-host registry-capability cache that returned[]on timeout/unsupported.None-vs-[]contract (None= discovery failed → fall back;[]= definitively no referrers → safe to fast-path UNSIGNED).Resolved by keeping both:
main's capability cache — it's consumed byretrieve_triage.pyto skip slow legacy probes on registries with no Referrers API (e.g. JFrog), so it must stay populated.Nonecontract — the consumer inscanner.pyrelies on it, and returning[]for an unsupported registry would wrongly fast-path those images to UNSIGNED. Everywheremainreturned[]it now returnsNone, while still callingmark_referrers_unsupported(...).All of
scanner_pycompiles; theNone/[]contract is consistent (the legacy_discover_referrersalias still normalizesNone→[]for old callers). The three resolution points are the parts most worth a close look.