podvm: fail loud when oras resolve returns empty digest#3092
Conversation
c441ba8 to
f2af6df
Compare
pull_agent_artifact and pull_gc_artifact in src/cloud-api-adaptor/podvm/Makefile.inc both call `oras resolve $OCI_IMAGE:$tag` and capture the result in OCI_DIGEST via make's $(shell) function. When the registry does not have the requested tag (for example, when versions.yaml pins a GUEST_COMPONENTS_REF or KATA_REF that does not have a corresponding published `<sha>-<arch>` or `<sha>-<tee>_<arch>` artifact), `oras resolve` exits non-zero and prints nothing to stdout. make captures the empty stdout into OCI_DIGEST, then constructs `OCI_REF := $(OCI_IMAGE)@` (trailing @ with no digest), and the downstream `oras pull` either fails opaquely or -- depending on local cache state -- pulls a stale layer from a previous successful build at a different commit. The resulting PodVM image then contains guest-components or kata-agent binaries that do not match the SHAs declared in versions.yaml, with no warning at build time. This is real. It was the underlying cause of multi-day diagnosis effort when a PodVM AMI built from CAA v0.20.0 contained an attestation-agent binary that spoke an older wire protocol than the commit pinned in versions.yaml shipped, leading to KBS-side `RcarAuthFailed: deserialize Request - missing field 'version'` rejections. The AMI build succeeded silently; the failure surfaced 1+ hours later at attestation handshake time, in a separate component (Trustee). This change adds a single guard between `oras resolve` and the subsequent `OCI_REF` construction: the shell output is wrapped in $(strip ...) (defensive against whitespace), and if OCI_DIGEST is empty afterwards, make's $(error) function aborts the build immediately with a message naming the missing image:tag and pointing at the versions.yaml field to investigate. oras's own stderr is left visible (no `2>/dev/null` redirection) so users see the underlying network / auth / 404 details alongside the make-level abort message. Behavior change: builds that previously produced a stale-binary PodVM image now fail at the make level with a clear error. Builds whose versions.yaml pins resolve correctly are unchanged. No CI workflow, script, or other consumer of this Makefile.inc relies on the silent- fallback behavior. A follow-up could add a CI workflow that pre-validates all versions.yaml::oci.*.reference SHAs resolve in their registries before a release tag is cut, but that is out of scope for this minimal-diff fix. Signed-off-by: Abhishek Agrawal <abhishek.yours4@gmail.com>
f2af6df to
9ac77f5
Compare
|
@mkulke both suggestions applied and threads resolved. Could you run ok-to-test when you have a moment? Thanks! |
@Agrek11 can you elaborate under which circumstances the Makefile.inc will produce "stale PodVM images". that should not happen, either with or without the change in the PR, so we might have a serious bug in the Makefile chain. My understanding from looking at the changes, the build should always fail hard, because "oras pull $image@" should result in a failure. AFAIU this PR improves diagnostics, but if we end up with an image that contains undesired (stale) artifacts, then we need to fix more. |
|
@mkulke you're right,thanks for pushing on this, i re-verified. With a missing tag,oras resolve fails inside $(shell) |
2 participants
Summary
pull_agent_artifactandpull_gc_artifactinsrc/cloud-api-adaptor/podvm/Makefile.incfail with a misleading error far from the root cause when the registry does not have the tag they ask for. This PR makesthem fail loud at the point of failure instead.
Both functions do
OCI_DIGEST := $(shell oras resolve $OCI_IMAGE:$tag), thenOCI_REF := $(OCI_IMAGE)@$(OCI_DIGEST). Whenoras resolveexits non-zero (missing tag),$(shell)swallows the exit code andcaptures empty stdout, so
OCI_REFbecomes$OCI_IMAGE@(trailing@). The subsequentoras pullthen fails with a generic usage error (Error: "<image>@": no tag or digest specified) — two steps removedfrom the actual problem, with nothing pointing at the missing tag or the
versions.yamlpin that caused it.This change adds
$(if $(OCI_DIGEST),,$(error ...))between the resolve and theOCI_REFconstruction. Empty digest → immediatemakeerror naming the exactimage:tagthat failed to resolve.Why this matters (real-world impact)
I hit this while iterating on
versions.yamlpins during a CAA v0.20.0 SEV-SNP peer-pod POC on AWS: pinning a guest-components SHA whose artifact tag had not yet been published by the publish workflow makes thebuild die on an oras usage error that names neither the missing tag nor the
versions.yamlfield to fix. With this guard, the same mistake is a single self-explanatory error line at the resolve step.Reproduction (without this PR)
src/cloud-api-adaptor/versions.yaml, setoci.guest-components.referenceto a 40-char SHA for which no<sha>-snp_amd64(or equivalent) tag has been published — for example, a fresh commit onguest-componentsmainthat hasn't yet been built and pushed by the publish workflow.TEE_PLATFORM=amd make image-debugfromsrc/cloud-api-adaptor/podvm-mkosi/.Error: "ghcr.io/confidential-containers/guest-components/attestation-agent@": no tag or digest specifiedfromoras pull— a usage error that does not mention the missing tag.With this PR applied, step 2 instead exits with:
What this PR does NOT do
versions.yamlpins resolve correctly — no-op in the happy path.oras resolve's own stderr — registry error details still print as before; the$(error)adds the make-level context on top.versions.yamlSHAs at PR / release time. That would be a useful follow-up but kept out of this PR to keep the diff minimal.Test plan
image:tagthat failed to resolvepodvm_mkosi_ubuntusmoke test + AWS e2e workflow should pass unchanged on this PRGUEST_COMPONENTS_REFto any 40-char SHA that lacks published artifacts and runningmake binariesHappy to follow this up with a CI workflow that pre-validates all
versions.yaml::oci.*.referenceSHAs resolve in their registries, if maintainers see value.