Skip to content

Commit 923b8d7

Browse files
christiangdaclaude
andcommitted
fix(ci): cosign sign container manifest by tag, not by local podman digest
The cosign step of the release workflow failed with MANIFEST_UNKNOWN on every release attempt after multi-arch builds were restored, e.g. on v0.45.0: Error: signing [ghcr.io/slashdevops/idp-scim-sync@sha256:2925...]: GET https://ghcr.io/v2/.../manifests/sha256:2925...: MANIFEST_UNKNOWN The previous logic resolved the digest to sign by piping `podman manifest inspect ghcr.io/...:TAG` into `jq -r '.digest // .manifests[0].digest'`. Two compounding bugs: 1. A manifest list's own JSON has no top-level `.digest` (the list's digest is computed by hashing the JSON, not stored inside it), so the `//` fallback always returned `.manifests[0].digest` -- which is the digest of the first per-arch image (arm64), not of the manifest list itself. 2. Podman re-serializes manifests on push (media-type conversion between Docker manifest.v2+json and OCI image.manifest.v1+json), so the locally computed digest does not match what GHCR stores. Cosign's lookup at that local digest therefore returned 404. Switch to `cosign sign --recursive ${IMAGE}:${TAG}`. Cosign internally HEAD-resolves the tag to the authoritative on-registry digest and signs that digest, so the resulting artifact is identical to what the broken code intended to produce. The classic "signing by tag races with concurrent pushes" caveat does not apply: this job exclusively owns the v<x.y.z> and latest tags and has just pushed them sequentially in the previous step. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 2cf16ba commit 923b8d7

2 files changed

Lines changed: 30 additions & 11 deletions

File tree

.github/workflows/container-image.yml

Lines changed: 15 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -90,16 +90,20 @@ jobs:
9090
echo "${{ secrets.GITHUB_TOKEN }}" \
9191
| cosign login ghcr.io --username "${{ github.actor }}" --password-stdin
9292
IMAGE="ghcr.io/${{ github.repository }}"
93-
# podman manifest inspect → resolve the multi-arch manifest tag we just
94-
# pushed to its content digest, then sign by digest. Cosign best-practice:
95-
# signing by tag races with subsequent pushes; signing by digest does not.
96-
# --recursive covers the manifest and every per-platform image under it.
93+
# Sign by tag — cosign internally HEADs the registry to resolve the
94+
# tag to its authoritative on-registry digest and signs that digest;
95+
# the signature is stored under that digest, not under the tag. We
96+
# previously resolved the digest locally with `podman manifest
97+
# inspect`, but (a) that JSON has no top-level `.digest` for a
98+
# manifest list and the `.manifests[0].digest` fallback returns the
99+
# *first per-arch* image's digest, not the list's, and (b) podman
100+
# re-serializes during push, so the local digest does not exist on
101+
# GHCR. Result: cosign got MANIFEST_UNKNOWN.
102+
# --recursive covers the manifest and every per-platform image
103+
# under it. The classic "signing by tag races with concurrent
104+
# pushes" caveat doesn't apply here: this job exclusively owns
105+
# these tags and has just pushed them.
97106
for TAG in "${GIT_VERSION}" "latest"; do
98-
DIGEST=$(podman manifest inspect "${IMAGE}:${TAG}" | jq -r '.digest // .manifests[0].digest')
99-
if [ -z "${DIGEST}" ] || [ "${DIGEST}" = "null" ]; then
100-
echo "::error::Could not resolve digest for ${IMAGE}:${TAG}"
101-
exit 1
102-
fi
103-
cosign sign --recursive "${IMAGE}@${DIGEST}" | tee -a $GITHUB_STEP_SUMMARY
104-
echo "**Signed:** \`${IMAGE}@${DIGEST}\` (tag: ${TAG})" >> $GITHUB_STEP_SUMMARY
107+
cosign sign --recursive "${IMAGE}:${TAG}" | tee -a $GITHUB_STEP_SUMMARY
108+
echo "**Signed:** \`${IMAGE}:${TAG}\`" >> $GITHUB_STEP_SUMMARY
105109
done

docs/Whats-New.md

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,21 @@ This document tracks notable changes, new features, and bug fixes across release
44

55
## Unreleased
66

7+
### CI fix: cosign now signs the published container manifest by tag (closes the v0.45.0 signing failure)
8+
9+
Fixes the `Cosign sign published container manifest (keyless / Sigstore)` step of the release workflow, which failed with **`MANIFEST_UNKNOWN: manifest unknown`** on every release attempt after the multi-arch build was restored (see ["CI fix: restore multi-arch container builds"](#ci-fix-restore-multi-arch-container-builds-in-the-release-workflow)).
10+
11+
**Root cause.** The step resolved the digest to sign by piping `podman manifest inspect` into `jq -r '.digest // .manifests[0].digest'`. Two compounding problems:
12+
13+
1. **A manifest list's own JSON has no top-level `.digest`** (its digest is computed by hashing the JSON, not stored inside it). So the `//` fallback always wins and returns `.manifests[0].digest` — the digest of the **first per-arch image** (arm64), not the manifest list.
14+
2. **Podman re-serializes manifests when pushing** (media-type conversion between Docker `vnd.docker.distribution.manifest.v2+json` and OCI `vnd.oci.image.manifest.v1+json`). The locally computed digest therefore does not match what GHCR stores, so cosign's lookup of `ghcr.io/…@sha256:<local-digest>` returns 404.
15+
16+
Result: cosign was asked to sign a digest that exists nowhere on the registry.
17+
18+
**Fix.** Sign by tag (`cosign sign --recursive ghcr.io/…:TAG`). Cosign internally HEAD-resolves the tag to its authoritative on-registry digest and signs that digest — the signature is still stored *by digest*, so the resulting artifact is identical to what the previous (broken) code intended to produce. The classic "signing-by-tag races with concurrent pushes" caveat does not apply here: this job exclusively owns the `v<x.y.z>` and `latest` tags and has just pushed them sequentially in the previous step.
19+
20+
No code or release-artifact changes.
21+
722
### CI fix: restore multi-arch container builds in the release workflow
823

924
Fixes the `Publish Container Images` job (failing since v0.44.1, surfaced again on the v0.45.0 release as ["Could not resolve digest for ghcr.io/slashdevops/idp-scim-sync:v0.45.0"](https://github.com/slashdevops/idp-scim-sync/actions/runs/26356807875/job/77585211704)).

0 commit comments

Comments
 (0)