ci: Add validation for multi-architecture image digests in Dockerfiles

[rm3l]

Description

This PR introduces automated validation to ensure that all container image digests used in Dockerfiles reference multi-architecture manifest lists rather than single-architecture manifests.

Which issue(s) does this PR fix or relate to

Follow-up to #1807 (comment)

PR acceptance criteria

• Tests
• Documentation

How to test changes / Special notes to the reviewer

• Example of failed validation: https://github.com/redhat-developer/rhdh-operator/actions/runs/19423759542/job/55566378465?pr=1882, which led to fixing the ubi9-minimal digests in use here: 706f86b (#1882)
• Example of successful validation after fixing the digests: https://github.com/redhat-developer/rhdh-operator/actions/runs/19424156598/job/55567649077?pr=1882#step:3:110

[rhdh-qodo-merge]

PR Reviewer Guide 🔍

(Review updated until commit fb5efd7)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪

🔒 No security concerns identified

⚡ Recommended focus areas for review Portability The script uses non-portable emoji and color escape sequences in logs which can clutter CI logs or fail in environments without ANSI support; consider a no-color fallback or CI-detection to disable styling. RED='\033[0;31m' GREEN='\033[0;32m' YELLOW='\033[1;33m' NC='\033[0m' # No Color # Functions log_success() { echo -e "${GREEN}✅ $*${NC}" } log_warning() { echo -e "${YELLOW}⚠️ $*${NC}" } log_error() { echo -e "${RED}❌ $*${NC}" } Robustness Grep-based parsing of FROM lines may miss valid patterns (e.g., multiline, ARG-substituted bases) or produce false positives; consider more robust parsing or Dockerfile-aware tools. # Get FROM statements with digests, excluding --platform lines from_lines=$(grep -E '^\s*FROM\s+.*@sha256:' "$dockerfile" | grep -v -E '^\s*FROM\s+--platform' || true) if [ -z "$from_lines" ]; then continue fi # Process each FROM line while IFS= read -r line; do if [[ -z "$line" ]]; then continue fi # Extract the image reference with digest image_ref=$(echo "$line" | sed -E 's/^\s*FROM\s+([^ ]+@sha256:[a-f0-9]+).*/\1/') if [ -n "$image_ref" ]; then # Mark that we've seen this digest digests_map["$image_ref"]=1 # Append this file to the list of files containing this digest if [ -n "${digests_files_map[$image_ref]:-}" ]; then digests_files_map["$image_ref"]="${digests_files_map[$image_ref]}|$dockerfile" else digests_files_map["$image_ref"]="$dockerfile" fi fi done <<< "$from_lines" done

📄 References redhat-developer/rhdh-operator/config/profile/rhdh/plugin-infra/plugin-infra.sh [1-49] redhat-developer/rhdh-operator/hack/db_copy.sh [1-23] redhat-developer/rhdh-operator/config/profile/rhdh/plugin-infra/gitops-secret-setup.sh [199-222] redhat-developer/rhdh-operator/config/profile/rhdh/plugin-infra/gitops-secret-setup.sh [133-182] redhat-developer/rhdh-operator/config/profile/rhdh/plugin-infra/gitops-secret-setup.sh [1-35] redhat-developer/rhdh-operator/config/profile/rhdh/plugin-infra/gitops-secret-setup.sh [37-62] redhat-developer/rhdh-operator/config/profile/rhdh/plugin-infra/gitops-secret-setup.sh [64-87] redhat-developer/rhdh-operator/config/profile/rhdh/plugin-infra/gitops-secret-setup.sh [183-197]

[rhdh-qodo-merge]

PR Type

(Describe updated until commit 706f86b)

Enhancement, Tests

---

Description

• Add automated validation script to ensure Dockerfile image digests reference multi-architecture manifest lists

• Implement GitHub Actions workflow to run digest validation on pull requests

• Update ubi9-minimal digests to use manifest list digests in Dockerfiles

• Integrate validation into Makefile for local testing

---

File Walkthrough

	Relevant files
Enhancement	validate-image-digests.sh Validation script for multi-arch image digests                      hack/validate-image-digests.sh New bash script that validates FROM statements with digests in Dockerfiles Uses skopeo to inspect container image manifests and verify they are manifest lists Checks multiple Dockerfiles including main Dockerfile and bundle variants Provides colored output with detailed error reporting and architecture information +202/-0  Makefile Makefile target for digest validation                                        Makefile Add new validate-image-digests target to Makefile Target calls the validation script with descriptive output Enables local testing of digest validation before pushing +6/-0
Configuration changes	validate-image-digests.yaml CI workflow for digest validation                                                .github/workflows/validate-image-digests.yaml New GitHub Actions workflow triggered on pull requests Runs on changes to Dockerfiles and validation script Executes the validate-image-digests script via Makefile target +22/-0
Bug fix	Dockerfile Update to multi-arch manifest list digest                                Dockerfile Update ubi9-minimal base image digest to manifest list digest Change from single-architecture digest to multi-architecture compatible digest +1/-1      Dockerfile Update to multi-arch manifest list digest                                .rhdh/docker/Dockerfile Update ubi9-minimal base image digest to manifest list digest Change from single-architecture digest to multi-architecture compatible digest +1/-1

---

[rhdh-qodo-merge]

PR Code Suggestions ✨

Latest suggestions up to fb5efd7

Category	Suggestion	Impact
General	Add retry with exponential backoff Implement a retry loop with exponential backoff for the skopeo inspect command to handle transient network or registry errors and improve CI stability. hack/validate-image-digests.sh [127-131] -# Inspect the manifest using skopeo -set +e # Temporarily disable exit on error -manifest_info=$(skopeo inspect --raw "docker://$image_ref_for_inspect" 2>&1) -inspect_status=$? -set -e # Re-enable exit on error +# Inspect the manifest using skopeo with retries +attempt=0 +max_attempts=3 +backoff=2 +manifest_info="" +inspect_status=1 +while (( attempt < max_attempts )); do + set +e + manifest_info=$(skopeo inspect --raw "docker://$image_ref_for_inspect" 2>&1) + inspect_status=$? + set -e + + if [ $inspect_status -eq 0 ]; then + break + fi + + # Retry on transient errors only + if echo "$manifest_info" | grep -Eqi '(timeout|temporarily unavailable|TLS handshake timeout|too many requests|server error|connection refused|i/o timeout)'; then + attempt=$((attempt + 1)) + if (( attempt < max_attempts )); then + log_warning "Transient error inspecting $image_ref_for_inspect (attempt $attempt/$max_attempts). Retrying in ${backoff}s..." + sleep "$backoff" + backoff=$((backoff * 2)) + continue + fi + fi + # Non-transient or max attempts reached: break + break +done + Apply / Chat Suggestion importance[1-10]: 7 __ Why: This suggestion significantly improves the script's reliability in a CI environment by adding a retry mechanism for skopeo inspect, making it more resilient to transient network or registry errors.	Medium
Possible issue	Add registry connectivity preflight check Add a pre-flight check for network connectivity to common container registries to prevent misleading failures caused by network outages. hack/validate-image-digests.sh [59-65] # Check if skopeo is available if ! command -v skopeo &> /dev/null; then log_error "skopeo not found in PATH" echo "Please install skopeo to run this validation" echo "See: https://github.com/containers/skopeo" exit 1 fi +# Basic network connectivity check to common registries used in Dockerfiles +check_host() { + local host="$1" + # try DNS resolution first + if ! getent hosts "$host" >/dev/null 2>&1; then + return 1 + fi + # then attempt a quick TCP connection on 443 if nc exists + if command -v nc >/dev/null 2>&1; then + nc -z -w 5 "$host" 443 >/dev/null 2>&1 || return 1 + fi + return 0 +} + +REGISTRY_HOSTS=("registry-1.docker.io" "quay.io" "ghcr.io" "gcr.io" "registry.redhat.io") +connectivity_ok=true +for h in "${REGISTRY_HOSTS[@]}"; do + if ! check_host "$h"; then + log_warning "Network/DNS check failed for $h; manifest validation may be unreliable." + connectivity_ok=false + fi +done +if [ "$connectivity_ok" = false ]; then + log_error "Registry connectivity checks failed. Aborting to avoid false validation errors." + exit 1 +fi + Apply / Chat Suggestion importance[1-10]: 6 __ Why: The suggestion improves the script's robustness by adding a pre-flight network check, which helps differentiate between actual digest validation failures and transient CI environment issues.	Low
Update

---

Previous suggestions

Suggestions up to commit a1f8df6

Category	Suggestion	Impact
Possible issue	Fix digest validation for platform-specific images Update the script to correctly parse and validate FROM statements that include the --platform flag by adjusting the grep and sed commands. hack/validate-image-digests.sh [74-85] -# Get FROM statements with digests, excluding --platform lines -from_lines=$(grep -E '^\s*FROM\s+.*@sha256:' "$dockerfile" | grep -v -E '^\s*FROM\s+--platform' || true) +# Get FROM statements with digests +from_lines=$(grep -E '^\s*FROM\s+.*@sha256:' "$dockerfile" || true) [ -z "$from_lines" ] && continue # Process each FROM line while IFS= read -r line; do [[ -z "$line" ]] && continue - # Extract the image reference with digest - image_ref=$(echo "$line" | sed -E 's/^\s*FROM\s+([^ ]+@sha256:[a-f0-9]+).*/\1/') + # Extract the image reference with digest, ignoring other parts of the line + image_ref=$(echo "$line" | sed -E 's/.*([^ ]+@sha256:[a-f0-9]+).*/\1/') if [ -n "$image_ref" ]; then Suggestion importance[1-10]: 9 __ Why: The suggestion correctly identifies a bug where FROM statements with a --platform flag are skipped, which undermines the script's core validation purpose. The proposed fix is accurate and makes the script more robust.	High
High-level	Use a pre-built container image Instead of installing skopeo on each CI run, use a pre-built container image like quay.io/skopeo/stable for the job. This change will make the workflow faster and more reliable. Examples: .github/workflows/validate-image-digests.yaml [11-25] validate-image-digests: name: Validate Image Digests runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5 - name: Install skopeo run: | ... (clipped 5 lines) Solution Walkthrough: Before: # .github/workflows/validate-image-digests.yaml jobs: validate-image-digests: runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@... - name: Install skopeo run: | sudo apt-get update sudo apt-get install -y skopeo - name: Run validation script run: make validate-image-digests After: # .github/workflows/validate-image-digests.yaml jobs: validate-image-digests: runs-on: ubuntu-latest container: image: quay.io/skopeo/stable steps: - name: Checkout code uses: actions/checkout@... - name: Run validation script run: make validate-image-digests Suggestion importance[1-10]: 7 __ Why: This is a valuable suggestion that improves the new CI workflow's efficiency and reliability by replacing a manual dependency installation with a standard container-based approach.	Medium

[github-actions]

PR images are available:

1. https://quay.io/rhdh-community/operator:0.9.0-pr-1882-a1f8df6
2. https://quay.io/rhdh-community/operator-bundle:0.9.0-pr-1882-a1f8df6
3. https://quay.io/rhdh-community/operator-catalog:0.9.0-pr-1882-a1f8df6

[rm3l]

/review

[rhdh-qodo-merge]

You are nearing your monthly Qodo Merge usage quota. For more information, please visit here.

Persistent review updated to latest commit fb5efd7

[github-actions]

PR images are available:

1. https://quay.io/rhdh-community/operator:0.9.0-pr-1882-fb5efd7
2. https://quay.io/rhdh-community/operator-bundle:0.9.0-pr-1882-fb5efd7
3. https://quay.io/rhdh-community/operator-catalog:0.9.0-pr-1882-fb5efd7

[rhdh-qodo-merge]

You are nearing your monthly Qodo Merge usage quota. For more information, please visit here.

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪

🔒 No security concerns identified

⚡ Recommended focus areas for review Robustness The grep-based parsing of FROM lines and sed extraction may miss valid edge cases (multi-line Dockerfile instructions, comments with '@sha256', or additional flags/aliases). Consider using a more resilient parser or handling continuations and AS clauses explicitly. for dockerfile in "${DOCKERFILES[@]}"; do if [ ! -f "$dockerfile" ]; then continue fi # Get FROM statements with digests, excluding --platform lines from_lines=$(grep -E '^\s*FROM\s+.*@sha256:' "$dockerfile" | grep -v -E '^\s*FROM\s+--platform' || true) if [ -z "$from_lines" ]; then continue fi # Process each FROM line while IFS= read -r line; do if [[ -z "$line" ]]; then continue fi # Extract the image reference with digest image_ref=$(echo "$line" | sed -E 's/^\s*FROM\s+([^ ]+@sha256:[a-f0-9]+).*/\1/') if [ -n "$image_ref" ]; then # Mark that we've seen this digest digests_map["$image_ref"]=1 # Append this file to the list of files containing this digest if [ -n "${digests_files_map[$image_ref]:-}" ]; then digests_files_map["$image_ref"]="${digests_files_map[$image_ref]}|$dockerfile" else digests_files_map["$image_ref"]="$dockerfile" fi fi done <<< "$from_lines" done Manifest Detection Media type detection relies on grep of raw manifest output; skopeo --raw may return OCI or Docker formats and error messages. Using jq on skopeo inspect (non-raw) when available would be more reliable; also consolidate duplicate skopeo calls to avoid extra network requests. # Check if the manifest is a manifest list (multi-architecture) if echo "$manifest_info" | grep -q '"mediaType".*"application/vnd.docker.distribution.manifest.list.v2+json"' || \ echo "$manifest_info" | grep -q '"mediaType".*"application/vnd.oci.image.index.v1+json"'; then # Valid manifest list log_success "Manifest list (multi-arch): $image_ref" echo " File(s):" for file in "${source_files[@]}"; do echo " - $file" done echo validated_count=$((validated_count + 1)) continue fi # Check if it's a single-architecture manifest if echo "$manifest_info" | grep -q '"mediaType".*"application/vnd.docker.distribution.manifest.v2+json"' || \ echo "$manifest_info" | grep -q '"mediaType".*"application/vnd.oci.image.manifest.v1+json"'; then # Extract the specific architecture if command -v jq &> /dev/null; then arch_info=$(skopeo inspect "docker://$image_ref_for_inspect" 2>/dev/null | jq -r '"\(.Os)/\(.Architecture)"' 2>/dev/null || echo "") if [ -n "$arch_info" ] && [ "$arch_info" != "/" ]; then log_error "Single-architecture manifest: $arch_info" else log_error "Single-architecture manifest" fi else log_error "Single-architecture manifest" fi

📚 Focus areas based on broader codebase context Scope Limitation The script only scans a hardcoded list of Dockerfiles and excludes FROM lines with --platform, which can miss multi-arch base usages in cross-build flows. Consider aligning with existing buildx/cross-Dockerfile patterns to ensure all build paths are validated. (Ref 2, Ref 4) # Extract all FROM statements with digests from specified Dockerfiles declare -A digests_map declare -A digests_files_map for dockerfile in "${DOCKERFILES[@]}"; do if [ ! -f "$dockerfile" ]; then continue fi # Get FROM statements with digests, excluding --platform lines from_lines=$(grep -E '^\s*FROM\s+.*@sha256:' "$dockerfile" | grep -v -E '^\s*FROM\s+--platform' || true) if [ -z "$from_lines" ]; then continue fi # Process each FROM line while IFS= read -r line; do if [[ -z "$line" ]]; then continue fi # Extract the image reference with digest image_ref=$(echo "$line" | sed -E 's/^\s*FROM\s+([^ ]+@sha256:[a-f0-9]+).*/\1/') if [ -n "$image_ref" ]; then # Mark that we've seen this digest digests_map["$image_ref"]=1 # Append this file to the list of files containing this digest if [ -n "${digests_files_map[$image_ref]:-}" ]; then digests_files_map["$image_ref"]="${digests_files_map[$image_ref]}|$dockerfile" else digests_files_map["$image_ref"]="$dockerfile" fi fi done <<< "$from_lines" done Reference reasoning: The repo builds images across multiple platforms and uses buildx and a generated Dockerfile.cross that injects --platform. The current script’s fixed file list and exclusion of --platform lines risks skipping those FROM statements, leading to incomplete validation compared to the multi-arch build approach shown in the Dockerfile and Makefile buildx flow.

📄 References redhat-developer/rhdh-operator/config/manager/deployment.yaml [25-47] redhat-developer/rhdh-operator/Dockerfile [1-34] redhat-developer/rhdh-operator/Makefile [225-248] redhat-developer/rhdh-operator/Makefile [249-256] redhat-developer/rhdh-operator/Makefile [327-329] redhat-developer/rhdh-operator/Makefile [331-365] redhat-developer/rhdh-operator/Makefile [29-44] redhat-developer/rhdh-operator/Makefile [310-326]

[rhdh-qodo-merge]

You are nearing your monthly Qodo Merge usage quota. For more information, please visit here.

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
High-level	Dynamically find Dockerfiles instead of hardcoding The validation script should dynamically find all Dockerfile and *.Dockerfile files using a command like find, instead of relying on a hardcoded list. This change would improve maintainability and ensure comprehensive validation across the repository. Examples: hack/validate-image-digests.sh [28-34] DOCKERFILES=( "Dockerfile" ".rhdh/docker/Dockerfile" ".rhdh/docker/bundle.Dockerfile" "bundle/rhdh/bundle.Dockerfile" "bundle/backstage.io/bundle.Dockerfile" ) Solution Walkthrough: Before: # hack/validate-image-digests.sh DOCKERFILES=( "Dockerfile" ".rhdh/docker/Dockerfile" ".rhdh/docker/bundle.Dockerfile" "bundle/rhdh/bundle.Dockerfile" "bundle/backstage.io/bundle.Dockerfile" ) for dockerfile in "${DOCKERFILES[@]}"; do if [ ! -f "$dockerfile" ]; then continue fi # ... validation logic ... done After: # hack/validate-image-digests.sh DOCKERFILES=() while IFS= read -r -d $'\0' file; do DOCKERFILES+=("$file") done < <(find . -type f \( -name "Dockerfile" -o -name "*.Dockerfile" \) -print0) for dockerfile in "${DOCKERFILES[@]}"; do # The check for file existence is no longer needed # as find only returns existing files. # ... validation logic ... done Suggestion importance[1-10]: 7 __ Why: The suggestion correctly identifies that the hardcoded list of Dockerfiles in hack/validate-image-digests.sh is brittle and may lead to incomplete validation as new Dockerfiles are added, which is a significant design improvement.	Medium
Possible issue	Fix regex to correctly handle ports Fix a greedy regex to correctly strip an image tag without removing the registry port from the image reference. hack/validate-image-digests.sh [119-125] # Skopeo doesn't support references with both tag and digest (image:tag@sha256:...) # Strip the tag if present, keeping only image@digest if [[ "$image_ref" =~ :[^@]+@sha256: ]]; then - image_ref_for_inspect=$(echo "$image_ref" | sed -E 's/:[^@]+(@sha256:)/\1/') + image_ref_for_inspect=$(echo "$image_ref" | sed -E 's/(.*):[^/:]+(@sha256:.*)/\1\2/') else image_ref_for_inspect="$image_ref" fi Apply / Chat Suggestion importance[1-10]: 7 __ Why: The suggestion correctly identifies a latent bug in the sed regex that would fail if an image reference contained a port, improving the script's robustness.	Medium
General	Avoid redundant API calls for efficiency Improve script efficiency by removing a redundant skopeo inspect call and reusing the already-fetched manifest data. hack/validate-image-digests.sh [163-172] if command -v jq &> /dev/null; then - arch_info=$(skopeo inspect "docker://$image_ref_for_inspect" 2>/dev/null | jq -r '"\(.Os)/\(.Architecture)"' 2>/dev/null || echo "") + arch_info=$(echo "$manifest_info" | jq -r '"\(.Os)/\(.Architecture)"' 2>/dev/null || echo "") if [ -n "$arch_info" ] && [ "$arch_info" != "/" ]; then log_error "Single-architecture manifest: $arch_info" else log_error "Single-architecture manifest" fi else log_error "Single-architecture manifest" fi Apply / Chat Suggestion importance[1-10]: 6 __ Why: The suggestion correctly points out a redundant skopeo inspect call and proposes reusing the previously fetched manifest, which improves script efficiency by avoiding an unnecessary network request.	Low
More

[github-actions]

PR images are available:

1. https://quay.io/rhdh-community/operator:0.9.0-pr-1882-706f86b
2. https://quay.io/rhdh-community/operator-bundle:0.9.0-pr-1882-706f86b
3. https://quay.io/rhdh-community/operator-catalog:0.9.0-pr-1882-706f86b

[github-actions]

PR images are available:

1. https://quay.io/rhdh-community/operator:0.9.0-pr-1882-143b579
2. https://quay.io/rhdh-community/operator-bundle:0.9.0-pr-1882-143b579
3. https://quay.io/rhdh-community/operator-catalog:0.9.0-pr-1882-143b579

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gazarenkov

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [gazarenkov]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment