Skip rebuilds when Docker Hub tag check is inconclusive (#357)

## Summary

The scheduled workflow's image-existence check silently defaults to
"rebuild" whenever the Docker Hub tags API returns anything other than a
parseable `{results: [...]}` body. Rate limits, 5xx, connection resets,
and non-JSON edge responses all collapse to the same "tag missing"
signal, so successful builds get re-queued an hour later even when the
upstream source hasn't moved.

Concrete example that triggered this PR: run
[24721350367](https://github.com/ethpandaops/eth-client-docker-image-builder/actions/runs/24721350367)
queued a full rebuild of `ethpandaops/nimbus-eth2:epbs-devnet-1-a23ebfd`
and its `-minimal` variant even though:
- `status-im/nimbus-eth2#epbs-devnet-1` HEAD hadn't moved in a week
(`a23ebfd`, committed 2026-04-14).
- Both tags were already on Docker Hub from a prior run the same day
(pushed 10:47 UTC).

## What changed

`.github/workflows/scheduled.yml` → `process_image()`:
- Captures HTTP status separately from body via `curl -w
'\n%{http_code}' --max-time 20`.
- Classifies the response:
- `200` with a well-formed `.results` array → authoritative, compute
`exists` from an **exact** `.name == $tag` match (was a substring match,
which could have masked a missing tag with a prefix-matching sibling).
- `404` with `{"message":"object not found"}` → repo doesn't exist yet
on Hub → build (first push).
- Anything else (non-JSON, 429, 5xx, timeout, empty body) → retry up to
3 times with 2/4/6s backoff.
- On persistent failure emits `exists:"unknown"` and logs a WARN,
instead of silently falling back to "rebuild".

Build-decision loop:
- Only queues a build when `images[$IMAGE] == "false"` (authoritative
empty).
- `"unknown"` and missing entries log a skip line and are passed over.

## Verification

Ran the new `process_image()` against the live Docker Hub API for six
scenarios:

| Case | Expected | Got |
|---|---|---|
| Exact tag present | `true` | `true` |
| Tag genuinely absent (200, empty results) | `false` | `false` |
| Repo missing on Hub (404 object not found) | `false` | `false` |
| Unreachable host | `unknown` after 3 retries | `unknown` |
| Non-JSON 4xx body | `unknown` after 3 retries | `unknown` |
| Substring-only match (would have fooled old logic) | `false` | `false`
|

Also re-ran the full check path for
`status-im/nimbus-eth2#epbs-devnet-1` and the `-minimal` variant against
the live API — both now correctly resolve to `exists:"true"` → skip
build, matching reality.

`bash -n` on the extracted embedded script passes.

## Test plan

- [ ] Next scheduled run completes without queuing nimbus-eth2
`epbs-devnet-1` (or any other already-present tag).
- [ ] If Docker Hub returns transient errors during a future run, log
shows `Skipping …: Docker Hub existence check was inconclusive.` instead
of a queued rebuild.
- [ ] First build for a brand-new client (Docker Hub repo not yet
created) still triggers correctly via the 404 path.

Signed-off-by: Barnabas Busa <busa.barnabas@gmail.com>

Author: barnabasbusa

.github/workflows/scheduled.yml

@@ -84,19 +84,55 @@ jobs:
               local LINE=$1
               local IMAGE=$2
               local URL=$3
+              local IMAGE_TAG=$4
 
               local imageOutput="${TEMP_DIR}/${LINE}_image.json"
               touch $imageOutput
-              local exists=$(curl -s $URL | jq '.results | length > 0')
 
-              # check if exists == true
-              if [ "$exists" == "true" ]; then
-                exists=true
-              else
-                exists=false
+              # Classify the Docker Hub response so we only trigger a build on an
+              # authoritative "tag missing" answer. Retries absorb transient
+              # failures (rate limits, 5xx, connection resets, non-JSON bodies).
+              local exists="unknown"
+              local attempt response http_code body count
+              for attempt in 1 2 3; do
+                response=$(curl -s --max-time 20 -w $'\n%{http_code}' "$URL")
+                if [ $? -ne 0 ] || [ -z "$response" ]; then
+                  sleep $((attempt * 2))
+                  continue
+                fi
+                http_code=$(printf '%s' "$response" | tail -n1)
+                body=$(printf '%s' "$response" | sed '$d')
+
+                if [ "$http_code" = "200" ]; then
+                  # Require a well-formed tags payload and exact-match the tag
+                  # name so a substring hit (e.g. `foo-abc` vs `foo-abcdef`)
+                  # can't mask a missing tag.
+                  count=$(echo "$body" | jq -e --arg tag "$IMAGE_TAG" '[.results[]? | select(.name == $tag)] | length' 2>/dev/null)
+                  if [ -n "$count" ]; then
+                    if [ "$count" -gt 0 ]; then
+                      exists=true
+                    else
+                      exists=false
+                    fi
+                    break
+                  fi
+                elif [ "$http_code" = "404" ]; then
+                  # Repository itself doesn't exist yet on Docker Hub — first build needs to push.
+                  if echo "$body" | jq -e '.message == "object not found"' >/dev/null 2>&1; then
+                    exists=false
+                    break
+                  fi
+                fi
+
+                echo "[LINE:$LINE] Docker Hub check attempt $attempt inconclusive for ${IMAGE} (http=$http_code), retrying." >&2
+                sleep $((attempt * 2))
+              done
+
+              if [ "$exists" = "unknown" ]; then
+                echo "[LINE:$LINE] WARN: Docker Hub check inconclusive for ${IMAGE} after retries, skipping build." >&2
               fi
 
-              echo "{\"line\": \"$LINE\", \"image\": \"$IMAGE\", \"exists\": $exists}" >> $imageOutput
+              echo "{\"line\": \"$LINE\", \"image\": \"$IMAGE\", \"exists\": \"$exists\"}" >> $imageOutput
           }
 
           # Get commit hashes for each configuration in parallel
@@ -126,7 +162,7 @@ jobs:
               IMAGE_TAG="${TARGET_TAG}-${COMMIT_HASH}"
               IMAGE="${TARGET_REPOSITORY}:${IMAGE_TAG}"
               URL="https://hub.docker.com/v2/repositories/${TARGET_REPOSITORY}/tags?page_size=25&page=1&ordering=&name=${IMAGE_TAG}"
-              process_image $LINE $IMAGE $URL &
+              process_image "$LINE" "$IMAGE" "$URL" "$IMAGE_TAG" &
           done < <(yq -r 'to_entries | map_values({"value":.value, "index":.key}) | .[] | [.index, .value.source.repository, .value.source.ref, .value.target.repository, .value.target.tag] | @tsv' "$CONFIG_FILE")
 
           wait
@@ -154,8 +190,13 @@ jobs:
               IMAGE="${TARGET_REPOSITORY}:${IMAGE_TAG}"
               CLIENT="${TARGET_REPOSITORY#*/}"
 
-              # Build if image doesn't exist OR if we couldn't determine existence (fail-safe)
-              if [ -z "${images[$IMAGE]}" ] || [ "${images[$IMAGE]}" == "false" ]; then
+              # Only build when Docker Hub authoritatively reported the tag as missing.
+              # "unknown" (check failed after retries) or missing entries are skipped to
+              # avoid burning runner time on rebuilds that the existence check couldn't verify.
+              if [ "${images[$IMAGE]}" != "true" ] && [ "${images[$IMAGE]}" != "false" ]; then
+                echo "[LINE:$LINE] Skipping ${IMAGE}: Docker Hub existence check was inconclusive."
+              fi
+              if [ "${images[$IMAGE]}" == "false" ]; then
                 # Handle platforms and runners, ensuring output files are created even if empty
                 platforms=$(yq e ".$CLIENT[]" "$PLATFORMS_FILE")
                 platformsArr=""