fix(software): retry truncated/empty downloads and report real checksum algorithm

[alex-au]

Description

Software downloads (pkg/software/downloader.go) failed the whole workflow when a
download came back truncated or empty, with no retry. Download did a single http.Get,
checked only StatusCode == 200, then io.Copy'd the body to disk — so a 200-with-empty-body,
a silently truncated copy, or a mid-stream reset produced a corrupt file that the checksum
verifier correctly rejected but the run could not recover from. Observed in the UAT Purge
Storage job on PR #785 during setup-crio:

software.checksum_error: checksum verification failed for file
  '/opt/solo/weaver/downloads/cri-o.amd64.v1.33.4.tar.gz' using algorithm 'unknown'
  [ expected = 8f6d1482...b54e50
    actual   = e3b0c442...b855 ]   # sha256 of a zero-byte file

Downloads are now self-healing for transient corruption. Download validates the response
before trusting it (rejects zero-byte bodies, and — when Content-Length is advertised — fails
on a byte-count mismatch instead of persisting a truncated file). A new
Downloader.DownloadAndVerify wraps download + checksum in a bounded retry loop: both download
errors and a subsequent checksum mismatch are treated as retryable, the bad file is deleted
between attempts, and it gives up after maxAttempts (default 3) with capped exponential
backoff. The three base_installer.go call sites now use it.

The misleading algorithm 'unknown' in checksum-mismatch errors is also fixed — the real
algorithm (sha256, etc.) is now threaded through the checksum helper.

Retry uses a small hand-rolled loop matching the repo's existing ProbeTCP style rather than
pulling in cenkalti/backoff (which is only a transitive go.sum entry, not a vendored/used
dependency).

Files changed

File	Change
pkg/software/downloader.go	Download rejects zero-byte / Content-Length-mismatched responses; new DownloadAndVerify retry loop + WithMaxAttempts/WithRetryDelay options
pkg/software/integrity_checker.go	Thread the real algorithm into the checksum helper so mismatch errors no longer report unknown
pkg/software/base_installer.go	Replace the three Download+VerifyChecksum pairs (archive/binary/config) with DownloadAndVerify
pkg/software/downloader_test.go	Tests: zero-byte + Content-Length-mismatch rejection, retry-then-succeed, retry-exhaustion
pkg/software/integrity_checker_test.go	Assert mismatch error names the real algorithm, never unknown

Review guide

Code review checklist

• downloader.go Download — zero-byte check fires before the Content-Length check; the Content-Length comparison is guarded on resp.ContentLength >= 0 so chunked/unknown-length responses (-1) are not falsely rejected.
• downloader.go DownloadAndVerify — the bad file is removed on every failed attempt (via base-path-validated os.Remove), so a corrupt partial never persists; the last error is returned once attempts are exhausted.
• downloader.go sleepBackoff — delay is capped at maxRetryDelay and the shift-overflow guard (delay < fd.retryDelay) holds; a zero retryDelay disables sleeping (used by tests).
• Truncation/empty errors reuse NewDownloadError wrapping an errorx.ExternalError cause — they stay software.download_error (retryable), no new errorx namespace.
• integrity_checker.go — all three checksum(...) calls pass the matching algorithm string; both NewChecksumError sites use it.
• base_installer.go — the pre-existing-file skip/delete logic is untouched; only the download+verify pair was collapsed.

Unit tests

# macOS-safe (pkg/software has no Linux-only deps):
go test -race -tags='!integration' ./pkg/software/... \
  -run 'Test_Downloader_Download_ZeroByte|Test_Downloader_Download_ContentLengthMismatch|Test_Downloader_DownloadAndVerify|Test_IntegrityChecker_VerifyChecksum'

# Or the full suite (run in the UTM VM for complete coverage):
task vm:test:unit

Manual UAT

The failure is inherently transient (CDN/network hiccup), so it is exercised by the
Test_Downloader_DownloadAndVerify_RetriesThenSucceeds test, which stands up an httptest
server that returns an empty body on the first request and the correct payload on the second,
and asserts the download recovers without failing. ..._ExhaustsAttempts asserts a
persistently-corrupt file still fails after 3 tries with the file removed.

Risks / rollback

A persistently-unreachable or invalid URL now takes up to 3 attempts (plus a few seconds of
backoff) before failing, instead of failing immediately — bounded, and only on the error path.
The change is additive to the download path; revert is dropping DownloadAndVerify and
restoring the original Download+VerifyChecksum call-site blocks.

Related Issues

• Closes Software downloads should retry on truncated/empty files instead of failing the run #786

[swirlds-automation]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[Copilot]

Pull request overview

This PR improves the resiliency and debuggability of pkg/software downloads by (1) detecting empty/truncated HTTP responses earlier, (2) adding a bounded retry loop that retries both download failures and checksum mismatches, and (3) ensuring checksum-mismatch errors report the actual algorithm used (e.g., sha256 instead of unknown).

Changes:

• Harden Downloader.Download to reject zero-byte bodies and advertised Content-Length mismatches.
• Add Downloader.DownloadAndVerify with capped exponential backoff and configurable retry settings.
• Thread the checksum algorithm through integrity checking so error messages report the real algorithm; update installer call sites and tests accordingly.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
pkg/software/downloader.go	Adds retry/backoff + download validation for empty/truncated responses.
pkg/software/integrity_checker.go	Propagates the actual checksum algorithm into mismatch errors.
pkg/software/base_installer.go	Switches archive/binary/config downloads to the new DownloadAndVerify path.
pkg/software/downloader_test.go	Adds unit tests covering zero-byte rejection and retry behavior.
pkg/software/integrity_checker_test.go	Adds assertions that mismatch errors name the real algorithm (not unknown).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[brunodam]

This is great!

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.