[CI] Harden Dockerfile wget calls with retry logic#7011
Closed
tenpercent wants to merge 5 commits into
Closed
Conversation
Add --tries=3 --retry-connrefused --waitretry=5 to all GitHub-facing wget calls to handle transient network failures that were causing Docker build failures (e.g. develop #864, wget exit code 8). Also fixes ninja download to use unzip instead of gunzip, since ninja-linux.zip is a zip archive, not gzip. Generated-by: Claude Code (Claude-Opus-4.6)
- Add --retry-on-http-error=502,503,504 to all 4 wget calls to handle transient GitHub HTTP errors (not just connection-refused) - Move dumb-init download to /tmp/ for consistency with other downloads Generated-by: Claude Code (Claude-Opus-4.6)
Add `optional: true` to the copyArtifacts call in generateAndArchiveBuildTraceVisualization so it silently succeeds when no trace file was produced (e.g. NINJA_BUILD_TRACE=false or gfx950 stage skipped). Previously the missing-artifact error was caught but copyArtifacts without optional: true throws before the catch can intercept it on some plugin versions, causing post-stage failures on every gfx950 branch build. Generated-by: Claude Code (claude-sonnet-4-6)
Set gc.autoDetach=true on the ref repo so git gc forks to background during remote update and cannot block or fail the checkout. Also clean up stale gc.log files older than 2 hours left by previously killed gc processes, so agents don't permanently stop repacking. Addresses recurring git_pack_objects_killed failures on ctr2-mlse-smc-alola-s98-* and jenkins-trex-vg-21 agents. Generated-by: Claude Code (claude-sonnet-4-6)
amd-anclark
approved these changes
May 4, 2026
Contributor
|
This pull request has been inactive for 25 days and will be marked as stale. If you would like to keep this PR open, please:
This PR will be automatically closed in 5 days if no further activity occurs. |
Contributor
|
This pull request has been inactive for 25 days and will be marked as stale. If you would like to keep this PR open, please:
This PR will be automatically closed in 5 days if no further activity occurs. |
Contributor
|
This pull request has been automatically closed due to inactivity (30 days with no updates). If you'd like to continue working on this, feel free to reopen the PR or create a new one. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--tries=3 --retry-connrefused --waitretry=5to all 4 GitHub-facingwgetcalls in the Dockerfile to handle transient network failuresunzipinstead ofgunzip—ninja-linux.zipis a zip archive, not gzip/tmpfor intermediate downloads and clean up temp filesoptional: truetocopyArtifactsingenerateAndArchiveBuildTraceVisualizationso the post-stage doesn't fail when no build trace JSON was produced (e.g.NINJA_BUILD_TRACE=falseor gfx950 stage skipped)gc.autoDetach=trueon the ref repo so git gc runs in the background duringgit remote updateand cannot block or fail the checkout. Clean up stalegc.logfiles older than 2 hours to prevent agents from permanently skipping gc after a killed pack-objectsMotivation
CI build develop #864 failed because
wgetgot exit code 8 (server error) downloading sccache from GitHub releases during Docker image build. No retry logic existed on any of the external downloads.Every branch build was hitting "Copied 0 artifacts" failures in the post-stage because
copyArtifacts(withoutoptional: true) throws when the filter matches nothing, andck_build_trace_*.jsonis only produced whenNINJA_BUILD_TRACE=true.Recurring
git_pack_objects_killedfailures onctr2-mlse-smc-alola-s98-*andjenkins-trex-vg-21agents caused bygit gcrunning in the foreground during ref repo update, getting killed by SIGTERM (timeout/resource limit), and cascading to 26-28 stage failures.Test plan
--build-arg compiler_version=therockNINJA_BUILD_TRACE=false🤖 Generated with Claude Code