Skip to content

Commit 95a802b

Browse files
committed
ci: make download-and-unpack retry-safe against truncated responses
`wget -qO- | tar xzf -` can't recover from a half-finished download — the partial bytes are already in tar's stdin by the time wget's retry logic could fire, and tar bails with "gzip: stdin: unexpected end of file" before anything restarts. We've seen this hit one cell of a 22-cell matrix run on libcurl 8.17.0's tarball (run 26722785133), with the other 21 cells succeeding on the same input — classic curl.se / GitHub-releases flake. Stage the download to a temp file first so wget can retry until the file is whole, then untar from the file. Also pass --tries, --timeout and --retry-connrefused so transient network errors don't fall through to a hard failure on the first attempt. This is also incidentally happening as I write this commit — curl.se is serving the tarball at ~1MB/s today, which would make the previous script's per-attempt timeout window much more likely to clip.
1 parent aec2b99 commit 95a802b

1 file changed

Lines changed: 16 additions & 4 deletions

File tree

scripts/ci/download-and-unpack.sh

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,22 @@ set -euo pipefail
55
# download tar gz file from source_url and unpack it to destination
66
# download_and_upack <source_url> <destination>
77
download_and_unpack() {
8-
mkdir -p $2
9-
# User agent for Edge on macOS
10-
wget -U "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36 Edg/94.0.992.38" \
11-
-qO- $1 | tar xzf - -C $2
8+
mkdir -p "$2"
9+
# Download to a temp file first instead of streaming straight into tar.
10+
# The previous `wget -qO- | tar xzf -` pipe couldn't recover from a
11+
# truncated response (e.g. curl.se/GitHub-release occasional hiccups) —
12+
# tar consumes the partial bytes immediately and fails with
13+
# "gzip: stdin: unexpected end of file" before wget's own retry can fire.
14+
# With a temp file, wget can retry until the file is whole, and we only
15+
# touch tar once.
16+
local tmpfile
17+
tmpfile=$(mktemp)
18+
trap "rm -f \"$tmpfile\"" RETURN
19+
# User agent for Edge on macOS
20+
wget --tries=5 --waitretry=3 --timeout=30 --retry-connrefused \
21+
-U "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36 Edg/94.0.992.38" \
22+
-qO "$tmpfile" "$1"
23+
tar xzf "$tmpfile" -C "$2"
1224
}
1325

1426
if [ "${1}" != "--source-only" ]; then

0 commit comments

Comments
 (0)