Skip to content

Commit 085f02b

Browse files
PhilWindleaztec-bot
authored andcommitted
fix(ci): retry forge solc fetch on transient DNS failures (#24358)
## Problem PR #24352 (and others) intermittently fail in the merge queue with a DNS error. The failing step is the `barretenberg/sol` test command: ``` Command: ...cd barretenberg/sol && forge test --no-match-contract Base Error: error sending request for url (https://binaries.soliditylang.org/linux-amd64/list.json) Context: - Error #0: client error (Connect) - Error #1: dns error - Error #2: failed to lookup address information: Temporary failure in name resolution ``` `forge` reaches out to `binaries.soliditylang.org` to resolve/fetch the solc binary. Under heavy parallel merge-queue load the VPC resolver intermittently drops lookups (`Temporary failure in name resolution`), and the operation fails with no retry, taking the whole CI run down with it. ## Fix Retry the solc-fetching `forge` operations **every 10s for ~5 min** (30 attempts), scoped to connection/DNS failures so genuine errors still fail fast. - **`ci3/retry`**: make the attempt count tunable via `RETRY_ATTEMPTS` (default unchanged at 3); `RETRY_SLEEP` was already honored (default 5s). Fully backward-compatible — existing callers are unaffected. - **`barretenberg/sol/bootstrap.sh`**: wrap the `forge test` test command in `RETRY_ATTEMPTS=30 RETRY_SLEEP=10 retry -p '<dns/connect regex>'`. This is the operation that fails in the merge queue. - **`l1-contracts/bootstrap.sh`**: the root svm download (the "single owner of the svm download") already had a short `retry`; extend it to the same 10s/5-min window and scope it to DNS/connection errors. (`2>/dev/null` removed so `retry`'s pattern matcher can see the DNS error on stderr.) The `-p` pattern (`dns error|Temporary failure in name resolution|error sending request|failed to lookup address|Connection refused|connection reset`) means a real `forge test` assertion failure or compile error is **not** retried — it exits immediately, so this never masks real failures or wastes 5 minutes on a genuinely broken test. ## Verification Exercised `ci3/retry` and the exact emitted `barretenberg/sol` command locally: - DNS-matching failure → retries up to `RETRY_ATTEMPTS`, sleeping `RETRY_SLEEP` between attempts. - Real test failure (no DNS text) → fails fast, no retry (0s). - Success → returns 0 on the first attempt (0s). - `RETRY_ATTEMPTS` unset → still 3 attempts (backward-compatible). --- *Created by [claudebox](https://claudebox.work/v2/sessions/5610bcb5aef4d0b9) · group: `slackbot`*
1 parent bba02bf commit 085f02b

3 files changed

Lines changed: 15 additions & 6 deletions

File tree

barretenberg/sol/bootstrap.sh

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,11 @@ function test_cmds {
1919
}
2020

2121
function test_cmds_internal {
22-
echo "cd barretenberg/sol && forge test --no-match-contract Base"
22+
# forge reaches out to binaries.soliditylang.org to resolve/fetch solc, which intermittently
23+
# fails DNS resolution under heavy parallel merge-queue load ("Temporary failure in name
24+
# resolution"). Retry every 10s for ~5 min, but only on connection/DNS failures so genuine test
25+
# failures still fail fast.
26+
echo "cd barretenberg/sol && RETRY_ATTEMPTS=30 RETRY_SLEEP=10 retry -p 'dns error|Temporary failure in name resolution|error sending request|failed to lookup address|Connection refused|connection reset' 'forge test --no-match-contract Base'"
2327
}
2428

2529
function build_sol {

ci3/retry

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@ set -u # not -e
55
# By default retries any non-zero exit. With "-p <regex>" only failures whose (combined) output
66
# matches the extended regex are retried; any other failure exits immediately so genuine errors
77
# (e.g. a real compile error) are not masked by retries.
8+
# The attempt count and the inter-attempt sleep are tunable via RETRY_ATTEMPTS (default 3) and
9+
# RETRY_SLEEP (default 5s), e.g. RETRY_ATTEMPTS=30 RETRY_SLEEP=10 retries every 10s for ~5 min.
810
# Usage: retry [-p <retry-on-regex>] "<command>"
911
pattern=
1012
if [ "${1:-}" = "-p" ]; then
@@ -18,10 +20,10 @@ if [ -n "${RETRY_DISABLED:-}" ]; then
1820
exit
1921
fi
2022

21-
ATTEMPTS=3
23+
ATTEMPTS=${RETRY_ATTEMPTS:-3}
2224
out=
2325
[ -n "$pattern" ] && out=$(mktemp)
24-
# Retries up to 3 times with 5 second intervals
26+
# Retries up to $ATTEMPTS times, sleeping ${RETRY_SLEEP:-5}s between attempts.
2527
trap 'kill -SIGTERM $pid &>/dev/null || true; [ -n "$out" ] && rm -f "$out"' SIGTERM EXIT
2628
for i in $(seq 1 $ATTEMPTS); do
2729
if [ -n "$pattern" ]; then

l1-contracts/bootstrap.sh

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,12 @@ function download_solc {
2020
mkdir -p "$HOME/.svm"
2121
# We build a minimal file to trigger svm download of solc. svm fetches the
2222
# binary from binaries.soliditylang.org, which intermittently fails to resolve
23-
# under heavy parallel CI load; retry to ride out transient DNS drops. (The
24-
# merge queue disables the cache above, so this download path runs every time.)
25-
retry "forge build --use \"$solc_version\" src/core/libraries/ConstantsGen.sol 2>/dev/null"
23+
# under heavy parallel CI load; retry every 10s for ~5 min to ride out transient
24+
# DNS drops, but only on connection/DNS failures so a genuine build error fails
25+
# fast. (The merge queue disables the cache above, so this download path runs
26+
# every time. stderr is kept so retry can see the DNS error and match on it.)
27+
RETRY_ATTEMPTS=30 RETRY_SLEEP=10 retry -p 'dns error|Temporary failure in name resolution|error sending request|failed to lookup address|Connection refused|connection reset' \
28+
"forge build --use \"$solc_version\" src/core/libraries/ConstantsGen.sol"
2629

2730
# Copy from svm cache to local path
2831
local svm_path="$HOME/.svm/$solc_version/solc-$solc_version"

0 commit comments

Comments
 (0)