Skip to content

Commit 84bc75a

Browse files
joshuacolvin0claude
joshuacolvin0
and
claude
committed
Overhaul CI setup caching and harden build reproducibility
Rework the ci-setup composite action to add proper caching for node_modules, solidity build artifacts, Go build/modules, and cbrotli, with validation and corruption detection on cache restore. Key changes: - Add dedicated cache steps for node_modules, solidity, Go, and cbrotli with carefully designed cache keys that include tool versions and all relevant source/config files via hashFiles() - Add self-validating cache key inputs: extract hashFiles() patterns from the action YAML and verify each pattern matches at least one file - Add cache-hit validation that checks cached directories exist and are non-empty before touching Make sentinels to skip rebuilds - Handle solidity/node_modules cache split-brain: purge solidity artifacts when node_modules cache is evicted to ensure consistent rebuilds - Add RETRY macro for npm/yarn installs to handle transient registry errors, with optional node_modules cleanup between attempts - Pin cbindgen version and skip reinstall when cached version matches - Use frozen lockfiles (npm ci / yarn --frozen-lockfile) in CI - Centralize node_modules and solidity directory lists as single source of truth consumed by both cache steps and validation - Add lint step that warns about cache-sensitive files not covered by any hashFiles() pattern - Harden install-rust action with set -euo pipefail and quoted outputs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent e89bc08 commit 84bc75a

4 files changed

Lines changed: 359 additions & 17 deletions

File tree

‎.github/actions/ci-setup/action.yml‎

Lines changed: 332 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -7,25 +7,94 @@ runs:
77

88
steps:
99
- name: Install dependencies
10-
run: sudo apt update && sudo apt install -y wabt gotestsum
10+
run: |
11+
set -euo pipefail
12+
sudo apt-get update && sudo apt-get install -y wabt gotestsum
13+
shell: bash
14+
15+
- name: Set Node.js version
16+
id: node-version
17+
run: echo "major=24" >> "$GITHUB_OUTPUT"
18+
shell: bash
19+
20+
- name: Set Foundry version
21+
id: foundry-version
22+
run: echo "version=v1.0.0" >> "$GITHUB_OUTPUT"
23+
shell: bash
24+
25+
# Single source of truth for node_modules directories. Consumed by
26+
# cache-node-modules and the validation step. Add new workspaces ONLY here.
27+
- name: Set node_modules directory list
28+
run: |
29+
set -euo pipefail
30+
dirs=(
31+
contracts/node_modules
32+
contracts-legacy/node_modules
33+
safe-smart-account/node_modules
34+
)
35+
{
36+
echo "NODE_MODULES_CACHE_PATHS<<NM_EOF"
37+
printf '%s\n' "${dirs[@]}"
38+
echo "NM_EOF"
39+
} >> "$GITHUB_ENV"
40+
echo "NODE_MODULES_DIRS=${dirs[*]}" >> "$GITHUB_ENV"
41+
shell: bash
42+
43+
# Single source of truth for solidity cache directories. Consumed by
44+
# cache-solidity (SOLIDITY_CACHE_PATHS) and the validate/purge step
45+
# (SOLIDITY_DIRS). Add new solidity cache directories ONLY here.
46+
- name: Set solidity cache directory list
47+
run: |
48+
set -euo pipefail
49+
# Space-separated; used by SOLIDITY_DIRS (word splitting) and
50+
# SOLIDITY_CACHE_PATHS (newline-separated for actions/cache).
51+
dirs=(
52+
contracts/build
53+
contracts/out
54+
contracts-legacy/build
55+
contracts-legacy/out
56+
contracts-local/out
57+
safe-smart-account/build
58+
solgen/go
59+
)
60+
{
61+
echo "SOLIDITY_CACHE_PATHS<<SOLIDITY_EOF"
62+
printf '%s\n' "${dirs[@]}"
63+
echo "SOLIDITY_EOF"
64+
} >> "$GITHUB_ENV"
65+
echo "SOLIDITY_DIRS=${dirs[*]}" >> "$GITHUB_ENV"
1166
shell: bash
1267

68+
# setup-node only caches yarn's global tarball cache, not node_modules.
69+
# No restore-keys: for safe-smart-account, npm ci wipes and reinstalls
70+
# node_modules (wasting any partial restore); for yarn workspaces,
71+
# additive installs over a stale restore could leave orphaned packages.
72+
- name: Cache node_modules
73+
id: cache-node-modules
74+
uses: actions/cache@v5
75+
with:
76+
path: ${{ env.NODE_MODULES_CACHE_PATHS }}
77+
key: ${{ runner.os }}-node-modules-node${{ steps.node-version.outputs.major }}-${{ hashFiles('contracts/yarn.lock', 'contracts-legacy/yarn.lock', 'safe-smart-account/package-lock.json') }}
78+
1379
- name: Setup Node.js
1480
uses: actions/setup-node@v5
1581
with:
16-
node-version: "24"
82+
node-version: ${{ steps.node-version.outputs.major }}
1783
cache: yarn
1884
cache-dependency-path: "**/yarn.lock"
1985

2086
- name: Setup Go
87+
id: setup-go
2188
uses: actions/setup-go@v6
2289
with:
2390
go-version-file: "go.mod"
91+
cache: false # See "Cache Go build and modules" step for restore-keys fallback
2492

2593
- name: Install wasm-ld
2694
run: |
27-
sudo apt-get update && sudo apt-get install -y lld-14
28-
sudo ln -s /usr/bin/wasm-ld-14 /usr/local/bin/wasm-ld
95+
set -euo pipefail
96+
sudo apt-get install -y lld-14
97+
sudo ln -sf /usr/bin/wasm-ld-14 /usr/local/bin/wasm-ld
2998
shell: bash
3099

31100
- name: Install rust
@@ -35,20 +104,209 @@ runs:
35104
- name: Setup Foundry
36105
uses: foundry-rs/foundry-toolchain@v1
37106
with:
38-
cache: false
39-
version: v1.0.0
107+
cache: true
108+
version: ${{ steps.foundry-version.outputs.version }}
40109

41-
- name: Install cbindgen
42-
run: cargo install --force cbindgen
110+
- name: Validate cache key inputs
111+
env:
112+
GO_VERSION: ${{ steps.setup-go.outputs.go-version }}
113+
RUST_VERSION: ${{ steps.install-rust.outputs.version }}
114+
FOUNDRY_VERSION: ${{ steps.foundry-version.outputs.version }}
115+
NODE_VERSION: ${{ steps.node-version.outputs.major }}
116+
run: |
117+
set -euo pipefail
118+
# Version strings used in cache keys must not be empty
119+
for pair in "Go:$GO_VERSION" "Rust:$RUST_VERSION" "Foundry:$FOUNDRY_VERSION" "Node:$NODE_VERSION"; do
120+
name="${pair%%:*}"; val="${pair#*:}"
121+
if [ -z "$val" ]; then
122+
echo "ERROR: $name version is empty -- cache keys will collide across versions"; exit 1
123+
fi
124+
if ! [[ "$val" =~ ^[a-zA-Z0-9._v-]+$ ]]; then
125+
echo "ERROR: $name version looks malformed: '$val' -- cache keys may not work correctly"; exit 1
126+
fi
127+
done
128+
# Auto-extract hashFiles() arguments from this file so the validation
129+
# stays in sync without maintaining a separate list.
130+
shopt -s globstar nullglob
131+
self="${GITHUB_ACTION_PATH}/action.yml"
132+
args=$(grep -oP "hashFiles\(\K[^)]+(?=\))" "$self" | tr "," "\n" | sed "s/^[[:space:]]*'//;s/'[[:space:]]*$//" | sort -u)
133+
# Fail-closed: if the regex stops matching (e.g., multi-line reformat),
134+
# surface it immediately rather than silently skipping validation.
135+
if [ -z "$args" ]; then
136+
echo "ERROR: no hashFiles() patterns extracted from $self -- grep regex may need updating"; exit 1
137+
fi
138+
# The line-by-line regex silently misses multi-line hashFiles() calls.
139+
# Guard against this by comparing call count vs extracted group count.
140+
# Only count non-comment YAML lines with ${{ to exclude shell and comments.
141+
expected=$(grep -P '^\s*[^#].*\$\{\{.*hashFiles\(' "$self" | grep -cP "hashFiles\(")
142+
actual=$(grep -oP "hashFiles\(\K[^)]+(?=\))" "$self" | wc -l)
143+
if [ "$actual" -ne "$expected" ]; then
144+
echo "ERROR: found $expected hashFiles() calls but only extracted $actual -- possible multi-line hashFiles() call"; exit 1
145+
fi
146+
# Export for the lint step so the regex is not duplicated.
147+
{
148+
echo "HASHFILES_PATTERNS<<HASHFILES_EOF"
149+
echo "$args"
150+
echo "HASHFILES_EOF"
151+
} >> "$GITHUB_ENV"
152+
# Disable globbing during iteration so nullglob does not silently
153+
# remove unmatched patterns from the list; re-enable inside the
154+
# body for intentional expansion.
155+
set -f
156+
for pattern in $args; do
157+
set +f
158+
# shellcheck disable=SC2206 # intentional glob expansion (globstar enabled above)
159+
matches=( $pattern )
160+
set -f
161+
if [ ${#matches[@]} -eq 0 ]; then
162+
echo "ERROR: hashFiles pattern '$pattern' matches nothing -- update .github/actions/ci-setup/action.yml"; exit 1
163+
fi
164+
done
165+
set +f
43166
shell: bash
44167

45-
- name: Cache Go build
168+
# Best-effort lint: warn about cache-sensitive files not covered by
169+
# any hashFiles() pattern. Produces ::warning annotations only --
170+
# never fails the build. Separate from validation because this is
171+
# advisory, not a correctness gate.
172+
- name: Lint cache key coverage
173+
run: |
174+
set -euo pipefail
175+
shopt -s globstar nullglob
176+
# Reuse patterns extracted by the validate step (avoid duplicating the regex).
177+
args="${HASHFILES_PATTERNS:-}"
178+
if [ -z "$args" ]; then
179+
echo "::warning::HASHFILES_PATTERNS is empty -- validate step may have failed or been skipped"; exit 0
180+
fi
181+
# shellcheck disable=SC2317 # function used below
182+
file_covered_by() {
183+
local file="$1"; shift
184+
for hp in "$@"; do
185+
set +f
186+
# shellcheck disable=SC2206 # intentional glob expansion (globstar enabled)
187+
local expanded=( $hp )
188+
set -f
189+
local m
190+
for m in "${expanded[@]}"; do
191+
if [ "$m" = "$file" ]; then return 0; fi
192+
done
193+
done
194+
return 1
195+
}
196+
skip_re='^(.*node_modules.*|.*/lib/.*|.*/build/.*|.*/out/.*|.*/dist/.*|nitro-testnode/.*|go-ethereum/.*|crates/.*|.*typechain.*|.*/certora/.*)$'
197+
# Pre-expand args into an array with globbing off so patterns are
198+
# preserved as literals for file_covered_by to expand on demand.
199+
set -f
200+
# shellcheck disable=SC2206 # intentional word splitting
201+
args_arr=( $args )
202+
set +f
203+
# Check lockfiles, configs, and build files.
204+
for file in **/yarn.lock **/package-lock.json **/foundry.toml **/hardhat.config.ts; do
205+
[[ "$file" =~ $skip_re || "$file" == */src/* ]] && continue
206+
file_covered_by "$file" "${args_arr[@]}" ||
207+
echo "::warning::$file is not in any hashFiles() cache key in .github/actions/ci-setup/action.yml -- may need adding"
208+
done
209+
# Check for .sol source files not covered by any hashFiles() glob.
210+
sol_patterns=()
211+
for p in "${args_arr[@]}"; do
212+
[[ "$p" == *.sol ]] && sol_patterns+=("$p")
213+
done
214+
if [ ${#sol_patterns[@]} -gt 0 ]; then
215+
uncovered_sol=()
216+
for file in **/*.sol; do
217+
[[ "$file" =~ $skip_re || "$file" == */test/* ]] && continue
218+
file_covered_by "$file" "${sol_patterns[@]}" || uncovered_sol+=("$file")
219+
done
220+
if [ ${#uncovered_sol[@]} -gt 0 ]; then
221+
echo "::warning::${#uncovered_sol[@]} .sol file(s) not covered by any hashFiles() sol glob -- may need adding: ${uncovered_sol[*]}"
222+
fi
223+
fi
224+
shell: bash
225+
226+
# restore-keys prefix fallback: Go build cache is content-addressed
227+
# (stale entries ignored), module cache self-heals by re-downloading.
228+
- name: Cache Go build and modules
229+
id: cache-go
230+
uses: actions/cache@v5
231+
with:
232+
path: |
233+
~/.cache/go-build
234+
~/go/pkg/mod
235+
key: ${{ runner.os }}-go-${{ steps.setup-go.outputs.go-version }}-${{ hashFiles('go.mod', 'go.sum') }}
236+
restore-keys: |
237+
${{ runner.os }}-go-${{ steps.setup-go.outputs.go-version }}-
238+
239+
# No restore-keys: partial Solidity restores (stale Go bindings with
240+
# new ABIs) would produce inconsistent artifacts.
241+
- name: Cache contract build artifacts
242+
id: cache-solidity
46243
uses: actions/cache@v5
47244
with:
48-
path: ~/.cache/go-build
49-
key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
245+
path: ${{ env.SOLIDITY_CACHE_PATHS }}
246+
key: ${{ runner.os }}-solidity-go${{ steps.setup-go.outputs.go-version }}-node${{ steps.node-version.outputs.major }}-foundry-${{ steps.foundry-version.outputs.version }}-${{ hashFiles('contracts/src/**/*.sol', 'contracts-legacy/src/**/*.sol', 'contracts-local/src/**/*.sol', 'contracts/foundry.toml', 'contracts-legacy/foundry.toml', 'contracts-local/foundry.toml', 'contracts/hardhat.config.ts', 'contracts-legacy/hardhat.config.ts', 'safe-smart-account/hardhat.config.ts', 'contracts-local/Makefile', 'safe-smart-account/contracts/**/*.sol', 'safe-smart-account/package.json', 'contracts/yarn.lock', 'contracts-legacy/yarn.lock', 'safe-smart-account/package-lock.json', 'solgen/gen.go', 'go-ethereum/accounts/abi/abigen/**/*.go', 'go.mod', 'go.sum') }}
247+
248+
# On solidity cache hit: validate artifacts, then either touch Make
249+
# sentinels (if node_modules also hit) or purge and rebuild (if not).
250+
- name: Validate and mark contract builds as up-to-date on cache hit
251+
if: steps.cache-solidity.outputs.cache-hit == 'true'
252+
env:
253+
NODE_MODULES_HIT: ${{ steps.cache-node-modules.outputs.cache-hit }}
254+
run: |
255+
set -euo pipefail
256+
257+
set -f
258+
for dir in $SOLIDITY_DIRS; do
259+
if [ ! -d "$dir" ]; then
260+
echo "ERROR: cached directory missing: $dir"; exit 1
261+
fi
262+
if [ -z "$(ls -A "$dir")" ]; then
263+
echo "ERROR: cached directory empty: $dir"; exit 1
264+
fi
265+
done
266+
set +f
267+
# Spot-check a known solgen output file to catch partial cache restores
268+
# (e.g., interrupted extraction) that pass the non-empty dir check above.
269+
if [ ! -f "solgen/go/bridgegen/bridgegen.go" ]; then
270+
echo "ERROR: solgen/go output looks incomplete despite cache hit -- possible partial restore"; exit 1
271+
fi
50272
273+
# Touch Make sentinels only when both solidity AND node_modules hit.
274+
mkdir -p .make
275+
if [ "$NODE_MODULES_HIT" = "true" ]; then
276+
if [ -z "${NODE_MODULES_DIRS:-}" ]; then
277+
echo "ERROR: NODE_MODULES_DIRS is empty or unset"; exit 1
278+
fi
279+
set -f
280+
for dir in $NODE_MODULES_DIRS; do
281+
test -d "$dir" || { echo "ERROR: $dir missing despite node_modules cache hit"; exit 1; }
282+
if [ -z "$(ls -A "$dir")" ]; then
283+
echo "ERROR: $dir is empty despite node_modules cache hit -- cache may be corrupted"; exit 1
284+
fi
285+
done
286+
set +f
287+
touch .make/yarndeps .make/solidity .make/solgen
288+
echo "INFO: All sentinels touched; Make will skip solidity/solgen/yarndeps targets"
289+
else
290+
# Solidity cache hit but node_modules missed -- purge cached
291+
# solidity artifacts so Make rebuilds with fresh node_modules.
292+
# One-time cost: node_modules saves on this run, so next run
293+
# both caches hit.
294+
echo "::warning::node_modules cache missed (likely evicted) -- purging cached solidity artifacts to rebuild (one-time cost)"
295+
if [ -z "${SOLIDITY_DIRS:-}" ]; then
296+
echo "ERROR: SOLIDITY_DIRS is empty or unset"; exit 1
297+
fi
298+
set -f # disable globbing for safe word splitting on $SOLIDITY_DIRS
299+
for dir in $SOLIDITY_DIRS; do
300+
rm -rf "$dir"
301+
done
302+
set +f
303+
fi
304+
shell: bash
305+
306+
# ORDERING: Must appear before "Cache cbrotli" -- the Rust cache
307+
# includes all of target/ which overlaps cbrotli paths.
51308
- name: Cache Rust build
309+
id: cache-rust
52310
uses: actions/cache@v5
53311
with:
54312
path: |
@@ -59,7 +317,23 @@ runs:
59317
target/etc/initial-machine-cache/
60318
/home/runner/.rustup/toolchains/
61319
key: ${{ runner.os }}-cargo-${{ steps.install-rust.outputs.version }}-${{ hashFiles('Cargo.lock') }}
320+
restore-keys: |
321+
${{ runner.os }}-cargo-${{ steps.install-rust.outputs.version }}-
322+
323+
- name: Install cbindgen
324+
run: |
325+
set -euo pipefail
326+
CBINDGEN_VERSION="0.24.3"
327+
actual=$(cbindgen --version 2>/dev/null) || true
328+
if [ "$actual" != "cbindgen $CBINDGEN_VERSION" ]; then
329+
echo "Installing cbindgen $CBINDGEN_VERSION (found: '${actual:-not installed}')"
330+
cargo install cbindgen --version "$CBINDGEN_VERSION" --force
331+
else
332+
echo "cbindgen $CBINDGEN_VERSION already installed"
333+
fi
334+
shell: bash
62335

336+
# ORDERING: Must appear after "Cache Rust build" -- see comment there.
63337
- name: Cache cbrotli
64338
id: cache-cbrotli
65339
uses: actions/cache@v5
@@ -70,7 +344,17 @@ runs:
70344
target/lib/libbrotlicommon-static.a
71345
target/lib/libbrotlienc-static.a
72346
target/lib/libbrotlidec-static.a
73-
key: ${{ runner.os }}-brotli-${{ hashFiles('scripts/build-brotli.sh') }}
347+
key: ${{ runner.os }}-brotli-rust${{ steps.install-rust.outputs.version }}-${{ hashFiles('scripts/build-brotli.sh') }}
348+
349+
# A partial Rust restore may place stale cbrotli artifacts in target/.
350+
# Remove them before rebuilding.
351+
- name: Clean stale cbrotli paths on cache miss
352+
if: steps.cache-cbrotli.outputs.cache-hit != 'true'
353+
run: |
354+
set -euo pipefail
355+
rm -rf target/include/brotli target/lib-wasm
356+
rm -f target/lib/libbrotli{common,enc,dec}-static.a
357+
shell: bash
74358

75359
- name: Build cbrotli-local
76360
if: steps.cache-cbrotli.outputs.cache-hit != 'true'
@@ -81,3 +365,39 @@ runs:
81365
if: steps.cache-cbrotli.outputs.cache-hit != 'true'
82366
run: ./scripts/build-brotli.sh -w -d
83367
shell: bash
368+
369+
# Runs after both cache-restore and build paths. Do not split into
370+
# separate "validate on hit" / "validate after build" steps -- identical
371+
# validation applies regardless of source, and splitting invites drift.
372+
- name: Validate cbrotli artifacts
373+
run: |
374+
set -euo pipefail
375+
for f in target/lib/libbrotli{common,enc,dec}-static.a \
376+
target/lib-wasm/libbrotli{common,enc,dec}-static.a; do
377+
[ -f "$f" ] && [ -s "$f" ] || { echo "ERROR: $f missing or empty"; exit 1; }
378+
done
379+
for f in target/include/brotli/encode.h target/include/brotli/decode.h; do
380+
test -f "$f" || { echo "ERROR: $f missing"; exit 1; }
381+
done
382+
shell: bash
383+
384+
# cache-hit is 'true' only on exact key match. For Go/Rust
385+
# (restore-keys enabled), 'false' could mean a partial restore
386+
# or a full miss.
387+
- name: Cache status summary
388+
env:
389+
NODE_MODULES_HIT: ${{ steps.cache-node-modules.outputs.cache-hit }}
390+
GO_HIT: ${{ steps.cache-go.outputs.cache-hit }}
391+
SOLIDITY_HIT: ${{ steps.cache-solidity.outputs.cache-hit }}
392+
RUST_HIT: ${{ steps.cache-rust.outputs.cache-hit }}
393+
CBROTLI_HIT: ${{ steps.cache-cbrotli.outputs.cache-hit }}
394+
run: |
395+
set -euo pipefail
396+
echo "=== Cache Status ==="
397+
printf " %-14s %s\n" \
398+
"node_modules" "${NODE_MODULES_HIT:-MISS}" \
399+
"go" "${GO_HIT:-MISS}" \
400+
"solidity" "${SOLIDITY_HIT:-MISS}" \
401+
"rust" "${RUST_HIT:-MISS}" \
402+
"cbrotli" "${CBROTLI_HIT:-MISS}"
403+
shell: bash

0 commit comments

Comments
 (0)