Skip to content

Overhaul CI setup caching and harden build reproducibility#4502

Draft
joshuacolvin0 wants to merge 1 commit into
masterfrom
improve-ci-caching
Draft

Overhaul CI setup caching and harden build reproducibility#4502
joshuacolvin0 wants to merge 1 commit into
masterfrom
improve-ci-caching

Conversation

@joshuacolvin0
Copy link
Copy Markdown
Member

@joshuacolvin0 joshuacolvin0 commented Mar 12, 2026

● Summary

  • Overhaul CI setup action with comprehensive caching for node_modules, Solidity builds, Go build/modules, Rust, and cbrotli
  • Add toolchain versions (Go, Rust, Foundry, Node) to all cache keys to prevent cross-version collisions
  • Add restore-keys prefix fallback for Go and Rust caches (safe: content-addressed); omit for Solidity and node_modules (unsafe: stale artifacts)
  • Pin Node.js and Foundry versions as single-source-of-truth steps with runtime verification
  • Validate cache key inputs (no empty versions), hashFiles() patterns (match real files), and cached artifacts (non-empty)
  • Handle Solidity/node_modules cache split-brain: purge solidity artifacts on node_modules eviction to force consistent rebuild
  • Skip cbindgen recompile when cached version matches
  • Add cache status summary with ::warning annotations for full misses
  • Harden shell scripts: set -euo pipefail, apt-get, ln -sf, path traversal guards
  • Use npm ci / yarn --frozen-lockfile in CI for reproducible installs

Test plan

  • CI passes on this branch (lint + build + changelog)
  • Verify cache status summary appears in GHA job logs
  • Cold run: all caches MISS, full build succeeds
  • Warm run: all caches HIT, sentinels touched, Make skips solidity/solgen/yarndeps
  • Partial: bump a lockfile, verify node_modules + solidity caches invalidate together
  • Partial: bump Cargo.lock, verify Rust cache falls back to restore-key prefix match
  • Verify Foundry version check fails if pinned version diverges from installed

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com

@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 12, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 29.16%. Comparing base (e89bc08) to head (019c56c).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4502      +/-   ##
==========================================
- Coverage   32.93%   29.16%   -3.78%     
==========================================
  Files         495      495              
  Lines       58648    58648              
==========================================
- Hits        19317    17103    -2214     
- Misses      35912    38443    +2531     
+ Partials     3419     3102     -317     

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 12, 2026

❌ 17 Tests Failed:

Tests completed Failed Passed Skipped
4423 17 4406 0
View the top 3 failed tests by shortest run time
TestEndToEnd_ManyEvilValidators
Stack Traces | -0.000s run time
... [CONTENT TRUNCATED: Keeping last 20 lines]
created by github.com/offchainlabs/nitro/bold/containers/events.(*Producer[...]).Broadcast in goroutine 241271
	/home/runner/work/nitro/nitro/bold/containers/events/producer.go:106 +0xca

goroutine 3852822 [select]:
github.com/offchainlabs/nitro/bold/containers/events.(*Producer[...]).Broadcast.func1()
	/home/runner/work/nitro/nitro/bold/containers/events/producer.go:107 +0xc5
created by github.com/offchainlabs/nitro/bold/containers/events.(*Producer[...]).Broadcast in goroutine 241271
	/home/runner/work/nitro/nitro/bold/containers/events/producer.go:106 +0xca

goroutine 3852823 [select]:
github.com/offchainlabs/nitro/bold/containers/events.(*Producer[...]).Broadcast.func1()
	/home/runner/work/nitro/nitro/bold/containers/events/producer.go:107 +0xc5
created by github.com/offchainlabs/nitro/bold/containers/events.(*Producer[...]).Broadcast in goroutine 241271
	/home/runner/work/nitro/nitro/bold/containers/events/producer.go:106 +0xca

goroutine 3852783 [select]:
github.com/offchainlabs/nitro/bold/containers/events.(*Producer[...]).Broadcast.func1()
	/home/runner/work/nitro/nitro/bold/containers/events/producer.go:107 +0xc5
created by github.com/offchainlabs/nitro/bold/containers/events.(*Producer[...]).Broadcast in goroutine 241265
	/home/runner/work/nitro/nitro/bold/containers/events/producer.go:106 +0xca
TestEndToEnd_ManyEvilValidators/honest_essential_edges_confirmed_by_challenge_win
Stack Traces | -0.000s run time
... [CONTENT TRUNCATED: Keeping last 20 lines]
created by github.com/offchainlabs/nitro/bold/containers/events.(*Producer[...]).Broadcast in goroutine 241271
	/home/runner/work/nitro/nitro/bold/containers/events/producer.go:106 +0xca

goroutine 3852822 [select]:
github.com/offchainlabs/nitro/bold/containers/events.(*Producer[...]).Broadcast.func1()
	/home/runner/work/nitro/nitro/bold/containers/events/producer.go:107 +0xc5
created by github.com/offchainlabs/nitro/bold/containers/events.(*Producer[...]).Broadcast in goroutine 241271
	/home/runner/work/nitro/nitro/bold/containers/events/producer.go:106 +0xca

goroutine 3852823 [select]:
github.com/offchainlabs/nitro/bold/containers/events.(*Producer[...]).Broadcast.func1()
	/home/runner/work/nitro/nitro/bold/containers/events/producer.go:107 +0xc5
created by github.com/offchainlabs/nitro/bold/containers/events.(*Producer[...]).Broadcast in goroutine 241271
	/home/runner/work/nitro/nitro/bold/containers/events/producer.go:106 +0xca

goroutine 3852783 [select]:
github.com/offchainlabs/nitro/bold/containers/events.(*Producer[...]).Broadcast.func1()
	/home/runner/work/nitro/nitro/bold/containers/events/producer.go:107 +0xc5
created by github.com/offchainlabs/nitro/bold/containers/events.(*Producer[...]).Broadcast in goroutine 241265
	/home/runner/work/nitro/nitro/bold/containers/events/producer.go:106 +0xca
TestEndToEnd_ManyEvilValidators
Stack Traces | -0.000s run time
... [CONTENT TRUNCATED: Keeping last 20 lines]
created by github.com/offchainlabs/nitro/bold/containers/events.(*Producer[...]).Broadcast in goroutine 230774
	/home/runner/work/nitro/nitro/bold/containers/events/producer.go:106 +0x10a

goroutine 5541752 [select]:
github.com/offchainlabs/nitro/bold/containers/events.(*Producer[...]).Broadcast.func1()
	/home/runner/work/nitro/nitro/bold/containers/events/producer.go:107 +0xd9
created by github.com/offchainlabs/nitro/bold/containers/events.(*Producer[...]).Broadcast in goroutine 230774
	/home/runner/work/nitro/nitro/bold/containers/events/producer.go:106 +0x10a

goroutine 5541730 [select]:
github.com/offchainlabs/nitro/bold/containers/events.(*Producer[...]).Broadcast.func1()
	/home/runner/work/nitro/nitro/bold/containers/events/producer.go:107 +0xd9
created by github.com/offchainlabs/nitro/bold/containers/events.(*Producer[...]).Broadcast in goroutine 230777
	/home/runner/work/nitro/nitro/bold/containers/events/producer.go:106 +0x10a

goroutine 5541733 [select]:
github.com/offchainlabs/nitro/bold/containers/events.(*Producer[...]).Broadcast.func1()
	/home/runner/work/nitro/nitro/bold/containers/events/producer.go:107 +0xd9
created by github.com/offchainlabs/nitro/bold/containers/events.(*Producer[...]).Broadcast in goroutine 230777
	/home/runner/work/nitro/nitro/bold/containers/events/producer.go:106 +0x10a

📣 Thoughts on this report? Let Codecov know! | Powered by Codecov

@joshuacolvin0 joshuacolvin0 force-pushed the improve-ci-caching branch 2 times, most recently from 17732ab to 84bc75a Compare March 12, 2026 21:44
Rework the ci-setup composite action to add proper caching for
node_modules, solidity build artifacts, Go build/modules, and cbrotli,
with validation and corruption detection on cache restore. Key changes:

- Add dedicated cache steps for node_modules, solidity, Go, and cbrotli
  with carefully designed cache keys that include tool versions and all
  relevant source/config files via hashFiles()
- Add self-validating cache key inputs: extract hashFiles() patterns from
  the action YAML and verify each pattern matches at least one file
- Add cache-hit validation that checks cached directories exist and are
  non-empty before touching Make sentinels to skip rebuilds
- Handle solidity/node_modules cache split-brain: purge solidity artifacts
  when node_modules cache is evicted to ensure consistent rebuilds
- Add RETRY macro for npm/yarn installs to handle transient registry errors,
  with optional node_modules cleanup between attempts
- Pin cbindgen version and skip reinstall when cached version matches
- Use frozen lockfiles (npm ci / yarn --frozen-lockfile) in CI
- Centralize node_modules and solidity directory lists as single source
  of truth consumed by both cache steps and validation
- Add lint step that warns about cache-sensitive files not covered by
  any hashFiles() pattern
- Harden install-rust action with set -euo pipefail and quoted outputs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant