Commit 938aee3
committed
circleci: restructure pipeline to fix fan-in and reduce io/network cost
Problem: fan-in = multiple upstream jobs persist same paths to workspace.
Origin/master had massive IO/network cost from rebuilding everything.
Thanks CircleCI free tier for open source projects!
Mitigation for circleci.canny.io/cloud-feature-requests/p/support-wildcards-in-savecachepaths
Changes vs origin/master:
- Add 'heads-docker' executor (centralized docker config, image v0.2.9)
- Add glossary explaining fan-in, workspace chain, cache layers
- Split build_and_persist into separate jobs per stage:
build (parallel board builds), x86_coreboot (per fork), x86_musl_cross_make,
x86_blobs, x86_save_modules_cache, ppc64_talos_2
- Add create_hashes job (creates cache hashes, shared by all arches)
- Replace save_cache with per-job cache saving (eliminates fan-in)
- Each job saves cache immediately before persisting to workspace
- x86 chain: create_hashes -> x86_blobs -> x86_musl_cross_make ->
x86_coreboot seeds -> parallel builds + x86_save_modules_cache
- ppc64 chain: create_hashes -> ppc64_talos_2 (linear)
- Build: only clean logs dir, not entire board dir (performance)
- Cache hierarchy: modules > coreboot+musl > musl > blobs (x86 only, workspace)
- Fix ppc64 target: 'UNTESTED_talos-2'
- Add checkout step to all jobs
- Rename cache keys: modules-and-patches -> modules
Signed-off-by: Thierry Laurion <insurgo@riseup.net>1 parent 67c2b28 commit 938aee3
1 file changed
+264
-158
lines changed
0 commit comments