Skip to content

Commit d3659c7

Browse files
julek-wolfssljulek-wolfssl
committed
CI: move Arduino cores from actions/cache to ghcr bundles
arduino.yml's per-core actions/cache layer stored the installed cores and toolchains (~/.arduino15) - several GB, dominated by the esp32 and mbed cores - in the 10 GB Actions cache. For esp32 it was also ineffective: the disk-cleanup step deletes the esp32 toolchain before actions/cache saves it, so esp32 re-downloaded every run anyway. - New arduino-cores-image workflow resolves each of the 9 distinct cores and publishes a tar of ~/.arduino15 + ~/Arduino/libraries to ghcr.io/<owner>/wolfssl-ci-arduino:<core>. It runs monthly: esp32, the fastest-moving core, releases ~monthly and the rest far less often. - New install-arduino-core composite action restores that bundle offline and verifies the core is present, falling back to `arduino-cli core install` when the bundle is unavailable - so nothing breaks until the image is first published and made public. - arduino.yml calls the action in place of the inline core install and the actions/cache step. This takes the flaky espressif / esp8266.com / pjrc.com downloads off the PR critical path and frees the Actions cache of the largest binaries it held.
1 parent 94a671b commit d3659c7

3 files changed

Lines changed: 235 additions & 68 deletions

File tree

‎.github/actions/install-arduino-core/action.yml‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
name: 'Install Arduino core'
2+
description: >
3+
Make an Arduino core (and the shared CI libraries) available, preferring a
4+
prebuilt bundle pulled from ghcr (published by the arduino-cores-image
5+
workflow) and falling back to `arduino-cli core install` when the bundle is
6+
unavailable or stale. Assumes arduino-cli is already on PATH.
7+
inputs:
8+
core-id:
9+
description: 'vendor:arch core to make available, e.g. esp32:esp32'
10+
required: true
11+
board-manager-url:
12+
description: >
13+
Optional third-party board_manager index URL, used only on the
14+
online-install fallback (the ghcr bundle already carries its own).
15+
required: false
16+
default: ''
17+
libs:
18+
description: 'Space-separated Arduino libraries to ensure are present'
19+
required: false
20+
default: 'ArduinoJson WiFiNINA Ethernet Bridge'
21+
runs:
22+
using: 'composite'
23+
steps:
24+
# Preferred path: restore ~/.arduino15 (the core + toolchain) and the
25+
# shared libraries from a prebuilt tarball pulled from ghcr, so the flaky
26+
# board_manager / toolchain downloads are off the PR critical path. The
27+
# bundle is published only under the wolfssl org (gated below), so fork PRs
28+
# read the public upstream image too. Best-effort: any failure leaves
29+
# "satisfied" unset and the online install below runs unchanged.
30+
- name: Restore Arduino core from ghcr bundle
31+
id: ghcr
32+
shell: bash
33+
run: |
34+
set -u
35+
command -v docker >/dev/null 2>&1 || { echo "::notice::docker unavailable; installing core online"; exit 0; }
36+
command -v arduino-cli >/dev/null 2>&1 || { echo "::notice::arduino-cli not on PATH; installing core online"; exit 0; }
37+
CORE_ID='${{ inputs.core-id }}'
38+
TAG=$(echo "$CORE_ID" | tr ':' '-')
39+
IMG="ghcr.io/wolfssl/wolfssl-ci-arduino:$TAG"
40+
if ! docker pull -q "$IMG" >/dev/null 2>&1; then
41+
echo "::notice::ghcr bundle $IMG unavailable; installing core online"
42+
exit 0
43+
fi
44+
cid=$(docker create "$IMG" 2>/dev/null) || { echo "::notice::cannot open bundle; installing core online"; exit 0; }
45+
rm -f "$RUNNER_TEMP/arduino-core.tar"
46+
docker cp "$cid:/arduino-core.tar" "$RUNNER_TEMP/arduino-core.tar" >/dev/null 2>&1 || true
47+
docker rm "$cid" >/dev/null 2>&1 || true
48+
test -f "$RUNNER_TEMP/arduino-core.tar" || { echo "::notice::bundle had no tarball; installing core online"; exit 0; }
49+
# Entries are stored relative to $HOME (.arduino15/..., Arduino/libraries/...).
50+
tar -C "$HOME" -xf "$RUNNER_TEMP/arduino-core.tar" || { echo "::notice::could not unpack bundle; installing core online"; exit 0; }
51+
rm -f "$RUNNER_TEMP/arduino-core.tar"
52+
if arduino-cli core list 2>/dev/null | awk 'NR>1 {print $1}' | grep -Fxq "$CORE_ID"; then
53+
echo "satisfied=true" >> "$GITHUB_OUTPUT"
54+
echo "Restored $CORE_ID from $IMG"
55+
else
56+
echo "::notice::bundle did not yield $CORE_ID; installing core online"
57+
fi
58+
59+
- name: Install Arduino core online
60+
if: steps.ghcr.outputs.satisfied != 'true'
61+
shell: bash
62+
run: |
63+
set -euo pipefail
64+
CORE_ID='${{ inputs.core-id }}'
65+
BM_URL='${{ inputs.board-manager-url }}'
66+
retry() { local i; for i in 1 2 3 4 5; do "$@" && return 0; sleep $((2**i)); done; "$@"; }
67+
68+
arduino-cli config init --overwrite
69+
# Wait up to 10 minutes for the big toolchain downloads.
70+
arduino-cli config set network.connection_timeout 600s
71+
# Scope third-party indexes to the one core that needs them: arduino-cli
72+
# re-reads every configured index on each call and fails if any is
73+
# unreachable, so an unconditional URL makes all jobs depend on it.
74+
if [ -n "$BM_URL" ]; then
75+
arduino-cli config add board_manager.additional_urls "$BM_URL"
76+
fi
77+
retry arduino-cli core update-index
78+
retry arduino-cli core install "$CORE_ID"
79+
for lib in ${{ inputs.libs }}; do
80+
retry arduino-cli lib install "$lib"
81+
done

‎.github/workflows/arduino-cores-image.yml‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
name: Arduino cores image
2+
3+
# Builds the prebuilt Arduino core bundles that arduino.yml restores offline
4+
# (see .github/actions/install-arduino-core). Each bundle is a tar of
5+
# ~/.arduino15 (the installed core + toolchain) and ~/Arduino/libraries (the
6+
# shared CI libraries) for one vendor:arch core, published to
7+
# ghcr.io/<owner>/wolfssl-ci-arduino:<core> (':' in the core id becomes '-').
8+
#
9+
# Why: the core/toolchain downloads (espressif, esp8266.com, pjrc.com) are
10+
# large and chronically flaky from runner egress, and the old actions/cache
11+
# layer both pressed on the 10 GB cache cap and - for esp32 - was deleted by
12+
# arduino.yml's disk cleanup before it was ever saved. Resolving each core ONCE
13+
# here and pulling it from ghcr on every PR keeps those downloads off the PR
14+
# critical path. ghcr storage/bandwidth is free for public images.
15+
#
16+
# ONE-TIME SETUP: after the first successful run, make the package
17+
# `wolfssl-ci-arduino` PUBLIC (repo/org > Packages > Package settings >
18+
# Change visibility). Anonymous `docker pull` then works from fork PRs too;
19+
# until then install-arduino-core simply installs the core online (no breakage).
20+
21+
on:
22+
schedule:
23+
# Monthly (1st). esp32 - the fastest-moving core - releases roughly monthly
24+
# and the rest far less often, so a monthly unconditional rebuild tracks
25+
# them closely enough; between rebuilds install-arduino-core installs any
26+
# newer core online. Each run republishes every bundle.
27+
- cron: '0 4 1 * *'
28+
workflow_dispatch:
29+
30+
concurrency:
31+
group: arduino-cores-image-${{ github.ref }}
32+
cancel-in-progress: true
33+
34+
permissions:
35+
contents: read
36+
packages: write
37+
38+
jobs:
39+
build:
40+
name: build ${{ matrix.core_id }}
41+
if: github.repository_owner == 'wolfssl'
42+
# teensy:avr's index lives at pjrc.com, chronically unreachable from runner
43+
# egress; let it fail without blocking the other eight bundles.
44+
continue-on-error: ${{ matrix.core_id == 'teensy:avr' }}
45+
runs-on: ubuntu-24.04
46+
timeout-minutes: 30
47+
strategy:
48+
fail-fast: false
49+
matrix:
50+
include:
51+
# Distinct vendor:arch cores behind arduino.yml's board matrix. The
52+
# esp32 and mbed_* cores are the GB-scale toolchains; the AVR/SAM/SAMD
53+
# cores are small. board_url is set only for cores whose index is not
54+
# in the default board manager.
55+
- core_id: arduino:avr
56+
- core_id: arduino:samd
57+
- core_id: arduino:sam
58+
- core_id: arduino:mbed_edge
59+
- core_id: arduino:mbed_portenta
60+
- core_id: arduino:renesas_uno
61+
- core_id: esp32:esp32
62+
- core_id: esp8266:esp8266
63+
board_url: https://arduino.esp8266.com/stable/package_esp8266com_index.json
64+
- core_id: teensy:avr
65+
board_url: https://www.pjrc.com/teensy/package_teensy_index.json
66+
steps:
67+
- name: Free disk space
68+
shell: bash
69+
run: |
70+
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL
71+
sudo apt-get clean
72+
df -h
73+
74+
- name: Install Arduino CLI
75+
shell: bash
76+
run: |
77+
set -euo pipefail
78+
mkdir -p "$HOME/bin"
79+
echo "$HOME/bin" >> "$GITHUB_PATH"
80+
curl -fsSL --retry 5 --retry-delay 10 \
81+
https://raw.githubusercontent.com/arduino/arduino-cli/master/install.sh \
82+
| BINDIR="$HOME/bin" sh
83+
"$HOME/bin/arduino-cli" version
84+
85+
- name: Install the core and shared libraries
86+
shell: bash
87+
run: |
88+
set -euo pipefail
89+
CORE_ID='${{ matrix.core_id }}'
90+
BM_URL='${{ matrix.board_url }}'
91+
retry() { local i; for i in 1 2 3 4 5; do "$@" && return 0; sleep $((2**i)); done; "$@"; }
92+
93+
arduino-cli config init --overwrite
94+
arduino-cli config set network.connection_timeout 600s
95+
if [ -n "$BM_URL" ]; then
96+
arduino-cli config add board_manager.additional_urls "$BM_URL"
97+
fi
98+
retry arduino-cli core update-index
99+
retry arduino-cli core install "$CORE_ID"
100+
# Mirror arduino.yml's always-installed libraries so consumers get a
101+
# complete bundle.
102+
for lib in ArduinoJson WiFiNINA Ethernet Bridge; do
103+
retry arduino-cli lib install "$lib"
104+
done
105+
mkdir -p "$HOME/Arduino/libraries"
106+
107+
- name: Pack the bundle tarball
108+
shell: bash
109+
run: |
110+
set -euo pipefail
111+
mkdir -p "$RUNNER_TEMP/ctx"
112+
# Paths relative to $HOME so install-arduino-core can `tar -C $HOME -x`
113+
# straight back. Drop the staging area and any wolfssl lib (arduino.yml
114+
# always installs the latest wolfssl itself).
115+
tar --exclude='.arduino15/staging' --exclude='Arduino/libraries/wolfssl' \
116+
-C "$HOME" -cf "$RUNNER_TEMP/ctx/arduino-core.tar" .arduino15 Arduino/libraries
117+
echo "Tarball size: $(du -h "$RUNNER_TEMP/ctx/arduino-core.tar" | cut -f1)"
118+
119+
- name: Log in to ghcr
120+
shell: bash
121+
run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${{ github.actor }}" --password-stdin
122+
123+
- name: Build and push bundle
124+
shell: bash
125+
run: |
126+
set -euo pipefail
127+
OWNER=$(echo "${{ github.repository_owner }}" | tr '[:upper:]' '[:lower:]')
128+
TAG=$(echo "${{ matrix.core_id }}" | tr ':' '-')
129+
IMG="ghcr.io/$OWNER/wolfssl-ci-arduino:$TAG"
130+
# Tiny busybox base so the consumer can `docker cp` the tarball out;
131+
# the base size is negligible next to the toolchain.
132+
printf 'FROM busybox\nCOPY arduino-core.tar /arduino-core.tar\n' > "$RUNNER_TEMP/ctx/Dockerfile"
133+
docker build -t "$IMG" "$RUNNER_TEMP/ctx"
134+
docker push "$IMG"
135+
echo "Pushed $IMG"

‎.github/workflows/arduino.yml‎

Lines changed: 19 additions & 68 deletions
Original file line numberDiff line numberDiff line change
@@ -197,53 +197,25 @@ jobs:
197197
run: |
198198
CORE_ID="$(echo '${{ matrix.fqbn }}' | cut -d: -f1-2)"
199199
echo "CORE_ID=$CORE_ID" >> "$GITHUB_ENV"
200-
201-
- name: Setup Arduino CLI
202-
run: |
203-
arduino-cli config init
204-
205-
# wait 10 minutes for big downloads (or use 0 for no limit)
206-
arduino-cli config set network.connection_timeout 600s
207-
208-
# Only add third-party board_manager URLs for matrix entries that actually need them.
209-
# arduino-cli re-reads every configured index on each invocation and fails the whole
210-
# step if any one is unreachable, so adding these unconditionally makes all jobs
211-
# depend on pjrc.com and esp8266.com -- a single outage there cascades into total
212-
# CI failure. Scope each URL to the one CORE_ID that uses it.
213-
if [ "$CORE_ID" = "teensy:avr" ]; then
214-
arduino-cli config add board_manager.additional_urls https://www.pjrc.com/teensy/package_teensy_index.json
215-
fi
216-
if [ "$CORE_ID" = "esp8266:esp8266" ]; then
217-
arduino-cli config add board_manager.additional_urls https://arduino.esp8266.com/stable/package_esp8266com_index.json
218-
fi
219-
arduino-cli core update-index
220-
221-
echo "CORE_ID: $CORE_ID"
222-
arduino-cli core install "$CORE_ID"
223-
224-
# The above is instead of:
225-
# arduino-cli core install esp32:esp32 # ESP32
226-
# arduino-cli core install arduino:avr # Arduino Uno, Mega, Nano
227-
# arduino-cli core install arduino:sam # Arduino Due
228-
# arduino-cli core install arduino:samd # Arduino Zero
229-
# arduino-cli core install teensy:avr # PJRC Teensy
230-
# arduino-cli core install esp8266:esp8266 # ESP8266
231-
# arduino-cli core install arduino:mbed_nano # nanorp2040connect
232-
# arduino-cli core install arduino:mbed_portenta # portenta_h7_m7
233-
# arduino-cli core install arduino:mbed_edge
234-
# arduino-cli core install arduino:renesas_uno
235-
236-
# For reference:
237-
238-
# mbed nano not yet tested
239-
# sudo "/home/$USER/.arduino15/packages/arduino/hardware/mbed_nano/4.2.4/post_install.sh"
240-
241-
# Always install networking (not part of FQBN matrix)
242-
# The first one also creates directory: /home/runner/Arduino/libraries
243-
arduino-cli lib install "ArduinoJson" # Example dependency
244-
arduino-cli lib install "WiFiNINA" # ARDUINO_SAMD_NANO_33_IOT
245-
arduino-cli lib install "Ethernet" # Install Ethernet library
246-
arduino-cli lib install "Bridge" # Pseudo-network for things like arduino:samd:tian
200+
# Third-party board_manager index for the cores that need one. Scoped
201+
# to the one CORE_ID that uses it: arduino-cli re-reads every index on
202+
# each call and fails if any is unreachable, so an unconditional URL
203+
# would make all jobs depend on pjrc.com / esp8266.com. Used only on
204+
# the online-install fallback; the ghcr bundle already carries it.
205+
case "$CORE_ID" in
206+
teensy:avr) echo "BM_URL=https://www.pjrc.com/teensy/package_teensy_index.json" >> "$GITHUB_ENV" ;;
207+
esp8266:esp8266) echo "BM_URL=https://arduino.esp8266.com/stable/package_esp8266com_index.json" >> "$GITHUB_ENV" ;;
208+
*) echo "BM_URL=" >> "$GITHUB_ENV" ;;
209+
esac
210+
211+
# Restore the core + toolchain + shared libraries from the prebuilt ghcr
212+
# bundle (arduino-cores-image), falling back to `arduino-cli core install`
213+
# when it is unavailable. Replaces the old per-core actions/cache layer.
214+
- name: Install Arduino core and libraries
215+
uses: ./.github/actions/install-arduino-core
216+
with:
217+
core-id: ${{ env.CORE_ID }}
218+
board-manager-url: ${{ env.BM_URL }}
247219

248220
- name: Set Job Environment Variables
249221
run: |
@@ -270,27 +242,6 @@ jobs:
270242
# WOLFSSL_EXAMPLES_ROOT is the repo root, not example location
271243
echo "WOLFSSL_EXAMPLES_ROOT = $WOLFSSL_EXAMPLES_ROOT"
272244
273-
- name: Cache Arduino Packages
274-
uses: actions/cache@v5
275-
with:
276-
path: |
277-
~/.arduino15
278-
~/.cache/arduino
279-
# Exclude staging directory from cache to save space
280-
!~/.arduino15/staging
281-
282-
# Arduino libraries
283-
# Specific to Arduino CI Build (2 of 4) Arduinbo Release wolfSSL for Local Examples
284-
# Include all libraries, as the latest Arduino-wolfSSL will only change upon release.
285-
~/Arduino/libraries
286-
# Ensure wolfssl is not cached, we're always using the latest. See separate cache.
287-
!~/Arduino/libraries/wolfssl
288-
key: arduino-${{ runner.os }}-${{ env.CORE_ID }}-${{ hashFiles('Arduino/sketches/board_list.txt') }}
289-
290-
restore-keys: |
291-
arduino-${{ runner.os }}-${{ env.CORE_ID }}-
292-
arduino-${{ runner.os }}-
293-
294245
- name: Get wolfssl-examples
295246
run: |
296247
# Fetch Arduino examples from the wolfssl-examples repo

0 commit comments

Comments
 (0)