Skip to content

Commit 9a7f5e6

Browse files
mudlermudler
committed
ci(darwin): add native caches to backend_build_darwin
macOS runners can't use the registry-backed BuildKit cache (no Docker daemon), so every darwin matrix run was paying full cost for brew installs, Go module downloads, llama.cpp recompiles and Python wheel resolution. Wires actions/cache@v4 into the reusable workflow for four caches: - Go modules + build cache (setup-go cache: true), shared across matrix - Homebrew downloads + selected /opt/homebrew/Cellar entries, with HOMEBREW_NO_AUTO_UPDATE so restored Cellar paths stay stable - ccache for the llama-cpp CMake variants, keyed on the pinned LLAMA_VERSION; CMAKE_*_COMPILER_LAUNCHER is exported via GITHUB_ENV so backend/cpp/llama-cpp/Makefile picks it up without script changes - Python uv + pip wheel cache, keyed by backend + ISO week — same one-cold-rebuild-per-week cadence as the Linux DEPS_REFRESH Read/write semantics match the existing BuildKit policy: every run restores, only master/tag pushes save, so PRs can't pollute master's warm cache. Documents the new caches and the macOS-specific constraints in .agents/ci-caching.md. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-7[1m] [Claude Code]
1 parent 6b63b47 commit 9a7f5e6

2 files changed

Lines changed: 153 additions & 2 deletions

File tree

‎.agents/ci-caching.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,30 @@ Eviction is rarely needed in normal operation — `DEPS_REFRESH` handles weekly
7676

7777
- The "Free Disk Space" / "Release space from worker" steps run on every job — these reclaim ~6 GB on `ubuntu-latest` runners. They are runner-state cleanup, not Docker, and BuildKit caches don't apply.
7878
- Intermediate artifacts of `Build and push (PR)` are not pushed anywhere — PRs only build for verification.
79+
- Darwin builds (see below) — macOS runners have no Docker daemon, so the registry-backed BuildKit cache cannot apply.
80+
81+
## Darwin native caches
82+
83+
`backend_build_darwin.yml` runs natively on `macOS-14` GitHub-hosted runners — there is no Docker, no BuildKit, no cross-job registry cache. Instead, the reusable workflow uses `actions/cache@v4` for four native caches that mirror the spirit of the Linux cache (warm by default, weekly refresh for unpinned Python deps, PRs read-only).
84+
85+
| Cache | Path(s) | Key | Scope |
86+
|---|---|---|---|
87+
| Go modules + build | `~/go/pkg/mod`, `~/Library/Caches/go-build` | `go.sum` (managed by `actions/setup-go@v5` `cache: true`) | All darwin jobs |
88+
| Homebrew | `~/Library/Caches/Homebrew/downloads`, selected `/opt/homebrew/Cellar/*` | hash of `backend_build_darwin.yml` | All darwin jobs |
89+
| ccache (llama.cpp CMake) | `~/Library/Caches/ccache` | pinned `LLAMA_VERSION` from `backend/cpp/llama-cpp/Makefile` | `inputs.backend == 'llama-cpp'` only |
90+
| Python wheels (uv + pip) | `~/Library/Caches/pip`, `~/Library/Caches/uv` | `inputs.backend` + ISO week (`+%Y-W%V`) + hash of that backend's `requirements*.txt` | `inputs.lang == 'python'` only |
91+
92+
Read/write semantics match the BuildKit cache: `actions/cache/restore` runs every time, `actions/cache/save` is gated on `github.event_name != 'pull_request'`. PRs read master's warm cache but never write back.
93+
94+
The Python wheel cache uses the same ISO-week cache-buster as the Linux `DEPS_REFRESH` build-arg — same problem (unpinned `torch`/`mlx`/`diffusers`/`transformers` resolve to fresh wheels weekly), same ~one-cold-rebuild-per-week solution.
95+
96+
The brew Cellar cache requires `HOMEBREW_NO_AUTO_UPDATE=1` and `HOMEBREW_NO_INSTALL_CLEANUP=1` (set as job-level env). Without those, `brew install` would mutate the very directories that were just restored, defeating the cache.
97+
98+
For ccache, the workflow exports `CMAKE_ARGS=… -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache` via `$GITHUB_ENV` before running `make build-darwin-go-backend`. The Makefile in `backend/cpp/llama-cpp/` already forwards `CMAKE_ARGS` through to each variant build (`fallback`, `grpc`, `rpc-server`), so no script changes are needed. The three variants share most TUs, so ccache dedupes object files across them.
99+
100+
### Cache budget on Darwin
101+
102+
GitHub Actions caches are limited to 10 GB per repo. Steady-state worst case: ~800 MB Go cache + ~2 GB brew Cellar + up to 2 GB ccache + ~1.5 GB × 5 python backends. If the cap is hit, prefer collapsing the per-backend Python keys into a shared `pyenv-darwin-shared-<week>` key (accepts more cross-backend churn for a smaller footprint) before reducing other caches.
79103

80104
## Touching the cache pipeline
81105

‎.github/workflows/backend_build_darwin.yml‎

Lines changed: 129 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,13 @@ jobs:
4848
strategy:
4949
matrix:
5050
go-version: ['${{ inputs.go-version }}']
51+
env:
52+
# Keep the brew Cellar stable across cache restores. Without these,
53+
# `brew install` would auto-update brew itself and re-link formulas,
54+
# mutating the very paths the cache just restored.
55+
HOMEBREW_NO_AUTO_UPDATE: '1'
56+
HOMEBREW_NO_INSTALL_CLEANUP: '1'
57+
HOMEBREW_NO_ANALYTICS: '1'
5158
steps:
5259
- name: Clone
5360
uses: actions/checkout@v6
@@ -58,21 +65,141 @@ jobs:
5865
uses: actions/setup-go@v5
5966
with:
6067
go-version: ${{ matrix.go-version }}
61-
cache: false
68+
# Caches ~/go/pkg/mod and ~/Library/Caches/go-build keyed on go.sum.
69+
# Shared across every darwin matrix entry — first job in a run warms
70+
# it, the rest hit warm.
71+
cache: true
6272

6373
# You can test your matrix by printing the current Go version
6474
- name: Display Go version
6575
run: go version
6676

77+
# ---- Homebrew cache ----
78+
# macOS runners have no Docker daemon, so the BuildKit registry cache used
79+
# for Linux backend images (see .agents/ci-caching.md) doesn't apply here.
80+
# We cache the brew downloads + Cellar entries for the formulas we install
81+
# below. Read on every run, write only on master/tag pushes — same policy
82+
# as the Linux registry cache.
83+
- name: Restore Homebrew cache
84+
id: brew-cache
85+
uses: actions/cache/restore@v4
86+
with:
87+
path: |
88+
~/Library/Caches/Homebrew/downloads
89+
/opt/homebrew/Cellar/protobuf
90+
/opt/homebrew/Cellar/grpc
91+
/opt/homebrew/Cellar/protoc-gen-go
92+
/opt/homebrew/Cellar/protoc-gen-go-grpc
93+
/opt/homebrew/Cellar/libomp
94+
/opt/homebrew/Cellar/llvm
95+
/opt/homebrew/Cellar/ccache
96+
key: brew-${{ runner.os }}-${{ runner.arch }}-v1-${{ hashFiles('.github/workflows/backend_build_darwin.yml') }}
97+
6798
- name: Dependencies
6899
run: |
69-
brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm
100+
# ccache is always installed (used by the llama-cpp variant build) so
101+
# the brew cache content stays stable across every backend in the
102+
# matrix — they all share one cache key.
103+
brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm ccache
104+
105+
- name: Save Homebrew cache
106+
if: github.event_name != 'pull_request' && steps.brew-cache.outputs.cache-hit != 'true'
107+
uses: actions/cache/save@v4
108+
with:
109+
path: |
110+
~/Library/Caches/Homebrew/downloads
111+
/opt/homebrew/Cellar/protobuf
112+
/opt/homebrew/Cellar/grpc
113+
/opt/homebrew/Cellar/protoc-gen-go
114+
/opt/homebrew/Cellar/protoc-gen-go-grpc
115+
/opt/homebrew/Cellar/libomp
116+
/opt/homebrew/Cellar/llvm
117+
/opt/homebrew/Cellar/ccache
118+
key: brew-${{ runner.os }}-${{ runner.arch }}-v1-${{ hashFiles('.github/workflows/backend_build_darwin.yml') }}
119+
120+
# ---- ccache for llama.cpp CMake builds ----
121+
# Three CMake variants (fallback, grpc, rpc-server) compile the same
122+
# llama.cpp source tree with overlapping flags — ccache dedupes object
123+
# files across them. Key on the pinned LLAMA_VERSION so a pin bump
124+
# invalidates cleanly; restore-keys fall back to the latest entry for the
125+
# same pin so unchanged TUs stay warm even when the cache is fresh.
126+
- name: Compute llama.cpp version
127+
if: inputs.backend == 'llama-cpp'
128+
id: llama-version
129+
run: |
130+
version=$(grep '^LLAMA_VERSION' backend/cpp/llama-cpp/Makefile | head -1 | cut -d= -f2 | cut -d'?' -f1 | tr -d ' ')
131+
echo "version=${version}" >> "$GITHUB_OUTPUT"
132+
133+
- name: Restore ccache
134+
if: inputs.backend == 'llama-cpp'
135+
id: ccache-cache
136+
uses: actions/cache/restore@v4
137+
with:
138+
path: ~/Library/Caches/ccache
139+
key: ccache-llama-${{ runner.arch }}-${{ steps.llama-version.outputs.version }}-${{ github.run_id }}
140+
restore-keys: |
141+
ccache-llama-${{ runner.arch }}-${{ steps.llama-version.outputs.version }}-
142+
143+
- name: Configure ccache
144+
if: inputs.backend == 'llama-cpp'
145+
run: |
146+
mkdir -p "$HOME/Library/Caches/ccache"
147+
ccache -M 2G
148+
ccache -z
149+
# llama-cpp-darwin.sh reads CMAKE_ARGS / CCACHE_DIR from env.
150+
{
151+
echo "CMAKE_ARGS=${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache"
152+
echo "CCACHE_DIR=$HOME/Library/Caches/ccache"
153+
} >> "$GITHUB_ENV"
154+
155+
# ---- Python wheel cache (uv + pip) ----
156+
# Mirrors the Linux DEPS_REFRESH cadence (see .agents/ci-caching.md): the
157+
# ISO-week segment of the cache key forces at most one cold rebuild per
158+
# backend per week, automatically picking up newer wheels for unpinned
159+
# deps (torch, mlx, diffusers, …). Restore-keys fall back to the most
160+
# recent build of the same backend so off-week PRs still hit warm.
161+
- name: Compute weekly cache bucket
162+
if: inputs.lang == 'python'
163+
id: weekly
164+
run: echo "bucket=$(date -u +%Y-W%V)" >> "$GITHUB_OUTPUT"
165+
166+
- name: Restore Python wheel cache
167+
if: inputs.lang == 'python'
168+
id: pyenv-cache
169+
uses: actions/cache/restore@v4
170+
with:
171+
path: |
172+
~/Library/Caches/pip
173+
~/Library/Caches/uv
174+
key: pyenv-darwin-${{ inputs.backend }}-${{ steps.weekly.outputs.bucket }}-${{ hashFiles(format('backend/python/{0}/requirements*.txt', inputs.backend)) }}
175+
restore-keys: |
176+
pyenv-darwin-${{ inputs.backend }}-
70177
71178
- name: Build ${{ inputs.backend }}-darwin
72179
run: |
73180
make protogen-go
74181
BACKEND=${{ inputs.backend }} BUILD_TYPE=${{ inputs.build-type }} USE_PIP=${{ inputs.use-pip }} make build-darwin-${{ inputs.lang }}-backend
75182
183+
- name: ccache stats
184+
if: inputs.backend == 'llama-cpp'
185+
run: ccache -s
186+
187+
- name: Save ccache
188+
if: inputs.backend == 'llama-cpp' && github.event_name != 'pull_request'
189+
uses: actions/cache/save@v4
190+
with:
191+
path: ~/Library/Caches/ccache
192+
key: ccache-llama-${{ runner.arch }}-${{ steps.llama-version.outputs.version }}-${{ github.run_id }}
193+
194+
- name: Save Python wheel cache
195+
if: inputs.lang == 'python' && github.event_name != 'pull_request' && steps.pyenv-cache.outputs.cache-hit != 'true'
196+
uses: actions/cache/save@v4
197+
with:
198+
path: |
199+
~/Library/Caches/pip
200+
~/Library/Caches/uv
201+
key: pyenv-darwin-${{ inputs.backend }}-${{ steps.weekly.outputs.bucket }}-${{ hashFiles(format('backend/python/{0}/requirements*.txt', inputs.backend)) }}
202+
76203
- name: Upload ${{ inputs.backend }}.tar
77204
uses: actions/upload-artifact@v7
78205
with:

0 commit comments

Comments
 (0)