ci(sccache): make the fork-PR/intermittent Depot 403 a green uncached build

A fork PR runs with secrets withheld, so SCCACHE_WEBDAV_TOKEN is empty and
Depot rejects the unauthenticated server-startup .sccache_check with 403
Forbidden; sccache treats a failed startup check as fatal, redding every TU.
The symptom was mac-only only because the macOS jobs ran 'brew install sccache'
unconditionally while the Linux/dockcross/aarch64 jobs fetch sccache only when
a token is present, so mac was the one platform with sccache on PATH to misfire.

Two independent guards:
- Gate every 'Install sccache' step on env.SCCACHE_WEBDAV_TOKEN != '' so a
  tokenless fork PR never installs sccache (mac now matches Linux).
- build.sh retries the build once without the launcher when the build fails AND
  the output shows an sccache cache error (sccache: error / Server startup
  failed / cache storage failed). Gated on that signature so a real compile
  error still fails fast. Also covers an intermittent 403 hitting a valid-token
  job mid-build, which the one-shot probe cannot foresee. The uncached -O3
  rebuild is content-identical and release-safe.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V9ZHtf3xz9pB5L9KWgWdWG

Author: claude

.github/build.sh

@@ -104,10 +104,37 @@ else
 fi
 
 cmake -Bbuild $LAUNCH $@ || exit 1
-cmake --build build --config Release -j"${JOBS}" || exit 1
+
+# Build. The pre-build probe only proves the cache was reachable at one instant; it cannot
+# foresee a cache outage that strikes *during* the build. When sccache is the launcher and its
+# backend fails mid-build — e.g. an intermittent Depot 403 on the server-startup .sccache_check,
+# or the tokenless 403 a fork PR hits because secrets are withheld — sccache makes every TU fatal
+# and reds the whole build. sccache exposes no "ignore backend errors" switch for that startup
+# check, so recover by retrying the build once WITHOUT the launcher: a from-scratch uncached -O3
+# build is content-identical and release-safe, so the cache can never red the build. The retry is
+# gated on the failure output actually showing an sccache cache error, so a genuine compile error
+# still fails fast (and is reported) instead of triggering a wasteful uncached rebuild.
+build_log="$(mktemp 2>/dev/null || echo "/tmp/jllama-build.$$.log")"
+cmake --build build --config Release -j"${JOBS}" 2>&1 | tee "$build_log"
+build_rc=${PIPESTATUS[0]}
+if [ "$build_rc" -ne 0 ]; then
+  if [ -n "$LAUNCH" ] && grep -qiE 'sccache: error|Server startup failed|cache storage failed' "$build_log"; then
+    echo "build.sh: build failed via an sccache cache error — retrying WITHOUT cache (clean reconfigure)."
+    rm -f "$build_log"
+    rm -rf build && mkdir -p build
+    cmake -Bbuild $@ || exit 1
+    cmake --build build --config Release -j"${JOBS}" || exit 1
+    LAUNCH=""  # cache disabled for this run; skip the stats query below
+  else
+    rm -f "$build_log"
+    exit 1
+  fi
+fi
+rm -f "$build_log"
 
 # Only query stats when sccache was actually used as the launcher; if the probe rejected a
-# crashing sccache, re-invoking it here would just repeat the crash output (harmless but noisy).
+# crashing sccache (or the mid-build retry disabled it), re-invoking it here would just repeat
+# the crash output (harmless but noisy).
 if [ -n "$LAUNCH" ] && command -v sccache >/dev/null 2>&1; then
   sccache --show-stats || true
 fi

.github/workflows/publish.yml

@@ -415,7 +415,7 @@ jobs:
           echo "=== Processor Details ==="
           system_profiler SPHardwareDataType
       - name: Install sccache (shared compiler cache)
-        if: env.USE_CACHE == 'true'
+        if: env.USE_CACHE == 'true' && env.SCCACHE_WEBDAV_TOKEN != ''
         continue-on-error: true
         run: brew install sccache
       - name: Build libraries
@@ -460,7 +460,7 @@ jobs:
           echo "=== Processor Details ==="
           system_profiler SPHardwareDataType
       - name: Install sccache (shared compiler cache)
-        if: env.USE_CACHE == 'true'
+        if: env.USE_CACHE == 'true' && env.SCCACHE_WEBDAV_TOKEN != ''
         continue-on-error: true
         run: brew install sccache
       - name: Build libraries
@@ -578,7 +578,7 @@ jobs:
         with:
           arch: x64
       - name: Install sccache (shared compiler cache)
-        if: env.USE_CACHE == 'true'
+        if: env.USE_CACHE == 'true' && env.SCCACHE_WEBDAV_TOKEN != ''
         continue-on-error: true
         shell: pwsh
         run: |
@@ -632,7 +632,7 @@ jobs:
         with:
           arch: x86
       - name: Install sccache (shared compiler cache)
-        if: env.USE_CACHE == 'true'
+        if: env.USE_CACHE == 'true' && env.SCCACHE_WEBDAV_TOKEN != ''
         continue-on-error: true
         shell: pwsh
         run: |
@@ -723,7 +723,7 @@ jobs:
           echo "=== Processor Details ==="
           system_profiler SPHardwareDataType
       - name: Install sccache (shared compiler cache)
-        if: env.USE_CACHE == 'true'
+        if: env.USE_CACHE == 'true' && env.SCCACHE_WEBDAV_TOKEN != ''
         continue-on-error: true
         run: brew install sccache
       - name: Build libraries

CLAUDE.md

@@ -290,6 +290,25 @@ sccache as the launcher: it compiles a trivial TU *through* sccache, and only se
 when a job sets one) for diagnosis but never reds the build. This closes the gap the original
 absent-only guard left.
 
+**The fork-PR `.sccache_check` 403 (mac-only symptom) and its two guards.** A fork PR (e.g.
+`vaiju1981/java-llama.cpp` → upstream) runs with secrets withheld, so `SCCACHE_WEBDAV_TOKEN`
+(`= secrets.DEPOT_TOKEN`) is **empty**. Depot rejects the unauthenticated server-startup
+`.sccache_check` with **403 Forbidden** (`PermissionDenied (temporary) … Forbidden`), and
+because sccache treats a failed startup check as fatal, *every* TU dies. The symptom looked
+**mac-only** purely because of an asymmetry in how sccache reaches `PATH`: the macOS jobs ran
+`brew install sccache` **unconditionally** (`if: USE_CACHE == 'true'`), whereas the
+Linux/dockcross/aarch64 jobs only **fetch** sccache when a token is present (the `[ -n
+"$SCCACHE_WEBDAV_TOKEN…" ]` guard in `build.sh`'s fetch block) — so on a tokenless fork PR
+mac was the only platform with sccache on `PATH` to misfire. Two independent guards now prevent
+it: **(1)** every `Install sccache` step is gated `if: env.USE_CACHE == 'true' && env.SCCACHE_WEBDAV_TOKEN
+!= ''`, so a tokenless fork PR never even installs sccache (mac now matches Linux); and **(2)**
+`build.sh`'s build step **retries once without the launcher** when the build fails *and* the
+output shows an sccache cache error (`sccache: error` / `Server startup failed` / `cache storage
+failed`) — a clean uncached `-O3` rebuild that is content-identical and release-safe. The retry
+is gated on that error signature so a genuine compile error still fails fast and is reported
+(no wasteful uncached rebuild). Guard (2) also covers an *intermittent* 403 that strikes a
+valid-token job mid-build, which the one-shot probe cannot foresee.
+
 **Rollout.** **Phase 1 — DONE & proven: the 3 macOS build jobs** (slowest + OOM-prone) —
 `brew install sccache` + the env above + `BUILD_JOBS: 2`. macOS build dropped **~40 min → ~6 min**
 with a warm cache. **Phase 2 — DONE: all 5 dockcross cross-compile jobs** now have the same