Skip to content

fix(sycl): stop building the ggml CPU variant matrix with icpx - #11321

Merged
mudler merged 1 commit into
masterfrom
fix/sycl-skip-cpu-all
Aug 3, 2026
Merged

fix(sycl): stop building the ggml CPU variant matrix with icpx#11321
mudler merged 1 commit into
masterfrom
fix/sycl-skip-cpu-all

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

Why

Since #11255 and #11276, every GPU image also builds ggml's CPU_ALL_VARIANTS matrix so that partial offload uses the host's SIMD kernels. That works everywhere except SYCL: backend/cpp/llama-cpp/Makefile compiles the whole tree with icpx -fsycl, and icpx never finishes ggml-cpu/arch/x86/repack.cpp at -march=sapphirerapids.

The result is that every sycl_f16 / sycl_f32 job for llama-cpp and turboquant runs into GitHub's 6h job limit. From run 30765516644, both the f16 and f32 llama-cpp jobs stop at the identical point:

21:59:06  [ 48%] Building CXX object .../ggml-cpu-sapphirerapids.dir/ggml-cpu/arch/x86/repack.cpp.o
22:12:01  [ 59%] Built target ggml-sycl          <- last line in the whole log
   ...    (5h30m of silence, one compile process in flight)
03:43:28  The job has exceeded the maximum execution time of 6h0m0s

The turboquant f16 job lost its runner outright at ~4h ("The hosted runner lost communication with the server").

This is compile-time pathology in icpx, not a resource problem:

  • Free disk space runs with swap-storage: true, so there is no swap. A compile that did not fit would be SIGKILLed with an explicit error within minutes, not go silent for 5h30m.
  • After ggml-sycl completes at 22:12 exactly one compile process remains on the runner, so there is no contention left to blame.
  • gcc compiles the same file in seconds on the same runner class in the vulkan job of the same run (389.2 Building ... repack.cpp.o -> 389.3 Built target ggml-cpu-sapphirerapids), and the CPU llama-cpp job builds the whole 14-variant matrix in ~17 minutes.

It reproduces deterministically, and it is not caused by any particular llama.cpp pin. History on master:

master commit contains sycl llama-cpp sycl turboquant
Jul 27 / Jul 28 neither fix ok, ~50 min ok, ~50 min
ad2be8a85 (Aug 1 16:06) #11255 only, #11276 merged 23s later 6h timeout ok, 50 min
8a80830f3 (Aug 2, #11283) both 6h timeout 6h timeout

Jobs that look green today finish in 7-8 minutes because they are pure registry-cache hits. Any cache miss, such as a llama.cpp pin bump, re-exposes the stall, and master's sycl merge jobs for #11283 already failed.

What

Route SYCL back to the portable fallback binary, which is exactly what these images shipped before #11255. Every other GPU build keeps the CPU variant matrix, so the intent of #11255 / #11276 is preserved for cublas, hipblas and vulkan.

run.sh already defaults to *-fallback and only upgrades to *-cpu-all when that file is present, and package.sh bundles ggml-shared-libs only when the directory exists, so nothing else needed changing.

Tradeoff: SYCL images lose the per-microarch CPU kernels for host-side work that they had between Aug 1 and now, back to the pre-#11255 behaviour. The alternative is getting icpx to compile that translation unit, which is an upstream compiler issue.

Testing

  • make test-build-scripts passes, which is what the build-scripts lint job runs.
  • The new assertions are red against master's selectors (amd64 sycl_f16 returns llama-cpp-cpu-all) and green with this change.
  • shellcheck clean on all four touched scripts.
  • Verified that the separate build-type: 'intel' matrix entries are all Dockerfile.python backends, so they never reach these selectors.

🤖 Generated with Claude Code

Since #11255 and #11276 every GPU image also builds ggml's CPU_ALL_VARIANTS
matrix, so a partial offload uses the host's SIMD kernels. That works
everywhere except SYCL, where the Makefile compiles the whole tree with
icpx -fsycl: icpx never finishes ggml-cpu/arch/x86/repack.cpp at
-march=sapphirerapids. In run 30765516644 both sycl_f16 and sycl_f32 stopped
at that translation unit and sat there for 5h30m with a single compile in
flight until GitHub killed the job at its 6h limit, and turboquant's f16 job
lost its runner outright. gcc compiles the same file in seconds in the vulkan
and CPU jobs of the same run, so the CPU variant matrix is only unbuildable
under icpx.

Route SYCL back to the portable fallback binary, which is what these images
shipped before #11255. run.sh already prefers *-cpu-all when present and falls
back otherwise, so nothing else has to change.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler
mudler merged commit a0f7faa into master Aug 3, 2026
1 of 22 checks passed
@mudler
mudler deleted the fix/sycl-skip-cpu-all branch August 3, 2026 17:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants