fix(sycl): stop building the ggml CPU variant matrix with icpx - #11321
Merged
Conversation
Since #11255 and #11276 every GPU image also builds ggml's CPU_ALL_VARIANTS matrix, so a partial offload uses the host's SIMD kernels. That works everywhere except SYCL, where the Makefile compiles the whole tree with icpx -fsycl: icpx never finishes ggml-cpu/arch/x86/repack.cpp at -march=sapphirerapids. In run 30765516644 both sycl_f16 and sycl_f32 stopped at that translation unit and sat there for 5h30m with a single compile in flight until GitHub killed the job at its 6h limit, and turboquant's f16 job lost its runner outright. gcc compiles the same file in seconds in the vulkan and CPU jobs of the same run, so the CPU variant matrix is only unbuildable under icpx. Route SYCL back to the portable fallback binary, which is what these images shipped before #11255. run.sh already prefers *-cpu-all when present and falls back otherwise, so nothing else has to change. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Since #11255 and #11276, every GPU image also builds ggml's
CPU_ALL_VARIANTSmatrix so that partial offload uses the host's SIMD kernels. That works everywhere except SYCL:backend/cpp/llama-cpp/Makefilecompiles the whole tree withicpx -fsycl, and icpx never finishesggml-cpu/arch/x86/repack.cppat-march=sapphirerapids.The result is that every
sycl_f16/sycl_f32job forllama-cppandturboquantruns into GitHub's 6h job limit. From run 30765516644, both the f16 and f32 llama-cpp jobs stop at the identical point:The turboquant f16 job lost its runner outright at ~4h ("The hosted runner lost communication with the server").
This is compile-time pathology in icpx, not a resource problem:
Free disk spaceruns withswap-storage: true, so there is no swap. A compile that did not fit would be SIGKILLed with an explicit error within minutes, not go silent for 5h30m.ggml-syclcompletes at 22:12 exactly one compile process remains on the runner, so there is no contention left to blame.389.2 Building ... repack.cpp.o->389.3 Built target ggml-cpu-sapphirerapids), and the CPU llama-cpp job builds the whole 14-variant matrix in ~17 minutes.It reproduces deterministically, and it is not caused by any particular llama.cpp pin. History on master:
ad2be8a85(Aug 1 16:06)8a80830f3(Aug 2, #11283)Jobs that look green today finish in 7-8 minutes because they are pure registry-cache hits. Any cache miss, such as a llama.cpp pin bump, re-exposes the stall, and master's sycl merge jobs for #11283 already failed.
What
Route SYCL back to the portable fallback binary, which is exactly what these images shipped before #11255. Every other GPU build keeps the CPU variant matrix, so the intent of #11255 / #11276 is preserved for cublas, hipblas and vulkan.
run.shalready defaults to*-fallbackand only upgrades to*-cpu-allwhen that file is present, andpackage.shbundlesggml-shared-libsonly when the directory exists, so nothing else needed changing.Tradeoff: SYCL images lose the per-microarch CPU kernels for host-side work that they had between Aug 1 and now, back to the pre-#11255 behaviour. The alternative is getting icpx to compile that translation unit, which is an upstream compiler issue.
Testing
make test-build-scriptspasses, which is what thebuild-scriptslint job runs.amd64 sycl_f16returnsllama-cpp-cpu-all) and green with this change.shellcheckclean on all four touched scripts.build-type: 'intel'matrix entries are allDockerfile.pythonbackends, so they never reach these selectors.🤖 Generated with Claude Code