Skip to content

opencl: add int8 dp4 dense and MoE prefill optimization for Adreno GPUs#25537

Merged
max-krasnyansky merged 2 commits into
ggml-org:masterfrom
qualcomm:hq/dp4a-prefill-moe
Jul 11, 2026
Merged

opencl: add int8 dp4 dense and MoE prefill optimization for Adreno GPUs#25537
max-krasnyansky merged 2 commits into
ggml-org:masterfrom
qualcomm:hq/dp4a-prefill-moe

Conversation

@wanghqc

@wanghqc wanghqc commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Overview

  • This is to add int8 (dp4a) prefill MatMul path alongside the f16 GEMMs: activations quantized to q8_1 once and do int8 dot-product with weights, which may be re-quantized to int8 if needed.
  • This applies to many flavors of dense GEMMs, including q4_K/q6_K/q5_K/q8_0/q5_0/IQ4_NL (+ weight-as-texture).
  • MoE: add added generic per-quant MoE expert GEMM + dedicated q4_0/mxfp4 MoE kernels.
  • Ragged MoE with DP4A: added ragged tiling (skip fully padded per-expert token tiles), and MoE-combine fusion.

Additional information

Here is a quick perf on X2-90, pp512, ON vs OFF same binary, without QPM bin kernels:

  • Qwen3-30B Q4_K +66.7%
  • gpt-oss MXFP4 +54.5%
  • Qwen3-30B Q4_0 +70.2%
  • Token generation neutral (prefill only)

For X1-85: q4_K/q6_K MoE X1 gate wins +20–23%, dense tile +56%.

Right now, it is default-on for Adreno X2E, and there are some regresses on X1's narrower ALU / lower occupancy. X1 keeps the f16 path mostly.

Requirements

@wanghqc wanghqc requested a review from a team as a code owner July 10, 2026 19:03
@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning OpenCL Issues specific to the OpenCL backend labels Jul 10, 2026
@lhez lhez force-pushed the hq/dp4a-prefill-moe branch 2 times, most recently from 1a5011f to 7fd1258 Compare July 10, 2026 21:07
@lhez lhez changed the title opencl: add DP4A (int8) and MoE ragged prefill optimization for Adreno GPUs opencl: add int8 dp4 dense and MoE prefill optimization for Adreno GPUs Jul 10, 2026
@lhez lhez force-pushed the hq/dp4a-prefill-moe branch from 7fd1258 to c9d5ec1 Compare July 10, 2026 23:57
@lhez lhez force-pushed the hq/dp4a-prefill-moe branch from c9d5ec1 to 5149b40 Compare July 11, 2026 00:19
@lhez lhez requested a review from max-krasnyansky July 11, 2026 03:32
@max-krasnyansky max-krasnyansky merged commit 1d1d9a9 into ggml-org:master Jul 11, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning OpenCL Issues specific to the OpenCL backend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants