[TRITON] MXFP4 Quantization Gluon implementation for gfx950 and gfx1250#4054
[TRITON] MXFP4 Quantization Gluon implementation for gfx950 and gfx1250#4054NimitPtl wants to merge 1 commit into
Conversation
NimitPtl
force-pushed
the
quant_gluon
branch
3 times, most recently
from
206dfc7 to
afa241b
Compare
cagrikymk
changed the title
|
@NimitPtl The target for this PR is https://github.com/ROCm/aiter/tree/quant_gluon branch, is that on purpose or you want to merge to main? |
| from aiter.ops.triton.gluon.quant_mxfp4 import _dynamic_mxfp4_quant_kernel_gluon | ||
| from aiter.ops.triton.gluon.quant_mxfp4 import _dynamic_mxfp4_quant_kernel_gluon_950 | ||
| from aiter.ops.triton.gluon.quant_mxfp4 import _dynamic_mxfp4_quant_kernel_gluon_1250 | ||
| from aiter.jit.utils.chip_info import get_gfx |
There was a problem hiding this comment.
I would use this https://github.com/ROCm/aiter/blob/main/aiter/ops/triton/utils/_triton/arch_info.py which is Triton/Gluon specific
| - "even" (default): `even_round` in `quark.torch.quantization.utils`. | ||
| - etc. | ||
| use_gluon: Whether to use Gluon kernel (default: True). Only available on MI350 (gfx950). | ||
| use_gluon: Whether to use Gluon kernel (default: True). Only available on (gfx950/1250). |
There was a problem hiding this comment.
I think we should use a standardized approach across all kernels and op wrappers. Right now this is all over the place.
My suggestion is that we should add a backend field to each op which can take values "triton", "gluon", or None(default).
if backend is None:
call the best performing kernel we know of for this op on this arch whether gluon or triton. This is hardcoded by the kernel/op developer or whoever tunes the kernel later on.
elif backend == "gluon":
call the gluon kernel if available on that arch(some helper in the https://github.com/ROCm/aiter/blob/main/aiter/ops/triton/utils/_triton/arch_info.py) . Otherwise, issue a warning and fall back to Triton kernel
else:
call the Triton kernel(assumption is there is always a Triton kernel for an op, I think that's a reasonable one to make.)
There was a problem hiding this comment.
https://github.com/ROCm/aiter/pull/2994/changes
this is the base change. This choses triton/gluon based on the user preference.
let me know if this is a satisfactory approach.
|
Can you post the benchmark numbers as well for both gfx950 and gfx1250? |
have posted the same in Test Result in description. |
I think it should be main, however while creating the PR, the older patch was pushed before the repository changes went into affect. |
|
closing the moving this to #4193 |
3 participants
Motivation
Using gluon instead of triton for performance improvement
Technical Details
Gluon kernel, with sizes derived from the TTGIR of Triton kernel.
Test Plan
Test suite as well as benchmark script for testing accuracy and speed of quant kernel.
Test Result
Marginal improvement for gfx950 and around 10% improvement with usage of LDS in gfx1250
Submission Checklist