perf: exl3 decode kernel optimization experiments#1655
Conversation
Signed-off-by: AlpinDale <alpindale@gmail.com>
AlpinDale
force-pushed
the
perf/exl3-decode-kernel-experiments
branch
from
d872c50 to
6fe5c1c
Compare
AlpinDale
changed the title
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6fe5c1c6ea
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| self.quantization == "exl3" | ||
| and isinstance(requested_dtype, str) | ||
| and requested_dtype.lower() == "auto" | ||
| and self.dtype != torch.float16 | ||
| and "moe" in self.hf_config.model_type.lower() |
There was a problem hiding this comment.
The new EXL3 auto-dtype override is gated by "moe" in self.hf_config.model_type.lower(), which misses valid MoE models whose model_type does not contain that substring (for example, MixtralForCausalLM maps to mixtral in the model registry). In those cases quantization=exl3 with dtype=auto will keep bf16 and skip this new fp16 default, causing inconsistent decode behavior/performance across MoE architectures. This condition should use a structural check like self.is_moe/get_num_experts() instead of name matching.
Useful? React with 👍 / 👎.
Signed-off-by: AlpinDale <alpindale@gmail.com>
1 participant
Fixed a regression from #1652
Trinity-Nano-Preview-4.0bpw: