Fix Qwen3.6-MoE low MTP acceptance rate

[ikawrakow]

This PR fixes the low acceptance rate reported in #1814.

The first draft token always uses the embeddings computed by the main model. But subsequent draft tokens use the embeddings computed by the MTP layer while generating the previous draft token. The issue was that the embedding prior to RMS norm got used, with the effect that any attempt to use more than one draft token resulted in significant performance drop.

Interestingly enough the same issue exists in the dense Qwen3.6-27B model, but for some reason there it makes no observable difference (difference is still within statistical noise after 10 repetitions of the same prompt).

With this change I finally see a small improvement of MTP performance on CUDA. For Qwen3.6-35B-A3B about 10% (IQ4_XS quantization) or even 20% (Q8_0 quantization) when using --draft-max 2 --draft-p-min 0.

Running CPU only, where we had some improvement due to MTP prior to this PR, I now observe a much more significant speedup for Qwen3.6-35B-A3B - about 65% for --draft-max 4 --draft-p-min 0.5.

[SamuelOliveirads]

The fix is also relevant for GLM, at least tried with GLM 5.0 and its now giving a more natural accept rate:

Mode	Value
Post-merge baseline smoke	11.1 +/- 0.1 t/s
Post-hidden-fix --draft-max 1 smoke	10.0 +/- 0.4 t/s @ 88.2% +/- 9.7%
Post-hidden-fix --draft-max 2 smoke	10.6 +/- 0.9 t/s @ 75.9% +/- 14.2%
Post-hidden-fix --draft-max 3 smoke	10.0 +/- 1.6 t/s @ 59.8% +/- 16.3%

So PR #1513 is becoming closer to non-mtp.

[ikawrakow]

The llama.cpp MTP PR has been merged. I was curious to see what it does CPU-only for Qwen3.6-A35B-A3B in comparison to here, where we now have a very significant speedup. For the "Write a quicksort implementation in python" prompt, I see 24.7 t/s without MTP, and 16.7 t/s with MTP and --spec-draft-n-max 4 --draft-p-min 0.5. In comparison, with ik_llama.cpp I get ~40 t/s without MTP and ~66 t/s with MTP.

[jdvpro]

Adding my numbers on EPYC 9454P (48c/96t, Zen 4) with 377GB DDR5-4800 in 12-channel config, running CPU-only.
Qwen3.6-35B-A3B (IQ4_XS, MTP Q8_0):

• Without MTP: ~63 t/s
• With MTP (--draft-max 1): ~82 t/s
• Acceptance rate: 0.856 (304/355)

Observations on tuning:

The --draft-p-min parameter consistently produced slightly worse results than the default setting, regardless of the value used.
With --draft-max set to 2, 3, or 4, throughput was also slightly lower compared to 1, although the difference was minor (around 3–5 t/s).

If needed, I can run any additional tests within my capabilities.

[ikawrakow]

@jdvpro

Thank you for these results. Can you try --draft-max 2 --draft-p-min 0 --mtprot iq4_ks? Thanks.

[jdvpro]

@ikawrakow

I don't currently have support for --mtprot iq4_ks in my build, so I can't run this test at the moment.

version: 4507 (c35189d8)
built with cc (Debian 12.2.0-14+deb12u1) 12.2.0 for x86_64-linux-gnu

If you can point me to the branch or commit with --mtprot iq4_ks support, I'll build it and run the test.

[ikawrakow]

I'm out and writing from the phone, but it is for sure on the latest main branch. It is -mtprot, not --mtprot (single dash)

[jdvpro]

@ikawrakow

You're right, it was a single dash issue.
But now with --draft-max 2 --draft-p-min 0 -mtprot iq4_ks — wow!

• TG: 92.8 t/s !!!
• draft acceptance rate: 0.74151 ( 393 accepted / 530 generated)

I've tried various combinations with -mtprot iq4_ks and it seems only --draft-max 4 --draft-p-min 0 -mtprot iq4_ks gives a slightly higher result — barely noticeable, within 1–2 t/s.
Also worth noting: --draft-p-min 0 is now mandatory. As soon as I remove it, speed drops by 5 t/s immediately.

[ikawrakow]

TG: 92.8 t/s !!!

Who needs a GPU?

[jdvpro]

Some more numbers. Ran two other models with -mtp:

Qwen3-Next-80B-A3B-Instruct iq4_xs:

• Without MTP: ~51.3 t/s
• With MTP (--draft-max 4 --draft-p-min 0.4 -mtprot iq4_xs): ~80.04 t/s
• Acceptance rate: 0.84146 (345/410)

Qwen3.5-122B-A10B iq4_xs:

• Without MTP: ~28.8 t/s
• With MTP (--draft-max 4 --draft-p-min 0.6 -mtprot iq4_xs): ~45.88 t/s
• Acceptance rate: 0.82803 (260/314)

Qwen3.6-35B-A3B iq4_xs (found even better params):

• Without MTP: ~63 t/s
• With MTP (--draft-max 4 --draft-p-min 0.4 -mtprot iq4_xs): ~96.95 t/s
• Acceptance rate: 0.81166 (362/446)

P.S. Had to write a GGUF converter that includes the MTP tensors and port your qwen35moe MTP logic to qwen3next in the engine. The architectures are very similar, so reusing your implementation was mostly straightforward.