Single 3090 with Unsloth Qwen 3.6 27B Q4, enabling MTP reduced my context size by almost 90% (or about 120k). Is this normal? (latest build)

[regunakyle]

Note: Latest version of llama.cpp (b4c0549)

My run command:

llama-server \
--temp 0.6 \
--top-p 0.95 \
--top-k 20 \
--presence_penalty 0.0 \
--min-p 0.00 \
--gpu-layers all \
-m /home/eleung/huggingface/unsloth/Qwen3.6-27B-MTP-GGUF/Qwen3.6-27B-UD-Q4_K_XL.gguf \
-a llama.cpp \
--host 0.0.0.0 \
--cache-type-k q8_0 --cache-type-v q8_0 \
--chat-template-kwargs '{"preserve_thinking":true}' \
--flash-attn on

The built in web UI shows that context size is 137k.

By adding spec-type draft-mtp --spec-draft-n-max 2, the reported context size drops to 14k. Is this normal?
I was not expecting such a significant reduction in context.

[mikhail-shevtsov-wiregate]

I've observed exactly the same behavior - I couldn't use q8_0 cache-type any longer. So I've dropped to q4_0. I do have GB10 and it works with q8_0 - and honestly I can't see the difference. See my optimal settings (llama.cpp on Ubuntu server 24.04 inside of Docker)

Another huge memory eater is np argument - I use only 1 for VRAM economy.

[Kononnable]

MTP requires more VRAM. Additionally it has its own KV-cache which also increase the cost. You can use -spec-draft-cache-type* parameters to limit the cost of MTP KV-cache(by default f16), but it can't do miracles.

[regunakyle]

Fair, I just didn't expect such a massive decrease of context size. the spec draft cache type params do help a bit

[regunakyle]

I have updated my command with helps from people of r/LocalLLaMA:

llama-server \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --presence_penalty 0.0 \
    --min-p 0.00 \
    --gpu-layers all \
    -m /home/eleung/huggingface/unsloth/Qwen3.6-27B-MTP-GGUF/Qwen3.6-27B-UD-Q4_K_XL.gguf \
    -a llama.cpp \
    --host 0.0.0.0 \
    --cache-type-k q8_0 --cache-type-v q8_0 \
    --chat-template-kwargs '{"preserve_thinking":true}' \
    --flash-attn on \
    --fit-target 64 \
    --no-mmproj \
    --ui-mcp-proxy \
    --spec-type draft-mtp --spec-draft-n-max 1 \
    --jinja --chat-template-file /home/eleung/huggingface/unsloth/Qwen3.6-27B-MTP-GGUF/chat_template.jinja \
    --spec-draft-type-k q4_0 --spec-draft-type-v q4_0

Params that increased my context size (ordered by effectiveness):

1. --fit-target 64 (I feel like this is essential if you run your server headlessly, which I do)
2. --spec-draft-n-max 1 (from 2 to 1)
3. --spec-draft-type-k q4_0 --spec-draft-type-v q4_0 (f16 -> q8_0 has the biggest effect, q8_0 -> q4_0 is not as significant)

Now I have 97.7K context and 57t/s.

Possible improvement:

1. -np 1 improves context size significantly at the cost of parallelism, I think it might affect agent harness usage?
2. Further reduce the quant of kv cache

[clintmarshall]

does this mean that it is just not worth it to use MTP - I really value a decent kv cache size (200k q4)

[Kononnable]

It depends:

• in some situations MTP doesn't increase speed significantly (depending on where your bottlenecks are)
• it will require some additional VRAM - if you were using all available VRAM and barely hitting your desired context window you will loose some. How much depends on the model and quantization used. General rule - if context was small (in MB) you can loose large percentage of it.

As usual - you've got to test it yourself. Only testing can show if you loose 90% of your context or 5% and if it gives you generation speed advantage.

Note: There were some memory related optimizations after this discussion was started, so percentages like 90% are less likely to happen.

[lee-b]

Possibly the reporting is looking at the smaller draft model and so misreporting this?