Set max context length per request independently of unified kv cache total size?

[jettoblack]

Correct me if my understanding is wrong, but there doesn't seem to be a way to do this today. In llama-server -c / --ctx-size currently serves a dual purpose: It sets the total unified kv cache size that is shared by all slots during parallel processing, and also sets the max length for any single prompt whether parallel processing is in use or not.

You can set ctx-size to larger than the model's native max context length to get a bigger unified kv cache for more parallel requests, but this also sets the max prompt length. Prompts will still ultimately be capped to the model's max length, but there doesn't seem to be a way to limit prompts to a shorter length than that cap while also increasing the unified kv cache size beyond that.

I think a second parameter is needed for the unified kv cache size, which defaults to the same size as the context length to maintain current behavior, but can be made larger. Something like: --ctx-size 128k --unified-kv-cache-size 512k

Use case example: Using a model with a native 256k max context length, I want to configure a max prompt length of 128k and a unified cache big enough for 4 parallel requests at 128k. If I set -c 512k, then the unified cache size will be correct, but the max prompt length will end up at 256k, not 128k, and its possible 2 requests will saturate the unified cache. If I set -c 128k then the max prompt length will be 128k, but the unified cache will be too small for 4 parallel requests.

[tftranslate]

I fully agree that the dual semantics of --ctx-size is a huge problem. I would like to reserve 524288 KV cache so that who longish of 262144 can stay in VRAM in the KV cache in parallel. But because I use MTP, the docs tell me I need to set parallel=1 and enable unified KV. Without a separate flag that limits the context window per request to 262144, I then have a config in which a single chat can potentially exceed the max model context size of 262144 and I will run into undefined behavior. I therefore fully support the ieade that --ctx-size 128k should strictly limit the context size per request (and lead to either "context window exceeded" errors or to a rolling context window bheavior if the window is exceeded), and a separate parameter to set the total kv-cache-size is nedded.

[ggerganov]

But because I use MTP, the docs tell me I need to set parallel=1 and enable unified KV.

Which docs are saying that? You can use any -np with MTP.

[tftranslate]

Thanks for clarifying. I must admit that it was only Perplexity that told me that. I now checked Perplexity's sources and find no primary source. Secondary sources are https://www.glukhov.org/de/llm-performance/optimization/speculative-decoding/, https://x.com/TeksEdge/status/2055468277934051434. Funny how two random posts on the web can guide chatbot answers.

I take your word that this is bullshit or outdated and that as of today, I can safely use gemma-4 31b with model-draft an spec-type = draft-mtp and/or qwen-3.6-27b with spectype = deraft-mtp,ngram-mod and its integrated MTP heads and set -np 2? Even if two users will talk to the model in parallel this will not create any race conditition?

Still, I hold that it is quite counter-intuitive that ctx-size does not set the context window length but rather the total KV cache size. Took me quite some time to figure that one out ...

[ggerganov]

Using a model with a native 256k max context length, I want to configure a max prompt length of 128k and a unified cache big enough for 4 parallel requests at 128k.

For this use case you don't need a unified KV cache. These args should work exactly as you want:

llama serve ... -np 4 --no-kv-unified -c 524288