Skip to content

Commit 0b1bbc1

Browse files
committed
server: use full logical ctx size for slot.n_ctx when tiered KV is enabled
Hot tier shrinks n_ctx passed to llama init, so llama_n_ctx_seq() returns the hot tier size (~60% of --ctx-size). With tiering, slot.n_ctx must equal kv_tier_total_ctx so requests up to the full context window are accepted.
1 parent 1c4e38c commit 0b1bbc1

1 file changed

Lines changed: 3 additions & 1 deletion

File tree

tools/server/server-context.cpp

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -877,7 +877,9 @@ struct server_context_impl {
877877

878878
const int n_ctx_train = llama_model_n_ctx_train(model);
879879

880-
int n_ctx_slot = llama_n_ctx_seq(ctx);
880+
int n_ctx_slot = (params_base.kv_tiered_enabled && params_base.kv_tier_total_ctx > 0)
881+
? params_base.kv_tier_total_ctx
882+
: llama_n_ctx_seq(ctx);
881883
if (n_ctx_slot > n_ctx_train) {
882884
SRV_WRN("the slot context (%d) exceeds the training context of the model (%d) - using rope scaling to extend\n", n_ctx_slot, n_ctx_train);
883885
// Do not cap: caller has configured rope scaling (--rope-scale / --rope-scaling yarn) to handle extended context.

0 commit comments

Comments
 (0)