context overstated

[dmt4]

I just noticed this, significant overstatement of the context used, way over the actual supported by the model instance:

Despite it, the model does behave well, has sensible, coherent output.

Also note how the max length is reported as 200k when the /v1/models does provide some info to say it is 256k:

curl http://[::1]:8078/v1/models
{"models":[{"name":"unsloth/gemma-4-31B-it-UD-Q8_K_XL","model":"unsloth/gemma-4-31B-it-UD-Q8_K_XL","modified_at":"","size":"","digest":"","type":"model","description":"","tags":[""],"capabilities":["completion","multimodal"],"parameters":"","details":{"parent_model":"","format":"gguf","family":"","families":[""],"parameter_size":"","quantization_level":""}}],"object":"list","data":[{"id":"unsloth/gemma-4-31B-it-UD-Q8_K_XL","aliases":["unsloth/gemma-4-31B-it-UD-Q8_K_XL"],"tags":[],"object":"model","created":1783253170,"owned_by":"llamacpp","meta":{"vocab_type":2,"n_vocab":262144,"n_ctx":262144,"n_ctx_train":262144,"n_embd":5376,"n_params":30697345596,"size":35004205296,"ftype":"Q8_0"}}]}

The provider was added with:

jcode provider add local --base-url http://[::1]:8078/v1 --no-api-key -m "unsloth/gemma-4-31B-it-UD-Q8_K_XL" --set-default

commit: 7d5e840

[1jehuang]

Triage: two separate things in this report.

Max window misreported as 200K (fixed). llama.cpp's /v1/models reports the serving context only inside meta (n_ctx / n_ctx_train), which jcode's catalog parser did not read, so your 256K model fell back to the generic 200K default. Fixed on master in 60c1901: the parser now falls back to meta.n_ctx (then n_ctx_train), with a test using your exact /v1/models response shape. After the next release your model should report 262,144.

Note you can also pin it explicitly today via a named provider profile:

[[providers.local.models]]
id = "unsloth/gemma-4-31B-it-UD-Q8_K_XL"
context_window = 262144

Usage overstatement (needs more info). The screenshot shows usage well above the real window while output stays coherent. That figure comes from provider-reported stream usage; if llama.cpp is doing its own context shifting/truncation, cumulative token counts can exceed the window. If this persists after the release with the corrected window, could you share what usage the server returns on a long turn (e.g. prompt_tokens) so we can tell whether it's a server-side cumulative count vs a jcode aggregation bug?

Will close the window-detection part automatically on the next release.

[1jehuang]

Released in v0.37.0. Run jcode update to get it.