Unable to get `docker model package` to package with `--chat-template`

[lwsrbrts]

I'm trying to create a new model which has thinking disabled. My original issue, which I assumed a packaged model would resolve, is that using Open WebUI to run Qwen 3.5 9B on a remote GPU workstation results in it "thinking", no matter what. The Open WebUI option to disable thinking doesn't have any effect on the model.

Just as with the context size setting in Open WebUI not having any effect on the model's context, and since I'd had success extending the context size to 40960 using docker model package, my thought was that I could do the same and include a new chat template which specifically disables thinking.

Using Docker's native model called qwen3.5:9B-UD-Q4_K_XL, I've tried the following commands, which all result in the following error:

Failed to package model
package model: failed to load packaged model: loading model archive: load failed with status 500 Internal Server Error: error while loading model: write manifest: missing blob "sha256:a427b36b6993cda8fd07037a8ac96b06be12be49dedecd01e0c6cff13276ade8" for manifest - refusing to write unless all blobs exist

Note that adding or removing single or double-quotes has absolutely no effect on the outcome, so I've not included those examples. Line Feeds I've also tried CRLF (Windows) and LF (Unix) in the chat template, which I've tried as both .txt and .jinja extensions (not that it should make any difference):

Package with from and a chat template - doesn't work.

docker model package --from ai/qwen3.5:9B-UD-Q4_K_XL --chat-template D:\Docker\qwen3-nothinking.txt --context-size 40960 lwsrbrts/qwen35-instruct:9b-40k

I also resorted to downloading the Unsloth GGUF from Hugging Face, the mmproj, and still using my own chat template (adding nother more than {%- set enable_thinking = false %} at line 3), but I get the exact same issue about a missing blob. (Side note, Unsloth are adamant that thinking is disabled in the 9B model, but no matter what, it still thinks when accessed from Open WebUI)

Package from gguf, mmproj and a chat template - doesn't work.

docker model package --gguf D:\Docker\Qwen3.5-9B-UD-Q4_K_XL.gguf --chat-template D:\Docker\qwen3-nothinking.jinja --mmproj D:\Docker\mmproj-BF16.gguf --context-size 40960 lwsrbrts/qwen35-instruct:9b-40k

The GGUF package gives this output. All looks absolutely fine right up until the end (note that the Transferred info is updated when loading the GGUF and it does count up to 5.6GB, then it does the mmproj, which is 879MB):

PS D:\Docker> docker model package --gguf D:\Docker\Qwen3.5-9B-UD-Q4_K_XL.gguf --chat-template D:\Docker\qwen3-nothinking.jinja --mmproj D:\Docker\mmproj-BF16.gguf --context-size 40960 lwsrbrts/qwen35-instruct:9b-40k
Adding GGUF file from "D:\\Docker\\Qwen3.5-9B-UD-Q4_K_XL.gguf"
Setting context size 40960
Adding chat template file from "D:\\Docker\\qwen3-nothinking.jinja"
Adding multimodal projector file from "D:\\Docker\\mmproj-BF16.gguf"
Loading model to Model Runner...
Transferred: 879.01 MB
Failed to package model
package model: failed to load packaged model: loading model archive: load failed with status 500 Internal Server Error: error while loading model: write manifest: missing blob "sha256:a427b36b6993cda8fd07037a8ac96b06be12be49dedecd01e0c6cff13276ade8" for manifest - refusing to write unless all blobs exist

As mentioned, simply changing the context-size works as expected to create the model variant so all eyes are on the chat template but I have no indication what is wrong with the chat template. It is clearly not "missing" as the error suggests, nor is there anything wrong with the line feeds, name of the file, its location or other things I can think of.

[ilopezluna]

Hello @lwsrbrts,
I'll take a closer look at the packaging issue you're hitting.
Regarding the chat template: you likely don't need to switch templates for this. If your goal is to control how much the model "thinks" before answering, you can use the thinking_budget_tokens parameter instead.

Note that this is a llama.cpp-specific option (not part of the standard OpenAI API), and it allows you to limit the number of tokens used for the reasoning phase.

Let me know if that matches what you're trying to achieve, or if there's a specific behavior you're seeing with the template.

curl --request POST \
  --url http://localhost:12434/engines/v1/chat/completions \
  --header 'content-type: application/json' \
  --data '{
  "model": "qwen3.5:9B-UD-Q4_K_XL",
  "thinking_budget_tokens": 0,
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant."
    },
    {
      "role": "user",
      "content": "Hello!"
    }
  ]
}'

Update:
I just remembered OpenWebUI integration is using Ollama API and I just tried:

curl --request POST \
  --url http://localhost:13434/api/chat \
  --header 'content-type: application/json' \
  --data '{
  "model": "qwen3.5:9B-UD-Q4_K_XL",
  "messages": [
    {
      "role": "user",
      "content": "Hello"
    }
  ],
  "think": false,
  "stream": false
}'

And I get a correct answer (without "thinking"):

{
  "model": "qwen3.5:9B-UD-Q4_K_XL",
  "created_at": "2026-04-29T19:39:39.778856+02:00",
  "message": {
    "role": "assistant",
    "content": "Hello! How can I assist you today?"
  },
  "done": true
}

I will have a closer look into the OpenWebUI integration as well as model packaging command. I will let you know what I find!

[ericcurtin]

This API should also be possible:

curl http://localhost:12434/api/chat -d '{
  "model": "qwen3.5:9B-UD-Q4_K_XL",
  "messages": [
    {
      "role": "user",
      "content": "how many r in the word strawberry?"
    }
  ],
  "think": false,
  "stream": false
}'

if either doesn't work, it's a bug.

[ilopezluna]

Hello @lwsrbrts I've just tried to enable/disable thinking in OpenWebUI using ai/qwen3.5:9B-UD-Q4_K_XL and I can not reproduce your issue.

Without thinking:

With thinking:

Could you provide more details?

I have also tried changing context (note I'm using as the max_tokens field here to make it easier to validate, but as you can see the output is broken in the middle because the short context defined 100):

[lwsrbrts]

How frustrating. I get this with the same test, minus the num_ctx - which ironically doesn't matter because when the model thinks, it burns through 4096 tokens easily.

Thinking off:

Thinking on:

Perhaps the major differences are my setup?

Self-hosted Open WebUI instance in a docker container on Linux which I access via a Traefik container to give me TLS - typical reverse proxy setup really.

Docker Desktop on a separate Windows physical machine with an RTX 3070Ti, running at boot using --headless, so I don't have to log on. TCP access is enabled on all interfaces 0.0.0.0 via the standard port 12434 and setting CORS as the address of the self-hosted Open Web UI instance.

I then configure Open WebUI to point to the Docker Desktop IP and port by configuring it as an OpenAI endpoint, giving it a prefix so I can tell where the model is running:

Have I missed something in relation to how to connect Open WebUI to the Docker Desktop instance?

[lwsrbrts]

Riiiiiiiiigggggghhhhhhhhhttttt. facepalm

Now we're getting somewhere. You support both OpenAI API and Ollama API connection types to Docker Model Runner and I am (was) using the OpenAI API connection, which ignores the Ollama specific settings that Open WebUI is sending. That makes sense.

I've added the same IP and port, without the /engines/v1 to the Ollama API section, tested successfully and now...lo and behold, turning off think works just as intended.

It's worth mentioning that the suggestion to use thinking_budget_tokens as a workaround did actually resolve the issue as well so there's a double solution.

Thanks for the hints @ilopezluna @ericcurtin

Looking forward to an update about the --chat-template and --mmproj for docker model package :)

[lwsrbrts]

While I can now set thinking on or off successfully when using the Ollama API endpoint, I have a couple more observations about using the Ollama API over the OpenAI API endpoint.

• Token information is not shown when using the Ollama API endpoint.
  • I believe the information was shown with the OpenAI API connection though (and is why I ended on the path of re-packaging in the first place due to the 4096 context size default.
• The Ollama context size setting num_ctx sent by Open WebUI is ignored, requiring me to repackage to use a larger context size than the 4096 default.
  • I used this command line to create a copy of the model with a larger context size: docker model package --from ai/qwen3.5:9B-UD-Q4_K_XL --context-size 40960 lwsrbrts/qwen35:9b-40k

I'm now contemplating whether using the OpenAI API endpoint with thinking_budget_tokens set to 0 might be more useful since token information is helpful in some circumstances...

All this has been very helpful, the whole exercise was to see whether I could host a model on a remote (& headless) Windows workstation which could be accessed by Open WebUI. Open WebUI could technically present that model via its own OpenAI compatible endpoint to other solutions like n8n or Home Assistant (for which thinking ideally needs to be off for speed's sake for local LLMs).

[ilopezluna]

Token information is not shown when using the Ollama API endpoint.

This is interesting, we might be missing an Ollama endpoint or (more likely) some additional information in the response.
Would you like to investigate this? It could be a good first issue if you're interested.

The Ollama context size setting num_ctx sent by Open WebUI is ignored, requiring me to repackage to use a larger context size than the 4096 default.

That’s unexpected to me, I tried it and it worked. I wonder if we might be interfering with the model reload.
To give you a bit of context, when you define a different context size, the model needs to be reloaded. I’ve been doing some testing today and haven’t been able to reproduce that behavior. I’ll try to find more time to look into this next week.

I also found the issue you had when packaging the model and have already created a PR with a fix. In case you’d like to take a look, here it is: #898

[ericcurtin]

Thanks very much for persisting @lwsrbrts this auto-closed when I merged the PR, but feel free to re-open if we haven't fully addressed.

Many of the points you discussed are being actively worked on. For example in an upcoming version of DD, DMR will have the port accessible by default when DMR is on.

But many of the other points you stated are also being worked on.

Keep giving us feedback. And by the way if interested the fastest way of changing the behaviour is opening a PR :)

[lwsrbrts]

Thanks @ilopezluna - glad you found a solution for the packaging error.

For the context size, it might be that I walked into it because of Home Assistant setting a small context itself. More to investigate and test for me perhaps.

Of course, if I felt confident enough to raise a PR, I absolutely would. Thanks again.