Docker Model Runner cannot be connected to after installing a runner

[factor3]

I have been using Docker Model Runner for gguf- based models for some time now. I have been using the following URL to
access it:

http://192.168.2.42:12434/engines/llama.cpp/v1/

This URL has been working well using WebUI as well as my own chat application.

I have recently installed the diffusers runner and stable-diffusion onto my machine. Unfortunately, since installing them no application can connect to my Docker Model Runner instance.

The only way I can access DMR is by using a curl command provided to me by @ilopezluna::

 curl -s -X POST http://localhost:12434/engines/diffusers/v1/images/generations   -H "Content-Type: application/json"   -d '{
    "model": "ai/stable-diffusion",
    "prompt": "A cat sitting on a couch",
    "size": "512x512"
  }'
  

I believe that this returns an image (it does return a lot of data), though I cannot confirm that the data I am seeing is actually an image using curl.

This doesn't matter much, because if I move to another machine and use the curl to access my machine:

 curl -s -X POST http://192.168.2.42:12434/engines/diffusers/v1/images/generations   -H "Content-Type: application/json"   -d '{
    "model": "ai/stable-diffusion",
    "prompt": "A cat sitting on a couch",
    "size": "512x512"
  }'
  

I still cannot connect to the machine.

With WebUI, I have tried placing it on my machine and using:

http://localhost:12434/engines/llama.cpp/v1/

and

http://localhost:12434/engines/diffusers/v1/images/generations

and I am still getting connection errors.

I suspect that this is a bug in DMR's runner. At this point, I cannot use it.

[doringeman]

Hey @factor3,

I believe that this returns an image

Yes, it does, see the example of how to actually see the image: https://docs.docker.com/ai/model-runner/inference-engines/#generating-images-with-diffusers.

I still cannot connect to the machine.

Because by default DMR starts on localhost. Run docker model uninstall-runner, then install-runner --backend diffusers again with --host/--port.

docker model install-runner --help | grep -e --host -e --port
      --host string         Host address to bind Docker Model Runner (default "127.0.0.1")
      --port uint16         Docker container port for Docker Model Runner (default: 12434 for Docker Engine, 12435 for Cloud mode)

and I am still getting connection errors.

You need http://localhost:12434/engines/v1, see https://docs.docker.com/ai/model-runner/openwebui-integration/#docker-engine-linux.

Regarding using Open WebUI for generating images, me and @ilopezluna will come back with some steps on this, I don't have them handy right now. 🙌

[ilopezluna]

Hello @factor3

I believe that this returns an image (it does return a lot of data), though I cannot confirm that the data I am seeing is actually an image using curl.

The model returns an image in base64, you can store it as PNG and verify it's a cata :D

curl -s -X POST http://localhost:12434/engines/diffusers/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai/stable-diffusion",
    "prompt": "A cat sitting on a couch",
    "size": "512x512"
  }' | jq -r '.data[0].b64_json' | base64 -d > cat.png

I still have to check image generation support using OpenWebUI 👍

[factor3]

Okay, @doringeman:

First of all: thank you for the information -- especially for the correct WebUI container and the docker-compose.yml file. It turns out that I was using a different container and I wasn't using docker compose to start it. I had looked at the wrong instructions for using WebUI with DMR.

In any event, I have uninstalled the runner and reinstalled it:

$ docker model uninstall-runner
$ docker model install-runner --backend diffusers --host 192.168.2.42

The status, after that, actually looks good (!):

$ docker model status
Docker Model Runner is running

BACKEND    STATUS         DETAILS
diffusers  Running        diffusers v0.1.0-20260216-000000
llama.cpp  Running        llama.cpp 3191462
sglang     Error          Python not found
mlx        Not Installed  only supported on Apple Silicon
vllm       Not Installed  binary not found

I have made the appropriate changes in the docker-compose file:

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "9020:8080"
    environment:
      - OPENAI_API_BASE_URL=http://192.168.2.42:12434/engines/v1
      - OPENAI_API_KEY=not-needed
      - WEBUI_AUTH=false
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - open-webui:/app/backend/data

volumes:
  open-webui:

And, of course, I started up the WebUI system in the usual manner:

$ docker compose up -d && docker compose logs -f

The WebUI system does not connect to DMR.

The only way it actually connects to DMR is if I change the URL back to localhost:

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "9020:8080"
    environment:
      - OPENAI_API_BASE_URL=http://localhost:12434/engines/v1
      - OPENAI_API_KEY=not-needed
      - WEBUI_AUTH=false
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - open-webui:/app/backend/data

volumes:
  open-webui:

This despite the fact that I did change the host when I installed the runner.

A couple of things to note:

1. WebUI does send prompts to my GGUF LLMs and uses them without problems. Unfortunately, It generates weird errors when I attempt to send a prompt to the stable-diffusion LLM.

2. I cannot access DMR from anywhere except localhost -- which makes it impossible to (for example) test my own chatting code from another machine.

3. Even if I run my chatting code, or any other AI based software (like N8N or Openhands) on the same machine as DMR using the same localhost URL as WebUI, I am getting connection errors.

I now have limited use of DMR (through WebUI), but it is still not usable for any of my larger purposes.

[ilopezluna]

Hello @factor3 I just tried OpenWebUI to generate an image via DMR and it works

You have to configure OpenWebUI to use DMR for image generation:

1. Go to http://localhost:3000/admin/settings/general (or wherever you have OpenWebUI running)
2. Go to images

As you can see I have configured the DMR endpoint there (note I'm running DMR in 13434 port).
And that's pretty much it. In your chats you must enable the image integration:

So OpenWebUI will let the current model know that can generate images via that endpoint. Hope it helps!

[factor3]

Greetings again, @ilopezluna:

Thank you for the instructions. They were very helpful.

Below are my settings for image generation:

My prompt is below:

The result of sending my prompt is below:

Note the 500 failure below the picture. It prevents further conversations from taking place. If I try to add to the conversation -- like for example, "make the cat black", I get a brief message saying "OOPS! There was an error in the previous response".

Add to that the fact that I am still unable to access DMR from another machine for any model, meaning that I cannot continue development of my AI systems, and the fact that no other applications (N8N, my apps, or Openhands) can access DMR even if I put them on the same machine, and I still cannot use DMR for everything I need to use it for.

I am considering using Ollama to handle my local AI requirements...

[doringeman]

Can you try chatting with another model, not with the stable-diffusion?
It'll still use stable-diffusion for generating images.
@ilopezluna has been working on a step by step guide on how to set this up properly, he might share/publish it very soon.

Add to that the fact that I am still unable to access DMR from another machine

After running docker model install-runner --host 192.168.2.42, does it work to access it from the same machine using that IP? Has it successfully started? Is it listening on the expected IP:PORT?

docker ps --filter label=com.docker.desktop.service

Also see:

docker model logs -f

[factor3]

Greetings, @doringeman:

Answering your questions:

1. Actually, I did try chatting with another model, with and without switching on the images mode with the prompt. I noticed in @ilopezluna's post that he was not using stable-diffusion with his prompt, so I tried using another model inorder to a) make sure that I could access my other models and b) see what would happen with my images using them.

Without the imaging switch, the prompts work without problems -- as long as I do not prompt for images. This is good. Unfortunately, with the images switch on and with prompts for images, I get one of two responses. The first is a 500 Connection error. The other response (especially if I use llama3.2) says that the model is a text model and then it offers me a text based description of the picture. Please see below:

2. As I had said 3 days ago: I had, as @ilopezluna had instructed, uninstalled then reinstalled the runner:

$ docker model uninstall-runner
$ docker model install-runner --backend diffusers --host 192.168.2.42

Unfortunately, when I attempted to access any model using 192.168.2.42, I was unable to -- even when I was trying to doso from the same system. I can only access models using localhost, and then only using WebUI.

Please note, however, that in my settings->images setup (please see my previous post) I am using the following URL:

http://host.docker.internal:12434/engines/diffusers/v1

That is, apparently, the only URL that prevents 500 connection errors whenever I send a prompt.

And yes: I have monitored the DMR using the commands you suggested (except that I used "docker ps" because I am not using docker desktop), several times. Yes, the model and the WebUI container have started up. Unfortunately, whenever I am getting connection errors, there are no corresponding entries in the logs. In the logs for WebUI, I am getting 404 errors.

[ilopezluna]

Hello @factor3! When using Open WebUI, there are multiple calls to your models happening behind the scenes.

Open WebUI uses follow-up questions/suggestions (open-webui/open-webui#18579). This means that when you're chatting with a model, it may send additional messages to generate things like chat titles or follow-up questions. You can disable this behavior in the admin panel.

I mention this because those 500 errors you saw when using the Stable Diffusion model for chat are likely caused by this. Open WebUI was sending chat requests to Stable Diffusion, even though you were only trying to generate an image.

I'm not entirely sure how Open WebUI internally distinguishes between chat and image requests. I assume it lets the model decide whether to respond with text or trigger an image generation via another model (as a tool call).

Keep in mind that smaller models like Llama 3.2 don’t handle tool calls very well. Even if an image-generation tool is available, they may fail to use it correctly. That could explain why Llama 3.2 responds with something like “I can’t generate the image.”

I was using gpt-oss because, even though it's a small model, it’s more capable than Llama 3.2.

http://host.docker.internal:12434/engines/diffusers/v1

This URL allows a container (where Open WebUI is running) to access your local host (where DMR is running). So it’s essentially telling Open WebUI how to reach DMR. It’s not related to the 500 errors—those are coming from chat requests being sent to the Stable Diffusion model, which isn’t designed for chat.

I’m not sure why you can’t access DMR from outside localhost—we might have a bug there. We’re a small team, so any help is very welcome. You’re already helping a lot with your feedback, so thank you!

We’re planning to publish a blog post next week, and just so you know—you inspired it 🙂 Thanks again! I’ll share it with you as soon as it’s published.

[factor3]

Greetings, @ilopezluna:

Did your team update the DMR?

I had decided to (just for grins and giggles) reinstall the runner(s) in the manner suggested yesterday by @doringeman. After a bit of futzing around (trying to install runners that were already running), I ended up doing the following:

$ docker model reinstall-runner --host 192.168.2.42

This caused a downloading of the model runner container (which is why I am wondering if you folks updated it), and it had a few other consequences:

1. I can now access DMR models from outside of localhost! I tested this with WebUI (on another machine) and from my development environment for my AI projects. The WebUI works (for text models)and my AI software no longer returns connection errors.

2. I can no longer access the DMR command line commands on my machine. Any docker model command I use now returns the following error:

$ docker model status
Failed to get Docker Model Runner status: error querying /models: Get "http://localhost:12434/models": dial tcp 127.0.0.1:12434: connect: connection refused
$ docker model reinstall-runner --backend diffusers --host 192.168.2.42
Installing diffusers backend...
failed to install diffusers backend: error querying /engines/install-backend: Post "http://localhost:12434/engines/install-backend": dial tcp 127.0.0.1:12434: connect: connection refused

So: have I found another bug, or is there a way to access the DMR from the command line using my (new) host address?

Oh, by the way: I am honored that you are publishing a blog post inspired by me. I wish I can help with coding, but unfortunately between work and my business project I don't have time to work on anything else (at least, not now or in the immediate future). I hope that I can continue to help with my feedback, though.

[factor3]

Greetings again:

I have tested the imaging capability of WebUI using the following URL setting on the /admin/settings.images page:

http://192.168.2.42:12434/engines/diffusers/v1

Apparently, DMR automatically downloads the diffusers runner if it isn't already installed (I saw the download and installation in the logs).

The image is now generated without the 500 connection error:

Note that I did not pull the gpt-oss model because I cannot.

$ docker model pull gpt-oss
Retrying download (attempt 1/3) in 1s...
Retrying download (attempt 2/3) in 2s...
Retrying download (attempt 3/3) in 4s...
Failed to pull model: failed to download after 3 retries: error querying /models/create: Post "http://localhost:12434/models/create": dial tcp 127.0.0.1:12434: connect: connection refused

This thing with not connecting to localhost is becoming a real problem...

[ilopezluna]

Hello @factor3 ! You can use MODEL_RUNNER_HOST to point the cli to a specific host:

MODEL_RUNNER_HOST=http://192.168.2.42:12434 docker model pull gpt-oss

Let me know if you face any other issue 🙏

[factor3]

Thank you, @ilopezluna, for the information on the environment variable.

Is there a way to start up the model runner with MODEL_RUNNER_HOST set, so that I won't have to set it for every command I do? No big deal if there isn't, but it would be very convenient if there is...

[doringeman]

Hey @factor3, how would the command you're thinking about look like?

You could either have an alias in your .bashrc/.zshrc like alias dm='MODEL_RUNNER_HOST=http://192.168.2.42:12434 docker model'.

[factor3]

@doringeman

I have actually done that. I still think it would be more convenient if doing the alias wasn't necessary.

[ilopezluna]

We could use something like docker context, wdyt?