Skip to content

Commit 5462671

Browse files
mios-devclaude
andcommitted
mios-ollama-igpu: sibling Quadlet for the micro-LLM lane
Operator directive 2026-05-18: "iGPU model(s) isn't working" Root cause: ollama/ollama:latest is the CUDA-only image. Even with the AMD CDI device wired via /etc/containers/systemd/ollama.container.d/ 20-mios-igpu.conf (AddDevice=amd.com/gpu=all), ollama-CUDA can't compute on the AMD device -- it falls back to CUDA-only on the dGPU, so qwen3:1.7b and the rest of the micro-LLM lane shared VRAM with qwen2.5-coder:7b on the dGPU. The "iGPU lane" wasn't running anywhere. Fix: a sibling ollama instance using ollama/ollama:rocm, pinned to the AMD device only, on a distinct port (:11435). The original mios-ollama keeps the dGPU + CUDA for big models; this iGPU sibling owns the micro-LLM lane. mios-daemon + agent-pipe router clients can re-point at :11435 in a follow-up commit once we verify the new lane day-to-day (this commit just stands the service up; client re-points come next). NEW: usr/share/containers/systemd/mios-ollama-igpu.container Quadlet sibling to mios-ollama. Key differences: * Image=docker.io/ollama/ollama:rocm (vs :latest) * AddDevice=amd.com/gpu=all (NO nvidia) * OLLAMA_HOST=0.0.0.0:11435 (distinct port) * User=823 / Group=823 (new sysuser) * Network=host (same workaround as mios-surrealdb -- WSL podman-bridge port- mapping quirk fails with dual-bridge + PublishPort even though `podman port` reports the mapping) * NUM_PARALLEL=8 / MAX_LOADED_MODELS=6 (tighter slots since models are small) Shares /var/lib/ollama with the CUDA instance so seeded model blobs are reachable from both lanes; HOME stays per-instance at /var/lib/ollama-igpu so the ssh-keygen identity file doesn't collide. NEW: usr/lib/sysusers.d/50-mios-services.conf mios-ollama-igpu at uid 823 (next free after agent-pipe=822). NEW: usr/lib/tmpfiles.d/mios-ollama-igpu.conf d /var/lib/ollama-igpu 0750 823 823 - SSOT entries: mios.toml: [ports].ollama_igpu = 11435 [image.sidecars].ollama_igpu = docker.io/ollama/ollama:rocm [services.ollama_igpu] (user=mios-ollama-igpu uid=823 gid=823) tools/lib/userenv.sh: Slot map adds ports.ollama_igpu -> MIOS_PORT_OLLAMA_IGPU, services.ollama_igpu.* -> MIOS_OLLAMA_IGPU_USER/UID/GID, image.sidecars.ollama_igpu* -> MIOS_OLLAMA_IGPU_IMAGE/VERSION. automation/15-render-quadlets.sh: Allowlist gains MIOS_OLLAMA_IGPU_IMAGE/USER/UID/GID + MIOS_PORT_OLLAMA_IGPU so the .container placeholders resolve at image build time. Live-verified on podman-MiOS-DEV: systemctl is-active mios-ollama-igpu.service -> active curl http://localhost:11435/api/version -> {"version":"0.24.0"} qwen3:1.7b inference (api/generate, 5-token cap) -> 3.0s end-to-end Inference compute: id=cpu library=cpu (see "WSL caveat" below) WSL caveat (operator-confirmed via /dev/kfd + /dev/dri missing): The WSL kernel doesn't expose AMD device nodes (/dev/kfd, /dev/dri/renderD*), so ROCm inside ollama:rocm has no device to bind and falls back to CPU. The micro-LLM lane is still useful on WSL because it ISOLATES the small-model workload from the dGPU/CUDA queue -- micro-LLMs no longer fight qwen2.5-coder:7b for dGPU memory. On bare-metal MiOS-bootc with the AMD iGPU, ROCm would activate (kfd + dri exposed by the host kernel) and the lane would run on the iGPU proper. Same Quadlet, different capability surface per deployment type. Follow-ups (queued as separate tasks): * Repoint mios-daemon's MIOS_DAEMON_ENDPOINT and agent-pipe router's MIOS_AGENT_PIPE_ROUTER_ENDPOINT at :11435 (the iGPU lane) so micro-LLM traffic actually lands there. * Step 4/5 of the pipe migration (Hermes Discord through :8640). * Hermes CPU-delegation + kanban auto-dispatch for multi-task prompts (Tasks #153, queued). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 8339885 commit 5462671

6 files changed

Lines changed: 131 additions & 2 deletions

File tree

automation/15-render-quadlets.sh

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ _render_with_envsubst() {
6565
content="${content//${BASH_REMATCH[0]}/${_rep}}"
6666
done
6767
# shellcheck disable=SC2016
68-
printf '%s' "$content" | envsubst '${MIOS_K3S_IMAGE} ${MIOS_CEPH_IMAGE} ${MIOS_FORGE_IMAGE} ${MIOS_SEARXNG_IMAGE} ${MIOS_HERMES_IMAGE} ${MIOS_OPEN_WEBUI_IMAGE} ${MIOS_CODE_SERVER_IMAGE} ${MIOS_OLLAMA_IMAGE} ${MIOS_GUACAMOLE_IMAGE} ${MIOS_FORGE_RUNNER_IMAGE} ${MIOS_CROWDSEC_IMAGE} ${MIOS_POSTGRES_IMAGE} ${MIOS_GUACD_IMAGE} ${MIOS_PXE_HUB_IMAGE} ${MIOS_BIB_ALPINE_IMAGE} ${MIOS_SURREALDB_IMAGE} ${MIOS_PORT_SSH} ${MIOS_PORT_FORGE_HTTP} ${MIOS_PORT_FORGE_SSH} ${MIOS_PORT_COCKPIT} ${MIOS_PORT_COCKPIT_LINK} ${MIOS_PORT_OLLAMA} ${MIOS_PORT_SEARXNG} ${MIOS_PORT_HERMES} ${MIOS_PORT_OPEN_WEBUI} ${MIOS_PORT_CODE_SERVER} ${MIOS_K3S_API_PORT} ${MIOS_GUACAMOLE_PORT} ${MIOS_CEPH_DASHBOARD_PORT} ${MIOS_RDP_PORT} ${MIOS_PORT_SURREALDB} ${MIOS_FORGE_USER} ${MIOS_FORGE_UID} ${MIOS_FORGE_GID} ${MIOS_SEARXNG_USER} ${MIOS_SEARXNG_UID} ${MIOS_SEARXNG_GID} ${MIOS_CEPH_USER} ${MIOS_CEPH_UID} ${MIOS_CEPH_GID} ${MIOS_HERMES_USER} ${MIOS_HERMES_UID} ${MIOS_HERMES_GID} ${MIOS_OPEN_WEBUI_USER} ${MIOS_OPEN_WEBUI_UID} ${MIOS_OPEN_WEBUI_GID} ${MIOS_OLLAMA_USER} ${MIOS_OLLAMA_UID} ${MIOS_OLLAMA_GID} ${MIOS_SURREALDB_USER} ${MIOS_SURREALDB_UID} ${MIOS_SURREALDB_GID} ${MIOS_CODE_SERVER_UID} ${MIOS_CODE_SERVER_GID} ${MIOS_QUADLET_NETWORK} ${MIOS_QUADLET_SUBNET} ${MIOS_CORE_NET_SUBNET} ${MIOS_CORE_NET_GATEWAY} ${MIOS_AI_NET_SUBNET} ${MIOS_AI_NET_GATEWAY} ${MIOS_AI_DIR} ${MIOS_AI_MODELS_DIR} ${MIOS_AI_MCP_DIR} ${MIOS_DB_USER} ${MIOS_DB_PASS} ${MIOS_DB_BACKEND} ${MIOS_DB_DATA_DIR}'
68+
printf '%s' "$content" | envsubst '${MIOS_K3S_IMAGE} ${MIOS_CEPH_IMAGE} ${MIOS_FORGE_IMAGE} ${MIOS_SEARXNG_IMAGE} ${MIOS_HERMES_IMAGE} ${MIOS_OPEN_WEBUI_IMAGE} ${MIOS_CODE_SERVER_IMAGE} ${MIOS_OLLAMA_IMAGE} ${MIOS_OLLAMA_IGPU_IMAGE} ${MIOS_GUACAMOLE_IMAGE} ${MIOS_FORGE_RUNNER_IMAGE} ${MIOS_CROWDSEC_IMAGE} ${MIOS_POSTGRES_IMAGE} ${MIOS_GUACD_IMAGE} ${MIOS_PXE_HUB_IMAGE} ${MIOS_BIB_ALPINE_IMAGE} ${MIOS_SURREALDB_IMAGE} ${MIOS_PORT_SSH} ${MIOS_PORT_FORGE_HTTP} ${MIOS_PORT_FORGE_SSH} ${MIOS_PORT_COCKPIT} ${MIOS_PORT_COCKPIT_LINK} ${MIOS_PORT_OLLAMA} ${MIOS_PORT_OLLAMA_IGPU} ${MIOS_PORT_SEARXNG} ${MIOS_PORT_HERMES} ${MIOS_PORT_OPEN_WEBUI} ${MIOS_PORT_CODE_SERVER} ${MIOS_K3S_API_PORT} ${MIOS_GUACAMOLE_PORT} ${MIOS_CEPH_DASHBOARD_PORT} ${MIOS_RDP_PORT} ${MIOS_PORT_SURREALDB} ${MIOS_FORGE_USER} ${MIOS_FORGE_UID} ${MIOS_FORGE_GID} ${MIOS_SEARXNG_USER} ${MIOS_SEARXNG_UID} ${MIOS_SEARXNG_GID} ${MIOS_CEPH_USER} ${MIOS_CEPH_UID} ${MIOS_CEPH_GID} ${MIOS_HERMES_USER} ${MIOS_HERMES_UID} ${MIOS_HERMES_GID} ${MIOS_OPEN_WEBUI_USER} ${MIOS_OPEN_WEBUI_UID} ${MIOS_OPEN_WEBUI_GID} ${MIOS_OLLAMA_USER} ${MIOS_OLLAMA_UID} ${MIOS_OLLAMA_GID} ${MIOS_OLLAMA_IGPU_USER} ${MIOS_OLLAMA_IGPU_UID} ${MIOS_OLLAMA_IGPU_GID} ${MIOS_SURREALDB_USER} ${MIOS_SURREALDB_UID} ${MIOS_SURREALDB_GID} ${MIOS_CODE_SERVER_UID} ${MIOS_CODE_SERVER_GID} ${MIOS_QUADLET_NETWORK} ${MIOS_QUADLET_SUBNET} ${MIOS_CORE_NET_SUBNET} ${MIOS_CORE_NET_GATEWAY} ${MIOS_AI_NET_SUBNET} ${MIOS_AI_NET_GATEWAY} ${MIOS_AI_DIR} ${MIOS_AI_MODELS_DIR} ${MIOS_AI_MCP_DIR} ${MIOS_DB_USER} ${MIOS_DB_PASS} ${MIOS_DB_BACKEND} ${MIOS_DB_DATA_DIR}'
6969
}
7070

7171
# Bash-only fallback for hosts without envsubst. Walks the same allow-
@@ -95,6 +95,8 @@ _render_with_bash() {
9595
MIOS_HERMES_USER MIOS_HERMES_UID MIOS_HERMES_GID \
9696
MIOS_OPEN_WEBUI_USER MIOS_OPEN_WEBUI_UID MIOS_OPEN_WEBUI_GID \
9797
MIOS_OLLAMA_USER MIOS_OLLAMA_UID MIOS_OLLAMA_GID \
98+
MIOS_OLLAMA_IGPU_USER MIOS_OLLAMA_IGPU_UID MIOS_OLLAMA_IGPU_GID \
99+
MIOS_OLLAMA_IGPU_IMAGE MIOS_PORT_OLLAMA_IGPU \
98100
MIOS_SURREALDB_USER MIOS_SURREALDB_UID MIOS_SURREALDB_GID \
99101
MIOS_QUADLET_NETWORK MIOS_QUADLET_SUBNET \
100102
MIOS_CORE_NET_SUBNET MIOS_CORE_NET_GATEWAY \

tools/lib/userenv.sh

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -216,6 +216,7 @@ slots = [
216216
("ports.rdp", "MIOS_RDP_PORT"),
217217
("ports.surrealdb", "MIOS_PORT_SURREALDB"),
218218
("ports.agent_pipe", "MIOS_PORT_AGENT_PIPE"),
219+
("ports.ollama_igpu", "MIOS_PORT_OLLAMA_IGPU"),
219220
# legacy aliases for ports
220221
("ports.forge_http", "MIOS_FORGE_HTTP_PORT"),
221222
("ports.forge_ssh", "MIOS_FORGE_SSH_PORT"),
@@ -317,6 +318,11 @@ slots = [
317318
("services.agent_pipe.user", "MIOS_AGENT_PIPE_USER"),
318319
("services.agent_pipe.uid", "MIOS_AGENT_PIPE_UID"),
319320
("services.agent_pipe.gid", "MIOS_AGENT_PIPE_GID"),
321+
("services.ollama_igpu.user", "MIOS_OLLAMA_IGPU_USER"),
322+
("services.ollama_igpu.uid", "MIOS_OLLAMA_IGPU_UID"),
323+
("services.ollama_igpu.gid", "MIOS_OLLAMA_IGPU_GID"),
324+
("image.sidecars.ollama_igpu_version", "MIOS_OLLAMA_IGPU_VERSION"),
325+
("image.sidecars.ollama_igpu", "MIOS_OLLAMA_IGPU_IMAGE"),
320326
# ── agent_pipe (standalone router + refine + critic FastAPI) ─────────
321327
("agent_pipe.endpoint", "MIOS_AGENT_PIPE_ENDPOINT"),
322328
("agent_pipe.backend", "MIOS_AGENT_PIPE_BACKEND"),

usr/lib/sysusers.d/50-mios-services.conf

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,15 @@ u mios-surrealdb 821:mios-surrealdb "'MiOS' SurrealDB shared agent state" /var/l
6363
# (chown via /usr/lib/tmpfiles.d/mios-agent-pipe.conf at boot).
6464
g mios-agent-pipe 822
6565
u mios-agent-pipe 822:mios-agent-pipe "'MiOS' Agent Pipe (router + refine + critic FastAPI)" /var/lib/mios/agent-pipe /usr/sbin/nologin
66+
# 'MiOS' Ollama iGPU instance (micro-LLM lane, ollama:rocm against
67+
# AMD iGPU via wsl2-amd.yaml CDI). Sibling to mios-ollama (uid 815)
68+
# which keeps the dGPU + CUDA for big-model inference. Operator
69+
# directive 2026-05-18: "iGPU model(s) isn't working" -- root cause
70+
# was ollama:latest being CUDA-only; this Quadlet runs ollama:rocm
71+
# on a sibling port so micro-LLMs (qwen3:1.7b, qwen3:0.6b, ...) get
72+
# actual iGPU compute instead of getting parked on the dGPU.
73+
g mios-ollama-igpu 823
74+
u mios-ollama-igpu 823:mios-ollama-igpu "'MiOS' Ollama iGPU instance (micro-LLM ROCm lane)" /var/lib/ollama-igpu /usr/sbin/nologin
6675
# Supplementary groups: agent-pipe needs read access to /var/lib/mios/
6776
# scratch and the hermes session JSONs for the critic / structured-
6877
# tool-history features.
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# /usr/lib/tmpfiles.d/mios-ollama-igpu.conf
2+
# HOME + state dir for the iGPU Ollama instance (Quadlet at
3+
# /usr/share/containers/systemd/mios-ollama-igpu.container). Owned
4+
# by mios-ollama-igpu (uid 823, declared in
5+
# /usr/lib/sysusers.d/50-mios-services.conf). Shares the model
6+
# store at /var/lib/ollama with the CUDA mios-ollama instance --
7+
# only this iGPU-specific HOME dir lives here.
8+
d /var/lib/ollama-igpu 0750 823 823 -
Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
[Unit]
2+
Description='MiOS' Ollama (iGPU instance, micro-LLM lane via AMD ROCm)
3+
After=network-online.target mios-cdi-detect.service
4+
Wants=network-online.target
5+
# Operator directive 2026-05-18: "iGPU model(s) isn't working" --
6+
# diagnosis: ollama/ollama:latest is the CUDA-only image; even with
7+
# the AMD CDI device (wsl2-amd.yaml) wired in via the
8+
# /etc/containers/systemd/ollama.container.d/20-mios-igpu.conf
9+
# drop-in, ollama-CUDA can't actually compute on the AMD iGPU. The
10+
# fix is a SECOND ollama instance using ollama/ollama:rocm pinned to
11+
# the AMD device. The original mios-ollama keeps the dGPU + CUDA;
12+
# this instance gets the iGPU + ROCm. Sibling, not replacement.
13+
#
14+
# Operator directive 2026-05-17 (still in force): "iGPU's are ONLY
15+
# micro-llms" + "wire to ollama!! JUST ONLY USES Micro-llms in MiOS
16+
# stack". This Quadlet hosts only the small models (qwen3:1.7b,
17+
# qwen3:0.6b, etc.) -- mios-daemon + agent-pipe router + (future)
18+
# refine all point at :11435 for the micro-LLM lane. Big-model
19+
# inference stays on the dGPU/CUDA mios-ollama at :11434.
20+
21+
[Container]
22+
# ROCm variant -- only image that can compute on AMD GPUs. Pinned
23+
# from mios.toml [image.sidecars].ollama_igpu (rendered at build).
24+
Image=${MIOS_OLLAMA_IGPU_IMAGE:-docker.io/ollama/ollama:rocm}
25+
ContainerName=mios-ollama-igpu
26+
# Host networking -- same workaround as mios-surrealdb. Dual-bridge
27+
# (mios.network + ai-net.network) + PublishPort hits a podman-WSL
28+
# port-forwarder quirk where the host listener never binds even
29+
# though `podman port` reports the mapping. Host-net gives the iGPU
30+
# instance a reliable http://localhost:11435 for mios-daemon +
31+
# agent-pipe router; no isolation loss that matters at loopback.
32+
Network=host
33+
AutoUpdate=registry
34+
35+
# AMD-only device pinning. Crucially does NOT add nvidia.com/gpu --
36+
# this instance must stay off the dGPU so the dGPU/CUDA mios-ollama
37+
# can keep using it for big models without contention. The amd.com/
38+
# gpu CDI spec is the WSL2 hand-rolled wsl2-amd.yaml that
39+
# mios-cdi-detect.service writes to /run/cdi/ at boot.
40+
AddDevice=amd.com/gpu=all
41+
42+
# Identity: mios-ollama-igpu (uid 823) -- sibling sysuser to
43+
# mios-ollama (uid 815). Separate UID so the iGPU instance's data
44+
# dir ownership doesn't collide with the CUDA instance even on
45+
# hosts that share /var/lib/ollama/models.
46+
User=${MIOS_OLLAMA_IGPU_UID:-823}
47+
Group=${MIOS_OLLAMA_IGPU_GID:-823}
48+
49+
# Distinct port -- mios-daemon's MIOS_DAEMON_ENDPOINT and the
50+
# agent-pipe router's MIOS_AGENT_PIPE_ROUTER_ENDPOINT point at
51+
# :11435 for the micro-LLM lane. The CUDA mios-ollama keeps :11434.
52+
Environment=OLLAMA_HOST=0.0.0.0:${MIOS_PORT_OLLAMA_IGPU:-11435}
53+
Environment=OLLAMA_ORIGINS=*
54+
55+
# Share the model store with mios-ollama -- the build-baked seed at
56+
# /usr/share/ollama/models was hardlinked into /var/lib/ollama/models
57+
# on first boot, and the iGPU instance reuses the same blobs. Saves
58+
# disk + means a model pulled via either instance is immediately
59+
# visible to the other.
60+
Environment=OLLAMA_MODELS=/var/lib/ollama/models
61+
Environment=HOME=/var/lib/ollama-igpu
62+
63+
# Micro-LLM lane parallelism. Smaller models, smaller per-request
64+
# memory footprint -- can run more concurrent slots than the CUDA
65+
# instance. mios-daemon's classify + refusal + cron loops all hit
66+
# this lane; agent-pipe's router classifier hits it; future kanban-
67+
# auto-dispatched parallel CPU agents will saturate it.
68+
Environment=OLLAMA_NUM_PARALLEL=8
69+
Environment=OLLAMA_MAX_LOADED_MODELS=6
70+
71+
# Volumes:
72+
# /var/lib/ollama shared writable model store (read-write
73+
# so the iGPU instance can ALSO pull
74+
# models if asked; the small-model set
75+
# rarely changes day-to-day)
76+
# /var/lib/ollama-igpu iGPU-only HOME + ssh-key state
77+
# /usr/share/ollama immutable seed mounted RO
78+
Volume=/var/lib/ollama:/var/lib/ollama:Z
79+
Volume=/var/lib/ollama-igpu:/var/lib/ollama-igpu:Z
80+
Volume=/usr/share/ollama:/usr/share/ollama:ro,Z
81+
82+
# Host networking -- no PublishPort needed; the OLLAMA_HOST bind
83+
# above (0.0.0.0:${MIOS_PORT_OLLAMA_IGPU}) goes directly to the
84+
# host loopback + LAN interfaces.
85+
86+
Label=org.opencontainers.image.title=mios-ollama-igpu
87+
Label=org.opencontainers.image.url=http://localhost:11435/
88+
Label=io.podman_desktop.openInBrowser=http://localhost:11435/
89+
90+
[Service]
91+
Restart=on-failure
92+
RestartSec=10
93+
Delegate=yes
94+
95+
[Install]
96+
WantedBy=multi-user.target default.target

usr/share/mios/mios.toml

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2064,7 +2064,8 @@ forge_http = 3000 # Forgejo HTTP web UI
20642064
forge_ssh = 2222 # Forgejo git+ssh (non-22 to coexist with sshd)
20652065
cockpit = 9090 # host Cockpit web console
20662066
cockpit_link = 19090 # mios-cockpit-link Podman Desktop discovery shim
2067-
ollama = 11434 # alternate local LLM backend
2067+
ollama = 11434 # alternate local LLM backend (CUDA / dGPU lane)
2068+
ollama_igpu = 11435 # iGPU micro-LLM lane (ollama:rocm, AMD iGPU via wsl2-amd.yaml CDI)
20682069
searxng = 8888 # local privacy-respecting metasearch
20692070
hermes = 8642 # Hermes-Agent (canonical /v1; live MiOS agent at /)
20702071
hermes_dashboard = 9119 # Hermes Dashboard web UI (sessions, skills, stats)
@@ -2111,6 +2112,8 @@ code_server_version = "latest"
21112112
code_server = "ghcr.io/coder/code-server:latest"
21122113
ollama_version = "latest"
21132114
ollama = "docker.io/ollama/ollama:latest"
2115+
ollama_igpu_version = "rocm"
2116+
ollama_igpu = "docker.io/ollama/ollama:rocm"
21142117
guacamole_version = "latest"
21152118
guacamole = "docker.io/guacamole/guacamole:latest"
21162119
forge_runner_version = "7"
@@ -2170,6 +2173,11 @@ user = "mios-agent-pipe"
21702173
uid = 822
21712174
gid = 822
21722175

2176+
[services.ollama_igpu]
2177+
user = "mios-ollama-igpu"
2178+
uid = 823
2179+
gid = 823
2180+
21732181
# ----------------------------------------------------------------------------
21742182
# [paths] -- canonical FHS paths for runtime artifacts. Consumed by
21752183
# every script that needs to know where AI memory, scratch, journal,

0 commit comments

Comments
 (0)