Skip to content

Commit 9b47527

Browse files
mios-devclaude
andcommitted
ci: move podman graphroot to /mnt so the 21-image bound-images layer commit fits
After the storage-reset fix, CI got into the OCI build and baked all 21 bound-images (baked=21 failed=0), then died COMMITTING that layer: "copying layers and metadata ... storing layer ... to file: io: read/write on closed pipe", exit 125. Root cause: GHA runners have only ~21GB free on / (even after jlumbroso frees ~30GB), but the 21 baked images (~50GB incl the AI lanes) + the commit's layer copy exhaust it -- ENOSPC surfaces as a closed pipe in the streaming layer write. (The step comment already records a prior disk-blowout at the 16th image.) Fix: point podman graphroot at /mnt -- the runner's LARGE ephemeral disk (~65GB+ free) -- and route buildah TMPDIR there too, in both the build and smoke jobs. runroot stays on tmpfs /run (small runtime state). Local dev-VM builds are unaffected (224GB on M:). install-robustness 2026-06-21. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 2a0018c commit 9b47527

1 file changed

Lines changed: 26 additions & 14 deletions

File tree

.github/workflows/mios-ci.yml

Lines changed: 26 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -105,19 +105,24 @@ jobs:
105105
106106
- name: Configure host podman storage (disable metacopy)
107107
run: |
108-
sudo mkdir -p /etc/containers
109-
# runroot + graphroot are REQUIRED: writing a [storage] table with only
110-
# `driver` makes `podman system reset` abort with "runroot must be set"
111-
# (install-robustness 2026-06-21). Use the standard rootful paths.
112-
echo -e '[storage]\ndriver = "overlay"\ngraphroot = "/var/lib/containers/storage"\nrunroot = "/run/containers/storage"\n[storage.options.overlay]\nmountopt = "nodev"' | sudo tee /etc/containers/storage.conf
108+
sudo mkdir -p /etc/containers /mnt/tmp
109+
# runroot + graphroot are REQUIRED (a [storage] table with only `driver`
110+
# makes `podman system reset` abort with "runroot must be set"). Put
111+
# graphroot on /mnt -- the GHA runner's LARGE ephemeral disk (~65GB+
112+
# free) vs / (~21GB even after jlumbroso frees ~30GB). The MiOS image
113+
# bakes 21 large bound-images (~50GB incl AI lanes); committing that
114+
# layer on / exhausts the disk and the layer copy's pipe closes ("io:
115+
# read/write on closed pipe", exit 125, 2026-06-21). runroot stays on
116+
# tmpfs /run (small runtime state only). install-robustness 2026-06-21.
117+
echo -e '[storage]\ndriver = "overlay"\ngraphroot = "/mnt/containers-storage"\nrunroot = "/run/containers/storage"\n[storage.options.overlay]\nmountopt = "nodev"' | sudo tee /etc/containers/storage.conf
113118
# The GHA runner ships a pre-seeded containers store whose libpod DB
114119
# may record an empty/foreign graph driver. `podman system reset`
115120
# reads that DB FIRST and aborts with 'database graph driver "" does
116121
# not match our graph driver "overlay"' BEFORE it can wipe. Remove
117122
# the stale store (rootful + rootless) so reset starts from a clean
118123
# slate, then make reset itself non-fatal (install-robustness
119124
# 2026-06-21).
120-
sudo rm -rf /var/lib/containers/storage /run/containers/storage
125+
sudo rm -rf /var/lib/containers/storage /run/containers/storage /mnt/containers-storage
121126
rm -rf "${HOME}/.local/share/containers/storage" 2>/dev/null || true
122127
sudo podman system reset -f || true
123128
@@ -188,7 +193,9 @@ jobs:
188193
# has full UID range and skips the remap. Operator-confirmed
189194
# CI failure 2026-05-15 (qdrant + 13 other bound images failed
190195
# with this exact error mid-bake).
191-
sudo podman build \
196+
# TMPDIR on /mnt too -- buildah's commit scratch must not spill onto
197+
# the small / (install-robustness 2026-06-21).
198+
sudo TMPDIR=/mnt/tmp podman build \
192199
"${BUILD_ARGS[@]}" \
193200
-f Containerfile \
194201
-t "localhost/mios:latest" \
@@ -314,19 +321,24 @@ jobs:
314321
315322
- name: Configure host podman storage (disable metacopy)
316323
run: |
317-
sudo mkdir -p /etc/containers
318-
# runroot + graphroot are REQUIRED: writing a [storage] table with only
319-
# `driver` makes `podman system reset` abort with "runroot must be set"
320-
# (install-robustness 2026-06-21). Use the standard rootful paths.
321-
echo -e '[storage]\ndriver = "overlay"\ngraphroot = "/var/lib/containers/storage"\nrunroot = "/run/containers/storage"\n[storage.options.overlay]\nmountopt = "nodev"' | sudo tee /etc/containers/storage.conf
324+
sudo mkdir -p /etc/containers /mnt/tmp
325+
# runroot + graphroot are REQUIRED (a [storage] table with only `driver`
326+
# makes `podman system reset` abort with "runroot must be set"). Put
327+
# graphroot on /mnt -- the GHA runner's LARGE ephemeral disk (~65GB+
328+
# free) vs / (~21GB even after jlumbroso frees ~30GB). The MiOS image
329+
# bakes 21 large bound-images (~50GB incl AI lanes); committing that
330+
# layer on / exhausts the disk and the layer copy's pipe closes ("io:
331+
# read/write on closed pipe", exit 125, 2026-06-21). runroot stays on
332+
# tmpfs /run (small runtime state only). install-robustness 2026-06-21.
333+
echo -e '[storage]\ndriver = "overlay"\ngraphroot = "/mnt/containers-storage"\nrunroot = "/run/containers/storage"\n[storage.options.overlay]\nmountopt = "nodev"' | sudo tee /etc/containers/storage.conf
322334
# The GHA runner ships a pre-seeded containers store whose libpod DB
323335
# may record an empty/foreign graph driver. `podman system reset`
324336
# reads that DB FIRST and aborts with 'database graph driver "" does
325337
# not match our graph driver "overlay"' BEFORE it can wipe. Remove
326338
# the stale store (rootful + rootless) so reset starts from a clean
327339
# slate, then make reset itself non-fatal (install-robustness
328340
# 2026-06-21).
329-
sudo rm -rf /var/lib/containers/storage /run/containers/storage
341+
sudo rm -rf /var/lib/containers/storage /run/containers/storage /mnt/containers-storage
330342
rm -rf "${HOME}/.local/share/containers/storage" 2>/dev/null || true
331343
sudo podman system reset -f || true
332344
@@ -354,7 +366,7 @@ jobs:
354366
run: |
355367
# sudo: rootful podman avoids user-namespace UID exhaustion in
356368
# the bake step (same fix as the main build step above).
357-
sudo podman build -t mios:smoke -f Containerfile .
369+
sudo TMPDIR=/mnt/tmp podman build -t mios:smoke -f Containerfile .
358370
359371
- name: Smoke RUN (image runs; key MiOS components present + agent-pipe compiles)
360372
run: |

0 commit comments

Comments
 (0)