Skip to content

hosted-grow-root-disk.md

Latest commit

 

History

History
27 lines (19 loc) · 5.56 KB

hosted-grow-root-disk.md

File metadata and controls

27 lines (19 loc) · 5.56 KB
Raw

hosted-grow-root-disk — grow the root filesystem to fill the provider disk on boot

  • Status: done
  • Date: 2026-07-08
  • Specs touched: ENVIRONMENT.md (# Storage)

The hosted cloud image bakes a fixed 8 GiB root so the shipped raw stays small (sparse), but nothing grew it onto the (much larger) provider disk, so a provisioned box ran on ~8 GiB. Docker image storage and the brain's SQLite store share that one volume, so a single app install can fill it; once full the brain's first store write on the SSO landing fails and login returns a bare HTTP 500 — the box looks offline though it is up, and self-heals only if the failing install rolls back and frees space. This closes that gap by growing the root to fill the whole disk on every boot.

What was done

  • Runtime repart definition — dev/cloud/mkosi.extra/usr/lib/repart.d/50-malmo-grow-root.conf (Type=root, GrowFileSystem=yes, no size cap). systemd-repart matches it to the single existing root partition and, with no SizeMaxBytes, extends it to consume all free space; GrowFileSystem online-grows the ext4 (root is mounted rw). No Format=, so the populated ext4 is never treated as empty. This is deliberately separate from the build-time layout in dev/cloud/mkosi.repart/ (which pins the root at exactly 8 GiB so the shipped raw stays small): build-time pins, runtime uncaps.
  • malmo-grow-root.service — a Type=oneshot / RemainAfterExit=yes unit running systemd-repart --dry-run=no, ConditionVirtualization=!container, SuccessExitStatus=76 77 (76 = no root block device, 77 = no GPT table — the "nothing to grow" cases). Ordered Before=docker.service host-agent.service so the disk is full before anything writes to the shared volume. Enabled by a .wants symlink in mkosi.postinst.chroot (same pattern as malmo-metadata-firewall).
  • Fail-closed gating — Before= only orders starts; it does not stop the write-heavy services from starting if the grow fails. So two hosted-only drop-ins under the tracked mkosi.extra/ add Requires=malmo-grow-root.service: docker.service.d/10-malmo-grow-root.conf and host-agent.service.d/10-malmo-grow-root.conf. If the grow ever fails, docker and the brain refuse to start on an un-grown 8 GiB root rather than run into the disk-full 500. The gate is a drop-in (not an edit to the unit) on purpose: host-agent.service is shared with the appliance (dist/systemd/host-agent.service, also used by dev/test-qemu and the nspawn boot-chain), and the appliance has no malmo-grow-root.service, so the Requires= must be scoped to the hosted profile.
  • Lean package set — on trixie the systemd-repart binary is split out of the systemd package, so the lean image depends on systemd-repart explicitly; it pulls only libfdisk1 on top of what systemd already brings (no libcryptsetup, so the cryptsetup cut is preserved). Both are added to expected-packages.txt; the lean guard confirms the delta.
  • Boot-proof assertion — a new check in cloud-assertions.sh asserts systemd-repart is present in the lean image and that malmo-grow-root.service reached active (held by RemainAfterExit). The QEMU boot-proof disk is fixed-size with no spare space, so the grow itself is a harmless no-op there; the assertion proves the tool is present and the unit is wired, not that a real disk grew.
  • Spec — ENVIRONMENT.md # Storage documents the boot-time grow and that hosted v1 is deliberately one volume for everything (system, docker images, app data, store); splitting app/data onto its own volume is the external-drives follow-up (# Open questions), not this profile.

Known gaps & deviations

  • Real full-disk growth is not exercised in CI. The fixed-size QEMU boot-proof lane only exercises the no-op path (present + wired + active). Partition + ext4 actually extending onto a larger disk is the acceptance step on a real provider VM, not this lane.
  • Double execution, by design (for now). The systemd-repart package ships its own systemd-repart.service, statically enabled, which already runs the same /usr/lib/repart.d/*.conf at boot — so the grow runs twice. This is harmless (systemd-repart is idempotent; a second pass on an already-grown disk is a no-op). The custom malmo-grow-root.service exists only to add the docker/host-agent ordering and the fail-closed Requires= gate, which the stock unit cannot express. Consolidating onto a stock-unit drop-in (contributing only the ordering/gating, dropping the duplicate ExecStart) is a deliberate deferral, pending a live re-test of the growth path — the current mechanism is the one verified on a real provider box, so it is kept intact.
  • Pre-existing, unrelated boot-proof failure. make test-cloud-qemu currently fails at the unseeded /setup check (gets HTTP 503 via Caddy, expects 403). This reproduces on a clean build of main with this change stashed out, so it is pre-existing and independent of this change: the box's Caddy isn't serving the /api route to the brain at probe time (the brain unconditionally returns 403 for hosted /setup, internal/api/auth.go). Likely fallout from the recent hosted SSO / wildcard-TLS work; tracked separately, not introduced here.

What's next

  • Consolidate the grow onto a single unit (drop-in on the stock systemd-repart.service contributing only the ordering + gating) once the growth path can be re-tested live, removing the double execution.
  • Exercise real full-disk growth in an automated lane (a provider-shaped disk with spare space) so partition + ext4 extension is covered without a manual on-box acceptance step.