Skip to content

cloud-vm-boot-proof.md

Latest commit

 

History

History
74 lines (49 loc) · 15.2 KB

cloud-vm-boot-proof.md

File metadata and controls

74 lines (49 loc) · 15.2 KB
Raw

Hosted cloud VM image: emit + boot the control plane (C2)

  • Status: done — VM-boot acceptance PASSED on the maintainer env (sudo -E make test-cloud-qemu, 2026-06-20): the hosted cloud image builds, converts to qcow2, boots once under OVMF UEFI (no swtpm, no LUKS), and the in-VM self-check reports MALMO_CLOUD_ASSERTIONS: PASS — all four control-plane containers up, dashboard + /api served through Caddy, hosted /setup gate returning 503
  • Date: 2026-06-20
  • Specs touched: ENVIRONMENT.md (# Provisioning, # Networking & discovery, # Boot — realized, not changed), BUILD.md (# 2/# 6 qcow2 artifact, # 5 offline bundle — realized), TESTING.md (# Full-stack control-plane integration — reused)

C2 of #196 (the cloud-VM track), after C1a (#202, the profile marker + brain read), C1b (#203, the lean image), C1c (#204, the slim host-agent), and C3a (#206, the hosted /setup gate). It emits the hosted cloud image, converts it to the qcow2 cloud artifact, and boots it once in QEMU to prove the control plane comes up and serves — no swtpm, no LUKS, no installer (ENVIRONMENT.md # Provisioning: "the disk IS the installed system"). The cloud analogue of the medium lane's full-stack boot, minus everything the hosted profile drops. The cloud-init-style seed (C3), the trimmed wizard (C4), and the seed→wizard→dashboard end-to-end (C5) stay out of scope: C2 asserts the brain serves (pre-setup), not that an admin exists.

What was done

Integration fix: the containerized brain now learns its profile (Go)

Bringing the brain up on a real hosted image surfaced an unflagged gap that C3a's unit tests (which override MALMO_PROFILE_PATH) could not catch: the brain runs in a container that mounts only the agent socket dir + /var/lib/malmo, neither of which covers /etc/malmo — so profile.Read("/etc/malmo/profile") inside the container always returned Appliance (the no-op default) regardless of what the image stamped. An appliance-mode brain runs /setup open (first-caller-becomes-admin / 409-on-empty), which is exactly the public-by-default hole C2's security review forbids. C3a's 503 gate is real but only fires when the brain resolves hosted.

Fix (minimal, host-agent-side, no brain change):

  • internal/hostagent/brainlaunch — Config gains ProfileMarkerPath; runSpec mounts it read-only at the same path into the brain container when set, so the brain reads /etc/malmo/profile exactly as it would natively. Empty (an unmarked appliance box, make dev) skips the mount and the brain resolves appliance — unchanged.
  • cmd/host-agent-real — brainLaunchConfig resolves the marker path (MALMO_PROFILE_PATH, default profile.DefaultMarkerPath) and sets ProfileMarkerPath only when it exists as a regular file (a same-path bind of a missing source would make Docker auto-create a root-owned directory). Both the appliance and -tags hosted builds get it (shared, untagged).
  • Tests — brainlaunch_test.go: the marker mount is absent by default and present + read-only when ProfileMarkerPath is set.

The boot proof's /setup → 503 assertion is the end-to-end check that this works: an appliance-mode brain would answer 409/200 there.

The boot harness (tooling, dev/cloud/)

  • dev/cloud/test/ — the boot-proof image definition. Its mkosi.conf Include=s the lean dev/cloud/mkosi.conf (C1b) rather than copying it, so the booted image — and the qcow2 converted from it — is the lean image plus instrumentation, never a divergent rebuild. make build-cloud-image still builds dev/cloud/ lean with no postinst; this directory adds the boot machinery via its own auto-detected mkosi.postinst.chroot + mkosi.extra/ (staged at build time), without touching the lean config. OutputDirectory is redirected to .dev/cloud-boot so the two images never clobber.
  • dev/cloud/test/mkosi.postinst.chroot — enables docker + containerd, systemd-networkd, host-agent, the first-boot image loader, and the assertions oneshot (.wants symlinks); provisions the malmo/malmo-app/malmo-shared identities; brings the single virtio NIC up via systemd-networkd DHCP ([Match] Type=ether, the minimal cloud-native path — no NetworkManager); pre-commits machine-id + masks systemd-firstboot. No sshd (hosted cut).
  • dev/cloud/test/bootstrap.sh — preflight, builds the slim host-agent (go build -tags hosted, #204) + the control-plane image bundle (make control-plane-images, reused verbatim), stages mkosi.extra/ (the agent + host-agent.service + a control-plane env drop-in, the image bundle + load-control-plane-images.sh + malmo-load-images.service reused from the medium lane, the control-plane compose same-path at /var/lib/malmo/control-plane/, the PAM stack, the assertions oneshot + script), stages Docker's apt repo (trixie), and runs mkosi build. Canary-gated.
  • dev/cloud/run-cloud-tests.sh — calls the bootstrap, qemu-img converts raw → the malmo-<version>-amd64.qcow2 artifact (BUILD.md # 6), and boots it once under OVMF UEFI with one virtio NIC restrict=on (air-gapped — a stray pull hard-fails, proving the offline bundle is complete), snapshot=on (keeps the artifact pristine), serial-log capture + dump_serial on failure. No swtpm, no LUKS credential, no two-boot cycle, no SSH.
  • dev/cloud/cloud-assertions.sh + malmo-cloud-assertions.service — a first-boot oneshot runs the in-VM self-check and writes its verdict (MALMO_CLOUD_ASSERTIONS: PASS|FAIL) to the serial console, which the driver greps. Serial-driven, not SSH-driven, because hosted ships no sshd (ENVIRONMENT.md # Access & files); the checks run in-guest over localhost, which is the real control-plane-up proof regardless. The checks: systemd userspace converged with no failed control-plane unit; PSI live (/proc/pressure/memory non-empty — the psi=1 cmdline pairs with C1b); systemd-networkd active + NetworkManager absent; docker active + the four baked images loaded; the four control-plane containers running; the proxy boundary (brain has no raw docker.sock); the dashboard answers through Caddy (GET / 200, /api/v1/me 200|401); and the hosted /setup gate returns 503 (no seed — the open-/setup window stays closed).
  • make test-cloud-qemu — sudo -E ./dev/cloud/run-cloud-tests.sh, mirroring test-medium-qemu's shape, with help text.

Lean-image fix surfaced by the first real build

C1b (#203) validated the lean image via mkosi summary only — the full mkosi build was blocked on its env (#189), so C2 is the first actual build of the cloud image and surfaced two genuine gaps in the lean package set, both added to dev/cloud/mkosi.conf (neither is a cut-list package, so the lean assertion is unaffected):

  • passwd (the shadow utilities — useradd/groupadd/gpasswd/chpasswd). On the appliance/medium image these arrive transitively via the openssh-server/network-manager/avahi postinsts; the hosted image cuts all of those, so it lost them. Without passwd, docker-ce's postinst (groupadd docker) fails the build, and at runtime host-agent's user management (/setup + user CRUD) and the boot-proof postinst's malmo-identity provisioning would have no tools.
  • systemd-boot-efi — the actual EFI executables mkosi's Bootloader=systemd-boot installs to the ESP. On trixie these moved to a separate package that systemd-boot only Recommends, and mkosi builds with Install-Recommends=false, so the build failed with "a systemd-boot binary was not found". (The medium lane is bookworm, whose systemd-boot bundles the binary — why it never hit this.)

And two boot-path fixes, both latent in C1b (it was never booted):

  • Partitioning (dev/cloud/mkosi.repart/, new). C1b shipped no repart definitions on the assumption that mkosi's default is a sane bounded ESP+root. It is not — with no definitions, systemd-repart grows the root to fill the whole build disk (a ~1 TiB root → ~17 GiB of ext4 inode tables written at build time → "No space left on device"). Added a minimal unencrypted ESP (512 MiB) + ext4 root fixed at 8 GiB (sparse; GrowFileSystem fills the partition), same sizing as the medium lane so first boot's docker load of the baked images has headroom — no LUKS (hosted's encryption-at-rest is provider/KMS-keyed, ENVIRONMENT.md # Storage). The boot image inherits it through the Include.
  • Root discovery (root=PARTUUID on the cmdline). The lean cmdline had no root= and relied on systemd gpt-auto, which needs the boot loader to set the LoaderDevicePartUUID EFI variable — not set in this firmware/boot path, so the initrd's gpt-auto-generator logged "EFI loader partition unknown, exiting" and initrd-switch-root.service dropped to emergency mode. Pinned the root partition UUID in 10-root.conf and added root=PARTUUID=… to KernelCommandLine (the medium lane likewise passes an explicit root=, via rd.luks).

Boot-path fix surfaced by the first actual boot: networkd vs Docker's interfaces

The first green build still failed the boot proof — only malmo-brain and malmo-docker-proxy came up; malmo-caddy/malmo-ui never did, because the brain could not reach the socket-proxy through the malmo-ingress bridge (dial tcp 172.18.0.2:2375: connect: no route to host, ARP INCOMPLETE both directions). The bridge was NO-CARRIER while the veths had carrier — i.e. the per-container veths were not contributing carrier to the bridge.

Root cause: the single-NIC .network matches [Match] Type=ether, which also matches Docker's br-<id> bridges and veth* endpoints. Left managed, systemd-networkd resets them, so the bridge never forwards and inter-container ARP never resolves. NetworkManager (the appliance/medium lane) ignores Docker's interfaces by default; networkd does not — which is exactly why the medium lane works and the cloud lane did not, despite the identical control-plane bundle. (This is L2: the previous build cycles chasing Docker's iptables raw-table / br_netfilter knobs were the wrong layer — those never affect ARP.)

Fix (dev/cloud/test/mkosi.postinst.chroot): stage /etc/systemd/network/05-docker-unmanaged.network matching Name=docker* veth* br-* with [Link] Unmanaged=yes. Its lower filename prefix means networkd evaluates it before 20-dhcp.network, so it claims the virtual links as unmanaged first and the real NIC still falls through to DHCP. The assertions' diagnostics gained bridge link show + networkctl list to confirm the Docker links read unmanaged on a green boot.

How it maps to the specs

  • ENVIRONMENT.md # Provisioning — "no installer; the disk IS the installed system": a single UEFI boot of the disk image, no kiosk/installer, no disk-selection/confirm-wipe.
  • ENVIRONMENT.md # Networking — "one virtual NIC brought up by the minimal cloud-native path, no NetworkManager": the systemd-networkd Type=ether DHCP .network, asserted live (networkd active, nmcli absent).
  • ENVIRONMENT.md # Boot — "no TPM unseal, no NM, no physical-disk storage assembly": the medium lane's swtpm/LUKS/two-boot cycle and its NM/Avahi network-state assertions are all gone.
  • ENVIRONMENT.md # Admin bootstrap — "a box with no seed returns 503, never the appliance's open behavior": asserted on the booted VM, which is what makes the profile-plumbing fix load-bearing.
  • BUILD.md # 6 — the qcow2 cloud artifact, produced by qemu-img convert.
  • TESTING.md # Full-stack control-plane integration — the bake + offline-load mechanism (make control-plane-images → tarballs → first-boot docker load) reused verbatim; the air-gap (restrict=on) carried over.

Known gaps & deviations

  • VM-boot acceptance passed on the maintainer env (sudo -E make test-cloud-qemu, accel=kvm, air-gapped, 2026-06-20): build → qcow2 → single UEFI boot → MALMO_CLOUD_ASSERTIONS: PASS. The raw is 8.5 GiB sparse (consumes ~960 MiB). make check is green (incl. the brainlaunch tests) and bash -n is clean on all four scripts.
  • On a green run the serial log is not retained. run-cloud-tests.sh only copies the serial to .dev/cloud-boot/last-serial.log via dump_serial on failure, and cleans the run dir on success — so after a PASS, last-serial.log still holds the previous failing run. The PASS verdict is authoritative from the driver's stdout. A future tweak could always retain the last serial; left as-is to keep success runs side-effect-free.
  • The qcow2 carries the test assertions oneshot. Per the maintainer's call, the artifact is the boot-tested raw (one image, not a separate clean build). The malmo-cloud-assertions oneshot is therefore in the shipped qcow2; it is side-effect-free (reads only, no poweroff) and run-once (a marker short-circuits re-runs), so a real provisioned boot runs it once harmlessly. A later slice can split a production-packaging target from the boot-proof image if that becomes a concern.
  • Serial-driven, not SSH-driven. The issue's literal wording was SSH hostfwd, but baking a test sshd + key into a qcow2 that can boot on a public IP is a security smell, and the lean image excludes openssh-server. The verdict travels over the serial console instead; the air-gapped NIC (restrict=on) still proves the offline bundle is complete. (SSH on hosted VMs is on the roadmap as real keyed operator access at provision time — not a baked test key.)
  • The dashboard route host is malmo.local (the brain's MALMO_DASHBOARD_HOST default; host-agent does not override it), asserted as a Host-header route match over localhost. The real hosted host is malmo.<box-id>.malmo.network, which needs the seed + enrollment (C3/C3b) — out of scope here. No DNS/mDNS is involved in the assertion.
  • build-cloud-image (C1b) stays lean and separate. It builds dev/cloud/ (no control plane, no postinst) and keeps asserting the lean package cut. The full hosted image (lean + control plane + agent) is built only by the boot path in this slice; a non-test production-build target is a later concern.
  • MALMO_OFFLINE_INSTALL is intentionally not set in the 10-cloud-brain.conf drop-in, even though the boot proof runs air-gapped (restrict=on). The proof installs no apps, and the control-plane compose only references locally-present, explicitly-tagged images, so it never needs a registry. A real hosted VM has internet (the registry is reachable), so offline-install mode would be wrong there. The consequence — only relevant inside the air-gapped proof, never on a deployed box — is that a user-triggered app install in that QEMU run would fail to pull; the assertions deliberately don't cover app install (C2 scope is control-plane-up only).

What's next

  • Cloud-lane seed delivery (now unblocked by this slice). Inject seed.json into the booted VM (SMBIOS / systemd credential) and assert the C3a gate end-to-end: with a seed, /setup accepts the bootstrap secret and creates the first admin. This is the hosted-setup-gate.md follow-up that was blocked on the boot harness.
  • C3b — seeded enrollment + DNS-01 cert (*.<box-id>.malmo.network), then C4 (the trimmed hosted wizard) and C5 (seed→wizard→dashboard end-to-end), which build on this boot proof.