feat: add upgrade/switch/rollback e2e tests for all 4 boot variants#3
Merged
Conversation
Adds a three-boot e2e test sequence that exercises the full cbootc lifecycle: switch to a v2 image pulled from a local TAP-backed registry, verify the new deployment boots, roll back, and verify the original deployment is restored. All 4 boot variants are covered (GRUB, GRUB+SecureBoot, UKI, UKI+SecureBoot) for both Fedora and Ubuntu. fix: rollback.rs uses numeric GRUB index on Ubuntu/Debian Ubuntu's GRUB does not ship blscfg.mod and does not reliably match `set default=<128-char-SHA512>` against `menuentry --id` values in the generated grub.cfg. On non-grub2 systems, cbootc rollback now writes the numeric position (newest-first, matching write_grub_menuentry_cfg's sort order) as next_entry instead of the raw digest. Fedora/RHEL (grub2's blscfg.mod, native BLS ID matching) are unchanged. Tested locally: Ubuntu GRUB 9/9, Ubuntu GRUB+SB 12/12.
On Fedora, the autologin shell fires while udev is still renaming the virtio-net interface, so ip-link shows only lo at that point. The previous code ran immediately and got an empty $iface, causing ip-link-set to fail and the configure_guest_network step to FAIL. Result: 2 passed / 7 failed (coincidental pass of rolled_back_digest_active). Fix: poll up to 15 s for a non-loopback interface to appear, then tell NetworkManager to unmanage it (preventing it from overriding the static IP) and flush any stale addresses before setting ours.
bootctl set-default was added in upgrade.rs but Ubuntu UKI tests now show "No such file or directory (os error 2)" on both switch and rollback. Add _diag() calls before switch_to_v2 and rollback_succeeds (UKI only) to capture: bootctl path/version, mount table, EFI/Linux contents, efivars count, and bootctl status output for the next CI run.
…rors The previous approach copied disk.raw to /tmp with cp --sparse=always, then booted the copy without snapshot=on. When QEMU wrote to previously- sparse regions (ext4 journal commits during boot), the writes failed with BLK_STS_IOERR because /tmp (tmpfs) had insufficient space. Fix: pass snapshot=on,format=raw on the QEMU drive instead. QEMU creates its own thin qcow2 overlay (delta-only, starts empty, ~100-200 MB for a switch+rollback run) and persists it across guest reboots within the same QEMU process. The original disk.raw is never modified, and there is no full-disk copy to blow out /tmp.
bootctl set-default / set-next require either the bootctl binary in PATH
or a writable efivarfs (backing OVMF_VARS flash device). Both may be
absent: the Ubuntu container image does not guarantee bootctl is on PATH,
and test VMs often run with a read-only combined OVMF_CODE.fd that has no
separate writable VARS.
Fix: write `default <entry-id>` directly into /boot/efi/loader/loader.conf.
Systemd-boot reads this file at every boot and honours it as the default
entry when no EFI variable preference is set. The file is on the FAT ESP
which is always writable when cbootc runs.
upgrade.rs: replace `bootctl set-default` with set_loader_conf_default()
rollback.rs: replace `bootctl set-next` / set_next_entry_bootctl() with the
same helper (persistent rollback — stays on old entry until the
next explicit upgrade, which is the correct behaviour here)
tests/e2e.py: replace test_bootctl_next_entry_set (EFI var check) with
test_loader_conf_default_set (loader.conf check); drop _diag
…king On Fedora, nftables is the backend for firewalld; plain iptables rules are silently ignored, so the host firewall was blocking guest→host TCP even after 'iptables -I INPUT ACCEPT'. Switching to 'firewall-cmd --zone=trusted' puts the TAP in the trusted zone and actually opens the traffic. configure_guest_network now re-reads the interface name on every iteration instead of caching it once at the start. udev can rename the virtio-net device (eth0→enp0s3) between the name lookup and ip addr add, returning ENODEV; the retry loop recovers from both the rename race and the case where the device is not yet visible.
…SC sequences Ubuntu's bash emits OSC 3008 shell-integration sequences before command output (\x1b]3008;...;type=command;cwd=/root\x1b\). The String Terminator (\x1b\) is not a newline, so re.MULTILINE ^ never matches at the position where 'default ...' begins. Switching from '^default' to '\bdefault' matches regardless of what character precedes it.
After cbootc switch/upgrade creates a new BLS entry, GRUB's blscfg.mod selects the default based on the grubenv 'default' variable, not by entry creation order. Without updating grubenv, GRUB continues to boot the previously-set default entry regardless of the new BLS conf. Add set_grub_default() and call it from run() on the Fedora/RHEL path (has_grub2 = true). The Ubuntu path is unaffected: it writes a new grub.cfg via write_grub_menuentry_cfg() where index 0 is already the newest entry.
firewall-cmd is only present on Fedora/RHEL (firewalld). On Ubuntu (GitHub Actions runner) subprocess.run raises FileNotFoundError before it can even return a non-zero exit code, crashing setup_tap(). Catch FileNotFoundError and fall back to iptables, which works on Ubuntu and other systems without firewalld.
TAP networking required host-side firewall manipulation (firewall-cmd / iptables) that was unreliable across distros: firewalld-managed nftables silently ignores plain iptables rules, and firewall-cmd was crashing CI runners that lack firewalld. QEMU user-mode networking (SLIRP) avoids all of this: the guest reaches the host at 10.0.2.2 and QEMU forwards those connections to host loopback 127.0.0.1, where the registry container (--network=host, REGISTRY_HTTP_ADDR=0.0.0.0:5000) is listening. No TAP, no firewall rules, no root-level network setup needed. - Remove setup_tap / teardown_tap and all TAP constants - boot_with_network: -netdev tap → -netdev user - configure_guest_network: static IP assignment → wait for DHCP route - wait_for_network: ping → TCP connect to registry port (more reliable; host firewall may pass ICMP but block TCP) - REGISTRY_HOST: 192.168.200.1 → 10.0.2.2 (SLIRP gateway)
The Ubuntu test image may not have a DHCP client configured for the virtio-net interface, so waiting for a default route via DHCP would time out. Assign 10.0.2.100/24 statically (same approach as the old TAP code but in the SLIRP subnet). 10.0.2.2 is reachable via the automatic 10.0.2.0/24 link-local route — no explicit default route needed.
Raw QEMU serial console output was piped directly to stdout via pexpect's logfile_read, burying PASS/FAIL lines in escape sequences and kernel boot noise. - Redirect console output to e2e-console.log instead of stdout; announce the path with a tail -f hint so progress is still followable during long boots - Strip ANSI/control characters from run_cmd output so assertion failure messages are readable without post-processing - On test failure or timeout, print the last 40–50 cleaned console lines under the FAIL line with a pointer to the full log
gawk provides 'awk', which the e2e configure_guest_network helper relies on but was absent from the minimal base images. Install it in the example Containerfiles (the right layer for test tooling) and remove the unrelated vim/htop that was there before. Also add [diag] output to run_upgrade_sequence for UKI runs: captures loader.conf, EFI/Linux listing, and bootctl list/status around the first reboot so the upgraded_digest_active failure can be diagnosed.
Writing loader.conf alone is insufficient: something running during boot sets the LoaderEntryDefault EFI variable to the currently-running entry, which takes priority over loader.conf. This caused rollback to be silently ignored — Boot 3 would boot the post-upgrade entry again instead of the original. Fix: call bootctl set-default after writing loader.conf, mirroring what the GRUB path does with grub2-editenv. NotFound errors are ignored for container/install contexts without a writable efivarfs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a three-boot e2e test sequence that exercises the full cbootc lifecycle: switch to a v2 image pulled from a local TAP-backed registry, verify the new deployment boots, roll back, and verify the original deployment is restored. All 4 boot variants are covered (GRUB, GRUB+SecureBoot, UKI, UKI+SecureBoot) for both Fedora and Ubuntu.
fix: rollback.rs uses numeric GRUB index on Ubuntu/Debian
Ubuntu's GRUB does not ship blscfg.mod and does not reliably match
set default=<128-char-SHA512>againstmenuentry --idvalues in the generated grub.cfg. On non-grub2 systems, cbootc rollback now writes the numeric position (newest-first, matching write_grub_menuentry_cfg's sort order) as next_entry instead of the raw digest. Fedora/RHEL (grub2's blscfg.mod, native BLS ID matching) are unchanged.Tested locally: Ubuntu GRUB 9/9, Ubuntu GRUB+SB 12/12.