fix(windows): stabilize WHPX and reproducible firmware builds#4
Draft
ZhiXiao-Lin wants to merge 31 commits into
Draft
fix(windows): stabilize WHPX and reproducible firmware builds#4ZhiXiao-Lin wants to merge 31 commits into
ZhiXiao-Lin wants to merge 31 commits into
Conversation
Phase A-1 of native snapshot-fork. When KRUN_SNAPSHOT_MEM_FILE is set (unix, non-TEE), guest RAM regions are backed by that file with MAP_SHARED (vm-memory from_ranges_with_files → MAP_NORESERVE|MAP_SHARED), so the file always holds live RAM: snapshot = pause + msync; restore will MAP_PRIVATE the same file for page-level CoW. Backing file + region layout registered in the new vmm snapshot module for the later snapshot step.
Phase A-2. Vmm::pause_vcpus (wires the existing Pause event) and Vmm::snapshot_to: collect per-vCPU KVM state via a new VcpuEvent::SaveState (answered from the paused vcpu thread through a channel — only the vcpu thread may touch its VcpuFd), save VM state (existing PIT/clock/irqchip code), fsync the MAP_SHARED RAM file, and write a bincode state file (vm+vcpu states + RAM layout; kvm-bindings serde feature). krun_start_enter spawns a listener on KRUN_SNAPSHOT_SOCK serving 'snapshot <state_path>' (linux x86_64, non-TEE).
vstate running() now ignores SaveState (only valid when paused). CpuId/Msrs are FamStructWrapper which kvm-bindings serde does not cover; add #[serde(with)] modules serializing their entry slices (entries derive Serialize) and rebuilding via from_entries on restore.
…e (Phase A-3) Restore mode triggers when KRUN_RESTORE_FROM (state file) + KRUN_SNAPSHOT_MEM_FILE (RAM file) are both set: - create_guest_memory_regions maps each region MAP_PRIVATE from the snapshot RAM file at the saved layout — kernel page-level CoW (this is the fork) - create_guest_memory skips load_payload + firmware write (kernel already in RAM) - build_microvm skips load_cmdline + configure_system and instead restores the saved Vm + per-vCPU KVM state before start_vcpus - Vcpu::restore_state made pub for the builder
…riant - vm_memory::mmap::MmapRegionBuilder (not re-exported at crate root) - add StartMicrovmError::RestoreState(VstateError) for Vm/Vcpu restore_state (returns vstate::Error, not vmm::Error — Internal didn't fit)
…rupted snapshot RAM)
Phase A (RAM+CPU snapshot/restore) is plumbing-complete and CPU-state byte-perfect. Root-caused the restore failure to lost virtio device-ring state (host devices reset to index 0 while guest RAM holds advanced indices -> deterministic wild jump, KVM INTERNAL_ERROR DELIVERY_EV #PF). Documents the exact Phase B save/restore surface (MmioTransport device_status/acked_features + per-Queue state + re-activation) and the path to the 20ms/100-VM target.
…se B foundation) Plain-data snapshot of a Queue's driver-programmed ring state (indices, ring addrs, ready, event_idx) with Queue::save_state/restore_state, so a freshly-built device can resume at the guest's indices. First building block of Phase B device-state restore.
…ate (Phase B) Per-device snapshot of transport regs (device_status/features/queue_select) + acked features + per-queue state, with restore that re-applies and re-activates the device (mirrors the DRIVER_OK activation path) so a fresh host device resumes at the guest's ring indices. Next: enumerate devices in the device manager and wire save/restore into the snapshot state + build_microvm restore branch.
…estore (Phase B) - MMIODeviceManager::save_virtio_states/restore_virtio_states (enumerate bus devices, downcast to MmioTransport) - SnapshotState gains device_states; serde mirrors (QueueStateSer/MmioDeviceStateSer) - snapshot_to collects device states; build_microvm restore re-applies + re-activates them before start_vcpus - new StartMicrovmError::RestoreDeviceState + mmio Error variants
…captures it ROOT CAUSE of the deterministic 0x24470e0 restore fault (found via guest page-table walk on ram.img + source analysis): the kernel image region [kernel_guest_addr, +kernel_size) sits in the hole arch_memory_regions punches out of the file-backed RAM, and Payload::KernelMmap inserts it as a raw mmap over libkrunfw host pages AFTER create_guest_memory_regions. The snapshot only file-backs arch_mem_regions, so the kernel region — which holds the guest's live early page tables, IDT/GDT and kernel .data/.bss — was never captured. On restore that hole had no KVM slot, so the first interrupt (CPU reads IDT at gpa 0x2447000, inside the hole) faulted with KVM_INTERNAL_ERROR DELIVERY_EV at IDT+#PF-slot = 0x24470e0. Fix: in snapshot mode, Payload::KernelMmap copies the kernel into the snapshot RAM file, maps it MAP_SHARED (capturing the guest's runtime writes), and appends it to MEM_BACKING.layout. Restore re-maps it MAP_PRIVATE via the existing mem_layout loop — no restore-side change needed.
…nded last) from_regions requires address-sorted regions; the file-backed kernel region is pushed to mem_layout last (gpa 0x1000000 after 0x100000000), so sort before building GuestMemoryMmap. Fixes UnsortedMemoryRegions on restore.
…restore now correct
…lf.queues ROOT CAUSE of exec-into-restored-guest hang (found via save-time index dump): vsock clones rx/tx queues into the muxer thread at activate(); the muxer advances ITS copies (reflected in guest RAM: avail_idx=256 used_idx=7) while self.queues stays frozen at activate time (0,0). save_state read self.queues, so the snapshot captured next_used=0 while the guest had last-seen used_idx=7. On restore the muxer re-emitted used entries from index 0 → the guest saw the used ring jump backwards and ignored every host OP_REQUEST → no OP_RESPONSE → exec timed out. Fix: add VirtioDevice::save_queue_states() (default = self.queues) and override it in vsock to read the muxer's live queue_rx/queue_tx under their mutex. MmioTransport:: save_state now uses it. Restore is unchanged — it sets self.queues then activate() re-clones them into the muxer, so the correct indices flow through. Includes the VSOCK_RESTORE_DIAGNOSIS.md from the diagnostic workflow.
… all devices virtio-fs (like vsock) hands a CLONED queue to its worker thread, so save_state read stale self.queues indices (next_used=0 vs guest RAM used_idx=162) — the restored fs worker then read the ring at the wrong position and failed every FUSE message with DecodeMessage(UnexpectedEof). Generalize the vsock fix: in MmioTransport::save_state, reconcile each queue's next_used/next_avail up to the guest-RAM used_ring.idx, which is authoritative (the device writes it on every completion and it can never lag). Catches every worker-thread device without per-device handling.
… on restored guest) ROOT CAUSE (workflow-confirmed via FUSE errno trace 'do_getattr MAP-MISS inode=1 -> EBADF, map has 0 inodes'): the host virtio-fs PassthroughFs is rebuilt fresh on restore with an EMPTY inode map. The root inode (nodeid=1) is only inserted by the FUSE_INIT handler, which the restored (already-mounted) guest never re-sends — so every rootfs op the guest makes (execve reading /bin/sh) resolves a cached nodeid the host has forgotten and returns EBADF, surfacing as 'Failed to spawn command sh: Bad file descriptor'. Fix: serialize the inode map (nodeid -> host path, recovered via readlink(/proc/self/fd/N) on the live O_PATH fds while the template is paused-but-alive) and rehydrate it on restore by reopening each path under its original nodeid. Plumbing: VirtioDevice gains save_device_state/restore_device_state (opaque blob); Fs shares its PassthroughFs with the worker via Arc (Server now holds Arc<F>) so the device can snapshot it; the blob rides MmioDeviceState through the snapshot state and is reapplied after activate() before vCPUs run. Open file handles deferred (a fresh exec opens its own). macOS/Windows stubs no-op.
…n on restore The template and each fork mount their rootfs at different host overlay dirs, and the template's dir is removed before restore — so absolute saved paths failed to reopen (0 inodes restored). Store each path relative to the root inode's canonical path; on restore rejoin against this fork's root_dir.
VcpuState (KVM x86 vcpu state) and Vcpu::save_state/restore_state are already #[cfg(target_arch = "x86_64")], but the VcpuEvent::SaveState variant that carries Box<VcpuState> and the running()-state match arm that referenced it were not — so the vmm failed to compile on aarch64 (E0425: cannot find type VcpuState). Gate both to x86_64. Snapshot-fork is x86_64-only; aarch64 builds cold-boot only. Verified: cargo check --target aarch64-unknown-linux-gnu -p vmm now passes.
…a-20260722 fix(windows): preserve virtiofs POSIX metadata
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Why
The Windows backend had several coupled portability and lifecycle failures: incomplete WHPX device behavior, snapshot state that omitted live virtio/kernel data, path-dependent build metadata, and firmware artifacts without a strict source-validation chain. Together these caused flaky or non-restorable guests and made release provenance difficult to verify.
Impact
The Box runtime can use this branch as a fetchable libkrun revision for reliable WHPX workloads and snapshot operations. Release builds also carry deterministic binaries and corresponding-source evidence.
Validation