fix(windows): stabilize WHPX and reproducible firmware builds

.cargo/config.toml

@@ -1,5 +1,5 @@
 [target.x86_64-pc-windows-msvc]
-linker = "lld-link"
+linker = "rust-lld"
 
 [target.aarch64-pc-windows-msvc]
-linker = "lld-link"
+linker = "rust-lld"

.github/workflows/publish-prebuilt.yml

@@ -38,12 +38,14 @@ jobs:
         with:
           targets: x86_64-pc-windows-msvc
 
-      - name: Create a fake init
-        shell: pwsh
-        run: New-Item -ItemType File -Path "init/init" -Force | Out-Null
+      - name: Setup Zig
+        uses: mlugg/setup-zig@v2
+        with:
+          version: 0.16.0
 
-      - name: Build krun.dll
-        run: cargo build --release -p libkrun --target x86_64-pc-windows-msvc
+      - name: Build reproducible PDB-free krun.dll
+        shell: pwsh
+        run: powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts/build-windows.ps1 -Packages libkrun
 
       - name: Package artifacts
         shell: pwsh

.github/workflows/windows_ci.yml

@@ -67,10 +67,14 @@ jobs:
         with:
           targets: x86_64-pc-windows-msvc
 
-      - name: Create a fake init
+      - name: Setup Zig
+        uses: mlugg/setup-zig@v2
+        with:
+          version: 0.16.0
+
+      - name: Build Linux guest init
         shell: pwsh
-        run: |
-          New-Item -ItemType File -Path "init/init" -Force | Out-Null
+        run: powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts/build-windows-init.ps1
 
       - name: Build check (Windows target)
         run: cargo check -p utils -p polly -p devices -p vmm -p libkrun --target x86_64-pc-windows-msvc
@@ -99,12 +103,14 @@ jobs:
         with:
           targets: x86_64-pc-windows-msvc
 
-      - name: Create a fake init
-        shell: pwsh
-        run: New-Item -ItemType File -Path "init/init" -Force | Out-Null
+      - name: Setup Zig
+        uses: mlugg/setup-zig@v2
+        with:
+          version: 0.16.0
 
-      - name: Build krun.dll
-        run: cargo build --release -p libkrun --target x86_64-pc-windows-msvc
+      - name: Build reproducible PDB-free krun.dll
+        shell: pwsh
+        run: powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts/build-windows.ps1 -Packages libkrun
 
       - name: Prepare artifact
         shell: pwsh
@@ -132,10 +138,14 @@ jobs:
         with:
           targets: x86_64-pc-windows-msvc
 
-      - name: Create a fake init
+      - name: Setup Zig
+        uses: mlugg/setup-zig@v2
+        with:
+          version: 0.16.0
+
+      - name: Build Linux guest init
         shell: pwsh
-        run: |
-          New-Item -ItemType File -Path "init/init" -Force | Out-Null
+        run: powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts/build-windows-init.ps1
 
       - name: WHPX smoke suite
         shell: pwsh

.gitignore

@@ -23,6 +23,8 @@ test-prefix
 
 # libkrunfw-win: kernel binary fetched at build time, not committed
 src/libkrunfw-win/kernel/vmlinux
+src/libkrunfw-win/kernel/kernel.bundle
+src/libkrunfw-win/kernel/kernel.bundle.metadata
 # Test logs
 test_*.log
 trace.log

Cargo.toml

@@ -1,6 +1,6 @@
 [workspace]
 members = ["src/libkrun", "src/krun_input", "src/libkrunfw-win", "krun-sys-windows"]
-exclude = ["examples/gtk_display"]
+exclude = ["examples/gtk_display", "corresponding-source/2692169"]
 resolver = "2"
 
 [profile.dev]

README.md

@@ -320,17 +320,27 @@ sudo make [FEATURE_OPTIONS] install
 * **Windows Hypervisor Platform** enabled (Settings → Optional Features, or `DISM /Online /Enable-Feature /FeatureName:HypervisorPlatform`)
 * A working [Rust](https://www.rust-lang.org/) toolchain with the `x86_64-pc-windows-msvc` target (`rustup target add x86_64-pc-windows-msvc`)
 * MSVC build tools (Visual Studio Build Tools 2019 or later)
-* A Windows-capable x86_64 ELF guest kernel at `src/libkrunfw-win/kernel/vmlinux`
+* [Zig](https://ziglang.org/) 0.16.0 (`winget install --id zig.zig --exact --version 0.16.0`)
+* Either a Windows-capable x86_64 ELF guest kernel at
+  `src/libkrunfw-win/kernel/vmlinux`, or the validated raw bundle pair produced
+  by `scripts/extract_kernel.sh` on Linux
 
 #### Windows kernel setup
 
-The Windows companion library expects a local `vmlinux` file at:
+The Windows companion library accepts either a local ELF `vmlinux` at:
 
 `src/libkrunfw-win/kernel/vmlinux`
 
-That file is not stored in normal Git history because it is too large. Build it
-locally from the WSL2 kernel source, then copy it into place before building
-`libkrunfw-windows` / `libkrun`.
+or the raw `kernel.bundle` plus `kernel.bundle.metadata` pair generated by:
+
+```bash
+bash scripts/extract_kernel.sh
+```
+
+The extractor pins and verifies the official libkrunfw v5.5.0 x86_64 archive.
+Raw API output is never treated as ELF: its exact load/entry addresses, length,
+digest, and source identity are validated from extractor metadata during the
+Rust build. Generated kernel inputs are not stored in normal Git history.
 
 Setup guide:
 
@@ -339,10 +349,31 @@ Setup guide:
 #### Compiling
 
 ```powershell
-cargo build -p libkrunfw-windows --target x86_64-pc-windows-msvc --release
-cargo build -p libkrun --target x86_64-pc-windows-msvc --release
+powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts/build-windows-init.ps1
+powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts/build-windows.ps1 -Packages libkrunfw-windows,libkrun
 ```
 
+The first script cross-compiles the ignored `init/init` payload as a stripped,
+static x86_64 Linux ELF. The second script rebuilds it, applies host-path remaps
+to all Rust crates, enables rust-lld's `/Brepro` mode, and disables the
+path-bearing release PDB/CodeView directory with `/DEBUG:NONE`. It then verifies
+that no host paths or CodeView/PDB reference remain and creates bit-for-bit
+reproducible release DLLs. Generated `init/init` is never committed, and PDBs
+are intentionally excluded from these distributable builds.
+
+#### Windows diagnostic logs
+
+Host-side file logging is disabled by default. To opt in for one process, set an
+explicit directory before starting the caller:
+
+```powershell
+$env:LIBKRUN_WINDOWS_DEBUG_LOG_DIR = "C:\path\to\libkrun-logs"
+```
+
+All Windows diagnostic files are created below that directory. The setting is
+read once per process; unset it (or start a new process without it) to keep file
+logging disabled. It does not control guest files such as `/init-rust.log`.
+
 #### Running tests
 
 ```powershell
@@ -357,10 +388,15 @@ cargo test -p vmm --target x86_64-pc-windows-msvc --lib -- test_whpx_ --ignored
 | `krun_add_net_unixstream` | `krun_add_net_tcp` (TcpStream address or NULL for disconnected) |
 | `krun_add_vsock_port` | `krun_add_vsock_port_windows` (Named Pipe name for AF_UNIX) |
 | `krun_add_disk` | same (uses file-backed block device) |
-| libkrunfw (bundled kernel) | Build `libkrunfw-windows` with a local `src/libkrunfw-win/kernel/vmlinux`, ship `libkrunfw.dll` next to `krun.dll` for automatic kernel loading, or call `krun_set_kernel(ctx, path, KRUN_KERNEL_FORMAT_ELF, NULL, cmdline)` explicitly |
+| libkrunfw (bundled kernel) | Build `libkrunfw-windows` with a local ELF `kernel/vmlinux` or the extractor-generated raw bundle pair, ship `libkrunfw.dll` next to `krun.dll` for automatic kernel loading, or call `krun_set_kernel(ctx, path, KRUN_KERNEL_FORMAT_ELF, NULL, cmdline)` explicitly |
 
 TSI for AF_INET/AF_INET6 (TCP and UDP) is enabled automatically when no virtio-net device is added, identical to Linux/macOS behavior.
 
+Windows virtiofs reports guest-created and guest-updated POSIX mode, UID, and
+GID values from its in-memory inode table. These values last for the VM
+lifetime; callers that need them across VM generations must capture them in the
+guest before shutdown and replay them after the next boot.
+
 #### Known limitations
 
 * x86_64 only (no ARM64/WHPX support on Windows)
@@ -432,7 +468,7 @@ contributor availability.
 | Item | Platform | Notes |
 |------|----------|-------|
 | Windows multi-vCPU | Windows | WHPX supports multiple virtual processors; libkrun currently wires a single vCPU on Windows. SMP boot protocol (INIT/SIPI) needs to be implemented. |
-| libkrunfw-windows distribution | Windows | The companion library exists, but a stable binary distribution path for the Windows guest `vmlinux` is still needed so callers do not have to build and place `src/libkrunfw-win/kernel/vmlinux` manually. |
+| libkrunfw-windows distribution | Windows | Distributable DLLs must keep immutable kernel/wrapper source provenance and WHPX validation; local builds can use either an ELF `vmlinux` or the pinned extractor-generated raw bundle pair. |
 | virtio-snd real backend | Windows | The current Windows backend is a NullBackend (no audio). Wiring to Windows Audio Session API (WASAPI) is planned. |
 | virtiofs: mknod / link / copy_file_range | Windows | Three syscalls are not yet implemented in the Windows passthrough driver (`fs/windows/passthrough.rs`): `mknod` (special file creation), `link` (hard links), and `copy_file_range` (in-kernel file copy). |
 | virtio-net TSO on Windows | Windows | Packet segmentation for TCP Segment Offload (TSO) is not yet implemented in the Windows TcpStream backend (`net_windows.rs`). |

SNAPSHOT_PHASE_B.md

@@ -0,0 +1,126 @@
+# Snapshot-fork — Phase A/B status
+
+## RESOLVED (2026-06-13): restored guest now runs correctly
+
+The deterministic `0x24470e0` restore fault was **NOT** virtio device-ring state
+(that was a red herring — the fault was byte-identical with/without device-state
+restore, with 1/2 vCPUs, with `nosmp`, and with `clocksource=tsc no-kvmclock`).
+
+**Real root cause:** the libkrunfw **kernel-image region `[0x1000000, 0x2450000)`
+(~21 MB)** was dropped from the snapshot. `arch_memory_regions` punches a hole there;
+`Payload::KernelMmap` (builder.rs ~2983) inserts that region via
+`guest_mem.insert_region(build_raw(...))` over libkrunfw host pages **after**
+`create_guest_memory_regions`, so it was never in `arch_mem_regions`, never file-backed,
+never in `mem_layout`. The guest's IDT / early page tables / kernel `.data` live in that
+region, so on restore the hole had no KVM slot → the first interrupt reads the IDT at
+GPA `0x2447000` (the `#PF` gate at `0x2447000 + 0xe0 = 0x24470e0`) → `DELIVERY_EV`.
+
+**Fix** (commits `cfec086` + `b823f27`): in snapshot mode `Payload::KernelMmap` copies
+the kernel into the snapshot RAM file, maps it `MAP_SHARED` (capturing the guest's
+runtime writes), and appends it to `MEM_BACKING.layout`; the restore branch sorts
+`mem_layout` by guest address (`from_regions` requires sorted) and maps all 4 regions
+`MAP_PRIVATE`. **Verified on KVM:** restored guest survives 10 s+ (was <1 s), zero
+`INTERNAL_ERROR`, zero ERROR lines; `ram.img` grew by exactly `0x1450000`.
+
+Phase B virtio device-state save/restore (below) is **implemented and kept** — it is
+needed once the restored guest exercises virtio I/O (virtio-fs / vsock backends), but it
+was not the cause of the boot-time fault.
+
+## Status after Phase A (branch `feat/snapshot-restore`)
+
+Phase A (RAM + CPU snapshot/restore) is **plumbing-complete and verified**:
+
+- **Snapshot** (`KRUN_SNAPSHOT_MEM_FILE` + `KRUN_SNAPSHOT_SOCK`): guest RAM is
+  file-backed `MAP_SHARED`; on the `snapshot <path>` socket command the vCPUs are
+  paused, KVM VM + per-vCPU state is saved (`Vm::save_state` / `Vcpu::save_state`),
+  the RAM file is fsync'd, and a bincode `SnapshotState` is written. Produces a
+  1.09 GB `ram.img` + ~10–19 KB `state.bin`, reply `ok`.
+- **Restore** (`KRUN_RESTORE_FROM` + `KRUN_SNAPSHOT_MEM_FILE`): each RAM region is
+  remapped `MAP_PRIVATE` (kernel page-level CoW) at the saved layout; kernel/cmdline/
+  firmware load and boot vCPU config are skipped; `Vm::restore_state` +
+  `Vcpu::restore_state` re-apply KVM state before `start_vcpus`.
+- **CPU-state restore is byte-perfect** — verified save-time vs restore-time:
+  `rip`, `rsp`, `cr0/cr3/cr4`, `efer`, `cs.base/sel` identical on every vCPU.
+
+### The remaining blocker (root cause, confirmed)
+
+The restored guest resumes at the correct RIP, runs briefly, then takes a
+**deterministic wild jump**: `KVM_EXIT_INTERNAL_ERROR` suberror 3 (DELIVERY_EV),
+a #PF on instruction-fetch (error `0x31` = present + instruction-fetch) at CR2
+`0x24470e0` — the *same address* every run, with 1 or 2 vCPUs.
+
+- Not the clock: forcing `clocksource=tsc tsc=reliable no-kvmclock` on the template
+  boot did **not** fix it.
+- Not missing RAM: `memory_init` registers exactly the 3 file-backed regions; the
+  kernel image (at R1 base `0x2450000`) is captured. Region map:
+  `R0 gpa=0x0 size=0x1000000`, `R1 gpa=0x2450000 size=0x20000000`,
+  `R2 gpa=0x100000000 size=0x20000000`; save and restore layouts are identical.
+
+Root cause: **virtio device-ring state is lost on restore.** On restore the host
+re-creates every virtio device fresh — `MmioTransport.device_status = INIT`, queues
+at `next_avail/next_used = 0`, device not activated — while the guest's restored RAM
+has the drivers at `DRIVER_OK` with the ring indices advanced. The guest driver, on
+its next completion/interrupt, reads a stale used-ring entry and branches through a
+garbage pointer → the deterministic wild jump.
+
+## Phase B work surface
+
+Goal: after building the device manager but **before** `start_vcpus`, re-apply the
+saved virtio state so each fresh host device resumes at the guest's indices and is
+re-activated without the guest replaying the setup MMIO writes.
+
+### State to serialize (add to `SnapshotState`)
+
+Per MMIO virtio device, keyed by MMIO base address (stable across boots for the same
+device topology):
+
+- From `MmioTransport` (`src/devices/src/virtio/mmio.rs:96`):
+  `device_status`, `acked_features_select`, `queue_select`.
+- From the `VirtioDevice` (`src/devices/src/virtio/device.rs`):
+  `acked_features` (the negotiated feature bits) — needs an accessor.
+- Per `Queue` (`src/devices/src/virtio/queue.rs:324`):
+  `size`, `ready`, `desc_table`, `avail_ring`, `used_ring`,
+  `next_avail`, `next_used`, `event_idx_enabled`, `num_added`.
+  (Several are `pub(crate)`; add a `QueueState` getter/setter or make the snapshot
+  code part of the `devices` crate.)
+
+### Restore sequence (in `build_microvm`, restore branch, before `start_vcpus`)
+
+For each MMIO device in the saved set:
+1. `set_acked_features(saved.acked_features)` on the device.
+2. Overwrite the device's `queues` with the saved `QueueState`s.
+3. Set `MmioTransport.{device_status, acked_features_select, queue_select}`.
+4. If saved `device_status` has `DRIVER_OK`, call `device.activate(mem, interrupt)`
+   to respawn the worker thread + register ioeventfd/irqfd (mirrors the
+   `DRIVER_OK` path at `mmio.rs:344`), bypassing the guest MMIO handshake.
+
+### Devices used by an idle alpine box (minimum set to get a clean restore)
+
+- **virtio-fs** (root `rootfstype=virtiofs`) — `src/devices/src/virtio/fs/`
+- **vsock** (exec/control) — `src/devices/src/virtio/vsock/`
+- **console** (`console=hvc0`) — `src/devices/src/virtio/console/`
+- **rng**, **balloon** if attached — `rng/`, `balloon/`
+
+Each device needs: an `acked_features` accessor, a `queues_mut()` / state setter, and
+an idempotent re-activation path. virtio-fs and vsock are the hard ones (worker
+threads + backend fds); console/rng are simpler and a good first target to prove the
+re-activation sequence end-to-end.
+
+### Quiescing requirement
+
+Snapshot must be taken at a device-quiescent point (no in-flight requests, rings
+drained) so the saved indices are self-consistent. An **idle deferred-main template**
+(`BOX_DEFERRED_MAIN=1`, queues empty) is the intended snapshot subject — snapshot the
+template, fork children from it. Verify the avail/used indices are equal (no pending)
+at snapshot time before trusting the restore.
+
+## Path to the 20ms / 100-VM target (after correctness)
+
+Correct single-VM restore is expected at ~10–150 ms first. To approach 100-in-20 ms:
+1. **Strip the restore path** — skip device *construction* cost; pre-build a device
+   template and clone, not re-parse config per VM.
+2. **Parallel restore** across cores (the MAP_PRIVATE fault-in is lazy, so per-VM
+   wall cost is dominated by KVM ioctls + device worker spawn).
+3. **Shared RAM template page cache** — all forks `MAP_PRIVATE` the same warm file;
+   only divergent pages cost memory.
+4. Measure honestly and report the real per-VM and 100-VM numbers at each step.