|
| 1 | +# MSR state across restore |
| 2 | + |
| 3 | +## Requirement |
| 4 | + |
| 5 | +A snapshot restore must remove all model-specific register (MSR) state written |
| 6 | +after the snapshot. The guest must observe the MSR values saved with the |
| 7 | +restored state. |
| 8 | + |
| 9 | +## Reset set |
| 10 | + |
| 11 | +The reset set contains every MSR whose guest-written value can persist. Each |
| 12 | +running snapshot stores values for this set. A snapshot created from a guest |
| 13 | +binary has no saved MSR values, so restore uses the baseline captured when the |
| 14 | +VM was created. |
| 15 | + |
| 16 | +`MSR_TABLE` lists the MSRs that hold retained state restore must write. A |
| 17 | +write-only command MSR holds no state, so it is absent from the table. |
| 18 | + |
| 19 | +The resolved reset set contains the backend core set, required MTRRs, and the |
| 20 | +validated allow list. Hyperlight sorts and deduplicates the indices before |
| 21 | +capturing the initialization baseline. |
| 22 | + |
| 23 | +The required invariant is: |
| 24 | + |
| 25 | +```text |
| 26 | +guest-writable retained state => host-readable and host-writable state |
| 27 | +``` |
| 28 | + |
| 29 | +Host-readable state need not be guest-writable. Extra reset entries are safe. |
| 30 | +`EFER`, `APIC_BASE`, `FS_BASE`, and `GS_BASE` belong to the special-register |
| 31 | +state. |
| 32 | + |
| 33 | +The two halves are established differently. Resolution checks the |
| 34 | +host-readable half at run time: a candidate index enters the set only when the |
| 35 | +host read succeeds, so an unreadable MSR is dropped. Nothing checks the |
| 36 | +host-writable half at run time. It holds by construction. Every reset MSR is |
| 37 | +stored in VP register state that the Hyper-V host interface both reads and |
| 38 | +writes, except `KERNEL_GS_BASE`, a real register the host reads and writes |
| 39 | +directly. Round-trip tests plant a guest value, restore, and assert that it does |
| 40 | +not survive. Every future entry must remain host-readable and host-writable. |
| 41 | + |
| 42 | +## Reset set justification |
| 43 | + |
| 44 | +Each entry is grounded in how the Hyper-V hypervisor handles a guest access to |
| 45 | +that register, confirmed against the Hyper-V source. |
| 46 | + |
| 47 | +A register is reset when the guest can write it and Hyper-V keeps the written |
| 48 | +value. Hyper-V keeps it in one of two ways: it stores the value in the VP's |
| 49 | +saved register state, or it lets the guest write the real register directly. |
| 50 | +Only `FS_BASE`, `GS_BASE`, and `KERNEL_GS_BASE` are written directly. Every |
| 51 | +other register below is intercepted and stored. Hyper-V stores the value on both |
| 52 | +Intel and AMD hosts. |
| 53 | + |
| 54 | +Interception alone is not the test. Hyper-V also intercepts registers the guest |
| 55 | +can only read, or that return a host-derived value. Those keep no |
| 56 | +guest-controlled state and are not reset. Each row below names the guest state |
| 57 | +that persists. |
| 58 | + |
| 59 | +| MSR (index) | Retained guest state | |
| 60 | +| --- | --- | |
| 61 | +| SYSENTER CS, ESP, EIP (`0x174`-`0x176`) | Guest write retained. | |
| 62 | +| STAR, LSTAR, CSTAR, SFMASK (`0xC000_0081`-`0xC000_0084`) | Guest write retained (syscall targets). | |
| 63 | +| KERNEL_GS_BASE (`0xC000_0102`) | Guest write retained. Written to the real register, and reachable through `SWAPGS` without a `WRMSR`. | |
| 64 | +| PAT (`0x277`) | Guest write retained. | |
| 65 | +| DEBUGCTL (`0x1D9`) | Guest write retained. | |
| 66 | +| SPEC_CTRL (`0x48`) | Guest write retained. | |
| 67 | +| CET U_CET, S_CET, PL0-3_SSP, INTERRUPT_SSP_TABLE_ADDR (`0x6A0`, `0x6A2`, `0x6A4`-`0x6A8`) | Guest write retained. | |
| 68 | +| XSS (`0xDA0`) | Guest write retained. | |
| 69 | +| TSC (`0x10`) | Guest write retained. Hyper-V forbids intercepting its implemented TSC, so restore rewrites the captured value. | |
| 70 | +| TSC_ADJUST (`0x3B`) | Guest write retained, independent of TSC. | |
| 71 | +| TSC_AUX (`0xC000_0103`) | Guest write retained. | |
| 72 | +| MTRRs (`0x2FF`, `0x200`-`0x21F`, `0x250`, `0x258`-`0x259`, `0x268`-`0x26F`) | Guest write retained (memory-type state). | |
| 73 | +| TSX_CTRL (`0x122`) | Guest write retained. | |
| 74 | +| XFD, XFD_ERR (`0x1C4`, `0x1C5`) | Guest write retained. | |
| 75 | +| UMWAIT_CONTROL (`0xE1`) | Guest write retained. Intel only. | |
| 76 | +| TSC_DEADLINE (`0x6E0`) | Guest write retained. | |
| 77 | +| BNDCFGS (`0xD90`) | Guest write retained when the host supports MPX. A guest access faults otherwise. | |
| 78 | +| MPERF, APERF (`0xE7`, `0xE8`) | Guest write retained in per-VP counters. | |
| 79 | + |
| 80 | +Write-only command MSRs hold no state. A guest write performs an action and |
| 81 | +leaves nothing to restore, so they are absent from the reset table and cannot |
| 82 | +be allowed. |
| 83 | + |
| 84 | +| MSR (index) | Behavior | |
| 85 | +| --- | --- | |
| 86 | +| PRED_CMD (`0x49`) | Guest write issues a prediction barrier. | |
| 87 | +| FLUSH_CMD (`0x10B`) | Guest write flushes caches. | |
| 88 | + |
| 89 | +Some registers are deliberately absent because guest access cannot leave state |
| 90 | +outside another reset mechanism. |
| 91 | + |
| 92 | +| MSR (index) | Why excluded | |
| 93 | +| --- | --- | |
| 94 | +| MISC_ENABLE (`0x1A0`) | Intercepted, but Hyper-V discards a guest write and returns a fixed value. No retained state. On AMD the access faults. See below. | |
| 95 | +| FRED (`0x1CC`-`0x1D4`) | Retained only when the host exposes FRED, which Hyperlight does not. A guest access faults otherwise. | |
| 96 | +| PASID (`0xD93`) | MSHV exposes ENQCMD on capable Intel hosts. PASID is a supervisor XSAVE component, so XSAVE reset clears it. WHP does not expose ENQCMD. | |
| 97 | +| PMU: PMC0, PERFEVTSEL0, FIXED_CTR_CTRL, PERF_GLOBAL_CTRL (`0xC1`, `0x186`, `0x38D`, `0x38F`) | Heads of the performance-monitoring class. Hyper-V leaves these unimplemented for the guest and installs guest-accessible descriptors, sized to the CPU counter count, only when perfmon is enabled. Hyperlight enables no perfmon, so a guest access faults and retains nothing. | |
| 98 | +| LBR: LBR_SELECT, LASTBRANCH_TOS, LBR_CTL, LBR_DEPTH (`0x1C8`, `0x1C9`, `0x14CE`, `0x14CF`) | Last-branch registers, gated with perfmon. A guest access faults and retains nothing while perfmon stays off. | |
| 99 | + |
| 100 | +Hyper-V virtualizes `BNDCFGS`, `FRED`, and `PASID` only when the matching CPU |
| 101 | +feature is exposed. `BNDCFGS` is reset because MPX is exposed by default on |
| 102 | +capable hosts. `FRED` stays excluded because Hyperlight does not expose it. |
| 103 | +MSHV can expose ENQCMD, but its XSAVE state mask then includes PASID. Hyperlight |
| 104 | +clears PASID during XSAVE reset before restoring MSRs. The performance-monitoring |
| 105 | +and last-branch registers remain inaccessible while perfmon is off. |
| 106 | + |
| 107 | +## Snapshot validation |
| 108 | + |
| 109 | +Snapshot MSR entries are untrusted. A snapshot records the reset values and the |
| 110 | +capturing sandbox's allow list. `validate_snapshot` enforces two rules against |
| 111 | +the destination VM's reset set: |
| 112 | + |
| 113 | +* The snapshot's allow list must be a subset of the destination's. A |
| 114 | + destination that allows at least as much accepts the snapshot. |
| 115 | +* Every supplied index must belong to the destination reset set. |
| 116 | + |
| 117 | +Indices the destination resets but the snapshot omits take the destination's |
| 118 | +creation-time baseline. A rejected restore poisons the sandbox before the guest |
| 119 | +can run. Equivalent allow lists produce the same sorted reset set, regardless of |
| 120 | +insertion order. |
| 121 | + |
| 122 | +## Restore across allow lists |
| 123 | + |
| 124 | +A restore or `from_snapshot` succeeds when the destination allow list is a |
| 125 | +superset of the snapshot's, on every backend. The snapshot's allowed MSRs keep |
| 126 | +their captured values. An MSR the destination allows but the snapshot did not |
| 127 | +resets to the destination baseline. A non-superset allow list is rejected |
| 128 | +uniformly. |
| 129 | + |
| 130 | +The rule is backend independent even though each backend sizes its reset set |
| 131 | +differently. KVM derives its reset set from the allow list. MSHV and WHP reset |
| 132 | +the full host table. The allow-list subset check gates the restore before either |
| 133 | +reset set is applied, so a flow that succeeds on one backend succeeds on all. |
| 134 | + |
| 135 | +The superset check is the common rule across backends. MSHV and WHP accept any |
| 136 | +allow list on their own. The shared check gives every backend KVM's constraint. |
| 137 | + |
| 138 | +## Allow list |
| 139 | + |
| 140 | +`SandboxConfiguration::allow_msrs` adds indices to the requested allow list. It |
| 141 | +enforces capacity only. VM creation verifies that each index is resettable and |
| 142 | +supported by the selected backend. |
| 143 | + |
| 144 | +KVM requires the index in `KVM_GET_MSR_INDEX_LIST` and a successful host read |
| 145 | +and write. MSHV and WHP require a named-register mapping and a successful host |
| 146 | +read. |
| 147 | + |
| 148 | +At most 64 distinct MSRs may be requested. KVM also limits the resulting |
| 149 | +contiguous filter groups to 16. |
| 150 | + |
| 151 | +## KVM |
| 152 | + |
| 153 | +KVM installs a deny filter over the full MSR space. Allowed indices form the |
| 154 | +only guest `RDMSR` and `WRMSR` paths through that filter. A denied access exits |
| 155 | +to Hyperlight, injects `#GP`, and poisons the sandbox. The denied write stores |
| 156 | +no state. |
| 157 | + |
| 158 | +The KVM reset set contains the allow list plus `KERNEL_GS_BASE` and `TSC`. |
| 159 | +`KERNEL_GS_BASE` is required because `WRGSBASE` followed by `SWAPGS` changes it |
| 160 | +without `WRMSR`. `TSC` gives restore the same clock semantics on every backend. |
| 161 | + |
| 162 | +KVM does not filter x2APIC indices `0x800..=0x8FF`. Hyperlight keeps the APIC in |
| 163 | +xAPIC mode, where MSR access to that range raises `#GP`. `APIC_BASE` is not an |
| 164 | +allowable MSR, so a guest cannot enable x2APIC. Snapshots created by Hyperlight |
| 165 | +therefore retain `APIC_BASE.EXTD = 0`. File snapshots serialize `APIC_BASE` |
| 166 | +without semantic validation, so the caller must trust the snapshot source as |
| 167 | +required by the snapshot format. |
| 168 | + |
| 169 | +## MTRRs |
| 170 | + |
| 171 | +MSHV and WHP read `IA32_MTRRCAP` when the VM is created. The required set |
| 172 | +contains `MTRR_DEF_TYPE`, each variable pair reported by `VCNT`, and all fixed |
| 173 | +MTRRs. |
| 174 | + |
| 175 | +Hyper-V accepts fixed-MTRR writes even when `MTRRCAP.FIX` is clear. All fixed |
| 176 | +MTRRs are therefore required. Hyper-V supports at most 16 variable pairs. VM |
| 177 | +creation fails when the count is larger or a required MTRR cannot be read. |
| 178 | + |
| 179 | +## MSHV |
| 180 | + |
| 181 | +MSHV has no per-MSR filter. Hyper-V permits an MSR intercept only for an |
| 182 | +unimplemented index, which already faults for the guest, and cannot intercept |
| 183 | +the implemented MSRs that hold retained state. Isolation therefore comes from |
| 184 | +reset, not a deny filter. |
| 185 | + |
| 186 | +The MSHV reset set contains every table entry that has a Hyper-V |
| 187 | +register mapping and can be read, plus the allow list. |
| 188 | + |
| 189 | +`msr_to_hv_reg_name` determines which indices the get and set path can reach. |
| 190 | +The enumerated host index list does not identify retained state, so it does not |
| 191 | +define the reset set. |
| 192 | + |
| 193 | +MSHV maps `IA32_XSS` through `MSR_IA32_REGISTER_U_XSS`. It maps `IA32_MPERF` |
| 194 | +and `IA32_APERF` to the per-VP `MCount` and `ACount` registers. TSX control, |
| 195 | +XFD, MPX (`BNDCFGS`), WAITPKG (`UMWAIT_CONTROL`), and the TSC deadline timer |
| 196 | +enter the reset set when their host-register probes succeed. |
| 197 | + |
| 198 | +MSHV enables every host-supported processor feature unless the caller supplies |
| 199 | +an explicit disabled-feature mask. Hyperlight supplies no mask. On capable |
| 200 | +Intel hosts this can expose ENQCMD and its PASID MSR. MSHV reports PASID in the |
| 201 | +partition XSAVE state mask, and Hyperlight's XSAVE reset clears it. |
| 202 | + |
| 203 | +## WHP |
| 204 | + |
| 205 | +WHP has no per-MSR filter. Its reset set contains every table entry |
| 206 | +that has a WHP register name and can be read, plus the allow list. |
| 207 | + |
| 208 | +WHP uses Germanium compatibility. Speculation control is off in its default |
| 209 | +feature banks, and perfmon (the PMU and architectural LBR) is a separate |
| 210 | +property WHP leaves off. Experimental `DEBUGCTL` bits stay disabled. The WHP |
| 211 | +API defines no FRED feature and its supported feature mask omits ENQCMD, so WHP |
| 212 | +cannot expose FRED or PASID. |
| 213 | + |
| 214 | +Each guest MSR write is either captured for restore or unsupported by the |
| 215 | +partition. Unsupported writes store no state. |
| 216 | + |
| 217 | +## TSC |
| 218 | + |
| 219 | +MSHV and WHP expose `TSC` as a host-writable register. Hyper-V stores `TSC` and |
| 220 | +`TSC_ADJUST` independently, so restoring `TSC_ADJUST` cannot undo a guest |
| 221 | +`WRMSR(TSC)`. |
| 222 | + |
| 223 | +While time is running, Hyper-V preserves `TSC - TSC_ADJUST`: writing `TSC` |
| 224 | +adds the same delta to `TSC_ADJUST`, and writing `TSC_ADJUST` adds its delta to |
| 225 | +the internal TSC offset. Restoring `TSC` followed by `TSC_ADJUST` therefore |
| 226 | +cancels any guest-controlled delta. Freezing partition time is not required for |
| 227 | +isolation. |
| 228 | + |
| 229 | +Hyper-V does not permit an intercept for its implemented `TSC` MSR. Restore |
| 230 | +must therefore write the captured `TSC` value. KVM also restores `TSC` so all |
| 231 | +backends rewind guest time with the rest of the snapshot state. |
| 232 | + |
| 233 | +## Feature exposure |
| 234 | + |
| 235 | +On MSHV and WHP a guest reaches an MSR only when the hypervisor exposes that |
| 236 | +CPU feature to the partition. This gives three cases: |
| 237 | + |
| 238 | +* Not exposed. Features the partition does not enable, such as the |
| 239 | + performance-monitoring unit, last-branch records, and FRED. Hyper-V may still |
| 240 | + model the register, but a guest access faults and stores no state until the |
| 241 | + feature is exposed. |
| 242 | +* Exposed by default. Features the host CPU supports, such as TSC deadline, |
| 243 | + UMWAIT, TSX control, CET, `MPERF`/`APERF`, XFD, AMX, and MPX. Their MSRs |
| 244 | + must be in the reset set. |
| 245 | +* Reset through another state class. MSHV can expose ENQCMD and PASID on |
| 246 | + capable Intel hosts. PASID is cleared by XSAVE reset, so it is not duplicated |
| 247 | + in the MSR reset set. |
| 248 | + |
| 249 | +MSHV and WHP enable partition features differently. MSHV creates the partition |
| 250 | +without an explicit feature mask, so it enables every processor feature the host |
| 251 | +supports. WHP starts from the host-supported set with speculation control off. |
| 252 | +MSHV exposes the broader surface and determines which registers the reset set |
| 253 | +must cover. |
| 254 | + |
| 255 | +Perfmon is not part of either default. The performance-monitoring unit and the |
| 256 | +last-branch registers are a separate opt-in partition property, off by default |
| 257 | +on both backends. Hyperlight never enables it, so those registers stay |
| 258 | +unreachable regardless of the enable-everything processor-feature default. |
| 259 | + |
| 260 | +Only reachable, retained MSRs need coverage, and retained state is always held |
| 261 | +in a host-readable and writable register. The mapped registers therefore bound |
| 262 | +the reset set: coverage is complete when every mapped register is in the reset |
| 263 | +table and reset. |
| 264 | + |
| 265 | +## Host-addressable but not guest-writable |
| 266 | + |
| 267 | +A host register mapping does not imply the guest can write the MSR. |
| 268 | +`IA32_MISC_ENABLE` (`0x1A0`) is the notable case. Hyper-V emulates it, discards |
| 269 | +a guest write, and returns a fixed value to the guest regardless of what was |
| 270 | +written. A guest cannot change it to any value, so it retains no guest state |
| 271 | +and needs no reset. On AMD the guest access faults. |
| 272 | + |
| 273 | +## Failed access reporting |
| 274 | + |
| 275 | +A KVM-denied or Hyper-V-unsupported MSR access does not persist and poisons the |
| 276 | +sandbox. The error type and its detail differ by backend. |
| 277 | + |
| 278 | +* KVM traps the access at the deny filter. Hyperlight reports |
| 279 | + `MsrReadViolation` or `MsrWriteViolation`, naming the MSR index and, for a |
| 280 | + write, the value. The report is host-verified. |
| 281 | +* MSHV and WHP have no host MSR trap. An unsupported access faults inside the |
| 282 | + guest as a general protection fault from Hyper-V, so Hyperlight reports |
| 283 | + `GuestAborted`. The message records the fault and the faulting instruction but |
| 284 | + does not identify the MSR. An exposed MSR can succeed even when absent from |
| 285 | + the allow list. Its retained state must be in the reset table. |
| 286 | + |
| 287 | +Future work: the guest exception handler could decode a faulting `RDMSR` or |
| 288 | +`WRMSR` and report the index, promoting the abort to a typed MSR violation on |
| 289 | +MSHV and WHP. That index would be guest-reported and therefore advisory. It is |
| 290 | +not implemented. |
| 291 | + |
| 292 | +## Limitations |
| 293 | + |
| 294 | +KVM's security boundary is structural because its deny filter bounds guest |
| 295 | +writes. MSHV and WHP depend on the reset table and exposed processor |
| 296 | +features. |
| 297 | + |
| 298 | +The filterless backend tests run on one CPU model per runner. Model-specific |
| 299 | +state absent on that CPU is not exercised. A backend that exposes a new |
| 300 | +retained MSR feature needs a matching table entry before Hyperlight can use it |
| 301 | +safely. |
| 302 | + |
| 303 | +The ignored full-window audit probes fixed index ranges with a small set of |
| 304 | +values. It cannot prove that every vendor MSR or accepted value is covered. |
0 commit comments