|
| 1 | +# MSR state across restore |
| 2 | + |
| 3 | +## Requirement |
| 4 | + |
| 5 | +A snapshot restore must remove all model-specific register (MSR) state written |
| 6 | +after the snapshot. The guest must observe the MSR values saved with the |
| 7 | +restored state. |
| 8 | + |
| 9 | +## Reset set |
| 10 | + |
| 11 | +The reset set contains every MSR whose guest-written value can persist. Each |
| 12 | +running snapshot stores values for this set. A snapshot created from a guest |
| 13 | +binary has no saved MSR values, so restore uses the baseline captured when the |
| 14 | +VM was created. |
| 15 | + |
| 16 | +`MSR_TABLE` lists the MSRs that hold retained state restore must write. A |
| 17 | +write-only command MSR holds no state, so it is absent from the table. |
| 18 | + |
| 19 | +The resolved reset set contains the backend core set, required MTRRs, and the |
| 20 | +validated allow list. Hyperlight sorts and deduplicates the indices before |
| 21 | +capturing the initialization baseline. |
| 22 | + |
| 23 | +The required invariant is: |
| 24 | + |
| 25 | +```text |
| 26 | +guest-writable retained state => host-readable and host-writable state |
| 27 | +``` |
| 28 | + |
| 29 | +Host-readable state need not be guest-writable. Extra reset entries are safe. |
| 30 | +`EFER`, `APIC_BASE`, `FS_BASE`, and `GS_BASE` belong to the special-register |
| 31 | +state. |
| 32 | + |
| 33 | +The two halves are established differently. Resolution checks the |
| 34 | +host-readable half at run time: a candidate index enters the set only when the |
| 35 | +host read succeeds, so an unreadable MSR is dropped. Nothing checks the |
| 36 | +host-writable half at run time. It holds by construction. Every reset MSR is |
| 37 | +stored in VP register state that the Hyper-V host interface both reads and |
| 38 | +writes, except `KERNEL_GS_BASE`, a real register the host reads and writes |
| 39 | +directly. This was confirmed against the Hyper-V source for every entry, and |
| 40 | +the round-trip tests exercise the write-back by planting a guest value, |
| 41 | +restoring, and asserting it does not survive. A future entry that is |
| 42 | +guest-writable but host-read-only would break this and must not be added. |
| 43 | + |
| 44 | +## Reset set justification |
| 45 | + |
| 46 | +Each entry is grounded in how the Hyper-V hypervisor handles a guest access to |
| 47 | +that register, confirmed against the Hyper-V source. |
| 48 | + |
| 49 | +A register is reset when the guest can write it and Hyper-V keeps the written |
| 50 | +value. Hyper-V keeps it in one of two ways: it stores the value in the VP's |
| 51 | +saved register state, or it lets the guest write the real register directly. |
| 52 | +Only `FS_BASE`, `GS_BASE`, and `KERNEL_GS_BASE` are written directly. Every |
| 53 | +other register below is intercepted and stored. Hyper-V stores the value on both |
| 54 | +Intel and AMD hosts. |
| 55 | + |
| 56 | +Interception alone is not the test. Hyper-V also intercepts registers the guest |
| 57 | +can only read, or that return a host-derived value. Those keep no |
| 58 | +guest-controlled state and are not reset. Each row below names the guest state |
| 59 | +that persists. |
| 60 | + |
| 61 | +| MSR (index) | Retained guest state | |
| 62 | +| --- | --- | |
| 63 | +| SYSENTER CS, ESP, EIP (`0x174`-`0x176`) | Guest write retained. | |
| 64 | +| STAR, LSTAR, CSTAR, SFMASK (`0xC000_0081`-`0xC000_0084`) | Guest write retained (syscall targets). | |
| 65 | +| KERNEL_GS_BASE (`0xC000_0102`) | Guest write retained. Written to the real register, and reachable through `SWAPGS` without a `WRMSR`. | |
| 66 | +| PAT (`0x277`) | Guest write retained. | |
| 67 | +| DEBUGCTL (`0x1D9`) | Guest write retained. | |
| 68 | +| SPEC_CTRL (`0x48`) | Guest write retained. | |
| 69 | +| CET U_CET, S_CET, PL0-3_SSP, INTERRUPT_SSP_TABLE_ADDR (`0x6A0`, `0x6A2`, `0x6A4`-`0x6A8`) | Guest write retained. | |
| 70 | +| XSS (`0xDA0`) | Guest write retained. | |
| 71 | +| TSC (`0x10`) | Guest write retained. Hyper-V forbids intercepting its implemented TSC, so restore rewrites the captured value. | |
| 72 | +| TSC_ADJUST (`0x3B`) | Guest write retained, independent of TSC. | |
| 73 | +| TSC_AUX (`0xC000_0103`) | Guest write retained. | |
| 74 | +| MTRRs (`0x2FF`, `0x200`-`0x21F`, `0x250`, `0x258`-`0x259`, `0x268`-`0x26F`) | Guest write retained (memory-type state). | |
| 75 | +| TSX_CTRL (`0x122`) | Guest write retained. | |
| 76 | +| XFD, XFD_ERR (`0x1C4`, `0x1C5`) | Guest write retained. | |
| 77 | +| UMWAIT_CONTROL (`0xE1`) | Guest write retained. Intel only. | |
| 78 | +| TSC_DEADLINE (`0x6E0`) | Guest write retained. | |
| 79 | +| BNDCFGS (`0xD90`) | Guest write retained when the host supports MPX. A guest access faults otherwise. | |
| 80 | +| MPERF, APERF (`0xE7`, `0xE8`) | Guest write retained in per-VP counters. | |
| 81 | + |
| 82 | +Write-only command MSRs hold no state. A guest write performs an action and |
| 83 | +leaves nothing to restore, so they are absent from the reset table and cannot |
| 84 | +be allowed. |
| 85 | + |
| 86 | +| MSR (index) | Behavior | |
| 87 | +| --- | --- | |
| 88 | +| PRED_CMD (`0x49`) | Guest write issues a prediction barrier. | |
| 89 | +| FLUSH_CMD (`0x10B`) | Guest write flushes caches. | |
| 90 | + |
| 91 | +Some registers are deliberately absent because a guest cannot leave retained |
| 92 | +state in them. |
| 93 | + |
| 94 | +| MSR (index) | Why excluded | |
| 95 | +| --- | --- | |
| 96 | +| MISC_ENABLE (`0x1A0`) | Intercepted, but Hyper-V discards a guest write and returns a fixed value. No retained state. On AMD the access faults. See below. | |
| 97 | +| FRED (`0x1CC`-`0x1D4`) | Retained only when the host exposes FRED, which Hyperlight does not. A guest access faults otherwise. | |
| 98 | +| PASID (`0xD93`) | Retained only when the host exposes ENQCMD, which Hyperlight does not. A guest access faults otherwise. | |
| 99 | +| PMU: PMC0, PERFEVTSEL0, FIXED_CTR_CTRL, PERF_GLOBAL_CTRL (`0xC1`, `0x186`, `0x38D`, `0x38F`) | Heads of the performance-monitoring class. Hyper-V leaves these unimplemented for the guest and installs guest-accessible descriptors, sized to the CPU counter count, only when perfmon is enabled. Hyperlight enables no perfmon, so a guest access faults and retains nothing. | |
| 100 | +| LBR: LBR_SELECT, LASTBRANCH_TOS, LBR_CTL, LBR_DEPTH (`0x1C8`, `0x1C9`, `0x14CE`, `0x14CF`) | Last-branch registers, gated with perfmon. A guest access faults and retains nothing while perfmon stays off. | |
| 101 | + |
| 102 | +`BNDCFGS`, `FRED`, and `PASID` share a pattern: Hyper-V virtualizes them only |
| 103 | +when the matching CPU feature is exposed, and a guest access faults otherwise. |
| 104 | +`BNDCFGS` is reset because MPX is exposed by default on capable hosts. `FRED` |
| 105 | +and `PASID` stay excluded because Hyperlight does not expose their features. The |
| 106 | +performance-monitoring and last-branch registers are the same case under |
| 107 | +perfmon. A host that exposed any of these would retain guest writes, so each |
| 108 | +needs a reset entry before Hyperlight uses that feature. |
| 109 | + |
| 110 | +## Snapshot validation |
| 111 | + |
| 112 | +Snapshot MSR entries are untrusted. A snapshot records the reset values and the |
| 113 | +capturing sandbox's allow list. `validate_snapshot` enforces two rules against |
| 114 | +the destination VM's reset set: |
| 115 | + |
| 116 | +* The snapshot's allow list must be a subset of the destination's. A |
| 117 | + destination that allows at least as much accepts the snapshot. |
| 118 | +* Every supplied index must belong to the destination reset set. |
| 119 | + |
| 120 | +Indices the destination resets but the snapshot omits take the destination's |
| 121 | +creation-time baseline. A rejected restore poisons the sandbox before the guest |
| 122 | +can run. Equivalent allow lists produce the same sorted reset set, regardless of |
| 123 | +insertion order. |
| 124 | + |
| 125 | +## Restore across allow lists |
| 126 | + |
| 127 | +A restore or `from_snapshot` succeeds when the destination allow list is a |
| 128 | +superset of the snapshot's, on every backend. The snapshot's allowed MSRs keep |
| 129 | +their captured values. An MSR the destination allows but the snapshot did not |
| 130 | +resets to the destination baseline. A non-superset allow list is rejected |
| 131 | +uniformly. |
| 132 | + |
| 133 | +The rule is backend independent even though each backend sizes its reset set |
| 134 | +differently. KVM derives its reset set from the allow list. MSHV and WHP reset |
| 135 | +the full host table. The allow-list subset check gates the restore before either |
| 136 | +reset set is applied, so a flow that succeeds on one backend succeeds on all. |
| 137 | + |
| 138 | +The uniform rule exists so a snapshot flow behaves the same everywhere. The |
| 139 | +superset check is the common denominator across backends: MSHV and WHP accept |
| 140 | +any allow list on their own, but the shared rule holds them to the same bar as |
| 141 | +KVM. Without it, a flow validated on one hypervisor could fail on another, for |
| 142 | +example passing on a developer's MSHV host and failing in KVM production. |
| 143 | + |
| 144 | +## Allow list |
| 145 | + |
| 146 | +`SandboxConfiguration::allow_msrs` adds indices to the requested allow list. It |
| 147 | +enforces capacity only. VM creation verifies that each index is resettable and |
| 148 | +supported by the selected backend. |
| 149 | + |
| 150 | +KVM requires the index in `KVM_GET_MSR_INDEX_LIST` and a successful host read |
| 151 | +and write. MSHV and WHP require a named-register mapping and a successful host |
| 152 | +read. |
| 153 | + |
| 154 | +At most 64 distinct MSRs may be requested. KVM also limits the resulting |
| 155 | +contiguous filter groups to 16. |
| 156 | + |
| 157 | +## KVM |
| 158 | + |
| 159 | +KVM installs a deny filter over the full MSR space. Allowed indices form the |
| 160 | +only guest `RDMSR` and `WRMSR` paths through that filter. A denied access exits |
| 161 | +to Hyperlight, injects `#GP`, and poisons the sandbox. The denied write stores |
| 162 | +no state. |
| 163 | + |
| 164 | +The KVM reset set contains the allow list plus `KERNEL_GS_BASE` and `TSC`. |
| 165 | +`KERNEL_GS_BASE` is required because `WRGSBASE` followed by `SWAPGS` changes it |
| 166 | +without `WRMSR`. `TSC` gives restore the same clock semantics on every backend. |
| 167 | + |
| 168 | +KVM does not filter x2APIC indices `0x800..=0x8FF`. Hyperlight keeps the APIC in |
| 169 | +xAPIC mode, where MSR access to that range raises `#GP`. |
| 170 | + |
| 171 | +## MTRRs |
| 172 | + |
| 173 | +MSHV and WHP read `IA32_MTRRCAP` when the VM is created. The required set |
| 174 | +contains `MTRR_DEF_TYPE`, each variable pair reported by `VCNT`, and all fixed |
| 175 | +MTRRs. |
| 176 | + |
| 177 | +Hyper-V accepts fixed-MTRR writes even when `MTRRCAP.FIX` is clear. All fixed |
| 178 | +MTRRs are therefore required. Hyper-V supports at most 16 variable pairs. VM |
| 179 | +creation fails when the count is larger or a required MTRR cannot be read. |
| 180 | + |
| 181 | +## MSHV |
| 182 | + |
| 183 | +MSHV has no per-MSR filter. Hyper-V permits an MSR intercept only for an |
| 184 | +unimplemented index, which already faults for the guest, and cannot intercept |
| 185 | +the implemented MSRs that hold retained state. Isolation therefore comes from |
| 186 | +reset, not a deny filter. |
| 187 | + |
| 188 | +The MSHV reset set contains every table entry that has a Hyper-V |
| 189 | +register mapping and can be read, plus the allow list. |
| 190 | + |
| 191 | +`msr_to_hv_reg_name` determines which indices the get and set path can reach. |
| 192 | +The enumerated host index list does not identify retained state, so it does not |
| 193 | +define the reset set. |
| 194 | + |
| 195 | +MSHV maps `IA32_XSS` through `MSR_IA32_REGISTER_U_XSS`. It maps `IA32_MPERF` |
| 196 | +and `IA32_APERF` to the per-VP `MCount` and `ACount` registers. TSX control, |
| 197 | +XFD, MPX (`BNDCFGS`), WAITPKG (`UMWAIT_CONTROL`), and the TSC deadline timer |
| 198 | +enter the reset set when their host-register probes succeed. |
| 199 | + |
| 200 | +## WHP |
| 201 | + |
| 202 | +WHP has no per-MSR filter. Its reset set contains every table entry |
| 203 | +that has a WHP register name and can be read, plus the allow list. |
| 204 | + |
| 205 | +WHP uses Germanium compatibility. Speculation control is off in its default |
| 206 | +feature banks, and perfmon (the PMU and architectural LBR) is a separate |
| 207 | +property WHP leaves off. Experimental `DEBUGCTL` bits stay disabled. The WHP |
| 208 | +API defines no FRED feature, so WHP cannot expose it. |
| 209 | + |
| 210 | +Each guest MSR write is either captured for restore or unsupported by the |
| 211 | +partition. Unsupported writes store no state. |
| 212 | + |
| 213 | +## TSC |
| 214 | + |
| 215 | +MSHV and WHP expose `TSC` as a host-writable register. Hyper-V stores `TSC` and |
| 216 | +`TSC_ADJUST` independently, so restoring `TSC_ADJUST` cannot undo a guest |
| 217 | +`WRMSR(TSC)`. |
| 218 | + |
| 219 | +Hyper-V does not permit an intercept for its implemented `TSC` MSR. Restore |
| 220 | +must therefore write the captured `TSC` value. KVM also restores `TSC` so all |
| 221 | +backends rewind guest time with the rest of the snapshot state. |
| 222 | + |
| 223 | +## Feature exposure |
| 224 | + |
| 225 | +On MSHV and WHP a guest reaches an MSR only when the hypervisor exposes that |
| 226 | +CPU feature to the partition. This gives two cases: |
| 227 | + |
| 228 | +* Not exposed. Features the partition does not enable, such as the |
| 229 | + performance-monitoring unit, last-branch records, FRED, and ENQCMD. Hyper-V |
| 230 | + may still model the register, but a guest access faults and stores no state |
| 231 | + until the feature is exposed. |
| 232 | +* Exposed by default. Features the host CPU supports, such as TSC deadline, |
| 233 | + UMWAIT, TSX control, CET, `MPERF`/`APERF`, XFD, AMX, and MPX. Their MSRs |
| 234 | + must be in the reset set. |
| 235 | + |
| 236 | +MSHV and WHP enable partition features differently. MSHV creates the partition |
| 237 | +without an explicit feature mask, so it enables every processor feature the host |
| 238 | +supports. The driver also offers a newer creation path that takes an explicit |
| 239 | +mask, but Hyperlight does not use it. WHP starts from the same host-supported |
| 240 | +set but leaves speculation control off in its default feature banks. MSHV is |
| 241 | +therefore the broader surface, and it drives which registers the reset set must |
| 242 | +cover. |
| 243 | + |
| 244 | +Perfmon is not part of either default. The performance-monitoring unit and the |
| 245 | +last-branch registers are a separate opt-in partition property, off by default |
| 246 | +on both backends. Hyperlight never enables it, so those registers stay |
| 247 | +unreachable regardless of the enable-everything processor-feature default. |
| 248 | + |
| 249 | +Only reachable, retained MSRs need coverage, and retained state is always held |
| 250 | +in a host-readable and writable register. The mapped registers therefore bound |
| 251 | +the reset set: coverage is complete when every mapped register is in the reset |
| 252 | +table and reset. |
| 253 | + |
| 254 | +## Host-addressable but not guest-writable |
| 255 | + |
| 256 | +A host register mapping does not imply the guest can write the MSR. |
| 257 | +`IA32_MISC_ENABLE` (`0x1A0`) is the notable case. Hyper-V emulates it, discards |
| 258 | +a guest write, and returns a fixed value to the guest regardless of what was |
| 259 | +written. A guest cannot change it to any value, so it retains no guest state |
| 260 | +and needs no reset. It is therefore deliberately absent from the reset table. |
| 261 | +Among the registers Hyperlight tracks it is the only one whose guest write |
| 262 | +Hyper-V discards while still returning a stable value. On AMD the guest access |
| 263 | +faults outright. |
| 264 | + |
| 265 | +This is why `0x1A0` serves as the "host-probeable but not resettable" example in |
| 266 | +the allow-list rejection test: the host can read and write the register, but |
| 267 | +the guest cannot retain a value in it. |
| 268 | + |
| 269 | +## Denied access reporting |
| 270 | + |
| 271 | +A guest access to a denied or unsupported MSR is reported differently per |
| 272 | +backend. Both outcomes are safe. The access never persists and the sandbox is |
| 273 | +poisoned. Only the error type and its detail differ. |
| 274 | + |
| 275 | +* KVM traps the access at the deny filter. Hyperlight reports |
| 276 | + `MsrReadViolation` or `MsrWriteViolation`, naming the MSR index and, for a |
| 277 | + write, the value. The report is host-verified. |
| 278 | +* MSHV and WHP have no host MSR trap. The access faults inside the guest as a |
| 279 | + general protection fault from Hyper-V, so Hyperlight reports `GuestAborted`. |
| 280 | + The message records the fault and the faulting instruction but does not |
| 281 | + identify the MSR. |
| 282 | + |
| 283 | +Future work: the guest exception handler could decode the faulting `RDMSR` or |
| 284 | +`WRMSR` and report the index, promoting the abort to a typed MSR violation on |
| 285 | +MSHV and WHP. That index would be guest-reported and therefore advisory. It is |
| 286 | +not implemented. |
| 287 | + |
| 288 | +## Limitations |
| 289 | + |
| 290 | +KVM's security boundary is structural because its deny filter bounds guest |
| 291 | +writes. MSHV and WHP depend on the reset table and exposed processor |
| 292 | +features. |
| 293 | + |
| 294 | +The filterless backend tests run on one CPU model per runner. Model-specific |
| 295 | +state absent on that CPU is not exercised. A backend that exposes a new |
| 296 | +retained MSR feature needs a matching table entry before Hyperlight can use it |
| 297 | +safely. |
| 298 | + |
| 299 | +The ignored full-window audit probes fixed index ranges with a small set of |
| 300 | +values. It cannot prove that every vendor MSR or accepted value is covered. |
0 commit comments