Skip to content

Commit 85bd8c1

Browse files
ryanberg-aquentryanberg-aquent
authored
AB#:11681: [Doc New] Create troubleshooting article: VM won't boot after SCSI-to-NVMe change (#11691)
* Net-new P1 content * Update troubleshoot-nvme-boot-failure.md * Update troubleshoot-nvme-boot-failure.md * Update troubleshoot-nvme-boot-failure.md * Update troubleshoot-nvme-boot-failure.md * Update troubleshoot-nvme-boot-failure.md * Update troubleshoot-nvme-boot-failure.md * Update troubleshoot-nvme-boot-failure.md
1 parent 9bdb54a commit 85bd8c1

3 files changed

Lines changed: 323 additions & 0 deletions

File tree

‎support/azure/virtual-machines/linux/toc.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -162,6 +162,8 @@ items:
162162
href: troubleshoot-vm-boot-error.md
163163
- name: Troubleshoot VM boot issues due to disabled Hyper-V drivers
164164
href: linux-hyperv-issue.md
165+
- name: Troubleshoot VM boot failure after SCSI-to-NVMe disk controller change
166+
href: ../windows/troubleshoot-nvme-boot-failure.md?context=/troubleshoot/azure/virtual-machines/linux/context/context
165167
- name: Can't boot after a host node upgrade
166168
href: linux-kernel-panics-upgrade.md
167169
- name: Device names are changed

‎support/azure/virtual-machines/windows/toc.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -370,6 +370,8 @@ items:
370370
href: windows-could-not-configure-system.md
371371
- name: Troubleshoot Windows Update installation capacity
372372
href: windows-update-installation-capacity.md
373+
- name: Troubleshoot VM boot failure after SCSI-to-NVMe disk controller change
374+
href: troubleshoot-nvme-boot-failure.md
373375
- name: Not a bootable disk
374376
href: troubleshoot-guide-not-bootable-disk.md
375377
- name: Reboot loop

‎support/azure/virtual-machines/windows/troubleshoot-nvme-boot-failure.md‎

Lines changed: 319 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,319 @@
1+
---
2+
title: Troubleshoot VM boot failure after SCSI-to-NVMe disk controller change
3+
description: Fix VM boot failure after a SCSI-to-NVMe controller change, including Windows BSOD and Linux kernel panic scenarios; use the recovery steps now.
4+
services: virtual-machines
5+
author: JarrettRenshaw
6+
ms.service: azure-virtual-machines
7+
ms.collection: windows, linux
8+
ms.tgt_pltfrm: vm-windows
9+
ms.workload: infrastructure
10+
ms.topic: troubleshooting
11+
ms.date: 04/16/2026
12+
ms.author: glimoli
13+
ms.reviewer: saimsh, divargas, kageorge, scotro, ruedward, sukon, msaenzbo
14+
ms.custom: sap:My VM is not booting
15+
---
16+
17+
# Troubleshoot VM boot failure after SCSI-to-NVMe disk controller change
18+
19+
## Summary
20+
21+
This article helps you resolve boot failures that occur after an Azure virtual machine's (VM's) disk controller type changes from Small Computer System Interface (SCSI) to NVM Express (NVMe). This change usually happens due to a VM resize, manual conversion, or deployment on an NVMe-default VM size.
22+
23+
## Symptoms
24+
25+
After resizing a VM from a SCSI-based size (for example, v3, v4) to an NVMe-default size (for example, v5, v6, v7), or after manually changing the disk controller type to NVMe, the VM fails to boot.
26+
27+
### Windows symptoms
28+
29+
| Symptom | Stop Code | Description |
30+
|---------|-----------|-------------|
31+
| Blue screen (BSOD) | **0x7B — INACCESSIBLE_BOOT_DEVICE** | The Windows kernel can't find the boot disk because the NVMe storage driver (`stornvme.sys`) isn't loaded in the boot configuration. |
32+
| Blue screen (BSOD) | **0xED — UNMOUNTABLE_BOOT_VOLUME** | The operating system (OS) locates the boot device but can't mount the NTFS file system under the NVMe controller. |
33+
| Blue screen (BSOD) | **0xEF — CRITICAL_PROCESS_DIED** | The storage stack fails during early boot initialization. |
34+
| VM stuck in "Starting" | Not applicable (N/A) | The VM never reaches the running state and CRP reports `OSProvisioningTimedOut`. |
35+
| Black screen in boot diagnostics | Not applicable (N/A) | No output beyond Unified Extensible Firmware Interface (UEFI) or Basic Input/Output System (BIOS). The OS fails before producing console output. |
36+
37+
### Linux symptoms
38+
39+
| Symptom | Description |
40+
|---------|-------------|
41+
| Kernel panic — "VFS: Unable to mount root fs" | The initial RAM filesystem (initramfs) doesn't include `nvme.ko` or `nvme-core.ko` kernel modules. |
42+
| Dracut timeout | The boot loader can't find disks using the NVMe controller. |
43+
| VM stuck in "Starting" | The guest OS never signals readiness to the Azure platform. |
44+
| Missing data disks | The VM boots (OS disk works) but data disks aren't visible in `lsblk` or Disk Manager. |
45+
46+
## Cause
47+
48+
When a VM's disk controller type changes from SCSI to NVMe, the OS must have the appropriate NVMe storage driver loaded in its boot configuration as demonstrated in the following examples:
49+
50+
- **Windows**: `stornvme.sys` must be registered as a boot-start driver.
51+
- **Linux**: Include `nvme.ko` and `nvme-core.ko` in the initramfs or initial ramdisk (initrd).
52+
53+
If these drivers aren't present, the OS can't access the boot disk through the NVMe controller and fails immediately.
54+
55+
### Common triggers
56+
57+
| Trigger | Description |
58+
|---------|-------------|
59+
| **VM resize** | Resizing from `Standard_D4s_v3` (SCSI) to `Standard_D4s_v5` (NVMe default) and the disk controller type changes without notice. |
60+
| **Manual conversion** | Running `az vm update --disk-controller-type NVMe` or `$vm.StorageProfile.DiskControllerType = "NVMe"` without verifying driver readiness. |
61+
| **New VM from custom image** | Deploying a custom image (built on SCSI) to an NVMe-default VM size and the image lacks NVMe drivers in the boot path. |
62+
| **Conversion script** | Using a SCSI-to-NVMe conversion script without pre-flight driver validation. |
63+
64+
### Affected OS versions
65+
66+
| OS | NVMe Support |
67+
|----|-------------|
68+
| Windows Server 2022, 2025 | ✅ `stornvme.sys` included by default |
69+
| Windows Server 2019 | ✅ `stornvme.sys` included by default |
70+
| Windows Server 2016 | ✅ `stornvme.sys` included (might need Windows Update) |
71+
| **Windows Server 2012 R2** | ❌ **Not supported** — no native NVMe driver |
72+
| **Windows Server 2012, 2008** | ❌ **Not supported** |
73+
| Windows 10/11 | ✅ `stornvme.sys` included by default |
74+
| Ubuntu 18.04+ | ✅ NVMe modules included in default initramfs |
75+
| RHEL/CentOS 7.4+ | ✅ NVMe modules available (might need initramfs rebuild) |
76+
| SLES 12 SP3+ | ✅ NVMe modules available |
77+
| Older Linux kernels (<4.15) | ⚠️ Might require manual module installation |
78+
79+
## Resolution
80+
81+
### Option 1: Revert disk controller type to SCSI (fastest recovery)
82+
83+
If the VM isn't booting, revert the disk controller type back to SCSI. Use Azure CLI or PowerShell to run the following commands.
84+
85+
1. Stop the VM (if not already stopped).
86+
87+
**Azure CLI**
88+
89+
```azurecli
90+
az vm stop --resource-group myRG --name myVM
91+
az vm deallocate --resource-group myRG --name myVM
92+
```
93+
94+
2. Change the disk controller type back to SCSI.
95+
96+
**Azure CLI**
97+
98+
```azurecli
99+
az vm update --resource-group myRG --name myVM --disk-controller-type SCSI
100+
```
101+
102+
**PowerShell**
103+
104+
```powershell
105+
$vm = Get-AzVM -ResourceGroupName "myRG" -Name "myVM"
106+
$vm.StorageProfile.DiskControllerType = "SCSI"
107+
Update-AzVM -ResourceGroupName "myRG" -VM $vm
108+
```
109+
110+
3. Resize the SCSI-based VM size back to the original size (as needed).
111+
112+
**Azure CLI**
113+
114+
```azurecli
115+
az vm resize --resource-group myRG --name myVM --size Standard_D4s_v3
116+
```
117+
118+
4. Start the VM.
119+
120+
**Azure CLI**
121+
122+
```azurecli
123+
az vm start --resource-group myRG --name myVM
124+
```
125+
126+
### Option 2: Install NVMe drivers by using a rescue VM (Windows)
127+
128+
If you need to keep the VM on an NVMe-capable size, install the NVMe driver offline.
129+
130+
1. Stop and deallocate the failed VM.
131+
1. Create a rescue VM on a SCSI-based size (for example, `Standard_D2s_v3`).
132+
1. Attach the OS disk of the failed VM as a data disk on the rescue VM.
133+
1. On the rescue VM, use PowerShell to load the `stornvme.sys` driver into the attached disk's boot configuration.
134+
135+
```powershell
136+
# Identify the drive letter of the attached OS disk (e.g., F:)
137+
$offlineDrive = "F:"
138+
139+
# Load the offline registry hive
140+
reg load HKLM\OFFLINE "$offlineDrive\Windows\System32\config\SYSTEM"
141+
142+
# Check if stornvme is already registered
143+
reg query "HKLM\OFFLINE\ControlSet001\Services\stornvme"
144+
145+
# If not present, create the service entry
146+
reg add "HKLM\OFFLINE\ControlSet001\Services\stornvme" /v Start /t REG_DWORD /d 0 /f
147+
reg add "HKLM\OFFLINE\ControlSet001\Services\stornvme" /v Type /t REG_DWORD /d 1 /f
148+
reg add "HKLM\OFFLINE\ControlSet001\Services\stornvme" /v ErrorControl /t REG_DWORD /d 1 /f
149+
reg add "HKLM\OFFLINE\ControlSet001\Services\stornvme" /v ImagePath /t REG_EXPAND_SZ /d "System32\drivers\stornvme.sys" /f
150+
reg add "HKLM\OFFLINE\ControlSet001\Services\stornvme" /v Group /t REG_SZ /d "SCSI miniport" /f
151+
152+
# Unload the hive
153+
reg unload HKLM\OFFLINE
154+
```
155+
156+
5. Verify that `stornvme.sys` exists at `$offlineDrive\Windows\System32\drivers\stornvme.sys`. If not, copy it from `C:\Windows\System32\drivers\stornvme.sys` on the rescue VM.
157+
6. Detach the OS disk from the rescue VM.
158+
7. Reattach the rescue VM's OS disk as the OS disk on the original VM.
159+
8. Set the disk controller type to NVMe and start the VM.
160+
161+
### Option 3: Rebuild initramfs by using a rescue VM (Linux)
162+
163+
To fix a Linux VM, complete the following steps:
164+
165+
- Deallocate the failed VM.
166+
- Create a rescue VM that has access to the failed VM's OS disk.
167+
- Rebuild the initramfs image in a chroot environment.
168+
169+
#### Step 1: Deallocate the failed VM
170+
171+
Deallocate the failed VM before you attach its OS disk to a rescue VM.
172+
173+
For more information, see [What's the difference between deallocated and stopped?](/answers/questions/574969/whats-the-difference-between-deallocated-and-stopp)
174+
175+
#### Step 2: Create a rescue VM
176+
177+
Create a rescue VM and attach the failed VM's OS disk as a data disk. Use either of the following methods.
178+
179+
**Method 1: Create the rescue VM automatically (recommended)**
180+
181+
Run the `az vm repair create` command to automatically create a rescue VM and attach a copy of the failed VM's OS disk as a data disk.
182+
183+
```azurecli
184+
az vm repair create --verbose --resource-group <resource-group> --name <vm-name> --repair-username <username> --repair-password '<password>'
185+
```
186+
187+
For more information, see [Repair a Linux VM by using the Azure Virtual Machine repair commands](../linux/repair-linux-vm-using-azure-virtual-machine-repair-commands.md).
188+
189+
**Method 2: Create the rescue VM manually**
190+
191+
1. Create a rescue VM by using any available Linux VM.
192+
1. Attach the OS disk of the failed VM as a data disk.
193+
194+
#### Step 3: Rebuild initramfs in a chroot environment
195+
196+
1. Use a command-line interface (CLI) tool to mount the OS disk and enter the [`chroot`](../linux/chroot-environment-linux.md) environment. The following commands show the general setup for a Linux image that uses raw (non-Logical Volume Manager (LVM)) partitions. Adjust the partition devices (for example, `/dev/sdc2`, `/dev/sdc1`) to match your OS disk layout:
197+
198+
```bash
199+
mkdir /rescue
200+
mount -o nouuid /dev/sdc2 /rescue
201+
mount -o nouuid /dev/sdc1 /rescue/boot/
202+
mount /dev/sdc15 /rescue/boot/efi # Only if a separate EFI system partition exists. Adjust the device as needed.
203+
204+
mount -t proc /proc /rescue/proc
205+
mount -t sysfs /sys /rescue/sys
206+
mount -o bind /dev /rescue/dev
207+
mount -o bind /dev/pts /rescue/dev/pts
208+
mount -o bind /run /rescue/run
209+
chroot /rescue
210+
```
211+
212+
> [!NOTE]
213+
> Partition numbers and mount commands vary by Linux distribution and disk layout. If your OS disk uses LVM or a different distribution (for example, Ubuntu, Oracle Linux, or SUSE Linux Enterprise Server), see [Chroot environment in a Linux rescue VM](../linux/chroot-environment-linux.md) for the exact steps that apply to your scenario.
214+
215+
2. Rebuild the initramfs image so that it includes the NVMe modules. The exact steps depend on your Linux distribution.
216+
217+
**RHEL, CentOS, or SLES 12 SP3+**
218+
219+
1. Determine the kernel version of the failed VM's OS disk (not the rescue VM). Select the latest kernel version that's installed on the attached OS disk:
220+
221+
```bash
222+
KVER=$(ls -1 /lib/modules | sort -V | tail -n 1)
223+
echo "Using kernel: $KVER"
224+
```
225+
226+
Alternatively, if you know the exact kernel version in use, set it manually:
227+
228+
```bash
229+
KVER="<full-kernel-version>"
230+
```
231+
232+
2. Add the NVMe modules to a persistent dracut configuration file. This configuration ensures that the modules are included again whenever a future kernel update rebuilds the initramfs image:
233+
234+
```bash
235+
cat > /etc/dracut.conf.d/azure-nvme.conf <<'EOF'
236+
add_drivers+=" nvme nvme_core "
237+
EOF
238+
```
239+
240+
3. Rebuild the initramfs image for the selected kernel version.
241+
242+
```bash
243+
dracut --force "/boot/initramfs-${KVER}.img" "${KVER}"
244+
```
245+
246+
**Ubuntu or Debian**
247+
248+
1. Determine the kernel version of the failed VM's OS disk (not the rescue VM). Select the latest kernel version that's installed on the attached OS disk:
249+
250+
```bash
251+
KVER=$(ls -1 /lib/modules | sort -V | tail -n 1)
252+
echo "Using kernel: $KVER"
253+
```
254+
255+
Alternatively, if you know the exact kernel version, set it manually:
256+
257+
```bash
258+
KVER="<full-kernel-version>"
259+
```
260+
261+
2. Add the NVMe modules to the initramfs configuration so that they're included in current and future initramfs rebuilds:
262+
263+
```bash
264+
echo "nvme" >> /etc/initramfs-tools/modules
265+
echo "nvme_core" >> /etc/initramfs-tools/modules
266+
```
267+
268+
3. Rebuild the initramfs image for the selected kernel version.
269+
270+
```bash
271+
update-initramfs -u -k "${KVER}"
272+
```
273+
274+
1. Exit the chroot environment and unmount the OS disk.
275+
276+
```bash
277+
exit
278+
sudo umount /rescue/sys
279+
sudo umount /rescue/proc
280+
sudo umount /rescue/dev
281+
sudo umount /rescue/boot/efi
282+
sudo umount /rescue/boot
283+
sudo umount /rescue
284+
```
285+
286+
4. Swap the repaired OS disk to the original VM, and then start the VM. Use either of the following methods.
287+
288+
**Method 1: Use the Azure Virtual Machine repair commands (recommended)**
289+
290+
If you created the rescue VM by using the `az vm repair create` command, run the `az vm repair restore` command. This command swaps the repaired disk back to the original VM, starts the VM, and removes the rescue VM resources.
291+
292+
```azurecli
293+
az vm repair restore --verbose --resource-group <resource-group> --name <vm-name>
294+
```
295+
296+
**Method 2: Swap the disk manually**
297+
298+
1. Detach the repaired OS disk from the rescue VM.
299+
1. On the original VM, use the **Swap OS disk** option in the [Azure portal](https://portal.azure.com) to replace its OS disk with the repaired disk.
300+
1. Start the original VM.
301+
302+
## Prevention
303+
304+
To prevent this issue in the future:
305+
306+
- **Always verify NVMe driver readiness before resizing**. For more information, see [NVMe FAQ: Pre-resize checklist](/azure/virtual-machines/enable-nvme-remote-faqs).
307+
- **Create an OS disk snapshot** before any resize to an NVMe-capable size.
308+
- **Use the safe conversion script**: [Convert-AzVMToNVMe.ps1](https://github.com/Azure/azure-support-scripts). This script includes preflight validation and automatic snapshots.
309+
- **Don't deploy custom images to NVMe sizes** without testing them on an NVMe VM first.
310+
- **Windows Server 2012 R2 doesn't support NVMe**. Don't attempt conversion.
311+
312+
## References
313+
314+
- [NVMe overview for Azure VMs](/azure/virtual-machines/nvme-overview)
315+
- [NVMe FAQ](/azure/virtual-machines/enable-nvme-remote-faqs)
316+
- [NVMe for Linux VMs](/azure/virtual-machines/nvme-linux)
317+
- [VM sizes that support NVMe](/azure/virtual-machines/nvme-overview#supported-vm-sizes)
318+
- [Repair a Linux VM by using the Azure Virtual Machine repair commands](../linux/repair-linux-vm-using-azure-virtual-machine-repair-commands.md)
319+
- [Use the chroot environment to repair a Linux VM](../linux/chroot-environment-linux.md)

0 commit comments

Comments
 (0)