Skip to content

Commit 21502f2

Browse files
taha-autaha-au
committed
ops: Add Ansible playbooks for nested virtualization setup
Add operational tooling to enable and verify nested virtualization support on x86_64 hosts running cloud-hypervisor: - check_nested_virt.yml: Check nested virtualization status across hosts - enable_nested_virt.yml: Enable nested virtualization on all x86_64 hosts - inventory.ini: Ansible inventory with host configuration - NESTED_VIRT_README.md: Complete usage guide These playbooks automatically skip ARM hosts and handle both Intel and AMD CPUs. Useful for enabling Windows VMs to run Hyper-V/WSL2/Docker.
1 parent 2159ee8 commit 21502f2

4 files changed

Lines changed: 436 additions & 0 deletions

File tree

‎ops/NESTED_VIRT_README.md‎

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
# Nested Virtualization Setup for Cloud Hypervisor
2+
3+
This guide helps you enable and verify nested virtualization on your x86_64 hosts running cloud-hypervisor, allowing Windows VMs to run Docker containers and WSL2.
4+
5+
## Quick Start
6+
7+
### 1. Check Current Status
8+
9+
First, check which hosts have nested virtualization enabled:
10+
11+
```bash
12+
cd ops
13+
ansible-playbook -i inventory.ini check_nested_virt.yml
14+
```
15+
16+
This will show you for each x86_64 host:
17+
- CPU vendor (Intel/AMD)
18+
- Whether nested virtualization is enabled
19+
- Cloud Hypervisor installation status
20+
- Number of running VMs
21+
22+
**ARM hosts are automatically skipped** (only x86_64 hosts are checked).
23+
24+
### 2. Enable Nested Virtualization
25+
26+
To enable nested virtualization on all x86_64 hosts:
27+
28+
```bash
29+
cd ops
30+
ansible-playbook -i inventory.ini enable_nested_virt.yml
31+
```
32+
33+
This playbook will:
34+
- Create persistent configuration in `/etc/modprobe.d/kvm-nested.conf`
35+
- Attempt to reload KVM modules immediately (if no VMs are running)
36+
- Show which hosts need a reboot
37+
38+
### 3. Reboot Hosts (if required)
39+
40+
If VMs were running or module reload failed, you'll need to reboot:
41+
42+
```bash
43+
cd ops
44+
# Reboot specific hosts
45+
ansible staging -i inventory.ini -m reboot -b --limit "host1,host2"
46+
47+
# Or reboot all hosts (be careful!)
48+
ansible staging -i inventory.ini -m reboot -b
49+
```
50+
51+
### 4. Verify After Reboot
52+
53+
After rebooting, verify nested virtualization is working:
54+
55+
```bash
56+
cd ops
57+
ansible-playbook -i inventory.ini check_nested_virt.yml
58+
```
59+
60+
You should see `✓ NESTED VIRTUALIZATION: ENABLED` for all x86_64 hosts.
61+
62+
## What This Enables
63+
64+
Once nested virtualization is enabled:
65+
66+
1. **Cloud Hypervisor automatically exposes VMX/SVM to guest VMs** - no additional configuration needed
67+
2. **Windows VMs can enable Hyper-V** and run virtualized workloads
68+
3. **WSL2 will work** in Windows VMs (requires Hyper-V)
69+
4. **Docker Desktop can run** in Windows VMs using WSL2 backend
70+
5. **Linux containers in Docker** will work for your customers' test suites
71+
72+
## How It Works
73+
74+
### Intel CPUs
75+
- Sets `options kvm_intel nested=1` in `/etc/modprobe.d/kvm-nested.conf`
76+
- Exposes VMX (Virtual Machine Extensions) to guests
77+
78+
### AMD CPUs
79+
- Sets `options kvm_amd nested=1` in `/etc/modprobe.d/kvm-nested.conf`
80+
- Exposes SVM (Secure Virtual Machine) to guests
81+
82+
### In Cloud Hypervisor
83+
- Cloud Hypervisor uses KVM's `get_supported_cpuid()` function
84+
- If KVM has nested virtualization enabled, VMX/SVM features are automatically included
85+
- No changes needed to your cloud-hypervisor configuration
86+
- Windows VMs will automatically see the virtualization extensions
87+
88+
## Manual Verification on a Host
89+
90+
To manually check on a single host:
91+
92+
```bash
93+
# SSH to the host
94+
ssh ubuntu@100.123.214.111
95+
96+
# Check if enabled (Intel)
97+
cat /sys/module/kvm_intel/parameters/nested
98+
# Should show: Y or 1
99+
100+
# Check if enabled (AMD)
101+
cat /sys/module/kvm_amd/parameters/nested
102+
# Should show: Y or 1
103+
104+
# Verify in a running Windows VM
105+
# From Windows PowerShell in the VM:
106+
systeminfo | findstr /C:"Hyper-V Requirements"
107+
# Should show "Yes" for all requirements
108+
```
109+
110+
## Performance Considerations
111+
112+
- **L2 VMs are slower than L1 VMs** - this is expected with nested virtualization
113+
- For test suites, the performance is usually acceptable
114+
- Production workloads may see 10-30% performance degradation depending on workload
115+
- I/O-bound workloads (like Docker builds) are less affected than CPU-bound ones
116+
117+
## Troubleshooting
118+
119+
### Nested virtualization shows as disabled after reboot
120+
```bash
121+
# Check if config file exists
122+
cat /etc/modprobe.d/kvm-nested.conf
123+
124+
# Reload modules manually
125+
sudo rmmod kvm_intel # or kvm_amd
126+
sudo rmmod kvm
127+
sudo modprobe kvm
128+
sudo modprobe kvm_intel nested=1 # or kvm_amd
129+
```
130+
131+
### Windows VM doesn't show virtualization support
132+
```bash
133+
# In Windows VM, run PowerShell as Administrator:
134+
Get-VMHost # Should not error if Hyper-V is available
135+
```
136+
137+
### Docker Desktop fails to start in Windows VM
138+
1. Ensure WSL2 is installed in Windows
139+
2. Enable "Use WSL2 based engine" in Docker Desktop settings
140+
3. Check Windows Features: Hyper-V and WSL2 should be enabled
141+
142+
## Files
143+
144+
- `inventory.ini` - Ansible inventory with host IPs
145+
- `check_nested_virt.yml` - Playbook to check status
146+
- `enable_nested_virt.yml` - Playbook to enable nested virtualization
147+
- `/etc/modprobe.d/kvm-nested.conf` - Created on each host (persistent config)
148+
149+
## Your Inventory
150+
151+
**x86_64 Hosts (will be configured):**
152+
- US: 6 hosts (Intel/AMD)
153+
- EU: 2 hosts (Intel/AMD)
154+
155+
**ARM64 Hosts (automatically skipped):**
156+
- US: 2 ARM hosts
157+
- EU: 1 ARM host

‎ops/check_nested_virt.yml‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
---
2+
- name: Check and Verify Nested Virtualization Support
3+
hosts: all
4+
gather_facts: yes
5+
become: yes
6+
tasks:
7+
- name: Get CPU architecture
8+
shell: uname -m
9+
register: cpu_arch
10+
changed_when: false
11+
12+
- name: Skip ARM hosts
13+
meta: end_host
14+
when: cpu_arch.stdout != "x86_64"
15+
16+
- name: Get CPU vendor (Intel or AMD)
17+
shell: |
18+
if grep -q "GenuineIntel" /proc/cpuinfo; then
19+
echo "intel"
20+
elif grep -q "AuthenticAMD" /proc/cpuinfo; then
21+
echo "amd"
22+
else
23+
echo "unknown"
24+
fi
25+
register: cpu_vendor
26+
changed_when: false
27+
28+
- name: Check if virtualization is supported in CPU
29+
shell: grep -qE 'vmx|svm' /proc/cpuinfo && echo "yes" || echo "no"
30+
register: virt_support
31+
changed_when: false
32+
33+
- name: Check if KVM modules are loaded
34+
shell: lsmod | grep -E '^kvm' | awk '{print $1}'
35+
register: kvm_modules
36+
changed_when: false
37+
failed_when: false
38+
39+
- name: Check nested virtualization status (Intel)
40+
shell: |
41+
if [ -f /sys/module/kvm_intel/parameters/nested ]; then
42+
cat /sys/module/kvm_intel/parameters/nested
43+
else
44+
echo "not_loaded"
45+
fi
46+
register: intel_nested
47+
changed_when: false
48+
when: cpu_vendor.stdout == "intel"
49+
50+
- name: Check nested virtualization status (AMD)
51+
shell: |
52+
if [ -f /sys/module/kvm_amd/parameters/nested ]; then
53+
cat /sys/module/kvm_amd/parameters/nested
54+
else
55+
echo "not_loaded"
56+
fi
57+
register: amd_nested
58+
changed_when: false
59+
when: cpu_vendor.stdout == "amd"
60+
61+
- name: Check if cloud-hypervisor is installed
62+
shell: which cloud-hypervisor || echo "not_found"
63+
register: ch_installed
64+
changed_when: false
65+
failed_when: false
66+
67+
- name: Check cloud-hypervisor version
68+
shell: cloud-hypervisor --version 2>/dev/null || echo "not_available"
69+
register: ch_version
70+
changed_when: false
71+
failed_when: false
72+
when: ch_installed.stdout != "not_found"
73+
74+
- name: Check for running cloud-hypervisor VMs
75+
shell: ps aux | grep -v grep | grep cloud-hypervisor | wc -l
76+
register: ch_vms
77+
changed_when: false
78+
79+
- name: Display comprehensive report
80+
debug:
81+
msg:
82+
- "========================================"
83+
- "Host: {{ inventory_hostname }}"
84+
- "Public IP: {{ public_ip | default('N/A') }} | Region: {{ region | default('N/A') }}"
85+
- "========================================"
86+
- "CPU Vendor: {{ cpu_vendor.stdout | upper }}"
87+
- "Hardware Virtualization (VMX/SVM): {{ virt_support.stdout | upper }}"
88+
- "KVM Modules: {{ kvm_modules.stdout_lines | default(['none']) | join(', ') }}"
89+
- "{% if cpu_vendor.stdout == 'intel' %}Nested Virt (Intel): {{ intel_nested.stdout }}{% endif %}"
90+
- "{% if cpu_vendor.stdout == 'amd' %}Nested Virt (AMD): {{ amd_nested.stdout }}{% endif %}"
91+
- "Cloud Hypervisor: {{ 'Installed' if ch_installed.stdout != 'not_found' else 'Not Installed' }}"
92+
- "Running VMs: {{ ch_vms.stdout }}"
93+
- "========================================"
94+
- "{% if (cpu_vendor.stdout == 'intel' and intel_nested.stdout in ['Y', '1']) or (cpu_vendor.stdout == 'amd' and amd_nested.stdout in ['Y', '1']) %}✓ NESTED VIRTUALIZATION: ENABLED{% else %}✗ NESTED VIRTUALIZATION: DISABLED{% endif %}"
95+
- "========================================"
96+
97+
- name: Generate Summary Report
98+
hosts: localhost
99+
gather_facts: no
100+
run_once: true
101+
tasks:
102+
- name: Create summary
103+
debug:
104+
msg:
105+
- ""
106+
- "=========================================="
107+
- " SUMMARY OF ALL HOSTS"
108+
- "=========================================="
109+
- "Total hosts checked: {{ groups['all'] | length }}"
110+
- "Check the output above for detailed status"
111+
- "=========================================="

0 commit comments

Comments
 (0)