Skip to content

Commit fa31cb0

Browse files
simongdaviessimongdavies
committed
Initial commit
Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
1 parent c8c4a59 commit fa31cb0

37 files changed

Lines changed: 6144 additions & 0 deletions

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
hyperlight-app/target/**
2+
device-plugin/hyperlight-device-plugin

‎README.md‎

Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,255 @@
1+
# Hyperlight on Kubernetes
2+
3+
Run [Hyperlight](https://github.com/hyperlight-dev/hyperlight) sandboxes in Kubernetes **without privileged containers**.
4+
5+
## Quick Start
6+
7+
| Target | Commands |
8+
|--------|----------|
9+
| **Local (KIND)** | `just local-up && just plugin-build && just plugin-local-push && just plugin-local-deploy` |
10+
| **Azure (AKS)** | `just azure-up && just get-aks-credentials && just plugin-build && just plugin-acr-push && just plugin-azure-deploy` |
11+
12+
```bash
13+
# Check status
14+
just status
15+
16+
# View logs
17+
just logs
18+
```
19+
20+
## Verify Device Injection
21+
22+
Deploy a test pod to verify the hypervisor device is properly injected:
23+
24+
```bash
25+
# Deploy test pod (after device plugin is running)
26+
kubectl apply -f deploy/manifests/examples/test-pod-kvm.yaml
27+
28+
# Check it's running
29+
kubectl get pod hyperlight-test-kvm
30+
31+
# View logs - should show /dev/kvm exists
32+
kubectl logs hyperlight-test-kvm
33+
34+
# Cleanup
35+
kubectl delete pod hyperlight-test-kvm
36+
```
37+
38+
Expected output:
39+
```
40+
=== Hyperlight KVM Test Pod ===
41+
Checking for /dev/kvm...
42+
✓ /dev/kvm exists
43+
crw-rw---- 1 nobody nobody 10, 232 Jan 6 12:00 /dev/kvm
44+
45+
Environment variables:
46+
HYPERLIGHT_HYPERVISOR=kvm
47+
HYPERLIGHT_DEVICE_PATH=/dev/kvm
48+
49+
=== Test Complete ===
50+
```
51+
52+
## How It Works
53+
54+
A Kubernetes Device Plugin exposes hypervisor devices (`/dev/kvm` or `/dev/mshv`) to pods using the [Container Device Interface (CDI)](https://github.com/cncf-tags/container-device-interface). Pods request `hyperlight.dev/hypervisor` and get the device injected securely.
55+
56+
```mermaid
57+
flowchart LR
58+
subgraph Node["Kubernetes Node"]
59+
DP["Device Plugin<br/>(DaemonSet)"]
60+
K[Kubelet]
61+
C[containerd]
62+
P["Pod"]
63+
CDI["/var/run/cdi/<br/>hyperlight.json"]
64+
65+
DP -->|"1. Registers<br/>hyperlight.dev/hypervisor"| K
66+
DP -->|"2. Writes CDI spec"| CDI
67+
K -->|"3. Schedules pod"| C
68+
C -->|"4. Reads CDI spec"| CDI
69+
C -->|"5. Injects /dev/kvm or /dev/mshv device"| P
70+
end
71+
```
72+
73+
## Using Hyperlight in Your Pods
74+
75+
Request the `hyperlight.dev/hypervisor` resource and apply security best practices:
76+
77+
```yaml
78+
apiVersion: v1
79+
kind: Pod
80+
metadata:
81+
name: my-hyperlight-app
82+
spec:
83+
runtimeClassName: hyperlight-kvm # or hyperlight-mshv
84+
automountServiceAccountToken: false
85+
securityContext:
86+
runAsNonRoot: true
87+
runAsUser: 65534
88+
seccompProfile:
89+
type: RuntimeDefault
90+
containers:
91+
- name: app
92+
image: your-hyperlight-app:latest
93+
resources:
94+
limits:
95+
hyperlight.dev/hypervisor: "1"
96+
securityContext:
97+
allowPrivilegeEscalation: false
98+
readOnlyRootFilesystem: true
99+
capabilities:
100+
drop: ["ALL"]
101+
```
102+
103+
See [deploy/manifests/examples/](deploy/manifests/examples/) for complete examples.
104+
105+
## Example Hyperlight Application
106+
107+
The `hyperlight-app/` directory contains an example demonstrating best practices:
108+
109+
| Feature | Implementation |
110+
|---------|----------------|
111+
| **Minimal Image** | `scratch` base - just the binaries (~2.7MB) |
112+
| **Static Binary** | musl libc, no runtime dependencies |
113+
| **Non-root** | Runs as UID 65534 (nobody), no privilege escalation |
114+
| **Read-only FS** | `readOnlyRootFilesystem: true` |
115+
| **Seccomp** | `RuntimeDefault` profile |
116+
| **No Capabilities** | All capabilities dropped |
117+
| **No K8s API** | `automountServiceAccountToken: false` |
118+
| **No Host Access** | hostNetwork/hostPID/hostIPC disabled |
119+
| **Masked /proc** | `procMount: Default` |
120+
121+
```bash
122+
# Build and deploy to AKS
123+
just app-build && just app-acr-push && just app-azure-deploy
124+
125+
# View logs
126+
kubectl logs -l app=hyperlight-hello -f
127+
```
128+
129+
## Documentation
130+
131+
| Guide | Description |
132+
|-------|-------------|
133+
| [Command Reference](docs/commands.md) | All `just` commands explained |
134+
| [Local Development](docs/local-development.md) | Test with KIND + local registry |
135+
| [Azure Deployment](docs/azure-deployment.md) | Production on AKS + ACR |
136+
| [GHCR Publishing](docs/ghcr-publishing.md) | Publish images to GitHub |
137+
| [Architecture](docs/architecture.md) | How the device plugin works |
138+
139+
## Project Structure
140+
141+
```
142+
.
143+
├── device-plugin/ # Device plugin source code
144+
│ ├── main.go # Plugin implementation
145+
│ └── Dockerfile
146+
├── hyperlight-app/ # Example Hyperlight application
147+
│ ├── host/ # Host binary (runs in container)
148+
│ ├── guest/ # Guest binary (runs in VM)
149+
│ ├── k8s/ # App deployment manifests
150+
│ └── Dockerfile # Multi-stage build (scratch)
151+
├── deploy/
152+
│ ├── manifests/ # Production Kubernetes manifests
153+
│ │ ├── device-plugin.yaml
154+
│ │ └── examples/ # Test pods and deployments
155+
│ ├── local/ # KIND-specific manifests and setup
156+
│ │ ├── setup.sh
157+
│ │ ├── teardown.sh
158+
│ │ └── device-plugin.yaml
159+
│ └── azure/ # Azure deployment
160+
│ ├── setup.sh
161+
│ ├── teardown.sh
162+
│ └── config.env
163+
├── docs/ # Documentation
164+
├── scripts/ # Build and test scripts
165+
└── justfile # Build/deploy commands
166+
```
167+
168+
## Available Commands
169+
170+
```bash
171+
just --list # Show all commands
172+
173+
# Device Plugin
174+
just plugin-build # Build binary + image
175+
just plugin-local-push # Push to local registry
176+
just plugin-local-deploy # Deploy to KIND
177+
just plugin-acr-push # Push to Azure Container Registry
178+
just plugin-azure-deploy # Deploy to AKS
179+
just plugin-ghcr-push # Push to ghcr.io/hyperlight-dev
180+
181+
# Example Hyperlight App
182+
just app-build # Build app (scratch image)
183+
just app-local-deploy # Deploy to KIND
184+
just app-azure-deploy # Deploy to AKS
185+
186+
# Cluster Management
187+
just local-up # Create KIND cluster + registry
188+
just local-down # Tear down KIND
189+
just azure-up # Create Azure infrastructure
190+
just azure-stop # Stop AKS cluster
191+
just azure-start # Start AKS cluster
192+
just azure-down # Delete all Azure resources
193+
194+
# Utilities
195+
just status # Show device plugin status
196+
just logs # View device plugin logs
197+
just check # Verify prerequisites
198+
```
199+
200+
## Requirements
201+
202+
### Platform
203+
204+
| OS | Support | Notes |
205+
|----|---------|-------|
206+
| **Linux** | ✅ Full | Native support |
207+
| **Windows** | ✅ Via WSL2 | Run everything inside WSL2 (Ubuntu recommended) |
208+
209+
> **Windows users:** Install [WSL2](https://learn.microsoft.com/en-us/windows/wsl/install) first, then work entirely inside WSL2. Docker Desktop should be configured to use WSL2 backend.
210+
>
211+
> ```powershell
212+
> # PowerShell (as admin)
213+
> wsl --install -d Ubuntu
214+
> ```
215+
>
216+
> Then open Ubuntu and continue from there.
217+
218+
### All targets
219+
220+
| Tool | Purpose | Install |
221+
|------|---------|---------|
222+
| [just](https://github.com/casey/just) | Command runner | `cargo install just` or [binaries](https://github.com/casey/just/releases) |
223+
| [Docker](https://docs.docker.com/get-docker/) | Build images | See Docker docs (use WSL2 backend on Windows) |
224+
| [kubectl](https://kubernetes.io/docs/tasks/tools/) | Kubernetes CLI | `curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"` |
225+
| [Go](https://go.dev/dl/) 1.21+ | Device plugin | `apt install golang` or download |
226+
| [envsubst](https://www.gnu.org/software/gettext/) | Template substitution | `apt install gettext-base` (usually pre-installed) |
227+
228+
### Local development (KIND)
229+
230+
| Tool | Purpose | Install |
231+
|------|---------|---------|
232+
| [KIND](https://kind.sigs.k8s.io/) 0.20+ | Local K8s | `go install sigs.k8s.io/kind@latest` |
233+
| `/dev/kvm` | Hypervisor | See below |
234+
235+
> **Minimum versions:** Kubernetes 1.26+, KIND 0.20+. Older versions may have container runtime issues.
236+
237+
### Azure deployment
238+
239+
| Tool | Purpose | Install |
240+
|------|---------|---------|
241+
| [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) | Azure management | `curl -sL https://aka.ms/InstallAzureCLIDeb \| sudo bash` |
242+
243+
### Building Hyperlight apps (optional)
244+
245+
| Tool | Purpose | Install |
246+
|------|---------|---------|
247+
| [Rust](https://rustup.rs/) | Hyperlight apps | `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \| sh` |
248+
| cargo-hyperlight | Guest binaries | `cargo install --locked cargo-hyperlight` |
249+
250+
> **Note:** Rust is only needed if you're building Hyperlight applications locally. The example app builds inside Docker, so Rust isn't required on your host machine.
251+
252+
## License
253+
254+
Apache 2.0
255+
c

‎deploy/azure/config.env‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# Azure Deployment Configuration
2+
# All resources are created in the same resource group
3+
4+
# Resource Group (shared by all Azure resources)
5+
RESOURCE_GROUP="${RESOURCE_GROUP:-hyperlight-rg}"
6+
LOCATION="${LOCATION:-westus3}"
7+
8+
# Azure Container Registry
9+
ACR_NAME="${ACR_NAME:-hyperlightacr}"
10+
11+
# AKS Cluster
12+
CLUSTER_NAME="${CLUSTER_NAME:-hyperlight-cluster}"
13+
14+
# System node pool
15+
SYSTEM_NODE_COUNT="${SYSTEM_NODE_COUNT:-1}"
16+
SYSTEM_NODE_VM_SIZE="${SYSTEM_NODE_VM_SIZE:-Standard_D2s_v3}"
17+
18+
# KVM node pool (Ubuntu with nested virtualization)
19+
KVM_NODE_POOL_NAME="${KVM_NODE_POOL_NAME:-kvmpool}"
20+
KVM_NODE_COUNT="${KVM_NODE_COUNT:-2}"
21+
KVM_NODE_VM_SIZE="${KVM_NODE_VM_SIZE:-Standard_D4s_v3}"
22+
KVM_NODE_MIN_COUNT="${KVM_NODE_MIN_COUNT:-1}"
23+
KVM_NODE_MAX_COUNT="${KVM_NODE_MAX_COUNT:-5}"
24+
25+
# MSHV node pool (AzureLinux with /dev/mshv)
26+
MSHV_NODE_POOL_NAME="${MSHV_NODE_POOL_NAME:-mshvpool}"
27+
MSHV_NODE_COUNT="${MSHV_NODE_COUNT:-2}"
28+
MSHV_NODE_VM_SIZE="${MSHV_NODE_VM_SIZE:-Standard_D4s_v3}"
29+
MSHV_NODE_MIN_COUNT="${MSHV_NODE_MIN_COUNT:-1}"
30+
MSHV_NODE_MAX_COUNT="${MSHV_NODE_MAX_COUNT:-5}"
31+
32+
# Derived values
33+
ACR_LOGIN_SERVER="${ACR_NAME}.azurecr.io"

0 commit comments

Comments
 (0)