docs: add no-TEE swtpm development guide

Author: kvinwang

README.md

@@ -73,6 +73,9 @@ Setting up dstack on your own hardware? Start with the [self-hosted quick onboar
 
 Building or customizing the guest OS itself? Follow the [guest-OS build guide](./docs/building-guest-os.md).
 
+Developing without TEE hardware? Use a development image with
+[no-TEE mode and swtpm](./docs/development-without-tee.md).
+
 ## Architecture
 
 ![Architecture](./docs/assets/arch.png)

docs/development-without-tee.md

@@ -0,0 +1,297 @@
+# Develop with dstack without TEE hardware
+
+Development guest images can run dstack's normal guest setup on a KVM machine
+that has no TDX or SEV-SNP support. The VMM starts the guest with `no_tee`, the
+guest supplies the TDX ABI through `dstack-tee-simulator`, and `swtpm` provides
+persistent TPM-backed application keys. This is suitable for development and
+integration testing, not for production workloads or secrets.
+
+This guide covers both a normal KVM host and a nested libvirt lab. The nested
+path was tested with a 16 GiB Ubuntu 24.04 VM, QEMU 8.2.2, swtpm 0.7.3, and a
+`dstack-dev-0.6.0` image built from commit `4d014d3b`.
+
+## What this mode tests
+
+The development simulator provides configfs-tsm reports, RTMR extension files,
+and a CCEL event log. Guest preparation, measurements, application key setup,
+encrypted persistent storage, the guest agent, and Docker Compose therefore
+use their normal code paths.
+
+It does not provide hardware isolation or a valid hardware-signed quote. The
+host controls QEMU and swtpm, and a production verifier or KMS must reject the
+simulated quote.
+
+## Build the development image on the host
+
+TEE hardware is not needed for the build. Follow the prerequisites in
+[Build the dstack guest OS](building-guest-os.md), then run this from the
+repository root:
+
+```bash
+make os-deps
+cd os/yocto/repro-build
+RELEASE_FLAVORS="prod dev" ./repro-build.sh -n
+```
+
+The artifact used below is
+`os/yocto/repro-build/dist/dstack-dev-<version>.tar.gz`. Confirm that it really
+is a development image:
+
+```bash
+mkdir -p /tmp/dstack-dev-image
+tar -xzf os/yocto/repro-build/dist/dstack-dev-*.tar.gz \
+  -C /tmp/dstack-dev-image
+jq '{version, git_revision, is_dev}' \
+  /tmp/dstack-dev-image/dstack-dev-*/metadata.json
+```
+
+Expected output includes `"is_dev": true`. The VMM rejects `--no-tee` for a
+production image.
+
+## Optional: create a nested libvirt host
+
+Skip this section when the development machine already exposes `/dev/kvm`.
+The outer host must enable VT-x or AMD-V and load KVM with nested virtualization
+enabled.
+
+Create an Ubuntu VM with host CPU passthrough. This example assumes an Ubuntu
+cloud image and cloud-init seed already exist:
+
+```bash
+virt-install \
+  --name dstack-no-tee-lab \
+  --memory 16384 \
+  --vcpus 8 \
+  --cpu host-passthrough,cache.mode=passthrough \
+  --disk path=/var/lib/libvirt/images/dstack-no-tee-lab.qcow2,bus=virtio \
+  --disk path=/var/lib/libvirt/images/dstack-no-tee-lab-seed.img,device=cdrom \
+  --os-variant ubuntu24.04 \
+  --network network=default,model=virtio \
+  --graphics none \
+  --import --noautoconsole
+```
+
+Add a virtio-vsock device. The dstack guest uses vsock to obtain its launch
+configuration from the VMM:
+
+```bash
+cat >/tmp/dstack-vsock.xml <<'EOF'
+<vsock model="virtio"><cid auto="yes"/></vsock>
+EOF
+virsh attach-device dstack-no-tee-lab /tmp/dstack-vsock.xml --live --config
+```
+
+Inside the Ubuntu VM, install the host dependencies and verify nested KVM:
+
+```bash
+sudo apt-get update
+sudo apt-get install -y \
+  qemu-system-x86 libvirt-daemon-system libvirt-clients swtpm swtpm-tools
+sudo usermod -aG kvm,libvirt "$USER"
+
+test -r /dev/kvm && test -w /dev/kvm
+qemu-system-x86_64 --version
+swtpm --version
+lscpu | grep -E 'Virtualization|Hypervisor'
+```
+
+Log out and back in after changing groups. A tested nested host reported both
+`Virtualization: VT-x` and `Hypervisor vendor: KVM`.
+
+Copy the image built on the outer host into the nested host rather than
+rebuilding it there:
+
+```bash
+scp os/yocto/repro-build/dist/dstack-dev-*.tar.gz ubuntu@LAB_HOST:/tmp/
+ssh ubuntu@LAB_HOST 'mkdir -p ~/.dstack-vmm/image && \
+  tar -xzf /tmp/dstack-dev-*.tar.gz -C ~/.dstack-vmm/image'
+```
+
+## Install and configure the VMM
+
+Build the VMM and supervisor from the same checkout as the image:
+
+```bash
+cargo build --manifest-path dstack/Cargo.toml --release \
+  -p dstack-vmm -p supervisor
+```
+
+Install both binaries and the CLI on the machine that will run QEMU. You can
+also run them directly from the checkout during development:
+
+```bash
+install -Dm755 dstack/target/release/dstack-vmm ~/.local/bin/dstack-vmm
+install -Dm755 dstack/target/release/supervisor ~/.local/bin/supervisor
+install -Dm755 dstack/vmm/src/vmm-cli.py ~/.local/bin/dstack
+cp dstack/vmm/vmm.toml ~/.dstack-vmm/vmm.toml
+```
+
+Set these values in `~/.dstack-vmm/vmm.toml`:
+
+```toml
+[image]
+path = "/home/USER/.dstack-vmm/image"
+
+[cvm]
+qemu_path = "/usr/bin/qemu-system-x86_64"
+
+[cvm.networking]
+mode = "user"
+
+[supervisor]
+exe = "/home/USER/.local/bin/supervisor"
+```
+
+On bare metal, keep the default host API address `vsock:2`. An outer libvirt VM
+has its own non-2 local CID, so the nested setup must listen on any local CID:
+
+```toml
+[host_api]
+address = "vsock:4294967295"
+port = 10000
+```
+
+Start the VMM from a stable working directory because its default API socket is
+relative to that directory:
+
+```bash
+mkdir -p ~/.dstack-vmm/run
+cd ~/.dstack-vmm/run
+dstack-vmm -c ../vmm.toml
+```
+
+In another terminal, check that the development image is visible:
+
+```bash
+cd ~/.dstack-vmm/run
+dstack lsimage --json | jq '.[] | {name, version, is_dev}'
+```
+
+## Launch with no TEE and swtpm
+
+Create a small stateful workload:
+
+```yaml
+# docker-compose.yml
+services:
+  state-test:
+    image: alpine:3.20
+    command:
+      - sh
+      - -c
+      - |
+        date -Iseconds >> /data/boots
+        touch /data/persistent-marker
+        sleep infinity
+    volumes:
+      - state-data:/data
+volumes:
+  state-data: {}
+```
+
+Convert it to app-compose JSON and select the TPM key provider:
+
+```bash
+dstack compose \
+  --name swtpm-persistence \
+  --docker-compose docker-compose.yml \
+  --key-provider tpm \
+  --public-logs \
+  --public-sysinfo \
+  --output app-compose.json
+```
+
+Deploy with the development image, no KMS, and no TEE:
+
+```bash
+dstack deploy \
+  --name swtpm-persistence \
+  --image dstack-dev-<version> \
+  --compose app-compose.json \
+  --vcpu 2 --memory 3G --disk 10G \
+  --no-tee
+```
+
+Successful output contains a VM ID. `dstack info VM_ID` should eventually show
+`Boot Progress: done`. The QEMU command line should contain the swtpm frontend
+and no TDX guest object:
+
+```bash
+ps aux | grep '[q]emu-system' | grep -E -- '-tpmdev|tpm-tis'
+ps aux | grep '[s]wtpm socket'
+```
+
+The VMM stores both durable components below the VM work directory:
+
+```text
+~/.dstack-vmm/vm/VM_ID/hda.img
+~/.dstack-vmm/vm/VM_ID/swtpm/tpm2-00.permall
+```
+
+Do not delete either path if the VM must restart with the same state.
+
+## Verify encrypted storage and restart persistence
+
+First check the guest preparation log:
+
+```bash
+dstack logs VM_ID -n 300 | grep -E \
+  'Generating app keys from TPM|Filesystem options|LUKS2 header|Device mapper'
+```
+
+A successful first boot includes output like:
+
+```text
+Generating app keys from TPM
+Filesystem options: encryption=true, filesystem=Zfs
+Mounting tmpfs for in-memory LUKS header
+Loading the LUKS2 header
+Device mapper /dev/mapper/dstack_data_disk is ready
+```
+
+The LUKS2 header is kept in memory rather than written beside the ciphertext,
+so `cryptsetup luksDump hda.img` is not an applicable test. The log establishes
+that the guest opened `/dev/mapper/dstack_data_disk` and placed ZFS on that
+encrypted mapping rather than on the raw data partition.
+
+Record the instance ID, then perform a graceful restart:
+
+```bash
+dstack info VM_ID | grep 'Instance ID'
+dstack stop VM_ID
+dstack start VM_ID
+```
+
+Wait for `Boot Progress: done`, then inspect the recent events and boot log:
+
+```bash
+dstack info VM_ID
+dstack logs VM_ID -n 300 | grep -E \
+  'mounting data disk|Loading the LUKS2 header|Device mapper|Container .* Started'
+```
+
+The restart passes when all of the following hold:
+
+1. the instance ID is unchanged;
+2. the event stream says `mounting data disk`, not `initializing data disk`;
+3. the LUKS mapping becomes ready without reformatting the disk;
+4. Docker starts the existing container and volume, and
+   `/data/persistent-marker` and the earlier entries in `/data/boots` remain.
+
+In the nested test, the same instance ID
+`cbcc237cdf1f8134434f374988e776c7ddea2e07` appeared before and after restart.
+The second boot reported `mounting data disk`, loaded the detached LUKS2
+header, opened `/dev/mapper/dstack_data_disk`, and started the existing
+`dstack-state-test-1` container.
+
+## Common failures
+
+`tpm key provider requested but swtpm is not installed` means `swtpm` is not on
+the VMM user's `PATH`. If QEMU reports that KVM is unavailable, confirm that
+the outer hypervisor uses host CPU passthrough and that `/dev/kvm` is writable
+inside the Ubuntu VM.
+
+`Failed to bind host API: Cannot assign requested address` in a nested host
+usually means the VMM tried to bind `vsock:2`. Add a virtio-vsock device to the
+outer VM and use `vsock:4294967295` for the nested VMM. Do not make that change
+on a normal bare-metal host without a reason.