Merge remote-tracking branch 'origin/master' into codex/external-attestation-root-ca

# Conflicts:
#	dstack/Cargo.lock
#	dstack/kms/Cargo.toml
#	dstack/test-scripts/snp-e2e-smoke.sh

Author: kvinwang

CHANGELOG.md

@@ -5,6 +5,11 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Added
+- shared API authentication (`dstack-api-auth`) protecting the full VMM HTTP/pRPC/UI surface and unifying Gateway/KMS admin auth: bearer/`X-Admin-Token`/HTTP Basic/bcrypt htpasswd, constant-time verification (#796)
+
 ## [0.5.5] - 2025-10-20
 
 ### Added

README.md

@@ -73,6 +73,9 @@ Setting up dstack on your own hardware? Start with the [self-hosted quick onboar
 
 Building or customizing the guest OS itself? Follow the [guest-OS build guide](./docs/building-guest-os.md).
 
+Developing without TEE hardware? Use a development image with
+[no-TEE mode and swtpm](./docs/development-without-tee.md).
+
 ## Architecture
 
 ![Architecture](./docs/assets/arch.png)

docs/deployment.md

@@ -74,6 +74,11 @@ reuse = true
 image_path = "./images"
 run_path = "./run/vm"
 
+[auth]
+enabled = true
+# generate with: openssl rand -hex 32
+tokens = ["<paste output of: openssl rand -hex 32>"]
+
 [cvm]
 kms_urls = []
 gateway_urls = []
@@ -93,6 +98,10 @@ address = "vsock:2"
 port = 10000
 ```
 
+> **VMM API authentication (required for non-localhost binds).** Since [PR #796](https://github.com/Dstack-TEE/dstack/pull/796), the `[auth]` token guards the *entire* VMM surface — creating and stopping CVMs, the web UI, and all pRPC calls — not just `/logs`. Because the example above binds `tcp:0.0.0.0:9080` (reachable off the host), you MUST enable `[auth]`; otherwise the whole control API is exposed unauthenticated. Auth is fail-closed: with `enabled = true` and no usable credential, requests are rejected.
+>
+> Clients authenticate with `Authorization: Bearer <token>` (or the `X-Admin-Token: <token>` header). Instead of inline `tokens`, you can point to an Apache bcrypt htpasswd file with `htpasswd_file = "/etc/dstack/admin.htpasswd"` (create it with `htpasswd -B -c /etc/dstack/admin.htpasswd admin`). Only bcrypt (`-B`) entries are accepted.
+
 Download guest images from [dstack guest-OS releases](https://github.com/Dstack-TEE/dstack/releases) and extract to `./images/`.
 
 > For reproducible builds and verification, see the [Security Model](./security/security-model.md).
@@ -113,6 +122,8 @@ Production KMS requires:
 
 #### Auth Server Options
 
+> **Note:** the boot-authorization webhook below (auth-simple / auth-eth) is a *different* mechanism from the KMS admin-API authentication. The webhook allowlists which CVMs may boot and receive keys; the admin API (`[core.admin]` in `kms.toml`) guards operator RPCs. See [dstack-kms admin authentication](../dstack/kms/README.md#admin-api-authentication).
+
 | Server | Use Case | Configuration |
 |--------|----------|---------------|
 | [auth-simple](../dstack/kms/auth-simple/) | Config-file-based whitelisting | JSON config file |
@@ -428,6 +439,8 @@ curl http://<new-kms>:9203/finish
 >
 > If you skip this, `Onboard.Onboard` or later trusted RPCs will fail with KMS authorization errors.
 
+> **Admin authentication.** Onboarding itself is gated by attestation and the authorization backend above — not by a token. Separately, the KMS *admin* RPCs (for example `ClearImageCache`) are served on a dedicated `[core.admin]` listener behind the shared HTTP authenticator, just like the VMM and gateway: set `[core.admin] enabled = true` with an `auth_token` (or the `DSTACK_KMS_ADMIN_TOKEN` / `ADMIN_API_TOKEN` env vars), and clients send `Authorization: Bearer <token>` or `X-Admin-Token`. Enabled with neither `auth_token` nor `htpasswd_file` (and `insecure_no_auth = false`) fails closed — the KMS refuses to start. See [dstack-kms admin authentication](../dstack/kms/README.md#admin-api-authentication).
+
 ---
 
 ## Deploying Apps

docs/development-without-tee.md

@@ -0,0 +1,222 @@
+# Develop with dstack without TEE hardware
+
+Development guest images can run dstack's normal guest setup on a KVM machine
+that has no TDX or SEV-SNP support. The VMM starts the guest with `no_tee`, the
+guest supplies the TDX ABI through `dstack-tee-simulator`, and `swtpm` provides
+persistent TPM-backed application keys. This is suitable for development and
+integration testing, not for production workloads or secrets.
+
+## What this mode tests
+
+The development simulator provides configfs-tsm reports, RTMR extension files,
+and a CCEL event log. Guest preparation, measurements, application key setup,
+encrypted persistent storage, the guest agent, and Docker Compose therefore
+use their normal code paths.
+
+It does not provide hardware isolation or a valid hardware-signed quote. The
+host controls QEMU and swtpm, and a production verifier or KMS must reject the
+simulated quote.
+
+## Build the development image on the host
+
+TEE hardware is not needed for the build. Follow the prerequisites in
+[Build the dstack guest OS](building-guest-os.md), then run this from the
+repository root:
+
+```bash
+make os-deps
+cd os/yocto/repro-build
+RELEASE_FLAVORS="dev" ./repro-build.sh -n
+```
+
+The artifact used below is
+`os/yocto/repro-build/dist/dstack-dev-<version>.tar.gz`. Confirm that it really
+is a development image:
+
+```bash
+mkdir -p ~/.dstack-vmm/image
+tar -xzf os/yocto/repro-build/dist/dstack-dev-*.tar.gz \
+  -C ~/.dstack-vmm/image
+jq '{version, git_revision, is_dev}' \
+  ~/.dstack-vmm/image/dstack-dev-*/metadata.json
+```
+
+Expected output includes `"is_dev": true`. Use a development image for this
+workflow because production images do not include `dstack-tee-simulator`.
+
+## Install and configure the VMM
+
+Install QEMU and swtpm on the development host. On Ubuntu:
+
+```bash
+sudo apt-get update
+sudo apt-get install -y qemu-system-x86 swtpm swtpm-tools jq
+test -r /dev/kvm && test -w /dev/kvm
+```
+
+Add the current user to the `kvm` group and log in again if the `/dev/kvm`
+check fails because of its permissions.
+
+Build the VMM and supervisor from the same checkout as the image:
+
+```bash
+cargo build --manifest-path dstack/Cargo.toml --release \
+  -p dstack-vmm -p supervisor
+```
+
+Install both binaries and the CLI on the machine that will run QEMU. You can
+also run them directly from the checkout during development:
+
+```bash
+mkdir -p ~/.dstack-vmm
+install -Dm755 dstack/target/release/dstack-vmm ~/.local/bin/dstack-vmm
+install -Dm755 dstack/target/release/supervisor ~/.local/bin/supervisor
+install -Dm755 dstack/vmm/src/vmm-cli.py ~/.local/bin/dstack
+mkdir -p ~/.dstack-vmm
+cp dstack/vmm/vmm.toml ~/.dstack-vmm/vmm.toml
+```
+
+Set these values in `~/.dstack-vmm/vmm.toml`:
+
+```toml
+[image]
+path = "/home/USER/.dstack-vmm/image"
+
+[cvm]
+qemu_path = "/usr/bin/qemu-system-x86_64"
+
+[cvm.networking]
+mode = "user"
+
+[supervisor]
+exe = "/home/USER/.local/bin/supervisor"
+```
+
+Start the VMM from a stable working directory because its default API socket is
+relative to that directory:
+
+```bash
+mkdir -p ~/.dstack-vmm/run
+cd ~/.dstack-vmm/run
+dstack-vmm -c ../vmm.toml
+```
+
+In another terminal, check that the development image is visible:
+
+```bash
+cd ~/.dstack-vmm/run
+dstack lsimage --json | jq '.[] | {name, version, is_dev}'
+```
+
+## Launch with no TEE and swtpm
+
+Create a small stateful workload:
+
+```yaml
+# docker-compose.yml
+services:
+  state-test:
+    image: alpine:3.20
+    command:
+      - sh
+      - -c
+      - |
+        date -Iseconds >> /data/boots
+        touch /data/persistent-marker
+        sleep infinity
+    volumes:
+      - state-data:/data
+volumes:
+  state-data: {}
+```
+
+Convert it to app-compose JSON and select the TPM key provider:
+
+```bash
+dstack compose \
+  --name swtpm-persistence \
+  --docker-compose docker-compose.yml \
+  --key-provider tpm \
+  --public-logs \
+  --public-sysinfo \
+  --output app-compose.json
+```
+
+Deploy with the development image, no KMS, and no TEE:
+
+```bash
+dstack deploy \
+  --name swtpm-persistence \
+  --image dstack-dev-<version> \
+  --compose app-compose.json \
+  --vcpu 2 --memory 3G --disk 10G \
+  --no-tee
+```
+
+Successful output contains a VM ID. `dstack info VM_ID` should eventually show
+`Boot Progress: done`. The QEMU command line should contain the swtpm frontend
+and no TDX guest object:
+
+```bash
+ps aux | grep '[q]emu-system' | grep -E -- '-tpmdev|tpm-tis'
+ps aux | grep '[s]wtpm socket'
+```
+
+The VMM stores both durable components below the VM work directory:
+
+```text
+~/.dstack-vmm/vm/VM_ID/hda.img
+~/.dstack-vmm/vm/VM_ID/swtpm/tpm2-00.permall
+```
+
+Do not delete either path if the VM must restart with the same state.
+
+## Verify encrypted storage and restart persistence
+
+First check the guest preparation log:
+
+```bash
+dstack logs VM_ID -n 300 | grep -E \
+  'Generating app keys from TPM|Filesystem options|LUKS2 header|Device mapper'
+```
+
+A successful first boot includes output like:
+
+```text
+Generating app keys from TPM
+Filesystem options: encryption=true, filesystem=Zfs
+Mounting tmpfs for in-memory LUKS header
+Loading the LUKS2 header
+Device mapper /dev/mapper/dstack_data_disk is ready
+```
+
+Record the instance ID, then perform a graceful restart:
+
+```bash
+dstack info VM_ID | grep 'Instance ID'
+dstack stop VM_ID
+dstack start VM_ID
+```
+
+Wait for `Boot Progress: done`, then inspect the recent events and boot log:
+
+```bash
+dstack info VM_ID
+dstack logs VM_ID -n 300 | grep -E \
+  'mounting data disk|Loading the LUKS2 header|Device mapper|Container .* Started'
+```
+
+The restart passes when all of the following hold:
+
+1. the instance ID is unchanged;
+2. the event stream says `mounting data disk`, not `initializing data disk`;
+3. the LUKS mapping becomes ready without reformatting the disk;
+4. Docker starts the existing container and volume, and
+   `/data/persistent-marker` and the earlier entries in `/data/boots` remain.
+
+## Common failures
+
+`tpm key provider requested but swtpm is not installed` means `swtpm` is not on
+the VMM user's `PATH`. If QEMU reports that KVM is unavailable, confirm that
+hardware virtualization is enabled and that `/dev/kvm` is writable by the VMM
+user.

docs/dstack-gateway.md

@@ -63,3 +63,28 @@ Open `vmm.toml` and adjust dstack-gateway configuration in the `gateway` section
 
 - `base_domain`: Same as `base_domain` from `gateway.toml`'s `core.proxy` section
 - `port`: Same as `listen_port` from `gateway.toml`'s `core.proxy` section
+
+## Admin API authentication
+
+The gateway exposes a separate admin API (used for sync, WireGuard peer management, and other operator RPCs). Configure it in the `core.admin` section of `gateway.toml`:
+
+```toml
+[core.admin]
+enabled = true
+address = "0.0.0.0:9016"
+# generate with: openssl rand -hex 32
+admin_token = "<paste output of: openssl rand -hex 32>"
+# alternatively, an Apache bcrypt htpasswd file (htpasswd -B -c admin.htpasswd admin)
+# htpasswd_file = "/etc/dstack/gateway-admin.htpasswd"
+insecure_no_auth = false
+```
+
+- `enabled`: enable the admin API server.
+- `address`: bind address/port for the admin API.
+- `admin_token`: shared admin token. It can also be supplied via the environment variables `DSTACK_GATEWAY_ADMIN_TOKEN` or `ADMIN_API_TOKEN` instead of the config file.
+- `htpasswd_file`: path to an Apache bcrypt htpasswd file (create with `htpasswd -B -c admin.htpasswd admin`); only bcrypt entries are accepted. Can be used instead of, or alongside, `admin_token`.
+- `insecure_no_auth`: development-only escape hatch that disables admin authentication. Never enable it on a network-reachable admin interface.
+
+The admin server is fail-closed: if it is enabled with no `admin_token` and no `htpasswd_file`, and `insecure_no_auth` is `false`, it refuses to start rather than exposing an unauthenticated admin API.
+
+Clients authenticate by sending `Authorization: Bearer <token>` or the `X-Admin-Token: <token>` header.

docs/security/security-best-practices.md

@@ -93,6 +93,16 @@ The KMS TLS listener may keep `rpc.tls.mutual.mandatory = false` because bootstr
 
 App key release and KMS key handover still require verified caller attestation from the RA-TLS client certificate. Certificate signing verifies the CSR signature and embedded attestation before signing.
 
+## Management/admin API authentication
+
+The VMM, gateway, and KMS management surfaces must have authentication enabled in production:
+
+- VMM: set `[auth] enabled = true` with `tokens` (or `htpasswd_file`) — this guards the entire VMM HTTP/pRPC/UI surface. Never bind to a non-localhost address without it. Clients send `Authorization: Bearer <token>` or `X-Admin-Token`.
+- Gateway: set `[core.admin] admin_token` (or `htpasswd_file`) and keep `insecure_no_auth = false`. Clients send `Authorization: Bearer <token>` or `X-Admin-Token`.
+- KMS: enable `[core.admin]` with an `auth_token` (or `htpasswd_file`); the admin RPCs are served on a dedicated listener and clients send `Authorization: Bearer <token>` or `X-Admin-Token`. Enabled with no credential denies all admin RPCs (fail-closed).
+
+All three share the same HTTP authenticator: bcrypt-only htpasswd (via `htpasswd -B`), constant-time token comparison, and fail-closed behavior.
+
 ## Keep private material owner-only
 
 Secret-bearing files should be owner-only (`0600`) wherever possible, including app keys, decrypted env files, KMS root keys, gateway WireGuard/TLS keys, and ACME credentials. Preserve restrictive permissions when copying volumes, backing up `/etc/kms/certs`, or moving gateway and certbot state between hosts. Public issue [#606](https://github.com/Dstack-TEE/dstack/issues/606) tracks the remaining low-cost hardening work in dstack-managed file writes.

docs/testing-dstackup-image-pull.md

@@ -0,0 +1,39 @@
+# Test `dstackup image pull` with a local release API
+
+`--release-api-base-url` accepts plain HTTP URLs, including localhost. A small
+GitHub Releases API overlay is included for local testing. Locally published
+releases shadow GitHub; API requests that are not present locally are relayed to
+`https://api.github.com`.
+
+Start it:
+
+```bash
+./tools/mock-github-releases.py --port 8000
+```
+
+Publish a release and copy a local tarball into the mock asset store. The mock
+calculates and publishes its SHA-256 digest automatically:
+
+```bash
+curl -fsS -X POST \
+  http://localhost:8000/__admin/repos/Dstack-TEE/dstack/releases \
+  -H 'content-type: application/json' \
+  -d '{
+    "tag_name": "guest-os-v99.0.0",
+    "assets": [{"local_path": "/tmp/dstack-99.0.0.tar.gz"}]
+  }'
+```
+
+Pull the locally published latest release:
+
+```bash
+sudo dstackup image pull \
+  --release-api-base-url http://localhost:8000/repos \
+  --force
+```
+
+A release can instead reference an already hosted asset by supplying `name`,
+`browser_download_url`, and optionally `digest` in the asset object. State is
+in memory and resets when the mock exits; copied assets remain in `--asset-dir`.
+GitHub API rate limits still apply to relayed requests. Set an `Authorization`
+header on a direct request to the mock when testing authenticated relays.