docs: document management and admin API authentication (#796)

kvinwang · kvinwang · commit 513926e3f1b6 · 2026-07-20T02:15:46.000-07:00
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -5,6 +5,11 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Added
+- shared API authentication (`dstack-api-auth`) protecting the full VMM HTTP/pRPC/UI surface and unifying Gateway/KMS admin auth: bearer/`X-Admin-Token`/HTTP Basic/bcrypt htpasswd, constant-time verification (#796)
+
 ## [0.5.5] - 2025-10-20
 
 ### Added
diff --git a/docs/deployment.md b/docs/deployment.md
@@ -74,6 +74,11 @@ reuse = true
 image_path = "./images"
 run_path = "./run/vm"
 
+[auth]
+enabled = true
+# generate with: openssl rand -hex 32
+tokens = ["<paste output of: openssl rand -hex 32>"]
+
 [cvm]
 kms_urls = []
 gateway_urls = []
@@ -93,6 +98,10 @@ address = "vsock:2"
 port = 10000
 ```
 
+> **VMM API authentication (required for non-localhost binds).** Since [PR #796](https://github.com/Dstack-TEE/dstack/pull/796), the `[auth]` token guards the *entire* VMM surface — creating and stopping CVMs, the web UI, and all pRPC calls — not just `/logs`. Because the example above binds `tcp:0.0.0.0:9080` (reachable off the host), you MUST enable `[auth]`; otherwise the whole control API is exposed unauthenticated. Auth is fail-closed: with `enabled = true` and no usable credential, requests are rejected.
+>
+> Clients authenticate with `Authorization: Bearer <token>` (or the `X-Admin-Token: <token>` header). Instead of inline `tokens`, you can point to an Apache bcrypt htpasswd file with `htpasswd_file = "/etc/dstack/admin.htpasswd"` (create it with `htpasswd -B -c /etc/dstack/admin.htpasswd admin`). Only bcrypt (`-B`) entries are accepted.
+
 Download guest images from [dstack guest-OS releases](https://github.com/Dstack-TEE/dstack/releases) and extract to `./images/`.
 
 > For reproducible builds and verification, see the [Security Model](./security/security-model.md).
@@ -113,6 +122,8 @@ Production KMS requires:
 
 #### Auth Server Options
 
+> **Note:** the boot-authorization webhook below (auth-simple / auth-eth) is a *different* mechanism from the KMS admin-API authentication. The webhook allowlists which CVMs may boot and receive keys; the admin API (`admin_token_hash` in `kms.toml`) guards operator RPCs. See [dstack-kms admin authentication](../dstack/kms/README.md#admin-api-authentication).
+
 | Server | Use Case | Configuration |
 |--------|----------|---------------|
 | [auth-simple](../dstack/kms/auth-simple/) | Config-file-based whitelisting | JSON config file |
@@ -428,6 +439,8 @@ curl http://<new-kms>:9203/finish
 >
 > If you skip this, `Onboard.Onboard` or later trusted RPCs will fail with KMS authorization errors.
 
+> **Admin authentication.** Onboarding itself is gated by attestation and the authorization backend above — not by a token. Separately, the KMS *admin* RPCs (for example clearing the image cache) are guarded by an admin token: set `admin_token_hash` (SHA-256 hex of your token) in the KMS `kms.toml` and send the token on admin requests via `Authorization: Bearer <token>`. An empty hash denies all admin RPCs (fail-closed). See [dstack-kms admin authentication](../dstack/kms/README.md#admin-api-authentication).
+
 ---
 
 ## Deploying Apps
diff --git a/docs/dstack-gateway.md b/docs/dstack-gateway.md
@@ -63,3 +63,28 @@ Open `vmm.toml` and adjust dstack-gateway configuration in the `gateway` section
 
 - `base_domain`: Same as `base_domain` from `gateway.toml`'s `core.proxy` section
 - `port`: Same as `listen_port` from `gateway.toml`'s `core.proxy` section
+
+## Admin API authentication
+
+The gateway exposes a separate admin API (used for sync, WireGuard peer management, and other operator RPCs). Configure it in the `core.admin` section of `gateway.toml`:
+
+```toml
+[core.admin]
+enabled = true
+address = "0.0.0.0:9016"
+# generate with: openssl rand -hex 32
+admin_token = "<paste output of: openssl rand -hex 32>"
+# alternatively, an Apache bcrypt htpasswd file (htpasswd -B -c admin.htpasswd admin)
+# htpasswd_file = "/etc/dstack/gateway-admin.htpasswd"
+insecure_no_auth = false
+```
+
+- `enabled`: enable the admin API server.
+- `address`: bind address/port for the admin API.
+- `admin_token`: shared admin token. It can also be supplied via the environment variables `DSTACK_GATEWAY_ADMIN_TOKEN` or `ADMIN_API_TOKEN` instead of the config file.
+- `htpasswd_file`: path to an Apache bcrypt htpasswd file (create with `htpasswd -B -c admin.htpasswd admin`); only bcrypt entries are accepted. Can be used instead of, or alongside, `admin_token`.
+- `insecure_no_auth`: development-only escape hatch that disables admin authentication. Never enable it on a network-reachable admin interface.
+
+The admin server is fail-closed: if it is enabled with no `admin_token` and no `htpasswd_file`, and `insecure_no_auth` is `false`, it refuses to start rather than exposing an unauthenticated admin API.
+
+Clients authenticate by sending `Authorization: Bearer <token>` or the `X-Admin-Token: <token>` header.
diff --git a/docs/security/security-best-practices.md b/docs/security/security-best-practices.md
@@ -93,6 +93,16 @@ The KMS TLS listener may keep `rpc.tls.mutual.mandatory = false` because bootstr
 
 App key release and KMS key handover still require verified caller attestation from the RA-TLS client certificate. Certificate signing verifies the CSR signature and embedded attestation before signing.
 
+## Management/admin API authentication
+
+The VMM, gateway, and KMS management surfaces must have authentication enabled in production:
+
+- VMM: set `[auth] enabled = true` with `tokens` (or `htpasswd_file`) — this guards the entire VMM HTTP/pRPC/UI surface. Never bind to a non-localhost address without it.
+- Gateway: set `[core.admin] admin_token` (or `htpasswd_file`) and keep `insecure_no_auth = false`.
+- KMS: set `admin_token_hash` (SHA-256 hex of the admin token); an empty hash denies all admin RPCs.
+
+Only bcrypt htpasswd entries are accepted (create with `htpasswd -B`), and credential comparisons are constant-time. Clients authenticate with `Authorization: Bearer <token>` (or `X-Admin-Token`). All three are fail-closed.
+
 ## Keep private material owner-only
 
 Secret-bearing files should be owner-only (`0600`) wherever possible, including app keys, decrypted env files, KMS root keys, gateway WireGuard/TLS keys, and ACME credentials. Preserve restrictive permissions when copying volumes, backing up `/etc/kms/certs`, or moving gateway and certbot state between hosts. Public issue [#606](https://github.com/Dstack-TEE/dstack/issues/606) tracks the remaining low-cost hardening work in dstack-managed file writes.
diff --git a/docs/tutorials/attestation-verification.md b/docs/tutorials/attestation-verification.md
@@ -129,6 +129,7 @@ Verify you have a running CVM:
 
 ```bash
 cd ~/dstack/dstack/vmm
+export DSTACK_VMM_AUTH_USER=admin
 export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token)
 ./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm
 ```
@@ -141,6 +142,7 @@ The VMM provides a `/guest/Info` endpoint that proxies into the CVM and retrieve
 
 ```bash
 cd ~/dstack/dstack/vmm
+export DSTACK_VMM_AUTH_USER=admin
 export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token)
 
 # Get the VM UUID for hello-world
@@ -384,6 +386,7 @@ Compare the CVM's actual measurements against your expected values:
 # verify-measurements.sh
 
 cd ~/dstack/dstack/vmm
+export DSTACK_VMM_AUTH_USER=admin
 export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token)
 
 # Get VM UUID
@@ -576,6 +579,7 @@ echo "Image:    $IMAGE_VERSION"
 echo ""
 
 cd ~/dstack/dstack/vmm
+export DSTACK_VMM_AUTH_USER=admin
 export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token)
 
 # --- Step 1: Get VM UUID ---
diff --git a/docs/tutorials/gateway-build-configuration.md b/docs/tutorials/gateway-build-configuration.md
@@ -460,6 +460,7 @@ Now generate the VMM deployment manifest:
 ```bash
 cd ~/dstack/dstack/vmm
 
+export DSTACK_VMM_AUTH_USER=admin
 export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token)
 
 ./src/vmm-cli.py --url http://127.0.0.1:9080 compose \
diff --git a/docs/tutorials/gateway-service-setup.md b/docs/tutorials/gateway-service-setup.md
@@ -85,6 +85,7 @@ cd ~/gateway-deploy
 set -a; source .env; set +a
 
 cd ~/dstack/dstack/vmm
+export DSTACK_VMM_AUTH_USER=admin
 export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token)
 
 ./src/vmm-cli.py --url http://127.0.0.1:9080 deploy \
@@ -408,6 +409,7 @@ Navigate to the VMM directory first:
 
 ```bash
 cd ~/dstack/dstack/vmm
+export DSTACK_VMM_AUTH_USER=admin
 export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token)
 ```
 
diff --git a/docs/tutorials/hello-world-app.md b/docs/tutorials/hello-world-app.md
@@ -184,6 +184,7 @@ Use `vmm-cli.py compose` to generate the encrypted deployment manifest. The `--g
 
 ```bash
 cd ~/dstack/dstack/vmm
+export DSTACK_VMM_AUTH_USER=admin
 export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token)
 
 ./src/vmm-cli.py --url http://127.0.0.1:9080 compose \
@@ -265,6 +266,7 @@ ssh user@your-server
 
 ```bash
 cd ~/dstack/dstack/vmm
+export DSTACK_VMM_AUTH_USER=admin
 export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token)
 
 SRV_DOMAIN=$(grep ^SRV_DOMAIN ~/gateway-deploy/.env | cut -d= -f2)
@@ -373,6 +375,7 @@ Navigate to the VMM directory:
 
 ```bash
 cd ~/dstack/dstack/vmm
+export DSTACK_VMM_AUTH_USER=admin
 export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token)
 ```
 
diff --git a/docs/tutorials/kms-cvm-deployment.md b/docs/tutorials/kms-cvm-deployment.md
@@ -256,6 +256,7 @@ Use the VMM CLI tool to deploy the CVM:
 cd ~/dstack/dstack/vmm
 
 # Set VMM auth from saved token
+export DSTACK_VMM_AUTH_USER=admin
 export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token)
 
 # Generate app-compose.json with local key provider enabled
@@ -306,7 +307,7 @@ curl -s -H "Authorization: Bearer $(cat ~/.dstack/secrets/vmm-auth-token)" \
   "http://127.0.0.1:9080/logs?id=VM_ID&follow=true&ansi=false"
 ```
 
-> **Note:** The VMM logs endpoint requires Bearer token authentication. The `vmm-cli.py logs` command may not work with token auth — use curl directly as shown above.
+> **Note:** The VMM logs endpoint requires authentication. `vmm-cli.py logs` sends credentials automatically when `DSTACK_VMM_TOKEN` (or `--token`) is set, so it works against an auth-enabled VMM; the curl form above is an equivalent alternative.
 
 Look for these log messages indicating KMS entered onboard mode:
 ```
diff --git a/docs/tutorials/troubleshooting-first-application.md b/docs/tutorials/troubleshooting-first-application.md
@@ -84,6 +84,7 @@ If `/guest/Info` returns empty or errors, check that the CVM is running:
 
 ```bash
 cd ~/dstack/dstack/vmm
+export DSTACK_VMM_AUTH_USER=admin
 export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token)
 ./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm
 ```
diff --git a/docs/tutorials/troubleshooting-gateway-deployment.md b/docs/tutorials/troubleshooting-gateway-deployment.md
@@ -59,6 +59,7 @@ sudo systemctl restart dstack-vmm
 
 **"Authentication required"** — Set the auth token:
 ```bash
+export DSTACK_VMM_AUTH_USER=admin
 export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token)
 ```
 
@@ -72,6 +73,7 @@ export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token)
 
 ```bash
 cd ~/dstack/dstack/vmm
+export DSTACK_VMM_AUTH_USER=admin
 export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token)
 
 # Get KMS VM ID and remove it
diff --git a/docs/tutorials/troubleshooting-kms-deployment.md b/docs/tutorials/troubleshooting-kms-deployment.md
@@ -231,6 +231,7 @@ cat ~/kms-deployment/docker-compose.yml | grep ports -A2
 
 # Check CVM status via vmm-cli.py
 cd ~/dstack/dstack/vmm
+export DSTACK_VMM_AUTH_USER=admin
 export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token)
 ./src/vmm-cli.py --url http://127.0.0.1:9080 lsvm
 ```
diff --git a/docs/vmm-cli-user-guide.md b/docs/vmm-cli-user-guide.md
@@ -70,26 +70,43 @@ export DSTACK_VMM_URL=unix:/path/to/socket
 
 ### Authentication
 
-If your dstack-vmm server requires authentication, you can provide credentials using:
+When the dstack-vmm server has `[auth] enabled = true`, the token now guards the
+*entire* management surface (listing/creating/stopping VMs, deploys, logs, and
+the web UI) — not just the logs endpoint. Provide credentials in one of two
+forms.
 
-#### Environment Variables (Recommended)
+#### Bearer token (Recommended)
 
-```bash
-# Set authentication credentials
-export DSTACK_VMM_AUTH_USER=your-username
-export DSTACK_VMM_AUTH_PASSWORD=your-password
+Pass the VMM API token directly; it is sent as `Authorization: Bearer <token>`.
 
-# Then use CLI normally
+```bash
+# the token dstackup writes, or your configured `[auth] tokens` entry
+export DSTACK_VMM_TOKEN=$(cat ~/.dstack/secrets/vmm-auth-token)
 ./vmm-cli.py lsvm
+
+# or as a flag
+./vmm-cli.py --token "$DSTACK_VMM_TOKEN" lsvm
 ```
 
-#### Command Line Arguments
+#### HTTP Basic
+
+The server also accepts HTTP Basic, where the password may be the shared token
+(any username, e.g. `admin`) or an entry in the server's `htpasswd_file`.
 
 ```bash
-./vmm-cli.py --auth-user your-username --auth-password your-password lsvm
+export DSTACK_VMM_AUTH_USER=admin
+export DSTACK_VMM_AUTH_PASSWORD=$(cat ~/.dstack/secrets/vmm-auth-token)
+./vmm-cli.py lsvm
+
+# or as flags
+./vmm-cli.py --auth-user admin --auth-password "$DSTACK_VMM_AUTH_PASSWORD" lsvm
 ```
 
-**Note:** Environment variables take precedence over command line arguments for authentication.
+**Note:** A bearer token takes precedence over Basic when both are set.
+Environment variables take precedence over command line arguments. Setting only
+one half of a Basic credential (e.g. `DSTACK_VMM_AUTH_PASSWORD` without
+`DSTACK_VMM_AUTH_USER`) is rejected with an error rather than silently sending an
+unauthenticated request.
 
 
 ## Basic Commands
@@ -325,8 +342,7 @@ After successful deployment, verify your VM is running correctly:
 export DSTACK_VMM_URL=http://127.0.0.1:12000
 
 # If authentication is required
-export DSTACK_VMM_AUTH_USER=your-username
-export DSTACK_VMM_AUTH_PASSWORD=your-password
+export DSTACK_VMM_TOKEN=$(cat ~/.dstack/secrets/vmm-auth-token)
 
 # Create a basic docker-compose.yml
 cat > docker-compose.yml << 'EOF'
diff --git a/dstack/kms/README.md b/dstack/kms/README.md
@@ -30,6 +30,18 @@ CVMs running in dstack support three boot modes:
 - `key-provider` in RTMR: `{"type": "kms", "id": "<kms-root-pubkey>"}`
 - `app-id` is equal to the address of the deployed App Smart Contract.
 
+## Admin API authentication
+
+The KMS admin RPCs are guarded by `admin_token_hash` in `kms.toml`, which is the SHA-256 hex digest of the admin token. Generate a token and its hash with:
+
+```bash
+TOKEN=$(openssl rand -hex 32); echo -n "$TOKEN" | sha256sum
+```
+
+Put the resulting hash in `kms.toml` under `[core]` as `admin_token_hash`, and keep `$TOKEN` for clients. An empty hash denies all admin RPCs (fail-closed). Clients send the token via `Authorization: Bearer <token>`.
+
+This is separate from the on-chain / webhook authorization described below, which decides whether a CVM is allowed to boot and receive keys.
+
 ## KMS Implementation
 
 ### Components
