Merge remote-tracking branch 'origin/master' into feat/shared-api-auth

# Conflicts:
#	dstack/vmm/src/main.rs

Author: kvinwang

CONTRIBUTING.md

@@ -62,6 +62,12 @@ dstack deploy ./docker-compose.yml \
   --no-tee
 ```
 
+To exercise persistent TPM-backed app keys as well, install `swtpm` on the VMM
+host and select `key_provider=tpm` in the VMM console (or pass `--key-provider
+tpm` in tooling that exposes the compose option). The VMM keeps the software
+TPM state in the VM work directory so that seal/unseal survives guest restarts.
+The software TPM is host-controlled and does not provide hardware isolation.
+
 The simulator package is installed only in development images. This mode
 provides no hardware isolation and must never be used with production
 workloads or secrets. Real quote generation, hardware isolation, and KMS

docs/security/cvm-boundaries.md

@@ -44,7 +44,7 @@ This is the main configuration file for the application in JSON format:
 | init_script | 0.5.5 | string | Bash script that executed prior to dockerd startup |
 | storage_fs | 0.5.5 | string | Filesystem type for the data disk of the CVM. Supported values: "zfs", "ext4". default to "zfs". **ZFS:** Ensures filesystem integrity with built-in data protection features. **ext4:** Provides better performance for database applications with lower overhead and faster I/O operations, but no strong integrity protection. |
 | swap_size | 0.5.5 | string/integer | The linux swap size. default to 0. Can be in byte or human-readable format (e.g., "1G", "256M"). |
-| key_provider | 0.5.6 | string | Key provider type. Supported values: "none", "kms", "local", "tpm". `"tpm"` is only supported on platforms whose TPM is part of the platform trust model (GCP vTPM, AWS EC2 NitroTPM); on other platforms guest setup fails closed, since sealing to a host-provided software TPM (e.g. swtpm under bare QEMU) offers no protection against the host. |
+| key_provider | 0.5.6 | string | Key provider type. Supported values: "none", "kms", "local", "tpm". GCP vTPM and AWS EC2 NitroTPM are part of their platform trust models. The Dstack platform can use VMM-managed swtpm for seal/unseal and restart persistence, but it offers no protection against the host and is intentionally not accepted by remote verifiers. |
 
 
 The hash of this file content is extended as the dstack `compose-hash` launch event. On TDX-family platforms the launch event is measured into RTMR3. On AWS NitroTPM it is measured into non-resettable SHA384 PCR14 before the `system-ready` launch boundary. Remote verifiers extract and replay this event during attestation.
@@ -74,6 +74,7 @@ This file contains system configuration in JSON format:
 | kms_urls | array of string | List of KMS service URLs |
 | gateway_urls | array of string | List of gateway service URLs |
 | pccs_url | string | URL of the PCCS service (used when dstack components need to verify a remote TD CVM or SGX enclave) |
+| nvidia_attestation_proxy_url | string | Optional persistent OCSP and RIM cache used by NVIDIA local GPU attestation |
 | docker_registry | string | URL of the docker registry |
 | host_api_url | string | VSOCK URL of host API |
 | vm_config | string | JSON string of VM configuration (os_image_hash, cpu_count, memory_size) |
@@ -85,6 +86,7 @@ The hash of this file is not extended to any RTMR because each field has its own
 | kms_urls | URLs themselves aren't security-critical. The trust anchor is the KMS root public key, which is extended as the `key-provider` launch event. On TDX-family platforms this is RTMR3; on AWS NitroTPM this is PCR14. Keys obtained from KMS will either successfully decrypt/encrypt the disk or fail-and-abort. |
 | gateway_urls | URLs aren't security-critical. Trust is established through CA certificates from KMS. App CVM and dstack-gateway CVM verify each other's CA certificates to ensure they're under the same KMS authority. |
 | pccs_url | URL isn't security-critical. Trust is anchored by the root public key pinned in the attestation verification program. |
+| nvidia_attestation_proxy_url | The URL is not a collateral trust anchor. The measured guest verifies NVIDIA signatures and the signed OCSP validity window, and continues to require a fresh GPU evidence nonce. A bad endpoint can withhold collateral and cause a denial of service, but cannot forge a successful attestation or replay an expired `good` response. |
 | docker_registry | Docker daemon verifies image integrity using the pinned image hashes in the docker-compose file. |
 | host_api_url | Used only for reporting or encrypted sealing key transport. An incorrect URL doesn't create security vulnerabilities. |
 | vm_config | Informs the CVM to report virtual hardware info to KMS when requesting keys. KMS uses this info to calculate expected RTMRs and verify image hash. If tampered with, image hash verification would fail and no keys would be distributed. |

docs/tutorials/vmm-configuration.md

@@ -105,6 +105,8 @@ qemu_path = ""
 kms_urls = ["http://127.0.0.1:8081"]
 gateway_urls = ["http://127.0.0.1:8082"]
 pccs_url = "https://pccs.phala.network/sgx/certification/v4"
+# Optional. See "NVIDIA GPU attestation cache" below.
+# nvidia_attestation_proxy_url = "http://10.0.2.2:8090"
 docker_registry = ""
 cid_start = 1000
 cid_pool_size = 1000
@@ -212,6 +214,24 @@ Verify QGS is running:
 systemctl status qgsd
 ```
 
+### Optional: NVIDIA GPU attestation cache
+
+GPU images perform local NVIDIA attestation before app keys are provisioned.
+Without a cache this contacts NVIDIA's OCSP and RIM services during every cold
+boot. A fleet can run the persistent
+[`dstack-nvidia-attest-proxy`](../../dstack/nvidia-attest-proxy/README.md) and pass its
+URL to guests through sys-config:
+
+```toml
+[cvm]
+nvidia_attestation_proxy_url = "http://10.0.2.2:8090"
+```
+
+The example address is the host as seen from QEMU user-mode networking. The
+proxy must be reachable during `dstack-prepare`. It stores only NVIDIA-signed
+collateral and never becomes a signing trust anchor. OCSP entries are not
+served after their signed validity window.
+
 ### Step 7: Create Runtime Directories
 
 ```bash

dstack/Cargo.lock

@@ -59,6 +59,7 @@ members = [
     "serde-duration",
     "dstack-mr",
     "dstack-mr/cli",
+    "nvidia-attest-proxy",
     "verifier",
     "size-parser",
     "crates/dstack-cli-core",

dstack/Cargo.toml

@@ -710,6 +710,10 @@ pub struct SysConfig {
     #[serde(default, alias = "tproxy_urls")]
     pub gateway_urls: Vec<String>,
     pub pccs_url: Option<String>,
+    /// Optional NVIDIA attestation collateral proxy. When present, nvattest
+    /// fetches both OCSP responses and RIM documents through this endpoint.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub nvidia_attestation_proxy_url: Option<String>,
     pub docker_registry: Option<String>,
     pub host_api_url: Option<String>,
     /// MrConfigV3 document string for platform app/config binding.

dstack/dstack-types/src/lib.rs

@@ -29,6 +29,7 @@ sha2.workspace = true
 tokio = { workspace = true, features = ["full"] }
 tracing.workspace = true
 tracing-subscriber.workspace = true
+url.workspace = true
 x25519-dalek.workspace = true
 
 dstack-kms-rpc.workspace = true

dstack/dstack-util/Cargo.toml

@@ -1081,6 +1081,7 @@ mod gpu {
     const ATTESTATION_TIMEOUT: Duration = Duration::from_secs(300);
     const EVENT_VERSION: u32 = 2;
     const POLICY_ENTRYPOINT: &str = "data.policy.nv_match";
+    const TRUST_OUTPOST_POLICY: &str = "/usr/share/nvattest/policies/allow_trust_outpost_ocsp.rego";
     /// Bound Rego evaluation so a runaway application policy cannot hang boot.
     const POLICY_TIMEOUT: Duration = Duration::from_secs(10);
 
@@ -1368,11 +1369,61 @@ mod gpu {
         serde_json::to_vec(&event).context("failed to serialize GPU attestation event")
     }
 
-    /// Run local GPU attestation via nvattest with a fresh nonce and no custom
-    /// relying-party policy. nvattest's built-in appraisal still applies. The
-    /// complete JSON output is preserved under /run and retained with its
-    /// claims for the application policy and versioned RTMR3 summary.
-    pub(super) async fn attest_gpu(expected_devices: u32) -> Result<GpuAttestationResult> {
+    fn normalize_proxy_url(proxy_url: Option<&str>) -> Result<Option<String>> {
+        let Some(proxy_url) = proxy_url.map(str::trim).filter(|url| !url.is_empty()) else {
+            return Ok(None);
+        };
+        let parsed = url::Url::parse(proxy_url).context("invalid NVIDIA attestation proxy URL")?;
+        if !matches!(parsed.scheme(), "http" | "https") || parsed.host_str().is_none() {
+            bail!("NVIDIA attestation proxy must be an absolute HTTP(S) URL");
+        }
+        if parsed.query().is_some()
+            || parsed.fragment().is_some()
+            || !parsed.username().is_empty()
+            || parsed.password().is_some()
+            || parsed.path() != "/"
+        {
+            bail!(
+                "NVIDIA attestation proxy URL must not contain credentials, path, query, or fragment"
+            );
+        }
+        Ok(Some(parsed.as_str().trim_end_matches('/').to_string()))
+    }
+
+    fn nvattest_args(nonce: &str, proxy_url: Option<&str>) -> Result<Vec<String>> {
+        let mut args = vec![
+            "attest".to_string(),
+            "--device".to_string(),
+            "gpu".to_string(),
+            "--verifier".to_string(),
+            "local".to_string(),
+            "--nonce".to_string(),
+            nonce.to_string(),
+            "--format".to_string(),
+            "json".to_string(),
+        ];
+        if let Some(proxy_url) = normalize_proxy_url(proxy_url)? {
+            args.extend([
+                "--ocsp-url".to_string(),
+                format!("{proxy_url}/ocsp"),
+                "--rim-url".to_string(),
+                proxy_url,
+                "--relying-party-policy".to_string(),
+                TRUST_OUTPOST_POLICY.to_string(),
+            ]);
+        }
+        Ok(args)
+    }
+
+    /// Run local GPU attestation via nvattest with a fresh evidence nonce. If
+    /// sys-config selects a collateral proxy, both RIM and OCSP traffic is
+    /// routed through it and NVIDIA's Trust Outpost policy accepts cached OCSP
+    /// responses whose responder nonce no longer matches. The independent GPU
+    /// evidence nonce remains mandatory and is checked below.
+    pub(super) async fn attest_gpu(
+        expected_devices: u32,
+        proxy_url: Option<&str>,
+    ) -> Result<GpuAttestationResult> {
         if !Path::new(NVATTEST).exists() {
             bail!("nvattest is not available in this image");
         }
@@ -1382,22 +1433,14 @@ mod gpu {
             warn!("failed to step system clock: {err:?}");
         }
         let nonce = hex::encode(rand::thread_rng().gen::<[u8; 32]>());
-        let output = run_command(
-            NVATTEST,
-            &[
-                "attest",
-                "--device",
-                "gpu",
-                "--verifier",
-                "local",
-                "--nonce",
-                &nonce,
-                "--format",
-                "json",
-            ],
-            ATTESTATION_TIMEOUT,
-        )
-        .await?;
+        let args = nvattest_args(&nonce, proxy_url)?;
+        if args.iter().any(|arg| arg == "--relying-party-policy")
+            && !Path::new(TRUST_OUTPOST_POLICY).is_file()
+        {
+            bail!("NVIDIA attestation proxy is configured but {TRUST_OUTPOST_POLICY} is missing");
+        }
+        let args = args.iter().map(String::as_str).collect::<Vec<_>>();
+        let output = run_command(NVATTEST, &args, ATTESTATION_TIMEOUT).await?;
         if !output.stderr.is_empty() {
             info!("nvattest: {}", truncated_lossy(&output.stderr, 2048));
         }
@@ -1556,6 +1599,35 @@ mod gpu {
             assert_eq!(nvidia_gpu_count(nvidia).unwrap(), 2);
         }
 
+        #[test]
+        fn proxy_routes_ocsp_and_rim_and_selects_outpost_policy() {
+            let nonce = format!("test-nonce-{}", std::process::id());
+            let args = nvattest_args(&nonce, Some("http://10.0.2.2:8090/")).unwrap();
+            assert!(args
+                .windows(2)
+                .any(|args| args == ["--ocsp-url", "http://10.0.2.2:8090/ocsp"]));
+            assert!(args
+                .windows(2)
+                .any(|args| args == ["--rim-url", "http://10.0.2.2:8090"]));
+            assert!(args
+                .windows(2)
+                .any(|args| args == ["--relying-party-policy", TRUST_OUTPOST_POLICY]));
+
+            let direct = nvattest_args(&nonce, None).unwrap();
+            assert!(!direct.iter().any(|arg| arg == "--ocsp-url"));
+            assert!(!direct.iter().any(|arg| arg == "--relying-party-policy"));
+        }
+
+        #[test]
+        fn proxy_url_validation_is_fail_closed() {
+            let nonce = format!("test-nonce-{}", std::process::id());
+            assert!(nvattest_args(&nonce, Some("file:///tmp/proxy")).is_err());
+            assert!(nvattest_args(&nonce, Some("https://user@example.com")).is_err());
+            assert!(nvattest_args(&nonce, Some("https://example.com?q=1")).is_err());
+            assert!(nvattest_args(&nonce, Some("https://example.com/base")).is_err());
+            assert!(normalize_proxy_url(Some("  ")).unwrap().is_none());
+        }
+
         #[test]
         fn basic_policy_requires_cc_and_rejects_devtools_by_default() {
             let nonce = "44".repeat(32);
@@ -1874,7 +1946,14 @@ impl Stage0<'_> {
         }
         self.vmm.notify_q("boot.progress", "attesting GPU").await;
         info!("verifying GPU TEE attestation");
-        let attestation = gpu::attest_gpu(expected_devices).await?;
+        let attestation = gpu::attest_gpu(
+            expected_devices,
+            self.shared
+                .sys_config
+                .nvidia_attestation_proxy_url
+                .as_deref(),
+        )
+        .await?;
 
         let gpu_state = gpu::query_gpu_state(expected_devices)?;
         attestation

dstack/dstack-util/src/system_setup.rs

@@ -0,0 +1,36 @@
+# SPDX-FileCopyrightText: © 2026 Phala Network <dstack@phala.network>
+#
+# SPDX-License-Identifier: Apache-2.0
+
+[package]
+name = "dstack-nvidia-attest-proxy"
+version.workspace = true
+authors.workspace = true
+edition.workspace = true
+license.workspace = true
+
+[dependencies]
+anyhow.workspace = true
+base64.workspace = true
+bytes.workspace = true
+futures.workspace = true
+chrono.workspace = true
+clap = { workspace = true, features = ["env"] }
+dashmap.workspace = true
+hex = { workspace = true, features = ["alloc"] }
+http.workspace = true
+http-body-util.workspace = true
+humantime.workspace = true
+hyper = { workspace = true, features = ["server"] }
+hyper-util = { workspace = true, features = ["tokio"] }
+reqwest.workspace = true
+serde.workspace = true
+serde_json.workspace = true
+sha2.workspace = true
+tokio = { workspace = true, features = ["full"] }
+tracing.workspace = true
+tracing-subscriber.workspace = true
+url.workspace = true
+
+[dev-dependencies]
+tempfile.workspace = true