feat(guest): structure GPU policy settings

Author: kvinwang

docs/security/security-model.md

@@ -57,9 +57,14 @@ dstack supports NVIDIA H100, H200, and B200 GPUs in confidential compute mode fo
 
 GPUs are passed through via VFIO to the TEE-protected CVM. Before key provisioning, `dstack-util setup` inventories every VGA/3D-controller PCI function, compares that count with `vm_config.num_gpus`, and runs NVIDIA's local `nvattest` verifier over every NVIDIA-driver-visible GPU with a fresh nonce. dstack does not ship or pass a custom relying-party policy to this command; nvattest's built-in appraisal still applies. The complete JSON result (`result_code`, `result_message`, `claims`, and `detached_eat`) is saved at `/run/nvidia-gpu-attestation/attestation.out`.
 
-An application may additionally set `requirements.gpu_policy` to a Rego v0 script. Its input is the nvattest output's `claims` array, and it must define the boolean entrypoint `data.policy.nv_match` in `package policy`. Immediately after measuring `compose-hash`, dstack measures `sha256(gpu_policy UTF-8 bytes)` in a `gpu-policy` event and then evaluates the policy. A false, undefined, malformed, or non-boolean result stops boot before key provisioning. If no GPU is configured, the policy receives an empty array. `gpu_policy` cannot be combined with `attest_gpu: false`.
+An application may additionally set `requirements.gpu_policy` to an object with the following deny-unknown-fields schema:
 
-Separate fail-closed `nvidia-smi` queries require the current CC feature to be ON and DevTools mode to be OFF. Only after the default appraisal, optional application policy, and these checks succeed does dstack set the GPU ready state. The dstack CPU/guest boot chain is verified independently through measured boot; a GPU claim named `secboot` refers to the GPU appraisal, not UEFI Secure Boot in the CVM.
+- `rego` (optional string): a Rego v0 script evaluated with the nvattest output's `claims` array as `input`. It must define the boolean entrypoint `data.policy.nv_match` in `package policy`.
+- `allow_devtools` (boolean, default `false`): permit NVIDIA DevTools mode. Production applications should leave this disabled because DevTools removes the expected GPU memory-confidentiality guarantee.
+
+Immediately after measuring `compose-hash`, dstack applies field defaults, JCS-canonicalizes the complete `gpu_policy` structure, and measures its SHA-256 digest in a `gpu-policy` event. It then applies the basic settings and optional Rego policy. A false, undefined, malformed, or non-boolean Rego result stops boot before key provisioning. If no GPU is configured, Rego receives an empty array. `gpu_policy` cannot be combined with `attest_gpu: false`.
+
+Separate fail-closed `nvidia-smi` queries always require the current CC feature to be ON and require DevTools mode to be OFF unless the measured policy explicitly permits it. Only after the default appraisal, optional application policy, and these checks succeed does dstack set the GPU ready state. The dstack CPU/guest boot chain is verified independently through measured boot; a GPU claim named `secboot` refers to the GPU appraisal, not UEFI Secure Boot in the CVM.
 
 ### Dual Attestation
 
@@ -71,7 +76,7 @@ A verifier must replay the measured event log, require exactly one pre-`system-r
 
 The GPU gate assumes a malicious host/VMM and untrusted host-provided PCI topology, while trusting the CPU TEE, the measured dstack guest/kernel, NVIDIA hardware/firmware roots, and the cryptography used by both attestation chains. Availability is out of scope.
 
-The events make the following **boot-time** statement: immediately before key provisioning, all attached VGA/3D PCI functions were NVIDIA devices, their count matched the quote-bound VM configuration, nvattest returned one fresh successfully appraised claim for each device, the measured application policy accepted those claims when present, CC was ON, DevTools was OFF, and setting the GPU ready state succeeded. This closes these cases:
+The events make the following **boot-time** statement: immediately before key provisioning, all attached VGA/3D PCI functions were NVIDIA devices, their count matched the quote-bound VM configuration, nvattest returned one fresh successfully appraised claim for each device, the measured application policy accepted those claims and GPU state when present, CC was ON, DevTools complied with that policy (OFF by default), and setting the GPU ready state succeeded. This closes these cases:
 
 - A GPU-less launch cannot be presented as a GPU-verified launch: it has `num_gpus == 0` and no `gpu-attestation` event.
 - A mixed launch cannot attest only its TEE-capable subset. Non-NVIDIA display GPUs are rejected, and the sysfs, `vm_config`, and nvattest claim counts must all agree. A non-CC NVIDIA GPU either prevents evidence collection/appraisal or causes the default appraisal, application policy, or CC-state check to fail.
@@ -180,7 +185,7 @@ Use this checklist to verify a workload running in a dstack CVM.
 - [ ] `vm_config.num_gpus` is greater than zero and matches the expected deployment
 - [ ] If `requirements.gpu_policy` is required, exactly one `gpu-policy` event follows `compose-hash`, and its payload matches the expected SHA-256 digest
 - [ ] Exactly one `gpu-attestation` event appears before `system-ready`
-- [ ] The event device count matches `vm_config.num_gpus`, and its CC/DevTools fields indicate production mode
+- [ ] The event device count matches `vm_config.num_gpus`, CC is ON, and its DevTools field complies with the measured policy
 - [ ] The platform binds the event log to a quoted RTMR/PCR (do not accept it from current SEV-SNP evidence)
 
 **Key management verification:**

dstack/Cargo.lock

@@ -121,6 +121,19 @@ pub struct AppCompose {
     pub requirements: Option<Requirements>,
 }
 
+#[derive(Deserialize, Serialize, Debug, Clone, Default, PartialEq, Eq)]
+#[serde(default, deny_unknown_fields)]
+pub struct GpuPolicy {
+    /// Optional Rego v0 policy evaluated against NVIDIA nvattest's `claims`
+    /// array. It must define the boolean rule `data.policy.nv_match`.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub rego: Option<String>,
+    /// Permit NVIDIA DevTools mode. This defaults to false because DevTools
+    /// disables the GPU memory-confidentiality guarantees expected in
+    /// production.
+    pub allow_devtools: bool,
+}
+
 #[derive(Deserialize, Serialize, Debug, Clone, Default, PartialEq, Eq)]
 #[serde(default, deny_unknown_fields)]
 pub struct Requirements {
@@ -159,15 +172,12 @@ pub struct Requirements {
     /// are unaffected either way.
     #[serde(skip_serializing_if = "Option::is_none")]
     pub attest_gpu: Option<bool>,
-    /// Optional application-supplied Rego policy evaluated against the
-    /// `claims` array in NVIDIA nvattest's JSON output before key
-    /// provisioning. The policy must define the boolean rule
-    /// `data.policy.nv_match`.
+    /// Optional application GPU policy applied before key provisioning.
     ///
-    /// The SHA-256 digest of the exact UTF-8 policy bytes is emitted as the
-    /// `gpu-policy` launch event immediately after `compose-hash`.
+    /// Its default-expanded, JCS-canonicalized JSON SHA-256 digest is emitted
+    /// as the `gpu-policy` launch event immediately after `compose-hash`.
     #[serde(skip_serializing_if = "Option::is_none")]
-    pub gpu_policy: Option<String>,
+    pub gpu_policy: Option<GpuPolicy>,
 }
 
 impl Requirements {
@@ -466,7 +476,10 @@ mod app_compose_tests {
                 "platforms": ["dstack-gcp-tdx", "dstack-tdx"],
                 "tdx_measure_acpi_tables": true,
                 "launch_token_hash": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08",
-                "gpu_policy": "package policy\n\ndefault nv_match = false\n"
+                "gpu_policy": {
+                    "rego": "package policy\n\ndefault nv_match = false\n",
+                    "allow_devtools": true
+                }
             }
         }))
         .unwrap();
@@ -481,10 +494,12 @@ mod app_compose_tests {
             requirements.launch_token_hash.as_deref(),
             Some("9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08")
         );
+        let gpu_policy = requirements.gpu_policy.as_ref().unwrap();
         assert_eq!(
-            requirements.gpu_policy.as_deref(),
+            gpu_policy.rego.as_deref(),
             Some("package policy\n\ndefault nv_match = false\n")
         );
+        assert!(gpu_policy.allow_devtools);
 
         let err = serde_json::from_value::<AppCompose>(serde_json::json!({
             "manifest_version": "3",
@@ -555,13 +570,31 @@ mod app_compose_tests {
             "name": "test",
             "runner": "docker-compose",
             "requirements": {
-                "gpu_policy": "package policy\n\ndefault nv_match = false\n"
+                "gpu_policy": {
+                    "rego": "package policy\n\ndefault nv_match = false\n"
+                }
             }
         }))
         .unwrap();
         let requirements = gpu_policy.requirements.as_ref().unwrap();
-        assert!(requirements.gpu_policy.is_some());
+        let gpu_policy = requirements.gpu_policy.as_ref().unwrap();
+        assert!(gpu_policy.rego.is_some());
+        assert!(!gpu_policy.allow_devtools);
         assert!(!requirements.is_empty());
+
+        let err = serde_json::from_value::<AppCompose>(serde_json::json!({
+            "manifest_version": "3",
+            "name": "test",
+            "runner": "docker-compose",
+            "requirements": {
+                "gpu_policy": {
+                    "rego": "package policy",
+                    "allow_debug": true
+                }
+            }
+        }))
+        .unwrap_err();
+        assert!(err.to_string().contains("unknown field"));
     }
 
     #[test]

dstack/dstack-types/src/lib.rs

@@ -25,6 +25,7 @@ serde.workspace = true
 serde-human-bytes.workspace = true
 semver.workspace = true
 serde_json.workspace = true
+serde_jcs.workspace = true
 sha2.workspace = true
 tokio = { workspace = true, features = ["full"] }
 tracing.workspace = true

dstack/dstack-util/Cargo.toml

@@ -20,7 +20,7 @@ use dstack_types::{
         APP_COMPOSE, APP_KEYS, DECRYPTED_ENV, DECRYPTED_ENV_JSON, ENCRYPTED_ENV,
         HOST_SHARED_DIR_NAME, HOST_SHARED_DISK_LABEL, INSTANCE_INFO, SYS_CONFIG, USER_CONFIG,
     },
-    KeyProvider, KeyProviderInfo,
+    GpuPolicy, KeyProvider, KeyProviderInfo,
 };
 use fs_err as fs;
 use luks2::{
@@ -1114,16 +1114,17 @@ mod gpu {
 
     pub(super) struct GpuAttestationResult {
         claims: Vec<Value>,
-        event: Vec<u8>,
+        output: Vec<u8>,
+        devices: u32,
     }
 
     impl GpuAttestationResult {
         pub(super) fn claims(&self) -> &[Value] {
             &self.claims
         }
 
-        pub(super) fn event(&self) -> &[u8] {
-            &self.event
+        pub(super) fn event(&self, devtools: bool) -> Result<Vec<u8>> {
+            attestation_event(&self.output, self.devices, devtools)
         }
     }
 
@@ -1236,20 +1237,28 @@ mod gpu {
         parse_nvidia_smi_state(&output.stdout, name)
     }
 
-    /// Require the current system-wide GPU state to be production CC. NVAT's
-    /// NVML collector accepts protected-PCIe as well as CC and its default
-    /// appraisal does not reject DevTools, so a successful appraisal alone is
-    /// not a sufficient confidentiality check.
-    async fn verify_production_cc_mode() -> Result<()> {
-        if !query_nvidia_smi_state("-f", "CC feature state").await? {
+    fn validate_cc_state(cc_enabled: bool, devtools: bool, allow_devtools: bool) -> Result<()> {
+        if !cc_enabled {
             bail!("nvidia confidential compute mode is not enabled");
         }
-        if query_nvidia_smi_state("-d", "DevTools mode").await? {
+        if devtools && !allow_devtools {
             bail!("nvidia DevTools mode is enabled");
         }
         Ok(())
     }
 
+    /// Require the current system-wide GPU state to use CC. NVAT's NVML
+    /// collector accepts protected-PCIe as well as CC, so a successful
+    /// appraisal alone is not a sufficient confidentiality check. DevTools is
+    /// rejected by default but can be explicitly allowed by the measured app
+    /// policy.
+    pub(super) async fn verify_cc_mode(allow_devtools: bool) -> Result<bool> {
+        let cc_enabled = query_nvidia_smi_state("-f", "CC feature state").await?;
+        let devtools = query_nvidia_smi_state("-d", "DevTools mode").await?;
+        validate_cc_state(cc_enabled, devtools, allow_devtools)?;
+        Ok(devtools)
+    }
+
     /// Mark the GPU as ready to accept work. Only meaningful (and only
     /// succeeds) when the GPU runs in CC mode.
     pub(super) async fn set_gpu_ready_state() -> Result<()> {
@@ -1306,22 +1315,22 @@ mod gpu {
         Ok(output.claims)
     }
 
-    fn attestation_event(stdout: &[u8], devices: u32) -> Result<Vec<u8>> {
+    fn attestation_event(stdout: &[u8], devices: u32, devtools: bool) -> Result<Vec<u8>> {
         let event = GpuAttestationEvent {
             version: EVENT_VERSION,
             provider: "nvidia",
             devices,
             cc_mode: "on",
-            devtools: false,
+            devtools,
             evidence_sha256: hex::encode(sha256(stdout)),
         };
         serde_json::to_vec(&event).context("failed to serialize GPU attestation event")
     }
 
     /// Run local GPU attestation via nvattest with a fresh nonce and no custom
     /// relying-party policy. nvattest's built-in appraisal still applies. The
-    /// complete JSON output is preserved under /run, while its claims and a
-    /// versioned summary are returned for the application policy and RTMR3.
+    /// complete JSON output is preserved under /run and retained with its
+    /// claims for the application policy and versioned RTMR3 summary.
     pub(super) async fn attest_gpu(expected_devices: u32) -> Result<GpuAttestationResult> {
         if !Path::new(NVATTEST).exists() {
             bail!("nvattest is not available in this image");
@@ -1360,15 +1369,16 @@ mod gpu {
             );
         }
         let claims = validate_attestation_output(&output.stdout, &nonce, expected_devices)?;
-        verify_production_cc_mode().await?;
         Ok(GpuAttestationResult {
             claims,
-            event: attestation_event(&output.stdout, expected_devices)?,
+            output: output.stdout,
+            devices: expected_devices,
         })
     }
 
-    pub(super) fn policy_digest(policy: &str) -> [u8; 32] {
-        sha256(policy.as_bytes())
+    pub(super) fn policy_digest(policy: &GpuPolicy) -> Result<[u8; 32]> {
+        let canonical = serde_jcs::to_vec(policy).context("failed to canonicalize GPU policy")?;
+        Ok(sha256(&canonical))
     }
 
     /// Evaluate the app-provided Rego v0 policy using the same input shape as
@@ -1483,6 +1493,14 @@ mod gpu {
             assert!(parse_nvidia_smi_state(b"current: ON, pending: OFF\n", "CC").is_err());
         }
 
+        #[test]
+        fn basic_policy_requires_cc_and_rejects_devtools_by_default() {
+            validate_cc_state(true, false, false).unwrap();
+            assert!(validate_cc_state(false, false, false).is_err());
+            assert!(validate_cc_state(true, true, false).is_err());
+            validate_cc_state(true, true, true).unwrap();
+        }
+
         #[test]
         fn nvattest_output_requires_every_expected_gpu_and_fresh_nonce() {
             let nonce = "11".repeat(32);
@@ -1510,12 +1528,12 @@ mod gpu {
             let nonce = "22".repeat(32);
             let output = nvattest_output(&nonce, 1);
             let event: Value =
-                serde_json::from_slice(&attestation_event(&output, 1).unwrap()).unwrap();
+                serde_json::from_slice(&attestation_event(&output, 1, true).unwrap()).unwrap();
             assert_eq!(event["version"], EVENT_VERSION);
             assert_eq!(event["devices"], 1);
             assert!(event.get("policy").is_none());
             assert_eq!(event["cc_mode"], "on");
-            assert_eq!(event["devtools"], false);
+            assert_eq!(event["devtools"], true);
             assert_eq!(event["evidence_sha256"], hex::encode(sha256(&output)));
         }
 
@@ -1539,10 +1557,22 @@ mod gpu {
         }
 
         #[test]
-        fn policy_digest_uses_exact_utf8_bytes() {
-            let policy = "package policy\n\ndefault nv_match = true\n";
-            assert_eq!(policy_digest(policy), sha256(policy.as_bytes()));
-            assert_ne!(policy_digest(policy), policy_digest(policy.trim()));
+        fn policy_digest_commits_to_the_canonical_policy_structure() {
+            let policy = GpuPolicy {
+                rego: Some("package policy\n\ndefault nv_match = true\n".to_string()),
+                allow_devtools: true,
+            };
+            let canonical = serde_jcs::to_vec(&policy).unwrap();
+            assert_eq!(policy_digest(&policy).unwrap(), sha256(&canonical));
+
+            let production_policy = GpuPolicy {
+                allow_devtools: false,
+                ..policy.clone()
+            };
+            assert_ne!(
+                policy_digest(&policy).unwrap(),
+                policy_digest(&production_policy).unwrap()
+            );
         }
     }
 }
@@ -2240,27 +2270,34 @@ impl<'a> Stage0<'a> {
         emit_runtime_event("app-id", &instance_info.app_id)?;
         emit_runtime_event("compose-hash", &compose_hash)?;
 
-        if let Some(policy) = self
+        let gpu_policy = self
             .shared
             .app_compose
             .requirements
             .as_ref()
-            .and_then(|requirements| requirements.gpu_policy.as_deref())
-        {
-            emit_runtime_event("gpu-policy", &gpu::policy_digest(policy))?;
+            .and_then(|requirements| requirements.gpu_policy.as_ref());
+        if let Some(policy) = gpu_policy {
+            emit_runtime_event("gpu-policy", &gpu::policy_digest(policy)?)?;
             let claims = gpu_attestation
                 .map(gpu::GpuAttestationResult::claims)
                 .unwrap_or(&[]);
-            gpu::evaluate_policy(policy, claims).context("failed to apply GPU policy")?;
-            info!("application GPU policy accepted the attestation claims");
+            if let Some(rego) = policy.rego.as_deref() {
+                gpu::evaluate_policy(rego, claims).context("failed to apply GPU Rego policy")?;
+            }
         }
 
         if let Some(attestation) = gpu_attestation {
+            let allow_devtools = gpu_policy.is_some_and(|policy| policy.allow_devtools);
+            let devtools = gpu::verify_cc_mode(allow_devtools).await?;
             gpu::set_gpu_ready_state().await?;
-            emit_runtime_event("gpu-attestation", attestation.event())
+            let event = attestation.event(devtools)?;
+            emit_runtime_event("gpu-attestation", &event)
                 .context("failed to emit GPU attestation event")?;
             info!("GPU TEE attestation succeeded");
         }
+        if gpu_policy.is_some() {
+            info!("application GPU policy accepted the attestation claims and state");
+        }
 
         emit_runtime_event("instance-id", &instance_id)?;
         emit_runtime_event("boot-mr-done", &[])?;
@@ -2865,7 +2902,9 @@ fn test_gpu_policy_requires_gpu_attestation() {
         "runner": "docker-compose",
         "requirements": {
             "attest_gpu": false,
-            "gpu_policy": "package policy\n\ndefault nv_match = true\n"
+            "gpu_policy": {
+                "rego": "package policy\n\ndefault nv_match = true\n"
+            }
         }
     }))
     .unwrap();