Skip to content

attestation.md

Latest commit

 

History

History
235 lines (190 loc) · 9.09 KB

attestation.md

File metadata and controls

235 lines (190 loc) · 9.09 KB
Raw

Attestation Admission

This example can gate Consul client-agent join and Consul Connect service registration on dstack TEE attestation.

Enable it in clusters/patroni-demo/terraform.tfvars:

enable_attestation_admission = true
kms                          = "phala"
# admission_expected_kms_public_key defaults to the public key for the
# default phala KMS backend. Override it when changing kms.

The Consul ACL management token is not a Terraform input. The coordinators share one app_id, so bootstrap-secrets derives it in-TEE via GetKey (identical on every coordinator) and writes it to /run/secrets/consul-management-token. It never appears in terraform.tfstate or on the deploy host.

The verifier image reference lives in the coordinator compose file. Use a versioned verifier tag and pin it by digest for production. Do not rely on dstacktee/dstack-verifier:latest; the current release workflow publishes version tags and does not move latest. The verifier must support the GetQuote evidence shape used by current dstack guest agents: quote, event_log, and vm_config.

Trust Flow

  1. Terraform calls Phala preflight for the same app definitions it deploys and reads each expected compose_hash.
  2. Terraform renders ADMISSION_POLICY_JSON for coordinators. Policy rows are keyed by workload identity and optional peer_id; compose_hash is revision evidence, not the workload key.
  3. A worker starts mesh-conn, then requests a nonce from a coordinator admission broker.
  4. The worker builds a binding statement containing the claimed identity, peer id, dstack app id, instance id, and compose hash.
  5. The worker calls dstack GetQuote(SHA-256(binding || nonce)) and sends {identity, binding, nonce, quote, event_log, vm_config} to the broker.
  6. The broker verifies the quote through dstack-verifier, checks report-data binding, nonce freshness, and KMS key-provider identity, then matches identity, peer id, and compose hash against the Terraform policy.
  7. On success the broker mints a scoped Consul ACL token. Worker agents receive a node-identity token; service workloads receive service-identity tokens plus any declared Consul permissions.

Report Data

report_data is the 64-byte TDX REPORTDATA field. The structured admission data stays outside the quote in the binding statement. The first 32 bytes are a SHA-256 digest; the remaining 32 bytes are zero padding:

report_data = SHA-256(binding_statement_json || broker_nonce) || 32 zero bytes

The broker nonce is 32 random bytes encoded as hex. The binding statement is JSON generated by the workload sidecar:

{
  "version": 1,
  "identity": "spiffe://demo/postgres",
  "cluster": "demo",
  "peer_id": "worker-3",
  "app_id": "<dstack app id>",
  "instance_id": "<dstack instance id>",
  "compose_hash": "<dstack compose hash>",
  "issued_at": "2026-05-18T00:00:00Z"
}

The client checks that dstack GetQuote() returns the same report_data it requested. The broker independently recomputes the digest from the submitted binding statement and nonce, then compares that digest with the verifier-returned report_data.

Peer IDs

peer_id is the mesh node label, not a secret and not a hardware identity. In this example it is derived from the local role and ordinal, such as worker-3, and reused as the Consul node name.

The current binding is:

peer_id -> binding statement -> SHA-256 report_data -> TDX quote

The broker also requires the peer_id to match the Terraform policy row before it issues a node token or a service token scoped to that node. This prevents one attested workload row from directly claiming another row's Consul node identity.

For stronger production binding, the policy should also pin a platform-stable workload instance field, such as a Phala-provided instance id or app id when that can be supplied without introducing a deployment cycle. Until then, peer_id is best understood as an authorization namespace declared by Terraform and quoted by the CVM.

Evidence Policy

The broker treats dstack-verifier as the authority for quote and event-log semantics. The broker should require verifier outputs such as:

  • quote_verified == true
  • event_log_verified == true
  • report_data present and equal to the broker-computed digest
  • app_info.compose_hash present and policy-allowed
  • app_info.key_provider_info present and equal to the policy-pinned key provider
  • when production_profile = true, os_image_hash_verified == true and os_image_hash equals the Terraform preflight hash

Key lifecycle should be policy-controlled. It answers three questions: which KMS backend originates the key material, which app identity and path scope the GetKey() output, and which CVMs can derive the same role-local secret.

This example pins the long-term KMS public key in Terraform as admission_expected_kms_public_key and checks it against the verifier-returned app_info.key_provider_info. The verifier field is a hex-encoded JSON payload:

{
  "name": "kms",
  "id": "<public-key-hex>"
}

The pinned key must match the kms backend used in the Phala launch configuration. The Terraform default matches the example's default kms = "phala" backend; override it when changing KMS backend.

bootstrap-secrets derives role-local secrets inside the CVM with dstack GetKey(). The output is scoped by KMS backend, app identity, derivation path, and cluster name. Replicas in one phala_app share one app_id, so they can derive the same role-local secret. Different workload groups have different app_id values, so GetKey() does not produce the same value across roles. This is why the Consul management token and Patroni passwords can be derived in-TEE, while the current cross-role gossip key is still a Terraform-generated workaround.

For production profiles, OS image identity should also be checked. This example makes that an explicit profile: set production_profile = true and an os_image name such as dstack-0.5.7. Terraform uses that image in the Phala launch config, reads the resulting os_image_hash from preflight, and emits that SHA-256 hash into the admission policy. The broker then requires dstack-verifier to prove the same OS image hash before issuing Consul ACL tokens.

Runtime Checks

From a coordinator CVM, inspect Consul membership:

docker exec dstack-sidecar-1 sh -lc \
  'CONSUL_HTTP_ADDR=http://127.0.0.1:8500 consul members -token "$(cat /run/secrets/consul-management-token)"'

Check Raft health:

docker exec dstack-sidecar-1 sh -lc \
  'CONSUL_HTTP_ADDR=http://127.0.0.1:8500 consul operator raft list-peers -token "$(cat /run/secrets/consul-management-token)"'

Look for successful node and service admission in worker sidecar logs:

phala logs --cvm-id <worker-app-id> dstack-sidecar-1 -n 700 |
  rg 'admission-client-.*admission accepted'

Look for hard attestation failures:

phala logs --cvm-id <worker-app-id> dstack-sidecar-1 -n 700 |
  rg 'QUOTE_INVALID|VERIFIER_FAILED|REPORT_DATA_MISMATCH|ADMISSION_REJECTED'

No output from the failure grep is expected.

Production Phala OS images disable SSH access. For prod-profile runs, use Phala instance metadata, phala ps, and public logs to validate OS hash, container health, and admission success. Use a dev image only when the test specifically requires in-CVM consul or patronictl commands.

Current Limits

  • The low-level mesh rendezvous path still uses TURN HMAC admission. Attestation admission gates Consul/Connect identity, not rendezvous.
  • Broker-issued Consul tokens do not expire by default. Setting ADMISSION_TOKEN_TTL requires a renewal path for the Consul agent, Patroni DCS token, and Envoy xDS token.
  • The verifier downloads OS-image inputs and computes reference measurements on cache miss. Coordinators mount /tmp/dstack-runtime/verifier-cache into the verifier so image and measurement caches survive verifier container restarts on the same CVM.
  • The measured compose_hash comes from Phala's app-compose.json. Image binding should use digest-pinned image references in the Compose source. Environment variable values are not a sufficient image-binding policy.
  • Dynamic upgrade policy is future work. Phase 1 uses the static Terraform-generated allowlist; a future policy server can admit new compose hashes under a signed policy epoch.

Verification Cost

Measured on 2026-05-18 with a live tdx.small worker quote and dstacktee/dstack-verifier:0.5.9:

Mode Time
Oneshot, empty verifier cache 2.1s
Oneshot, image and measurement cache already populated 1.5s
Long-running HTTP server, warm cache 1.1-1.2s

The cold run downloaded a 20 MB OS-image bundle and computed the reference measurements. Larger OS images or slow network paths will increase the first request cost; the verifier download timeout is 300s by default. After that, the verifier reuses the cached image and cached measurements for the same VM config. Quote verification still costs about one second, so the broker should treat verification as a startup/admission cost, not something to repeat for every leaf-cert rotation.