Add attested Consul admission

Author: h4x3rotab

consul-postgres-ha/ATTESTATION.md

@@ -0,0 +1,111 @@
+# Attestation Admission
+
+This example can gate Consul client-agent join and Consul Connect
+service registration on dstack TEE attestation.
+
+Enable it in `cluster-example/terraform.tfvars`:
+
+```hcl
+enable_attestation_admission = true
+consul_management_token      = "<consul-acl-management-token>"
+dstack_verifier_image        = "dstacktee/dstack-verifier:0.5.9"
+```
+
+Use a versioned verifier tag and pin it by digest for production.
+Do not rely on `dstacktee/dstack-verifier:latest`; the current
+release workflow publishes version tags and does not move `latest`.
+The verifier must support the `GetQuote` evidence shape used by
+current dstack guest agents: `quote`, `event_log`, and `vm_config`.
+
+## Trust Flow
+
+1. Terraform calls Phala preflight for the same app definitions it
+   deploys and reads each expected `compose_hash`.
+2. Terraform renders `ADMISSION_POLICY_JSON` for coordinators. Policy
+   rows are keyed by workload identity and optional `peer_id`;
+   `compose_hash` is revision evidence, not the workload key.
+3. A worker starts `mesh-conn`, then requests a nonce from a
+   coordinator admission broker.
+4. The worker builds a binding statement containing the claimed
+   identity, peer id, dstack app id, instance id, and compose hash.
+5. The worker calls dstack `GetQuote(SHA-512(binding || nonce))` and
+   sends `{identity, binding, nonce, quote, event_log, vm_config}` to
+   the broker.
+6. The broker verifies the quote through `dstack-verifier`, checks
+   report-data binding and nonce freshness, then matches identity,
+   peer id, and compose hash against the Terraform policy.
+7. On success the broker mints a scoped Consul ACL token. Worker
+   agents receive a node-identity token; service workloads receive
+   service-identity tokens plus any declared Consul permissions.
+
+## Runtime Checks
+
+From a coordinator CVM, inspect Consul membership:
+
+```sh
+docker exec dstack-sidecar-1 sh -lc \
+  'CONSUL_HTTP_ADDR=http://127.0.0.1:8500 consul members -token "$CONSUL_MANAGEMENT_TOKEN"'
+```
+
+Check Raft health:
+
+```sh
+docker exec dstack-sidecar-1 sh -lc \
+  'CONSUL_HTTP_ADDR=http://127.0.0.1:8500 consul operator raft list-peers -token "$CONSUL_MANAGEMENT_TOKEN"'
+```
+
+Look for successful node and service admission in worker sidecar logs:
+
+```sh
+phala logs --cvm-id <worker-app-id> dstack-sidecar-1 -n 700 |
+  rg 'admission-client-(node|demo|webdemo).*admission accepted'
+```
+
+Look for hard attestation failures:
+
+```sh
+phala logs --cvm-id <worker-app-id> dstack-sidecar-1 -n 700 |
+  rg 'QUOTE_INVALID|VERIFIER_FAILED|REPORT_DATA_MISMATCH|ADMISSION_REJECTED'
+```
+
+No output from the failure grep is expected.
+
+## Current Limits
+
+- The low-level mesh rendezvous path still uses TURN HMAC admission.
+  Attestation admission gates Consul/Connect identity, not rendezvous.
+- Broker-issued Consul tokens do not expire by default. Setting
+  `ADMISSION_TOKEN_TTL` requires a renewal path for the Consul agent,
+  Patroni DCS token, and Envoy xDS token.
+- The verifier downloads OS-image inputs and computes reference
+  measurements on cache miss. Coordinators mount
+  `/tmp/dstack-runtime/verifier-cache` into the verifier so image and
+  measurement caches survive verifier container restarts on the same
+  CVM.
+- The measured `compose_hash` comes from Phala's `app-compose.json`.
+  Image binding should use digest-pinned image references in the
+  Compose source. Environment variable values are not a sufficient
+  image-binding policy.
+- Dynamic upgrade policy is future work. Phase 1 uses the static
+  Terraform-generated allowlist; a future policy server can admit new
+  compose hashes under a signed policy epoch.
+
+## Verification Cost
+
+Measured on 2026-05-18 with a live `tdx.small` worker quote and
+`dstacktee/dstack-verifier:0.5.9`:
+
+| Mode | Time |
+|---|---:|
+| Oneshot, empty verifier cache | 2.1s |
+| Oneshot, image and measurement cache already populated | 1.5s |
+| Long-running HTTP server, warm cache | 1.1-1.2s |
+
+The cold run downloaded a 20 MB OS-image bundle and computed the
+reference measurements. Larger OS images or slow network paths will
+increase the first request cost; the verifier download timeout is
+300s by default. After that, the verifier reuses the cached image and
+cached measurements for the same VM config. Quote verification still
+costs about one second, so the broker should treat verification as a
+startup/admission cost, not something to repeat for every leaf-cert
+rotation.

consul-postgres-ha/README.md

@@ -27,7 +27,7 @@ peers' ICE candidates; no data ever passes through it once peers
 connect. Per-CVM and per-cluster secrets are split: the TURN HMAC is
 derived per-CVM from dstack KMS, while cluster-wide-identical
 material (gossip key, Patroni passwords) is generated in Terraform
-and broadcast via env until attestation-rooted admission lands.
+and broadcast via env until the later TEE-derived secret phase lands.
 Connect CA uses Consul's built-in CA provider — root in Raft, no
 external derivation needed.
 
@@ -290,26 +290,29 @@ the platform plumbing untouched.
 
 | | |
 |---|---|
-| In-place env updates | Yes — change image tags or env values, `terraform apply`, CVMs update without losing pgdata. Requires provider `phala-network/phala 0.2.0-beta.3+`. |
+| In-place env updates | Yes — change image tags or env values, `terraform apply`, CVMs update without losing pgdata. Requires provider `phala-network/phala 0.2.0-beta.4+`. |
 | Failover RTO | ~24s soft-kill, ~33s hard-kill (default Patroni `ttl=30`). See [`FAILOVER.md`](FAILOVER.md). |
 | Cheap rejoin | Yes — a recovered ex-leader replays local WAL and rejoins as a streaming replica without pg_basebackup. |
 | Disk-loss rejoin | Yes — Patroni detects empty pgdata and runs full pg_basebackup over the QUIC overlay. |
 | Build provenance | Sigstore-attested via GitHub Build Provenance on every published image. Verify with `gh attestation verify oci://... --repo Dstack-TEE/dstack-examples`. |
+| Attested Consul admission | Optional with `enable_attestation_admission = true`. See [`ATTESTATION.md`](ATTESTATION.md). |
 
 ## Known limitations
 
 * Each `terraform apply` that fans out more than 1 `phala_app` create
   in parallel hits
   [`phala-cloud#247`](https://github.com/Phala-Network/phala-cloud/issues/247)
   — use `-parallelism=1` for now (~5 min × N to bring-up).
-* The mesh-conn admission story is **shared-secret based today**
-  (TURN HMAC), not attestation-based. Adding TEE attestation as the
-  admission credential is the next architectural step.
+* The low-level mesh-conn rendezvous story is **shared-secret based
+  today** (TURN HMAC), not attestation-based. Consul/Connect
+  admission can be gated by TEE attestation with
+  `enable_attestation_admission = true`; replacing rendezvous auth is
+  separate work.
 * The cluster's gossip key, Patroni superuser password, and
   replication password are **generated in Terraform and broadcast
   via env to every CVM** — a workaround, because those bytes live
   in `terraform.tfstate` and pass through whoever runs `apply`. The
-  attestation-rooted admission design
+  attestation-rooted secret design
   ([`design/attestation-admission.md`](design/attestation-admission.md))
   replaces this with TEE-derived material that no human ever sees.
 

consul-postgres-ha/admission-broker/main.go

@@ -25,7 +25,7 @@ const (
 	defaultListenAddr  = "127.0.0.1:8787"
 	defaultVerifierURL = "http://127.0.0.1:8080/verify"
 	defaultConsulURL   = "http://127.0.0.1:8500"
-	defaultTokenTTL    = time.Hour
+	defaultTokenTTL    = 0
 )
 
 func main() {
@@ -34,7 +34,7 @@ func main() {
 		verifierURL = flag.String("verifier-url", envOr("DSTACK_VERIFIER_URL", defaultVerifierURL), "dstack-verifier /verify URL")
 		consulURL   = flag.String("consul-url", envOr("CONSUL_HTTP_ADDR", defaultConsulURL), "Consul HTTP API URL")
 		policyPath  = flag.String("policy", os.Getenv("ADMISSION_POLICY_FILE"), "path to admission policy JSON")
-		tokenTTL    = flag.Duration("token-ttl", durationEnvOr("ADMISSION_TOKEN_TTL", defaultTokenTTL), "issued Consul ACL token TTL")
+		tokenTTL    = flag.Duration("token-ttl", durationEnvOr("ADMISSION_TOKEN_TTL", defaultTokenTTL), "issued Consul ACL token TTL; 0 disables expiration")
 	)
 	flag.Parse()
 
@@ -87,6 +87,7 @@ type Policy struct {
 type Workload struct {
 	WorkloadID            string            `json:"workload_id"`
 	Identity              string            `json:"identity"`
+	PeerID                string            `json:"peer_id,omitempty"`
 	ConsulService         string            `json:"consul_service"`
 	ConsulPermissions     ConsulPermissions `json:"consul_permissions,omitempty"`
 	AllowedComposeHashes  []string          `json:"allowed_compose_hashes"`
@@ -95,9 +96,10 @@ type Workload struct {
 }
 
 type ConsulPermissions struct {
-	KeyPrefixes   []string `json:"key_prefixes,omitempty"`
-	SessionWrite  bool     `json:"session_write,omitempty"`
-	AgentReadSelf bool     `json:"agent_read_self,omitempty"`
+	KeyPrefixes      []string `json:"key_prefixes,omitempty"`
+	SessionWrite     bool     `json:"session_write,omitempty"`
+	AgentReadSelf    bool     `json:"agent_read_self,omitempty"`
+	NodeIdentitySelf bool     `json:"node_identity_self,omitempty"`
 }
 
 func ParsePolicy(b []byte) (*Policy, error) {
@@ -133,8 +135,15 @@ func ParsePolicy(b []byte) (*Policy, error) {
 		if strings.TrimSpace(w.Identity) == "" {
 			return nil, fmt.Errorf("workloads[%d].identity is required", i)
 		}
-		if strings.TrimSpace(w.ConsulService) == "" {
-			return nil, fmt.Errorf("workloads[%d].consul_service is required", i)
+		w.PeerID = strings.TrimSpace(w.PeerID)
+		if w.PeerID != "" {
+			if err := validateConsulAgentName(w.PeerID); err != nil {
+				return nil, fmt.Errorf("workloads[%d].peer_id: %w", i, err)
+			}
+		}
+		w.ConsulService = strings.TrimSpace(w.ConsulService)
+		if w.ConsulService == "" && !w.ConsulPermissions.hasGrant() {
+			return nil, fmt.Errorf("workloads[%d] must define consul_service or consul_permissions", i)
 		}
 		for j, prefix := range w.ConsulPermissions.KeyPrefixes {
 			prefix = strings.Trim(prefix, "/")
@@ -160,16 +169,20 @@ func ParsePolicy(b []byte) (*Policy, error) {
 	return &p, nil
 }
 
-func (p *Policy) Match(identity, composeHash string) (*Workload, error) {
+func (p *Policy) Match(identity, composeHash, peerID string) (*Workload, error) {
 	normalized, err := normalizeComposeHash(composeHash)
 	if err != nil {
 		return nil, fmt.Errorf("invalid compose_hash from quote: %w", err)
 	}
+	peerID = strings.TrimSpace(peerID)
 	for _, w := range p.byIdentity[identity] {
-		if _, ok := w.allowedComposeHashSet[normalized]; ok {
+		if _, ok := w.allowedComposeHashSet[normalized]; ok && (w.PeerID == "" || w.PeerID == peerID) {
 			return &w, nil
 		}
 	}
+	if peerID != "" {
+		return nil, fmt.Errorf("identity %q is not allowed for compose_hash %s and peer_id %q", identity, normalized, peerID)
+	}
 	return nil, fmt.Errorf("identity %q is not allowed for compose_hash %s", identity, normalized)
 }
 
@@ -227,11 +240,11 @@ type AttestRequest struct {
 }
 
 type AttestResponse struct {
-	ConsulACLToken string    `json:"consul_acl_token"`
-	ExpiresAt      time.Time `json:"expires_at"`
-	WorkloadID     string    `json:"workload_id"`
-	ComposeHash    string    `json:"compose_hash"`
-	PolicyEpoch    int       `json:"policy_epoch"`
+	ConsulACLToken string     `json:"consul_acl_token"`
+	ExpiresAt      *time.Time `json:"expires_at,omitempty"`
+	WorkloadID     string     `json:"workload_id"`
+	ComposeHash    string     `json:"compose_hash"`
+	PolicyEpoch    int        `json:"policy_epoch"`
 }
 
 func (s *Server) handleAttest(w http.ResponseWriter, r *http.Request) {
@@ -264,10 +277,9 @@ func (s *Server) handleAttest(w http.ResponseWriter, r *http.Request) {
 	defer cancel()
 
 	verified, err := s.verifier.Verify(ctx, VerifyRequest{
-		Attestation: req.Attestation,
-		Quote:       req.Quote,
-		EventLog:    rawJSONAsString(req.EventLog),
-		VMConfig:    req.VMConfig,
+		Quote:    firstNonEmpty(req.Quote, req.Attestation),
+		EventLog: rawJSONAsString(req.EventLog),
+		VMConfig: req.VMConfig,
 	})
 	if err != nil {
 		writeError(w, http.StatusBadGateway, "VERIFIER_FAILED", err.Error())
@@ -282,17 +294,22 @@ func (s *Server) handleAttest(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 
-	workload, err := s.policy.Match(req.Identity, verified.ComposeHash)
+	workload, err := s.policy.Match(req.Identity, verified.ComposeHash, bindingStatement.PeerID)
 	if err != nil {
 		writeError(w, http.StatusForbidden, "ADMISSION_REJECTED", err.Error())
 		return
 	}
 
-	expiresAt := s.now().Add(s.tokenTTL)
+	var expiresAt *time.Time
+	if s.tokenTTL > 0 {
+		t := s.now().Add(s.tokenTTL)
+		expiresAt = &t
+	}
 	token, err := s.issuer.IssueToken(ctx, TokenRequest{
 		Description:       fmt.Sprintf("attested %s %s", workload.WorkloadID, verified.ComposeHash),
 		ConsulService:     workload.ConsulService,
 		ConsulPermissions: workload.ConsulPermissions,
+		Datacenter:        s.policy.Cluster,
 		PeerID:            bindingStatement.PeerID,
 		TTL:               s.tokenTTL,
 	})
@@ -413,10 +430,10 @@ func (s *NonceStore) gc(now time.Time) {
 }
 
 type VerifyRequest struct {
-	Attestation string `json:"attestation,omitempty"`
-	Quote       string `json:"quote"`
-	EventLog    string `json:"event_log"`
-	VMConfig    string `json:"vm_config"`
+	Attestation *string `json:"attestation"`
+	Quote       string  `json:"quote"`
+	EventLog    string  `json:"event_log"`
+	VMConfig    string  `json:"vm_config"`
 }
 
 type VerifiedQuote struct {
@@ -518,6 +535,7 @@ type TokenRequest struct {
 	Description       string
 	ConsulService     string
 	ConsulPermissions ConsulPermissions
+	Datacenter        string
 	PeerID            string
 	TTL               time.Duration
 }
@@ -533,20 +551,40 @@ type ConsulIssuer struct {
 }
 
 func (i *ConsulIssuer) IssueToken(ctx context.Context, req TokenRequest) (string, error) {
-	if req.ConsulService == "" {
-		return "", errors.New("consul service is required")
+	if req.TTL < 0 {
+		return "", errors.New("token TTL must not be negative")
 	}
 	payload := map[string]any{
-		"Description":   req.Description,
-		"ExpirationTTL": fmt.Sprintf("%ds", int(req.TTL.Seconds())),
-		"ServiceIdentities": []map[string]string{
+		"Description": req.Description,
+	}
+	if req.TTL > 0 {
+		payload["ExpirationTTL"] = fmt.Sprintf("%ds", int(req.TTL.Seconds()))
+	}
+	if req.ConsulService != "" {
+		payload["ServiceIdentities"] = []map[string]string{
 			{"ServiceName": req.ConsulService},
-		},
+		}
+	}
+	if req.ConsulPermissions.NodeIdentitySelf {
+		peerID := strings.TrimSpace(req.PeerID)
+		if err := validateConsulAgentName(peerID); err != nil {
+			return "", fmt.Errorf("node_identity_self requires attested peer_id: %w", err)
+		}
+		datacenter := strings.TrimSpace(req.Datacenter)
+		if err := validateConsulAgentName(datacenter); err != nil {
+			return "", fmt.Errorf("node_identity_self requires datacenter: %w", err)
+		}
+		payload["NodeIdentities"] = []map[string]string{
+			{"NodeName": peerID, "Datacenter": datacenter},
+		}
 	}
 	rules, err := consulACLRules(req.ConsulPermissions, req.PeerID)
 	if err != nil {
 		return "", err
 	}
+	if req.ConsulService == "" && !req.ConsulPermissions.NodeIdentitySelf && rules == "" {
+		return "", errors.New("token request must define consul service identity, node identity, or ACL rules")
+	}
 	if rules != "" {
 		policyName, err := i.EnsurePolicy(ctx, "attested-workload-"+shortSHA256(rules, 16), "Additional permissions for attested workloads", rules)
 		if err != nil {
@@ -705,6 +743,10 @@ func consulACLRules(p ConsulPermissions, peerID string) (string, error) {
 	return b.String(), nil
 }
 
+func (p ConsulPermissions) hasGrant() bool {
+	return len(p.KeyPrefixes) > 0 || p.SessionWrite || p.AgentReadSelf || p.NodeIdentitySelf
+}
+
 func validateConsulAgentName(name string) error {
 	if name == "" {
 		return errors.New("peer_id is empty")