Clarify mesh attestation tracking

h4x3rotab · h4x3rotab · commit 6178b2efa3df · 2026-05-18T07:30:15.000Z
diff --git a/consul-postgres-ha/ATTESTATION.md b/consul-postgres-ha/ATTESTATION.md
@@ -28,7 +28,7 @@ current dstack guest agents: `quote`, `event_log`, and `vm_config`.
    coordinator admission broker.
 4. The worker builds a binding statement containing the claimed
    identity, peer id, dstack app id, instance id, and compose hash.
-5. The worker calls dstack `GetQuote(SHA-512(binding || nonce))` and
+5. The worker calls dstack `GetQuote(SHA-256(binding || nonce))` and
    sends `{identity, binding, nonce, quote, event_log, vm_config}` to
    the broker.
 6. The broker verifies the quote through `dstack-verifier`, checks
@@ -38,6 +38,80 @@ current dstack guest agents: `quote`, `event_log`, and `vm_config`.
    agents receive a node-identity token; service workloads receive
    service-identity tokens plus any declared Consul permissions.
 
+## Report Data
+
+`report_data` is an unstructured SHA-256 digest. The structured
+admission data stays outside the quote in the binding statement:
+
+```text
+report_data = SHA-256(binding_statement_json || broker_nonce)
+```
+
+The broker nonce is 32 random bytes encoded as hex. The binding
+statement is JSON generated by the workload sidecar:
+
+```json
+{
+  "version": 1,
+  "identity": "spiffe://demo/postgres",
+  "cluster": "demo",
+  "peer_id": "worker-3",
+  "app_id": "<dstack app id>",
+  "instance_id": "<dstack instance id>",
+  "compose_hash": "<dstack compose hash>",
+  "issued_at": "2026-05-18T00:00:00Z"
+}
+```
+
+The client checks that dstack `GetQuote()` returns the same
+`report_data` it requested. The broker independently recomputes the
+digest from the submitted binding statement and nonce, then compares
+that digest with the verifier-returned `report_data`.
+
+## Peer IDs
+
+`peer_id` is the mesh node label, not a secret and not a hardware
+identity. In this example it is derived from the local role and
+ordinal, such as `worker-3`, and reused as the Consul node name.
+
+The current binding is:
+
+```text
+peer_id -> binding statement -> SHA-256 report_data -> TDX quote
+```
+
+The broker also requires the `peer_id` to match the Terraform policy
+row before it issues a node token or a service token scoped to that
+node. This prevents one attested workload row from directly claiming
+another row's Consul node identity.
+
+For stronger production binding, the policy should also pin a
+platform-stable workload instance field, such as a Phala-provided
+instance id or app id when that can be supplied without introducing
+a deployment cycle. Until then, `peer_id` is best understood as an
+authorization namespace declared by Terraform and quoted by the CVM.
+
+## Evidence Policy
+
+The broker treats `dstack-verifier` as the authority for quote and
+event-log semantics. The broker should require verifier outputs such
+as:
+
+- `quote_verified == true`
+- `event_log_verified == true`
+- `report_data` present and equal to the broker-computed digest
+- `app_info.compose_hash` present and policy-allowed
+
+KMS identity should be policy-controlled because it defines the key
+lifecycle. A production policy should pin the long-term KMS public
+key or an equivalent well-known KMS identity that is hard-coded in
+Terraform and checked against verified dstack evidence.
+
+For production profiles, OS image identity should also be checked.
+The expected OS image hash should be both a Terraform launch setting
+and an admission policy field, so a workload cannot satisfy the
+compose policy while running on an unexpected base image.
+
 ## Runtime Checks
 
 From a coordinator CVM, inspect Consul membership:
diff --git a/consul-postgres-ha/PROGRESS.md b/consul-postgres-ha/PROGRESS.md
@@ -0,0 +1,71 @@
+# Project Progress
+
+This tracks the service-mesh project as a whole, not only
+attestation. Keep it current when landing changes that alter the
+deployment model, trust model, or operational status.
+
+## Current Status
+
+- Terraform deploys a 3-coordinator / 3-worker real Phala Cloud
+  cluster with public provider `phala-network/phala v0.2.0-beta.4`.
+- `mesh-conn` direct connectivity is enabled by default; relay-only
+  configuration was removed.
+- Consul server quorum, Consul clients, Envoy Connect sidecars, and
+  Patroni HA Postgres are running through the mesh.
+- Attestation admission gates Consul node and service token issuance
+  through real dstack quotes and `dstack-verifier`.
+- Worker preflight compose hashes are generated by Terraform and used
+  as revision evidence in the admission policy.
+
+## Recently Validated
+
+- Full real-cluster Terraform apply with preflight-backed admission
+  policy.
+- Public Terraform provider path, with no local provider override.
+- Real attestation path on all workers for node, webdemo, and
+  postgres identities.
+- Negative admission test: a valid quote claiming the wrong `peer_id`
+  is rejected by policy.
+- Worker sidecar restart: node token, webdemo token, and postgres
+  token are re-issued through attestation before the worker rejoins.
+- Consul health after tests: zero critical checks, all coordinators
+  and workers alive.
+
+## Open Work
+
+- KMS policy: pin the long-term KMS public key or equivalent KMS
+  identity in Terraform and require the verifier-proven KMS evidence
+  to match it.
+- Production OS policy: when a production profile is enabled, include
+  the expected OS image hash in both Phala launch config and admission
+  policy.
+- Peer identity hardening: keep `peer_id` as the Terraform-declared
+  Consul node namespace, but bind it to a platform-stable instance
+  identifier when Phala exposes one without creating a deployment
+  cycle.
+- Token lifecycle: keep non-expiring tokens for the first integrated
+  version; next step is one active token per attested workload
+  instance, then bounded TTL plus renewal once Consul, Envoy, and
+  Patroni reload paths are explicit.
+- Upgrade policy: replace static compose-hash allowlists with a
+  policy server or on-chain bridge that admits new revisions under
+  signed policy epochs.
+- Secret lifecycle: move remaining cluster-wide-identical secrets
+  out of Terraform env into TEE-rooted generation/distribution.
+- Long soak: run the real cluster for at least one hour with zero ACL
+  authorization failures, zero attestation rejections, and stable
+  Patroni leader/replica health.
+
+## Decisions
+
+- Workload identity is the Terraform-declared identity; compose hash
+  is revision evidence, not the workload key.
+- `report_data` is `SHA-256(binding_statement_json || nonce)`.
+- The broker relies on `dstack-verifier` for quote and event-log
+  semantics. It does not parse raw RTMR event-log internals itself.
+- `peer_id` is the logical mesh/Consul node label, for example
+  `worker-3`. It is not a secret. It is quoted in `report_data` and
+  matched against Terraform policy before node-scoped tokens are
+  issued.
+- Production image policy should prefer digest-pinned Compose image
+  references.
diff --git a/consul-postgres-ha/README.md b/consul-postgres-ha/README.md
@@ -34,6 +34,10 @@ external derivation needed.
 For the full topology and layering walkthrough, see
 [`ARCHITECTURE.md`](ARCHITECTURE.md).
 
+For current project status and remaining work across Terraform,
+mesh transport, Consul, Patroni, and attestation, see
+[`PROGRESS.md`](PROGRESS.md).
+
 ### The three address spaces (you'll see these everywhere)
 
 Apps and operators run into three loopback ranges in this example. Knowing
@@ -110,6 +114,7 @@ consul-postgres-ha/
 ├── README.md             you are here
 ├── ARCHITECTURE.md       the three-layer stack, peer topology, port plan
 ├── FAILOVER.md           soft-kill / hard-kill / disk-loss recipes + measured RTO
+├── PROGRESS.md           project status, recently validated flows, open work
 ├── PUBLISHING.md         CI publish flow, manual ttl.sh shortcuts, hot-patch
 ├── ROBUSTNESS.md         where each layer breaks + mitigations
 │
diff --git a/consul-postgres-ha/admission-broker/main.go b/consul-postgres-ha/admission-broker/main.go
@@ -5,7 +5,6 @@ import (
 	"context"
 	"crypto/rand"
 	"crypto/sha256"
-	"crypto/sha512"
 	"encoding/hex"
 	"encoding/json"
 	"errors"
@@ -342,7 +341,7 @@ func reportDataHex(bindingHex, nonceHex string) (string, error) {
 	if len(nonce) != 32 {
 		return "", fmt.Errorf("nonce must be 32 bytes, got %d", len(nonce))
 	}
-	h := sha512.New()
+	h := sha256.New()
 	h.Write(binding)
 	h.Write(nonce)
 	return hex.EncodeToString(h.Sum(nil)), nil
diff --git a/consul-postgres-ha/admission-broker/main_test.go b/consul-postgres-ha/admission-broker/main_test.go
@@ -3,7 +3,7 @@ package main
 import (
 	"bytes"
 	"context"
-	"crypto/sha512"
+	"crypto/sha256"
 	"encoding/hex"
 	"encoding/json"
 	"net/http"
@@ -120,7 +120,7 @@ func TestReportDataHexBindsStatementAndNonce(t *testing.T) {
 	}
 	bindingBytes, _ := hex.DecodeString(binding)
 	nonceBytes, _ := hex.DecodeString(nonce)
-	h := sha512.New()
+	h := sha256.New()
 	h.Write(bindingBytes)
 	h.Write(nonceBytes)
 	want := hex.EncodeToString(h.Sum(nil))
diff --git a/consul-postgres-ha/admission-client/main.go b/consul-postgres-ha/admission-client/main.go
@@ -3,7 +3,7 @@ package main
 import (
 	"bytes"
 	"context"
-	"crypto/sha512"
+	"crypto/sha256"
 	"encoding/hex"
 	"encoding/json"
 	"errors"
@@ -237,7 +237,7 @@ func reportData(statementBytes []byte, nonceHex string) ([]byte, error) {
 	if len(nonce) != 32 {
 		return nil, fmt.Errorf("nonce must be 32 bytes, got %d", len(nonce))
 	}
-	h := sha512.New()
+	h := sha256.New()
 	h.Write(statementBytes)
 	h.Write(nonce)
 	return h.Sum(nil), nil
diff --git a/consul-postgres-ha/admission-client/main_test.go b/consul-postgres-ha/admission-client/main_test.go
@@ -3,7 +3,7 @@ package main
 import (
 	"bytes"
 	"context"
-	"crypto/sha512"
+	"crypto/sha256"
 	"encoding/hex"
 	"encoding/json"
 	"net/http"
@@ -23,7 +23,7 @@ func TestReportDataBindsStatementAndNonce(t *testing.T) {
 		t.Fatal(err)
 	}
 	nonceBytes, _ := hex.DecodeString(nonce)
-	h := sha512.New()
+	h := sha256.New()
 	h.Write(statement)
 	h.Write(nonceBytes)
 	if !bytes.Equal(got, h.Sum(nil)) {
diff --git a/consul-postgres-ha/design/attestation-admission.md b/consul-postgres-ha/design/attestation-admission.md
@@ -141,15 +141,15 @@ The broker validates:
 1. **Quote chain**: TDX → Intel PCS root, via a sibling
    `dstack-verifier` HTTP service.
 2. **`report_data` binding**: must equal
-   `SHA-512(binding_statement || nonce)`. Anything else means the
+   `SHA-256(binding_statement || nonce)`. Anything else means the
    quote was generated for a different handshake; reject. The broker
    also verifies that the statement identity equals the claimed
    identity.
 3. **Nonce freshness**: must be a nonce the broker issued
    recently and hasn't been consumed.
-4. **RTMR replay**: use the Go SDK's `ReplayRTMRs()` helper to
-   confirm the event log matches the quote's RTMRs. Otherwise the
-   event log could be forged.
+4. **Verifier event-log result**: require `event_log_verified` from
+   `dstack-verifier`. The broker forwards the event log to the
+   verifier but does not reimplement RTMR replay semantics itself.
 5. **Policy check**: evaluate the claimed `identity` against the
    generated admission policy. Phase 1 accepts a quote when the
    identity names a known Terraform workload and the quote's
@@ -258,11 +258,24 @@ single ordinary apply: Terraform already knows the workload
 definition, preflight yields the measured revision hash, and the
 broker receives a policy document derived from those same inputs.
 
-`app_id`, KMS details, TCB fields, and `app-compose.json` remain
-valuable evidence and audit material, but they are not the primary
-policy key in Phase 1. In particular, `app-compose.json` is useful
-for human review and transparency logs because it is the exact byte
-payload dstack measures and it embeds the Docker Compose string.
+`app_id`, instance id, KMS details, OS image hash, TCB fields, and
+`app-compose.json` remain valuable evidence and audit material, but
+they are not the primary policy key in Phase 1. In particular,
+`app-compose.json` is useful for human review and transparency logs
+because it is the exact byte payload dstack measures and it embeds
+the Docker Compose string.
+
+KMS identity should be promoted into the admission predicate. KMS
+public keys are long-term, so Terraform can carry a well-known KMS
+public key or equivalent KMS identity and the broker can compare it
+with verified dstack evidence. This ties service-mesh admission to
+the key lifecycle domain that protects workload secrets.
+
+OS image identity should be enforced when the Terraform deployment
+uses a production profile. The expected OS image hash should appear
+both in the real Phala CVM launch config and in the admission policy,
+so a valid compose hash cannot be admitted on an unexpected base
+image.
 
 ## Upgradeability policy server
 
@@ -315,7 +328,7 @@ client agent itself, then for each declared service identity.
 
 3. POST /v1/admission/challenge to broker. Receive nonce.
 
-4. report_data = SHA-512(binding_statement || nonce)
+4. report_data = SHA-256(binding_statement || nonce)
    Call dstack GetQuote(report_data). Receive quote, event_log,
    vm_config, and report_data.
 
