Validate slot-based attested mesh deployment

Author: h4x3rotab

ATTESTATION.md

@@ -109,6 +109,8 @@ as:
 - `app_info.compose_hash` present and policy-allowed
 - `app_info.key_provider_info` present and equal to the policy-pinned
   KMS provider id
+- when `production_profile = true`, `os_image_hash_verified == true`
+  and `os_image_hash` equals the Terraform preflight hash
 
 KMS identity should be policy-controlled because it defines the key
 lifecycle. This example pins the long-term KMS public key in
@@ -128,9 +130,13 @@ configuration. The Terraform default matches the example's default
 `kms = "phala"` backend; override it when changing KMS backend.
 
 For production profiles, OS image identity should also be checked.
-The expected OS image hash should be both a Terraform launch setting
-and an admission policy field, so a workload cannot satisfy the
-compose policy while running on an unexpected base image.
+This example makes that an explicit profile: set
+`production_profile = true` and an `os_image` name such as
+`dstack-0.5.7`. Terraform uses that image in the Phala launch config,
+reads the resulting
+`os_image_hash` from preflight, and emits that SHA-256 hash into the
+admission policy. The broker then requires `dstack-verifier` to prove
+the same OS image hash before issuing Consul ACL tokens.
 
 ## Runtime Checks
 
@@ -164,6 +170,12 @@ phala logs --cvm-id <worker-app-id> dstack-sidecar-1 -n 700 |
 
 No output from the failure grep is expected.
 
+Production Phala OS images disable SSH access. For prod-profile runs,
+use Phala instance metadata, `phala ps`, and public logs to validate
+OS hash, container health, and admission success. Use a dev image only
+when the test specifically requires in-CVM `consul` or `patronictl`
+commands.
+
 ## Current Limits
 
 - The low-level mesh rendezvous path still uses TURN HMAC admission.

PROGRESS.md

@@ -7,21 +7,30 @@ deployment model, trust model, or operational status.
 ## Current Status
 
 - Terraform deploys a 3-coordinator / 3-worker real Phala Cloud
-  cluster with public provider `phala-network/phala v0.2.0-beta.4`.
+  cluster with public provider `phala-network/phala v0.3.0-beta.1`
+  using the named-slot model.
 - `mesh-conn` direct connectivity is enabled by default; relay-only
   configuration was removed.
 - Consul server quorum, Consul clients, Envoy Connect sidecars, and
   Patroni HA Postgres are running through the mesh.
 - Attestation admission gates Consul node and service token issuance
   through real dstack quotes and `dstack-verifier`.
-- Worker preflight compose hashes are generated by Terraform and used
-  as revision evidence in the admission policy.
+- Worker role preflight compose hashes are generated by Terraform and
+  used as revision evidence in the admission policy.
+- Admission policy checks KMS key-provider evidence. When
+  `production_profile = true`, it also checks verifier-proven OS image
+  hash evidence against the Terraform preflight hash.
 
 ## Recently Validated
 
 - Full real-cluster Terraform apply with preflight-backed admission
   policy.
-- Public Terraform provider path, with no local provider override.
+- Public Terraform provider path, with no local provider override:
+  `terraform init -upgrade`, `apply`, no-op `plan`, and `destroy`
+  against `phala-network/phala v0.3.0-beta.1`.
+- Production-profile real-cluster apply on prod `dstack-0.5.7`, with
+  all six CVMs reporting `is_dev = false` and OS image hash
+  `761c05d282c81abeae2d1a8f6d5b1e039c8ce14cc95a6da020b9ed2ff1056816`.
 - Real attestation path on all workers for node, webdemo, and
   postgres identities.
 - Negative admission test: a valid quote claiming the wrong `peer_id`
@@ -30,23 +39,36 @@ deployment model, trust model, or operational status.
   token are re-issued through attestation before the worker rejoins.
 - Consul health after tests: zero critical checks, all coordinators
   and workers alive.
+- Broker unit tests cover KMS mismatch rejection and production OS
+  image hash mismatch rejection.
 
 ## Open Work
 
-- Terraform slot API: integrate the newly implemented Phala Cloud
-  slot API into the Terraform deployment flow. Tracking note:
-  `design/terraform-slot-api.md`.
-- KMS policy: Terraform now carries the selected KMS backend and the
-  expected long-term KMS public key; admission requires the verifier
-  returned key-provider evidence to match it.
-- Production OS policy: when a production profile is enabled, include
-  the expected OS image hash in both Phala launch config and admission
-  policy.
+- Terraform slot API: provider 0.3.0-beta.1 lands the named-slot
+  resource (`phala_app_instance` + `phala_app.members`) and
+  `cluster-example/cluster.tf` is migrated onto it. App-level
+  `docker_compose`/`env`/`image` updates on a phala_app in members
+  mode are intentionally refused at plan time by the provider until
+  the cloud's `/apps/{app_id}/revisions/{rev_id}/redeploy` fan-out
+  endpoint is wired into the provider — see
+  `design/terraform-slot-api.md` for the integration path.
+- Rollout driver: `cluster-example/rollout.sh` is paused for
+  in-place rolls. It still serves `--plan` and Consul-health
+  probing; the actual `terraform apply -target=...` loop now exits
+  with a clear "Unsafe update in members mode" guidance message
+  rather than half-mutating a cluster. The eventual slot-aware
+  rollout will drive `/apps/{app_id}/revisions/{rev_id}/redeploy`
+  with per-slot drains, reusing this script's leader-transfer +
+  wait-for-quorum scaffolding.
 - Secret lifecycle: move remaining cluster-wide-identical secrets
   out of Terraform env into TEE-rooted generation/distribution.
 - Long soak: run the real cluster for at least one hour with zero ACL
   authorization failures, zero attestation rejections, and stable
   Patroni leader/replica health.
+- Deep prod-profile operational validation: prod OS images disable SSH,
+  so use Phala logs/API evidence for admission and container health.
+  Run dev-image validation when direct `consul`/`patronictl` CLI access
+  is required.
 - Token lifecycle: keep non-expiring tokens for the first integrated
   version; next step is one active token per attested workload
   instance, then bounded TTL plus renewal once Consul, Envoy, and

README.md

@@ -80,8 +80,8 @@ $EDITOR terraform.tfvars   # set gateway_domain, image refs, external_*
 # deploy is silent until you spot it in the dashboard.
 export PHALA_CLOUD_API_KEY=phak_...
 
-terraform init
-terraform apply -parallelism=1   # phala-cloud#247 needs serial creates
+terraform init -upgrade
+terraform apply
 ```
 
 Once apply finishes, the cluster is HA Postgres on
@@ -295,7 +295,7 @@ the platform plumbing untouched.
 
 | | |
 |---|---|
-| In-place env updates | Yes — change image tags or env values, `terraform apply`, CVMs update without losing pgdata. Requires provider `phala-network/phala 0.2.0-beta.4+`. |
+| Slot add/remove | With provider `phala-network/phala 0.3.0-beta.1+`, replica count changes map to stable named slots; adding or removing a slot does not disturb the remaining slots. |
 | Failover RTO | ~24s soft-kill, ~33s hard-kill (default Patroni `ttl=30`). See [`FAILOVER.md`](FAILOVER.md). |
 | Cheap rejoin | Yes — a recovered ex-leader replays local WAL and rejoins as a streaming replica without pg_basebackup. |
 | Disk-loss rejoin | Yes — Patroni detects empty pgdata and runs full pg_basebackup over the QUIC overlay. |
@@ -304,10 +304,18 @@ the platform plumbing untouched.
 
 ## Known limitations
 
-* Each `terraform apply` that fans out more than 1 `phala_app` create
-  in parallel hits
-  [`phala-cloud#247`](https://github.com/Phala-Network/phala-cloud/issues/247)
-  — use `-parallelism=1` for now (~5 min × N to bring-up).
+* Provider `0.3.0-beta.1` named-slot support is indexed by Terraform
+  Registry and the cluster example has been validated with the public
+  provider. Keep the exact beta pin until the slot model has soaked
+  under real traffic.
+* App-level `docker_compose`, `env`, and `image` updates are blocked
+  in provider `0.3.0-beta.1` members mode. Slot add/remove is safe;
+  in-place rolling revisions need the cloud revision redeploy endpoint
+  wired into the provider.
+* Prod OS images disable SSH. Production-profile attestation tests
+  should validate through Phala instance metadata, `phala ps`, and
+  logs; use a dev image for tests that require direct `consul` or
+  `patronictl` commands inside the CVM.
 * The low-level mesh-conn rendezvous story is **shared-secret based
   today** (TURN HMAC), not attestation-based. Consul/Connect
   admission can be gated by TEE attestation with
@@ -326,12 +334,7 @@ the platform plumbing untouched.
 * [`Phala-Network/terraform-provider-phala#5`](https://github.com/Phala-Network/terraform-provider-phala/issues/5)
   — `storage_fs` triggers ForceNew when unset; we explicitly pin
   `storage_fs = "zfs"` in `cluster.tf`.
-* [`Phala-Network/phala-cloud#247`](https://github.com/Phala-Network/phala-cloud/issues/247)
-  — concurrent `phala_app` creates against the same workspace return
-  `400 "configuration parameters not compatible"`. Workaround:
-  `terraform apply -parallelism=1`.
-* [`Phala-Network/phala-cloud#242`](https://github.com/Phala-Network/phala-cloud/issues/242)
-  — `phala cvms list` collapses replicas to one entry.
 * [`Phala-Network/phala-cloud#243`](https://github.com/Phala-Network/phala-cloud/issues/243)
-  — per-instance Terraform resource + `update_policy` + lifecycle
-  hooks would let `cluster-example/rollout.sh` collapse into HCL.
+  — per-instance Terraform resource landed in provider
+  `0.3.0-beta.1`; revision rollout policy still needs provider support
+  for the cloud `/apps/{app_id}/revisions/{rev_id}/redeploy` path.

ROBUSTNESS.md

@@ -305,11 +305,12 @@ depth.
 
 ### Already shipped
 
-- **Three Consul servers** — landed in commit `17f4642`. The
-  coordinator app deploys with `replicas = 3` and Consul agents on
-  those CVMs run as servers with `bootstrap_expect = 3`. Workers
-  retry-join through every coordinator's serf port via mesh-conn,
-  so the single-coordinator-failure scenario stays operational.
+- **Three Consul servers** — landed in commit `17f4642`; the current
+  Terraform model deploys three stable coordinator slots, and Consul
+  agents on those CVMs run as servers with `bootstrap_expect = 3`.
+  Workers retry-join through every coordinator's serf port via
+  mesh-conn, so the single-coordinator-failure scenario stays
+  operational.
 - **Real registry** — Sigstore-attested GHCR images via
   `.github/workflows/consul-postgres-ha-publish.yml`. See
   `PUBLISHING.md`.
@@ -401,9 +402,8 @@ in attestation rather than handed in by the deployer.
   "Already shipped" for the failure analysis and the regression test.
 - **Auth-channel reconnect deadlock** — fixed via fresh
   `peerSession{}` per `dialICE` + drain-then-push on `authCh`.
-- **Three-server Consul** — coordinator deploys with
-  `replicas = 3`; Consul agents run as servers with
-  `bootstrap_expect = 3`.
+- **Three-server Consul** — coordinator deploys three stable named
+  slots; Consul agents run as servers with `bootstrap_expect = 3`.
 - **PEERS_JSON validation** — `validatePeers()` runs at startup
   with nine-case unit-test coverage.
 - **Real registry** — Sigstore-attested GHCR images via