Clean up service mesh docs and dead prototype

Author: h4x3rotab

ARCHITECTURE.md

@@ -90,9 +90,10 @@ remote peer over the QUIC link. With the 3-service example
 | 8300  | Consul server RPC (server-to-server, client-to-server)| TCP         |
 | 8301  | Consul serf-LAN gossip                                | UDP + TCP   |
 
-The allowlist is intentionally minimal — mesh-conn knows **peers,
-not services**. Apps never dial peer VIPs; only Envoy and Consul-agent
-do, and both speak well-known platform ports.
+The runtime allowlist is intentionally minimal — mesh-conn knows
+**peers, not services**. Apps never dial peer VIPs; only Envoy and
+Consul-agent do. Consul uses fixed platform ports; Envoy sidecar ports
+are generated from `local.services`.
 
 The allowlist is **platform-generated, not Go-const**: per-service
 Connect-sidecar ports come from `local.services` in `cluster.tf`, get
@@ -344,10 +345,10 @@ When a stream opens, the **first 3 bytes** carry a mesh-conn header:
 - `tag = 0x33` → **streamTCP** — per-connection raw TCP byte stream.
 
 The 16-bit `port` is the **receiver's local port to dial / write
-into**. The receiver validates it against the static allowlist
-(`{21000, 21001, 8300, 8301}`); anything else is rejected. There is
-no per-peer port-list lookup — ports are platform-level constants
-the same on every CVM.
+into**. The receiver validates it against `MESH_CONN_ALLOWLIST`; any
+port absent from that runtime allowlist is rejected. There is no
+per-peer port-list lookup — the platform emits the same allowlist on
+every CVM from `local.services` plus Consul's infrastructure ports.
 
 UDP-over-stream uses an explicit 2-byte big-endian length prefix per
 datagram, since QUIC streams are byte-oriented (like yamux was) and
@@ -440,10 +441,10 @@ instead of a generic TCP connect check.
   webdemo, Patroni all think they're on a flat loopback talking to
   services by name. Anything that runs against Consul today (Vault,
   Nomad, Boundary, custom apps) drops in unchanged.
-- **Layer 1 + 2 are a single component (mesh-conn, ~600 LoC Go
-  including the QUIC adapter) and have zero awareness of services.**
-  mesh-conn forwards bytes between peer VIPs on a static infra-port
-  allowlist; it does not know what Consul, Patroni, or Envoy are.
+- **Layer 1 + 2 are a single component (`mesh-conn`) and have zero
+  awareness of services.**
+  mesh-conn forwards bytes between peer VIPs on the runtime allowlist;
+  it does not know what Consul, Patroni, or Envoy are.
   It would equally well move Postgres replication, Redis Sentinel,
   Kafka, etc. — and in fact this example uses it for both Consul
   cluster gossip and Patroni-replication-via-Envoy.

ATTESTATION.md

@@ -158,7 +158,7 @@ Look for successful node and service admission in worker sidecar logs:
 
 ```sh
 phala logs --cvm-id <worker-app-id> dstack-sidecar-1 -n 700 |
-  rg 'admission-client-(node|demo|webdemo).*admission accepted'
+  rg 'admission-client-.*admission accepted'
 ```
 
 Look for hard attestation failures:

FAILOVER.md

@@ -17,7 +17,12 @@ back.
 
 ## Recipe
 
-Set up env (cluster IDs from `RESUME.md`):
+These recipes require shell access inside the CVMs, so run them on a
+dev-image cluster. Production Phala OS images disable SSH; for those,
+use Phala logs/API evidence unless a separate debug-access path exists.
+
+Set up env with the gateway domain and current worker IDs from your
+Terraform outputs or Phala instance list:
 
 ```bash
 GW=dstack-pha-prod5.phala.network

PUBLISHING.md

@@ -7,15 +7,16 @@ tag (or, better, by digest) and verify provenance with
 `gh attestation verify`.
 
 `mesh-sidecar` is the consolidated platform-plumbing image — a single
-container that runs bootstrap-secrets, mesh-conn, consul, and (on
-workers) envoy. It's the heaviest by a wide margin because it
-inherits from envoyproxy/envoy and bundles three more binaries on top.
+container that runs bootstrap-secrets, mesh-conn, attestation
+admission helpers, consul, and (on workers) envoy. It's the heaviest
+by a wide margin because it inherits from envoyproxy/envoy and bundles
+several more binaries on top.
 
 This doc covers the three paths you'll actually use:
 
 1. **CI publish** (the steady-state)
 2. **Manual one-off publish** (dev iteration / breaking glass)
-3. **Hot-patch on a live cluster** (debugging without a redeploy)
+3. **Hot-patch on a dev cluster** (debugging without a redeploy)
 
 ## 1. CI publish — the steady-state
 
@@ -25,9 +26,10 @@ change, and on PRs touching the same paths. Each run:
 
 - Builds all four images via a matrix job. The `mesh-sidecar` build
   uses `consul-postgres-ha/` as its docker context (instead of
-  `consul-postgres-ha/mesh-sidecar/`) so its Dockerfile can pull
-  `bootstrap-secrets/` and `mesh-conn/` Go sources from sibling
-  directories.
+  `consul-postgres-ha/mesh-sidecar/`) so its Dockerfile can pull the
+  embedded Go components from sibling directories:
+  `bootstrap-secrets/`, `mesh-conn/`, `admission-broker/`, and
+  `admission-client/`.
 - On `main`, pushes to `ghcr.io/dstack-tee/dstack-examples/consul-postgres-ha-<name>` with two tags: the long-form commit SHA (`sha-<40-hex>`) and `latest`.
 - Generates a GitHub Build Provenance attestation per image via
   `actions/attest-build-provenance@v2`. The attestation is signed by
@@ -78,9 +80,11 @@ docker build -t $TAG -f consul-postgres-ha/mesh-sidecar/Dockerfile consul-postgr
 docker push $TAG
 ```
 
-Then point the running cluster at it via `terraform.tfvars`'s
-`mesh_sidecar_image = ...` (and `terraform apply`), or hot-patch the
-running CVM (see §3). `ttl.sh` images expire 24h after push.
+Then use it in `terraform.tfvars` for a fresh dev cluster, or
+hot-patch one SSH-enabled dev CVM (see §3). Existing provider-0.3
+named-slot clusters cannot roll measured image changes in place until
+the revision-redeploy provider path lands. `ttl.sh` images expire 24h
+after push.
 
 ### b) Personal GHCR namespace (persistent, requires PAT)
 
@@ -97,16 +101,13 @@ These manual builds do **not** carry a build-provenance attestation —
 that comes from CI's OIDC identity. For anything user-facing, run the
 real CI workflow.
 
-## 3. Hot-patch on a live cluster — debugging without a redeploy
+## 3. Hot-patch on a dev cluster — debugging without a redeploy
 
-Sometimes you need to swap a binary on a running CVM right now —
-faster than re-running `terraform apply` (which propagates env updates
-correctly as of provider `0.2.0-beta.3`, but still goes per-CVM and
-takes a minute), useful for testing a fix on one CVM before rolling it
-cluster-wide, and the only option on clusters running the older
-`0.2.0-beta.2` provider where in-place env updates silently no-op'd
-(Phala-Network/phala-cloud#246; fixed by
-Phala-Network/terraform-provider-phala#8).
+Sometimes you need to swap a binary on one running CVM right now to
+debug a fix before building a proper image. This is a dev-only path.
+It requires SSH, so it works only on dev OS images; production Phala
+OS images disable SSH. It also bypasses dstack compose measurements,
+so do not use it for attestation or production validation.
 
 ```bash
 GW=dstack-pha-prod5.phala.network
@@ -133,16 +134,18 @@ your tfstate.
 
 ## What to bump after a CI publish
 
-When CI publishes a new `latest` and you want to roll it to a running
-cluster:
+When CI publishes new images:
 
 1. Decide whether you're pinning to `:latest` (drifts) or to the
    `:sha-...` tag from the new run (recommended). Find the new SHA by
    inspecting the workflow run's output or `gh run view`.
-2. Edit `consul-postgres-ha/cluster-example/terraform.tfvars`
-   to that pin.
-3. `terraform apply`. Per-CVM compose re-renders and the dstack agent
-   recreates each service. (Or hot-patch per §3 if you want to verify
-   on one CVM first.)
+2. For a new cluster, edit
+   `consul-postgres-ha/cluster-example/terraform.tfvars` to that pin
+   before `terraform apply`.
+3. For an existing provider-0.3 named-slot cluster, app-level image
+   and compose changes are blocked until the Cloud revision-redeploy
+   endpoint is wired into the provider. Today, use a new cluster or an
+   explicit destroy/recreate for measured changes; use the hot-patch
+   path only for dev smoke testing on a single SSH-enabled CVM.
 4. Verify with `gh attestation verify oci://...@<digest>` if you want
    to be sure the image you're pinning was built by this repo.

README.md

@@ -127,8 +127,7 @@ consul-postgres-ha/
 ├── mesh-conn/            Go source — QUIC-over-pion/ICE overlay (built into sidecar)
 ├── patroni/              Patroni + Postgres image
 ├── webdemo/              example workload sitting on the mesh
-├── signaling/            HTTP /publish + /poll broker for ICE auth/candidate exchange
-└── quic-on-ice/          standalone smoke test for the QUIC-over-ICE transport
+└── signaling/            HTTP /publish + /poll broker for ICE auth/candidate exchange
 ```
 
 ## Adapting to your own workload
@@ -164,8 +163,8 @@ Say you want a billing service on port 9090 that the existing
    and add a `billing` service to `compose/worker.yaml` (modelled on
    `webdemo` — it just binds `127.0.0.1:9090` and serves).
 
-3. **`terraform apply`**. The provider's in-place env update pushes
-   the new `SERVICES_JSON` to every CVM:
+3. **Deploy the new revision**. For a fresh cluster, `terraform apply`
+   creates CVMs with the new `SERVICES_JSON`:
 
    - `mesh-conn`'s allowlist extends to `{21000, 21001, 21002,
      8300, 8301}` automatically — `MESH_CONN_ALLOWLIST` is computed
@@ -176,6 +175,13 @@ Say you want a billing service on port 9090 that the existing
    - Coordinator-0 writes a default-allow intention for `billing`
      and any subset/redirect resolvers implied by the declaration.
 
+   On an existing provider-0.3 named-slot cluster, app-level
+   `docker_compose`, `env`, and `image` mutations are intentionally
+   blocked until the Cloud revision-redeploy endpoint is wired into
+   the Terraform provider. Today that means this kind of service-shape
+   change is a new-cluster or destroy/recreate operation, not an
+   in-place roll.
+
 4. **From any container on any peer**, `curl http://billing:9090/...`
    load-balances across all peers' `billing` instances over Connect
    mTLS. No application-side service-discovery code.

ROBUSTNESS.md

@@ -13,9 +13,9 @@ do we do about it?", and lands on a prioritised punch list.
 ```
   Layer 3   apps         Consul + Envoy + webdemo + Patroni
                          (HashiCorp / Lyft / Zalando code, well-trodden)
-  Layer 2   forwarder    mesh-conn ~600 LoC: peer VIPs (127.50.0.0/24),
-                         static infra-port allowlist {21000, 21001,
-                         8300, 8301}, 3-byte stream header
+  Layer 2   forwarder    mesh-conn: peer VIPs (127.50.0.0/24),
+                         runtime allowlist from SERVICES_JSON plus
+                         Consul infra ports, 3-byte stream header
   Layer 1   transport    pion/ice + QUIC: punched UDP path,
                          stream multiplex, flow control, keepalive
   Layer 0   rendezvous   coturn + signalling broker on a public box;
@@ -79,7 +79,7 @@ whose direct path goes down can't fail over to TURN until it's back".
 
 ### What's there
 
-- one ICE connection per peer-pair (6 in our 4-CVM cluster),
+- one ICE connection per peer-pair (15 in the default 6-CVM cluster),
   established via signalling broker + coturn
 - one `QUIC.Session` per ICE conn, with `EnableKeepAlive=true`
 - the streams flowing inside (one long-lived UDP-per-port,
@@ -201,8 +201,9 @@ whose direct path goes down can't fail over to TURN until it's back".
 ### What's there
 
 - peer VIPs `127.50.0.<vip>` (one per peer, in PEERS_JSON)
-- the static infra-port allowlist `{21000, 21001, 8300, 8301}`
-  (in `mesh-conn/main.go`, deliberately small — see ARCHITECTURE.md)
+- the runtime port allowlist from `MESH_CONN_ALLOWLIST`
+  (generated by `mesh-sidecar/entrypoint.sh`, deliberately small —
+  see ARCHITECTURE.md)
 - the 3-byte stream header (tag, port-uint16-BE)
 - the per-stream pumps (UDP length-prefix, TCP raw splice)
 - one accept-loop per peer pair to demux incoming streams
@@ -259,17 +260,20 @@ The mitigations are mostly testing discipline:
   out-of-range VIPs. A canonicalised `peersDigest()` is logged at
   startup so `grep` across all peers' logs immediately surfaces
   config drift. Unit tests in `validate_test.go` cover the cases
-  plus an `allowedPort` check pinning the static allowlist.
+  plus an `allowedPort` check pinning membership in the configured
+  allowlist.
 
 ## Layer 3 — Consul + Envoy + apps
 
 ### What's there
 
 - three Consul servers (Raft quorum) on the coordinator CVMs, three
   clients on the worker CVMs
-- Connect enabled, default CA, allow intention webdemo→webdemo
+- Connect enabled, default CA, generated default-allow intentions per
+  parent service
 - Envoy sidecars front-running each webdemo and Patroni
-- gossip key NOT set; RPC TLS NOT set
+- gossip key set from Terraform-generated cluster material; RPC TLS
+  not set
 
 ### What can break
 
@@ -426,8 +430,8 @@ Honest answer: not really. Each layer earns its place.
 - **Per-peer loopback VIPs** are the *one* clever-and-ours
   technique, and they exist because Consul's own protocol assumes
   every peer can be reached at a stable address. The peer-VIP plane
-  gives us that stable address without modifying Consul; the static
-  infra-port allowlist on top keeps mesh-conn workload-agnostic.
+  gives us that stable address without modifying Consul; the runtime
+  allowlist on top keeps mesh-conn workload-agnostic.
 
 The risk concentration isn't in the count of layers; it's in the
 **single piece of code we wrote ourselves** (mesh-conn). That's

cluster-example/cluster.tf

@@ -188,8 +188,8 @@ locals {
   # Peer VIPs: 127.50.0.<vip>/32. Allocated as ordinal+1 (so vip=1
   # for the first coordinator, never 0 — the validate_test in
   # mesh-conn rejects vip=0). PEERS_JSON shape: [{id, vip}], same
-  # on every CVM. mesh-conn binds the static infra-port allowlist
-  # ({21000, 21001, 8300, 8301}) on every other peer's VIP.
+  # on every CVM. mesh-conn binds the runtime allowlist emitted by
+  # mesh-sidecar/entrypoint.sh on every other peer's VIP.
   peers = concat(
     [
       for i in range(var.coordinator_replicas) : {

design/README.md

@@ -1,8 +1,8 @@
 # Design docs — open work, intentionally separate from the user-facing docs
 
-This directory holds design briefs for **planned but not-yet-implemented**
-work on `consul-postgres-ha`. Each doc is structured so an agent (or a
-person) can pick it up cold and start implementing.
+This directory holds design briefs and handoff notes for open or
+partially landed work on `consul-postgres-ha`. Each doc is structured
+so an agent (or a person) can pick it up cold and continue.
 
 The user-facing docs (`README.md`, `ARCHITECTURE.md`, `FAILOVER.md`,
 `PUBLISHING.md`, `ROBUSTNESS.md`) describe what's *shipping today*.
@@ -12,8 +12,8 @@ in their face.
 
 | Doc | What |
 |---|---|
-| [`attestation-admission.md`](attestation-admission.md) | Use dstack TEE attestation as the mesh-conn admission credential, replacing/augmenting the shared TURN HMAC. Phased plan: per-app-id first, Consul-KV-rooted policy later. |
-| [`terraform-slot-api.md`](terraform-slot-api.md) | Integrate the Phala Cloud slot API into Terraform so mesh members have stable logical identities across CVM replacement and future attestation policy binding. |
+| [`attestation-admission.md`](attestation-admission.md) | Phase 1 admission is implemented. Remaining work: token lifecycle, non-NAT secret distribution, upgrade policy, and rendezvous hardening. |
+| [`terraform-slot-api.md`](terraform-slot-api.md) | Provider slot resources are implemented and validated. Remaining work: slot-preserving revision rollout through the Cloud redeploy endpoint. |
 
 Each doc includes:
 
@@ -25,5 +25,5 @@ Each doc includes:
 - Success criteria
 - Hand-off instructions
 
-When a doc's work lands, delete the doc (the implementation + the
-user-facing docs are the surviving artifacts).
+When all open work in a doc lands, delete the doc; the implementation
+and user-facing docs are the surviving artifacts.