Remove relay-only mesh config

Author: h4x3rotab

consul-postgres-ha/ARCHITECTURE.md

@@ -307,8 +307,10 @@ recovery, congestion control, and an idle-timeout-driven liveness
 check. Crucially, it does not assume a reliable underlay — *unlike*
 yamux, which we tried first and gave up on. The earlier yamux build
 sustained ~3 KB on the dstack hairpin path before its keepalive /
-recv-window invariants tripped on dropped packets. QUIC sustains
-~25 MB/s on the same path.
+recv-window invariants tripped on dropped packets. QUIC gives us the
+loss recovery and congestion control needed for this underlay; see the
+real-env service mesh benchmark below for the current measured data
+point.
 
 ### Client / server roles
 
@@ -372,6 +374,46 @@ per ICE conn**, plus a 3-byte header per stream and a 2-byte length
 prefix per UDP datagram. That is the entirety of mesh-conn's wire
 format.
 
+### Real-env service mesh bandwidth
+
+On 2026-05-17 we measured a direct worker-to-worker service-mesh path
+on Phala Cloud `US-WEST-1` with:
+
+- 1 coordinator CVM + 2 worker CVMs, all `tdx.small`.
+- Default ICE candidate gathering, so direct candidates were available.
+- Worker-1 client -> local service VIP `127.10.0.13:5201` -> local
+  Envoy -> worker-2 peer VIP `127.50.0.3:21002` -> worker-2 Envoy ->
+  nginx bound on `127.0.0.1:5201`.
+- A Consul service-resolver pinned the benchmark service to
+  `peer=worker-2`, so the measured route did not use a local backend.
+
+The selected worker-to-worker ICE pair was direct UDP, not TURN relay:
+
+```text
+worker-1: host 10.0.2.10:55650 <-> srflx 66.220.6.105:50015 (proto=udp)
+worker-2: srflx 66.220.6.105:50015 <-> prflx 66.220.6.105:48294 (proto=udp)
+```
+
+Five 512 MiB HTTP downloads through the full Consul Connect path:
+
+| run | seconds | MiB/s | Mbps |
+| --- | ---: | ---: | ---: |
+| 1 | 8.499705 | 60.24 | 505.31 |
+| 2 | 7.780371 | 65.81 | 552.03 |
+| 3 | 7.347274 | 69.69 | 584.57 |
+| 4 | 7.439566 | 68.82 | 577.31 |
+| 5 | 8.585113 | 59.64 | 500.28 |
+
+Average: **64.84 MiB/s / 543.90 Mbps**. Median:
+**65.81 MiB/s / 552.03 Mbps**.
+
+The first benchmark attempt used `iperf3`, but Consul's generic TCP
+health check opens a plain TCP connection to the service port. That is
+valid for most TCP services, but `iperf3` treats the probe as a broken
+control session. For arbitrary TCP services whose accept path has
+protocol side effects, use a TTL check or a separate health endpoint
+instead of a generic TCP connect check.
+
 ## Trust boundaries
 
 - **App → Envoy** is plaintext on loopback. Same CVM, same TEE.

consul-postgres-ha/FAILOVER.md

@@ -166,11 +166,9 @@ detection (which sees worker-4's whole agent disappear, not just the
 Patroni lock) lines up with the Patroni leader-key TTL on this run,
 so neither signal extends the RTO.
 
-The post-restart rejoin path on dstack-worker pairs is occasionally
-flaky (the documented `MESH_CONN_RELAY_ONLY=1` escape hatch in
-`compose/worker.yaml` is exactly this case — flip it on if your
-deployment hits a wedged ICE re-handshake). The mesh-conn binary
-behavior is unchanged by the single-sidecar consolidation.
+The post-restart rejoin path on dstack-worker pairs used to be
+occasionally flaky when direct ICE re-handshake wedged. The mesh-conn
+binary behavior is unchanged by the single-sidecar consolidation.
 
 ### Things confirmed by the hard-kill that the soft-kill didn't exercise
 
@@ -213,9 +211,10 @@ T_streaming    17:35:43   streaming WAL on timeline 4, lag=0            +82s tot
 ```
 
 A few-MB pgdata transferred in ~18 seconds end-to-end. The dataset
-is small enough that handshake/startup overhead dominates — for a
-realistic throughput number, see the soft-kill section's pg_basebackup
-trace at ~25 MB/s sustained on the QUIC path.
+is small enough that handshake/startup overhead dominates, so this
+trace should be read as a recovery-path check rather than a bandwidth
+benchmark. For a current full service-mesh datapoint, see
+`ARCHITECTURE.md`.
 
 The path itself is the proof point: Patroni correctly detects empty
 pgdata, picks `bootstrap from leader` (not WAL replay), pulls the full

consul-postgres-ha/README.md

@@ -293,7 +293,7 @@ the platform plumbing untouched.
 | In-place env updates | Yes — change image tags or env values, `terraform apply`, CVMs update without losing pgdata. Requires provider `phala-network/phala 0.2.0-beta.3+`. |
 | Failover RTO | ~24s soft-kill, ~33s hard-kill (default Patroni `ttl=30`). See [`FAILOVER.md`](FAILOVER.md). |
 | Cheap rejoin | Yes — a recovered ex-leader replays local WAL and rejoins as a streaming replica without pg_basebackup. |
-| Disk-loss rejoin | Yes — Patroni detects empty pgdata, runs full pg_basebackup over the QUIC overlay (~25 MB/s sustained between dstack workers). |
+| Disk-loss rejoin | Yes — Patroni detects empty pgdata and runs full pg_basebackup over the QUIC overlay. |
 | Build provenance | Sigstore-attested via GitHub Build Provenance on every published image. Verify with `gh attestation verify oci://... --repo Dstack-TEE/dstack-examples`. |
 
 ## Known limitations

consul-postgres-ha/ROBUSTNESS.md

@@ -162,13 +162,11 @@ whose direct path goes down can't fail over to TURN until it's back".
   sustained-forge variant gated on the `onAttemptStarted` hook,
   which guarantees the forge lands inside the in-flight window.
 
-- **pion DisconnectedTimeout / FailedTimeout headroom for the
-  TURN-relay path.** With pion's defaults (5 s + 25 s = 30 s of no
-  inbound from the selected pair before the state callback fires
-  Failed), the relay-only path on `MESH_CONN_RELAY_ONLY=1` clusters
-  flapped on its own roughly every couple of minutes: brief
-  jitter / refresh-window noise at coturn was enough to accumulate
-  a 30 s consent-freshness gap, pion declared Failed,
+- **pion DisconnectedTimeout / FailedTimeout headroom.** With pion's
+  defaults (5 s + 25 s = 30 s of no inbound from the selected pair
+  before the state callback fires Failed), lossy or high-jitter selected
+  pairs can flap on their own: brief jitter / refresh-window noise is
+  enough to accumulate a 30 s consent-freshness gap, pion declares Failed,
   `OnConnectionStateChange` fired `abortAttempt`, QUIC's
   `accept`/pumps came back with `context canceled`, and `runPeerLink`
   re-established the link 2–5 s later. The link-pair recovered every
@@ -194,9 +192,9 @@ whose direct path goes down can't fail over to TURN until it's back".
   Regression net: `mesh-conn/main_test.go::TestSteadyStateNoFlap` —
   loopback two-peer steady-state observation. Default window 5 min;
   `STEADY_STATE_OBSERVE_SECONDS=N` overrides for the CI 50-trial
-  sweep at 30 s each (passes 50/50). Live verification: 6-CVM
-  relay-only cluster observed for 15+ min post-rollout with zero
-  `ice state: Failed` events on any peer's sidecar log.
+  sweep at 30 s each (passes 50/50). Live verification: a 6-CVM
+  cluster was observed for 15+ min post-rollout with zero `ice state:
+  Failed` events on any peer's sidecar log.
 
 ## Layer 2 — mesh-conn forwarder
 

consul-postgres-ha/cluster-example/cluster.tf

@@ -107,15 +107,6 @@ variable "external_turn_secret" {
   sensitive = true
 }
 
-# Force ICE to gather Relay candidates only — routes all peer traffic
-# through coturn instead of attempting NAT-hairpin direct paths. Set
-# this when worker↔worker direct-pair ICE handshakes are unstable
-# (the dstack provider NAT path is known-flaky for these pairs).
-variable "mesh_conn_relay_only" {
-  type    = string
-  default = ""
-}
-
 # Set to "1" to enable pion's verbose ICE debug logs (connectivity
 # checks, STUN attribute parsing). Useful when debugging hot-patch
 # / re-handshake flakiness; off by default because it's very chatty.
@@ -328,7 +319,6 @@ locals {
       GOSSIP_KEY              = random_bytes.gossip_key.base64
       PATRONI_SUPERUSER_PW    = random_bytes.patroni_superuser_pw.hex
       PATRONI_REPLICATION_PW  = random_bytes.patroni_replication_pw.hex
-      MESH_CONN_RELAY_ONLY    = var.mesh_conn_relay_only
       MESH_CONN_DEBUG_ICE     = var.mesh_conn_debug_ice
       ADMISSION_BROKER_ENABLE = var.enable_attestation_admission ? "1" : ""
       CONSUL_MANAGEMENT_TOKEN = var.consul_management_token
@@ -349,7 +339,6 @@ locals {
       MESH_SIDECAR_IMAGE      = var.mesh_sidecar_image
       DSTACK_VERIFIER_IMAGE   = var.dstack_verifier_image
       GOSSIP_KEY              = random_bytes.gossip_key.base64
-      MESH_CONN_RELAY_ONLY    = var.mesh_conn_relay_only
       MESH_CONN_DEBUG_ICE     = var.mesh_conn_debug_ice
       ADMISSION_BROKER_ENABLE = var.enable_attestation_admission ? "1" : ""
       # The env key is part of the compose allowlist, but the policy

consul-postgres-ha/compose/coordinator.yaml

@@ -69,8 +69,6 @@ services:
       # WORKAROUND: gossip key generated in Terraform and broadcast
       # identically to every CVM. See cluster.tf.
       - GOSSIP_KEY=${GOSSIP_KEY}
-      # See worker.yaml for the rationale on MESH_CONN_RELAY_ONLY.
-      - MESH_CONN_RELAY_ONLY=${MESH_CONN_RELAY_ONLY:-}
       - MESH_CONN_DEBUG_ICE=${MESH_CONN_DEBUG_ICE:-}
       - ADMISSION_BROKER_ENABLE=${ADMISSION_BROKER_ENABLE:-}
       - ADMISSION_POLICY_JSON=${ADMISSION_POLICY_JSON:-}

consul-postgres-ha/compose/worker.yaml

@@ -52,11 +52,6 @@ services:
       - GOSSIP_KEY=${GOSSIP_KEY}
       - ADMISSION_BROKER_ENABLE=${ADMISSION_BROKER_ENABLE:-}
       - CONSUL_MANAGEMENT_TOKEN=${CONSUL_MANAGEMENT_TOKEN:-}
-      # MESH_CONN_RELAY_ONLY=1 forces ICE to gather only Relay
-      # candidates, routing all peer traffic through the coturn server.
-      # Default off because direct candidates work; flip on if a
-      # deployment hits worker↔worker direct-pair instability.
-      - MESH_CONN_RELAY_ONLY=${MESH_CONN_RELAY_ONLY:-}
       - MESH_CONN_DEBUG_ICE=${MESH_CONN_DEBUG_ICE:-}
     volumes:
       - /var/run/dstack.sock:/var/run/dstack.sock:ro

consul-postgres-ha/mesh-conn/main.go

@@ -55,8 +55,8 @@
 // the root cause is dropped packets. QUIC has built-in loss recovery,
 // congestion control, and stream-multiplexing — it's exactly what a
 // lossy UDP underlay needs. The previous yamux build died at 3KB-260KB
-// depending on path; the QUIC build sustains 25-28 MB/s on the same
-// hairpin.
+// depending on path; current service-mesh bandwidth is documented in
+// ARCHITECTURE.md from real-env benchmarks.
 
 package main
 
@@ -1115,37 +1115,26 @@ func (m *Mesh) dialICE(remoteID string, sess *peerSession, attemptCtx context.Co
 		}
 	}
 
-	// MESH_CONN_RELAY_ONLY=1 restricts candidate gathering to Relay only.
-	// Use when direct (host/srflx/prflx) connectivity is unreliable — e.g.
-	// dstack worker-to-worker pairs where pion's connectivity check fails
-	// for every direct pair and the agent never gets to relay before
-	// timing out. Trades latency for guaranteed reachability via coturn.
 	candidateTypes := []ice.CandidateType{
 		ice.CandidateTypeHost,
 		ice.CandidateTypeServerReflexive,
 		ice.CandidateTypePeerReflexive,
 		ice.CandidateTypeRelay,
 	}
-	if os.Getenv("MESH_CONN_RELAY_ONLY") == "1" {
-		candidateTypes = []ice.CandidateType{ice.CandidateTypeRelay}
-	}
 	// MESH_CONN_DEBUG_ICE=1 turns on pion's verbose ICE-level logging
 	// (connectivity-check requests/responses, STUN attribute parsing,
 	// candidate-pair scoring). Off by default because it's chatty.
 	//
 	// Disconnected/Failed timeouts: pion defaults to 5 s + 25 s (= 30 s
 	// of no inbound from the selected pair before the state callback
 	// fires Failed and our outer code aborts the attempt). That is too
-	// tight for the TURN-relay path. The relay leg adds ~150 ms RTT,
-	// the periodic permission/allocation refreshes can briefly
+	// tight for high-jitter selected pairs. TURN relay candidates add
+	// RTT, the periodic permission/allocation refreshes can briefly
 	// reorder packet flow at coturn, and pion's own 2 s consent
-	// freshness has no retry budget — a sub-second jitter window
-	// where 1 STUN binding gets lost and another arrives late is
-	// enough to start the disconnected clock, and consecutive jitter
-	// events accumulate against the 30 s budget. We've observed the
-	// resulting "Connected → Disconnected → Failed → Checking →
-	// Connected" cycle every couple of minutes on freshly-deployed
-	// relay-only clusters, with no actual peer death behind it.
+	// freshness has no retry budget — a sub-second jitter window where
+	// 1 STUN binding gets lost and another arrives late is enough to
+	// start the disconnected clock, and consecutive jitter events
+	// accumulate against the 30 s budget.
 	//
 	// The fix is to give pion enough headroom that only a genuine
 	// relay-path outage (rather than refresh / jitter noise) trips

consul-postgres-ha/quic-on-ice/main.go

@@ -58,13 +58,12 @@ func main() {
 	turnHost := flag.String("turn-host", "", "TURN host (e.g. 155.138.146.255)")
 	turnSecret := flag.String("turn-secret", "", "TURN HMAC shared secret")
 	mb := flag.Int("mb", 10, "MB to transfer")
-	relayOnly := flag.Bool("relay-only", false, "force ICE relay-only candidates")
 	flag.Parse()
 	if *role == "" || *peerID == "" || *signal == "" {
 		log.Fatalf("need -role, -peer, -signal")
 	}
 
-	conn, err := dialICE(*role, *peerID, *signal, *turnHost, *turnSecret, *relayOnly)
+	conn, err := dialICE(*role, *peerID, *signal, *turnHost, *turnSecret)
 	if err != nil {
 		log.Fatalf("ice dial: %v", err)
 	}
@@ -125,8 +124,8 @@ func runQUICClient(pkt net.PacketConn, peerID string, mb int) {
 	// of "where to send" matches what the PacketConn does anyway.
 	remote := pkt.LocalAddr() // any addr — pkt.WriteTo ignores it
 	conn, err := quic.Dial(ctx, pkt, remote, tlsConf, &quic.Config{
-		KeepAlivePeriod:        5 * time.Second,
-		MaxIdleTimeout:         60 * time.Second,
+		KeepAlivePeriod:                5 * time.Second,
+		MaxIdleTimeout:                 60 * time.Second,
 		InitialStreamReceiveWindow:     4 << 20,
 		MaxStreamReceiveWindow:         16 << 20,
 		InitialConnectionReceiveWindow: 8 << 20,
@@ -264,7 +263,7 @@ var (
 	sessions   = map[string]*peerSession{}
 )
 
-func dialICE(self, remote, signalURL, turnHost, turnSecret string, relayOnly bool) (*ice.Conn, error) {
+func dialICE(self, remote, signalURL, turnHost, turnSecret string) (*ice.Conn, error) {
 	var urls []*stun.URI
 	if turnHost != "" {
 		user, pass := turnCreds(turnSecret, time.Hour)
@@ -280,9 +279,6 @@ func dialICE(self, remote, signalURL, turnHost, turnSecret string, relayOnly boo
 		ice.CandidateTypePeerReflexive,
 		ice.CandidateTypeRelay,
 	}
-	if relayOnly {
-		candTypes = []ice.CandidateType{ice.CandidateTypeRelay}
-	}
 	agent, err := ice.NewAgent(&ice.AgentConfig{
 		Urls:           urls,
 		NetworkTypes:   []ice.NetworkType{ice.NetworkTypeUDP4, ice.NetworkTypeTCP4},