Support STUN-only demo bootstrap

Author: h4x3rotab

ARCHITECTURE.md

@@ -11,11 +11,11 @@ on any L3 path. Every CVM NATs out to the same public IP, so even a
 "direct" peer-to-peer flow is hairpinned by the provider edge. The
 only thing the CVMs share is outbound internet egress.
 
-Plus one plain Linux box with a public IP — the **external
-coordinator** — running `coturn` (STUN/TURN) and a tiny HTTP
-signaling broker. This is NAT traversal bootstrap infrastructure only:
-once peers have ICE-handshaked, no data passes through it unless TURN
-relay fallback is required.
+Plus one public NAT traversal bootstrap endpoint running STUN and a tiny
+HTTP signaling broker. This is setup infrastructure only: once peers
+have ICE-handshaked, no data passes through it. A self-hosted
+coordinator can additionally enable TURN relay fallback, but the public
+demo service should leave TURN disabled.
 
 ```
                       coturn + signaling

README.md

@@ -41,13 +41,16 @@ $EDITOR terraform.tfvars
 
 Fill in the Phala Cloud gateway and NAT traversal bootstrap settings in
 `terraform.tfvars`. The easiest first-run path should use a public demo
-bootstrap service operated by Phala Cloud. That service lets CVMs
-exchange connection setup messages and discover usable network paths; it
-should not carry application traffic in the normal case.
+bootstrap service operated by Phala Cloud. That demo service should run
+signaling plus STUN only. It lets CVMs exchange connection setup
+messages and discover usable network paths, but it does not provide TURN
+relay capacity and should not carry application traffic.
 
 If a public demo endpoint is not available in your environment yet, use
 the self-hosted coordinator in [`coordinator/`](coordinator/) and paste
-its `external_*` values.
+its `external_*` values. Leave `external_turn_secret` empty for the
+public STUN-only demo service; set it only for a self-hosted coordinator
+where you intentionally enable TURN fallback.
 
 Then deploy:
 
@@ -232,7 +235,8 @@ docker build -t service-mesh-mesh-sidecar-check -f mesh-sidecar/Dockerfile .
 ## Current Limits
 
 - The easiest user experience depends on a hosted NAT traversal
-  bootstrap endpoint.
+  bootstrap endpoint. The public demo endpoint should be STUN-only; TURN
+  relay is a self-hosted or future credentialed service.
   Until that is available in your environment, use the self-hosted
   coordinator fallback in [`coordinator/`](coordinator/).
 - Provider `0.3.0-beta.1` supports stable instance management, but

ROBUSTNESS.md

@@ -40,28 +40,26 @@ ours, they're the parts that *must* be made robust by hand.
 
 ### What's there
 
-- one public-IP host running `coturn` (STUN+TURN UDP/TCP) and a Go HTTP
-  signalling broker
+- one public-IP host running STUN and a Go HTTP signalling broker; a
+  self-hosted coordinator may also enable TURN relay fallback
 - the dstack CVMs themselves, which sit behind Phala's provider NAT
 
 ### What can break
 
 | failure                           | impact                                                                | recovery |
 | ---                               | ---                                                                   | ---      |
 | Coordinator host dies             | New peers can't bootstrap. **Existing ICE pairs keep working** (no data flows through this box once handshake is done). New retries from existing peers fail until it's back. | bring it back; peers reconnect on their own. |
-| Coordinator ufw / network change  | Same as above.                                                       | restore ports 3478/udp+tcp, 5349/tcp, 7000/tcp, 49152-49999/udp. |
-| TURN shared secret leaks          | Anyone can use the box as an open TURN relay (cost / abuse risk).    | rotate `TURN_SHARED_SECRET` in coordinator + every CVM env, redeploy. |
+| Coordinator firewall / network change | Same as above.                                                    | restore the signaling and STUN ports; if self-hosted TURN is enabled, also restore the TURN relay port range. |
+| TURN shared secret leaks          | Applies only when self-hosted TURN is enabled. Anyone with the secret can use that box as a relay. | rotate `TURN_SHARED_SECRET` in coordinator + every CVM env, redeploy. |
 | Signalling broker is unauthenticated | Any external actor can publish/poll messages, spoof candidates, intercept ICE handshakes. Currently low-impact only because we're solo. | gate `/publish` + `/poll` on attestation-derived identity. |
-| dstack provider NAT changes type (e.g. cone → symmetric) | ICE picks TURN relay path. ~150 ms RTT instead of ~6 ms. **Functionality unchanged.** | none needed; coturn covers this fallback. |
+| dstack provider NAT changes type (e.g. cone to symmetric) | With the public STUN-only demo endpoint, direct connection may fail instead of relaying. With self-hosted TURN enabled, ICE can pick the relay path. | use self-hosted TURN or a future credentialed relay service for environments that need relay fallback. |
 | Underlying CVM dies               | That peer's services drop out. Consul will mark it `failed` after gossip timeout, Envoy LB removes it within seconds. | redeploy; the rest of the cluster is unaffected. |
 
 ### Risk shape
 
-Coordinator host is a **single point of failure for NEW joins** and
-a SPOF for the TURN-relay fallback path. It is **NOT a SPOF for
-established traffic** — established peers ICE-direct and don't
-touch it. So dying coordinator = "no new peers can join, and any pair
-whose direct path goes down can't fail over to TURN until it's back".
+Coordinator host is a **single point of failure for NEW joins**. If
+self-hosted TURN is enabled, it is also a SPOF for the relay fallback
+path. It is **NOT a SPOF for established direct traffic**.
 
 ### Recommended fixes
 
@@ -348,10 +346,11 @@ service_healthy } }` would silence them entirely.
 
 ### Time drift
 
-TURN credentials are time-bound (1-hour TTL in our derivation). If a
-CVM clock drifts more than ~minutes from the coordinator's, TURN auth
-fails. dstack CVMs run NTP so this isn't a real concern, but worth
-noting for the runbook.
+TURN credentials are used only when self-hosted TURN fallback is
+enabled. They are time-bound (1-hour TTL in our derivation). If a CVM
+clock drifts more than a few minutes from the coordinator's, TURN auth
+fails. dstack CVMs run NTP so this is not a real concern, but worth
+noting for that runbook.
 
 ### Configuration drift / inconsistency
 

cluster-example/cluster.tf

@@ -144,6 +144,7 @@ variable "external_coordinator_host" { type = string }
 variable "external_signaling_url" { type = string }
 variable "external_turn_secret" {
   type      = string
+  default   = ""
   sensitive = true
 }
 

cluster-example/terraform.tfvars.example

@@ -22,13 +22,13 @@ mesh_sidecar_image = "ghcr.io/dstack-tee/service-mesh/mesh-sidecar:latest"
 webdemo_image      = "ghcr.io/dstack-tee/service-mesh/webdemo:latest"
 patroni_image      = "ghcr.io/dstack-tee/service-mesh/patroni:latest"
 
-# NAT traversal bootstrap settings. In the productized path these should
-# come from Phala Cloud managed service-mesh infrastructure. Until that
-# is available in your environment, run coordinator/docker-compose.yaml
-# on a public-IP host and use its values here.
+# NAT traversal bootstrap settings. The public demo bootstrap service
+# should run signaling + STUN only, with TURN relay disabled. Leave
+# external_turn_secret empty for that path. Set it only when using a
+# self-hosted coordinator that intentionally enables TURN fallback.
 # external_coordinator_host  = "<managed-or-self-hosted-bootstrap-host>"
 # external_signaling_url     = "http://<managed-or-self-hosted-bootstrap-host>:7000"
-# external_turn_secret       = "<hex>"
+# external_turn_secret       = ""
 
 # Attestation admission is enabled by default. Terraform generates the
 # Consul management token when consul_management_token is empty.

compose/coordinator.yaml

@@ -61,8 +61,8 @@ services:
       # for retry-join (skipping self).
       - COORDINATOR_VIPS=${COORDINATOR_VIPS}
       # External coordinator path: coordinator mesh-conn uses the same
-      # public coturn + signaling endpoint that workers do, so peer-pair
-      # NAT traversal setup happens in a single shared place.
+      # public bootstrap endpoint that workers do, so peer-pair NAT
+      # traversal setup happens in a single shared place.
       - SIGNALING_URL=${SIGNALING_URL}
       - TURN_HOST=${TURN_HOST}
       - TURN_SHARED_SECRET=${TURN_SHARED_SECRET}

mesh-conn/main.go

@@ -144,9 +144,9 @@ func loadConfig() *Config {
 	//   - SELF identity comes from /run/instance/info.json written by
 	//     the bootstrap-secrets init container (which read it from the
 	//     dstack SDK's Info() call). Falls back to PEER_ID env.
-	//   - TURN_SHARED_SECRET comes from /run/secrets/turn (a hex blob
-	//     written by bootstrap-secrets via getKey()). Falls back to
-	//     the env value if the file isn't present.
+	//   - TURN_SHARED_SECRET is optional. When present, mesh-conn also
+	//     tries TURN relay candidates. When absent, mesh-conn uses the
+	//     bootstrap host for STUN only and will not relay traffic.
 	//   - PEERS_JSON still comes via env — cluster.tf computes it from
 	//     the `replicas` count and re-applies on topology change,
 	//     which propagates to every CVM via Phala's in-place compose
@@ -196,7 +196,7 @@ func readSelfID() string {
 	return mustEnv("PEER_ID")
 }
 
-// readTurnSecret resolves the TURN shared secret in priority order:
+// readTurnSecret resolves the optional TURN shared secret in priority order:
 //
 //  1. TURN_SHARED_SECRET env (set when using an external coturn whose
 //     static-auth-secret was configured out-of-band). When this is
@@ -206,16 +206,26 @@ func readSelfID() string {
 //     coordinator's coturn which reads the same file).
 //
 // Order matters: env beats file so that "use external coturn" can be
-// configured purely at the cluster.tf layer.
+// configured purely at the cluster.tf layer. An explicitly empty env
+// value is valid and means STUN-only; no TURN relay credentials are
+// configured.
 func readTurnSecret() string {
-	if v := os.Getenv("TURN_SHARED_SECRET"); v != "" {
+	return readTurnSecretFrom("/run/secrets/turn")
+}
+
+func readTurnSecretFrom(file string) string {
+	if v, ok := os.LookupEnv("TURN_SHARED_SECRET"); ok {
+		if v == "" {
+			log.Printf("turn shared secret explicitly empty; using STUN-only bootstrap")
+			return ""
+		}
 		log.Printf("turn shared secret loaded from TURN_SHARED_SECRET env (%d bytes)", len(v))
 		return v
 	}
-	if b, err := os.ReadFile("/run/secrets/turn"); err == nil {
+	if b, err := os.ReadFile(file); err == nil {
 		s := strings.TrimSpace(string(b))
 		if s != "" {
-			log.Printf("turn shared secret loaded from /run/secrets/turn (%d bytes)", len(s))
+			log.Printf("turn shared secret loaded from %s (%d bytes)", file, len(s))
 			return s
 		}
 	}
@@ -1135,11 +1145,15 @@ func (m *Mesh) dialICE(remoteID string, sess *peerSession, attemptCtx context.Co
 	cfg := m.cfg
 	var urls []*stun.URI
 	if cfg.TurnHost != "" {
-		user, pass := turnCreds(cfg.TurnSecret, time.Hour)
 		urls = []*stun.URI{
 			{Scheme: stun.SchemeTypeSTUN, Host: cfg.TurnHost, Port: 3478, Proto: stun.ProtoTypeUDP},
-			{Scheme: stun.SchemeTypeTURN, Host: cfg.TurnHost, Port: 3478, Proto: stun.ProtoTypeUDP, Username: user, Password: pass},
-			{Scheme: stun.SchemeTypeTURN, Host: cfg.TurnHost, Port: 3478, Proto: stun.ProtoTypeTCP, Username: user, Password: pass},
+		}
+		if cfg.TurnSecret != "" {
+			user, pass := turnCreds(cfg.TurnSecret, time.Hour)
+			urls = append(urls,
+				&stun.URI{Scheme: stun.SchemeTypeTURN, Host: cfg.TurnHost, Port: 3478, Proto: stun.ProtoTypeUDP, Username: user, Password: pass},
+				&stun.URI{Scheme: stun.SchemeTypeTURN, Host: cfg.TurnHost, Port: 3478, Proto: stun.ProtoTypeTCP, Username: user, Password: pass},
+			)
 		}
 	}
 

mesh-conn/validate_test.go

@@ -2,6 +2,8 @@ package main
 
 import (
 	"net"
+	"os"
+	"path/filepath"
 	"strings"
 	"testing"
 	"time"
@@ -141,6 +143,38 @@ func TestValidateAllowlist_PortOutOfRange(t *testing.T) {
 	}
 }
 
+func TestReadTurnSecret_EmptyEnvDisablesFileFallback(t *testing.T) {
+	t.Setenv("TURN_SHARED_SECRET", "")
+	dir := t.TempDir()
+	path := filepath.Join(dir, "turn")
+	if err := os.WriteFile(path, []byte("file-secret\n"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	if got := readTurnSecretFrom(path); got != "" {
+		t.Fatalf("empty TURN_SHARED_SECRET should force STUN-only, got %q", got)
+	}
+}
+
+func TestReadTurnSecret_UsesFileWhenEnvUnset(t *testing.T) {
+	old, had := os.LookupEnv("TURN_SHARED_SECRET")
+	t.Cleanup(func() {
+		if had {
+			os.Setenv("TURN_SHARED_SECRET", old)
+		} else {
+			os.Unsetenv("TURN_SHARED_SECRET")
+		}
+	})
+	os.Unsetenv("TURN_SHARED_SECRET")
+	dir := t.TempDir()
+	path := filepath.Join(dir, "turn")
+	if err := os.WriteFile(path, []byte("file-secret\n"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	if got := readTurnSecretFrom(path); got != "file-secret" {
+		t.Fatalf("got %q", got)
+	}
+}
+
 func TestValidatePeers_DigestStableUnderReorder(t *testing.T) {
 	a := []Peer{
 		{ID: "ctrl", Vip: 1},