From ad2082fa22add70ebab8f3966d1ba1d74a6f7a49 Mon Sep 17 00:00:00 2001 From: hsinhoyeh Date: Fri, 19 Jun 2026 14:03:26 +0800 Subject: [PATCH 1/2] docs: design for BYOC public HTTP ingress MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A box on a tunnel-joined BYOC host runs fine but its public subdomain returns an empty page — only the control path (peer-proxy) is wired; public app HTTP never crosses the tunnel to the box. This documents the gap and a phased design. Key finding: the transport already exists. The tunnel forwards :443 by default (pool join --ports 22,8080,443) and the sentinel SNI router already DialTunnels into a host for tunnel-PROMOTED primaries. BYOC ingress = teach the SNI router to resolve a BYOC box's subdomain to its host's tunnel and pass TLS through to the host's Caddy (which already terminates + routes the box). Missing links: an authoritative cloud→sentinel subdomain→host map (cloud is the authority, no host self-announce → anti-hijack), and a cert story (Phase 1: operator-host DNS-01; Phase 2: central cert sync or sentinel TLS-terminate + plaintext forward, gated on whether third-party BYOC ingress is in scope). Phase 1 is a path-only validation on an operator-owned lab host with zero new control APIs. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/BYOC-PUBLIC-INGRESS-DESIGN.md | 165 +++++++++++++++++++++++++++++ 1 file changed, 165 insertions(+) create mode 100644 docs/BYOC-PUBLIC-INGRESS-DESIGN.md diff --git a/docs/BYOC-PUBLIC-INGRESS-DESIGN.md b/docs/BYOC-PUBLIC-INGRESS-DESIGN.md new file mode 100644 index 00000000..2390478f --- /dev/null +++ b/docs/BYOC-PUBLIC-INGRESS-DESIGN.md @@ -0,0 +1,165 @@ +# BYOC public HTTP ingress (design) + +Status: draft / unapproved. Target: make a tenant box that runs on a +tunnel-joined BYOC host reachable on a public `*.example.com` subdomain +through the cloud edge — the same way a region-backend box already is. + +## Problem + +A box deployed onto a BYOC host (a tunnel-joined machine, reached via the +sentinel peer-proxy) runs fine and is fully reachable on its own host's LAN, +but its **public subdomain does not work**: + +- The cloud mints `-.example.com` and `GetWorkspaceAccess` hands it + back. DNS (wildcard) resolves it to the **sentinel/edge**. +- The edge has **no path to a box living on the BYOC host** (the host is behind + NAT, connected only by its outbound tunnel). The edge returns an empty `200`. +- Only the **control path** is wired today: `/peer//…` on the + sentinel forwards the *daemon API* to the host. There is no **data path** for + public app traffic (the box's HTTP port) to the box. + +A region-backend box works because the edge *is* its backend; a BYOC box is one +tunnel hop away and nothing carries public HTTP across it. + +## What already exists (the building blocks) + +The tunnel data-path is **already capable** — this design is mostly routing +metadata, not new transport. + +- **Tunnel multiplexes arbitrary ports.** The BYOC host dials the sentinel + outbound (yamux over `:443`); the sentinel can open a stream to any forwarded + port on the host. (`internal/sentinel/tunnel_client.go`, + `internal/sentinel/tunnel_server.go`.) +- **The sentinel SNI router already forwards public HTTPS into a tunnel** — for + hosts *promoted to primary*: it peeks the TLS SNI and, when the backend is a + tunnel, does `tunnelRegistry.DialTunnel(spotID, port)` + (`internal/sentinel/manager.go`, ~L865). This is the exact primitive BYOC + ingress needs; today it just isn't reachable for a non-primary BYOC box. +- **The BYOC host's daemon already runs Caddy and already gets the box route.** + The cloud's subdomain-route reconciler resolves the per-host (BYOC) driver and + calls `AddRoute` through the peer-proxy; the host's `RouteSyncJob` writes it + into the host's Caddy, with TLS. (`internal/app/route_sync.go`, + `internal/app/proxy.go`; cloud + `internal/sweeper/subdomain_route_reconciler.go`.) So the host's Caddy can + terminate TLS and reverse-proxy the subdomain to the box **once traffic + arrives**. + +## The missing links + +1. **The sentinel doesn't know `subdomain → BYOC host`.** Its SNI router + resolves a hostname to a *region primary* or a *promoted primary*. A BYOC + box's subdomain matches neither, so it falls through to the default (empty + `200`). +2. **The tunnel must forward the host's HTTPS port** (the host Caddy's `:443`) + so the sentinel can `DialTunnel(spotID, 443)`. +3. **TLS / cert ownership.** Either the BYOC host's Caddy terminates TLS for the + subdomain (needs a cert for `*.example.com` on the host), or the sentinel + terminates and forwards plaintext (needs an HTTP layer on the sentinel, which + it doesn't have today). +4. **The cloud-minted subdomain must match the route on the host.** The host's + daemon currently also mints its *own* `-workspace.example.com` route; + the cloud mints `-.example.com`. The route the reconciler pushes + must be the cloud-minted name (verify the reconciler targets the BYOC driver + for workspace subdomains). + +## Chosen approach: SNI pass-through to the host's Caddy + +Reuse the promoted-primary path. The sentinel peeks SNI; if the hostname maps to +a BYOC host, it `DialTunnel(spotID, 443)` and **passes the raw TLS bytes +through** to the host's Caddy, which terminates TLS and reverse-proxies to the +box. No TLS work on the sentinel, no new HTTP layer — identical to how a +tunnel-promoted primary is already served. + +``` +client ──TLS──> sentinel:443 + │ peek SNI = -.example.com + │ lookup → BYOC host spotID (NEW: authoritative map) + ▼ + DialTunnel(spotID, 443) ──yamux──> BYOC host Caddy:443 + terminates TLS (cert for the name) + reverse_proxy → box: (wsauth → app) +``` + +### The authoritative `subdomain → host` map (link 1) + +The **cloud is the authority** (it knows box → host placement); the host must +**not** self-announce subdomains (a compromised/abusive host could otherwise +hijack another tenant's hostname). So: + +- The cloud pushes BYOC subdomain bindings (`hostname → backend-id`) to the + sentinel as they're created/torn down — a small, defined API (proto-first, + per the API convention), not a hand-rolled handler. The sentinel keeps an + in-memory `byocRoute[hostname] = spotID` consulted by the SNI router before + the default fallback. +- Idempotent + reconciled (same shape as the existing route reconciler), so a + sentinel restart re-syncs. + +### Tunnel forwards `:443` (link 2) — already satisfied + +`pool join` defaults to `--ports 22,8080,443` (`internal/cmd/pool_join.go`), so +the host Caddy's HTTPS port is **already** carried by the tunnel — +`DialTunnel(spotID, 443)` will connect with no host-side change. (`8080` is also +forwarded, which enables the plaintext-forward cert variant in Phase 2.) + +### Cert strategy (link 3) — phased + +- **Phase 1 (operator-owned host, validate the path):** give the BYOC host's + Caddy DNS-01 capability for the wildcard (the same mechanism the edge uses), + so it serves a trusted cert for `-.example.com`. Acceptable when the + host is operator-owned (e.g. the lab GPU host). **Not** acceptable for a true + third-party BYOC host — handing it the DNS provider token is a credential leak. +- **Phase 2 (third-party BYOC):** the cloud mints the cert centrally and syncs + it to the host (a cert-push over the control channel), or the sentinel + terminates TLS with the wildcard it already holds and forwards **plaintext** + to the box port over the tunnel (`DialTunnel(spotID, )`, Host header + preserved). The plaintext-forward variant removes all cert handling from the + host but requires the sentinel to grow an HTTP reverse-proxy for tenant + subdomains — a larger sentinel change, deferred. + +### Cloud expose for BYOC (link 4) + +Confirm the subdomain-route reconciler, for a workspace/box on a BYOC host, +pushes the **cloud-minted** hostname (`-.example.com`) to the host's +daemon via the per-host driver (so the host Caddy terminates the right name). +The reconciler already resolves the BYOC driver for control ops; this is mostly +ensuring the workspace subdomain flows through the same path. + +## Security + +- **No host self-announce.** Only cloud-pushed mappings are honored; the + sentinel never trusts a host's claim to serve a hostname (anti-hijack). +- **Egress + cert.** In Phase 1 the operator host holds a DNS token — operator + hosts only. Phase 2 removes that for third-party hosts. +- **Isolation unchanged.** The box's own in-box auth proxy (basic-auth + + cookie) still gates access; ingress only carries traffic to it. + +## Phases + +1. **Path-only validation (operator host).** Phase-1 cert on the BYOC host + + manually register one `subdomain → spotID` mapping on the sentinel + ensure + the tunnel forwards `:443`. Confirm the workspace subdomain serves end to end + to a box on the lab host. Proves the SNI pass-through + tunnel:443 + host + Caddy chain with zero new control APIs. +2. **Authoritative map sync.** Define the cloud→sentinel binding-push API + (proto-first) + the sentinel `byocRoute` registry + SNI-router lookup. Wire + the cloud subdomain manager to push/retract on bind/unbind. +3. **Cloud expose alignment.** Ensure BYOC workspace subdomains push the + cloud-minted name through the per-host driver; reconcile on a loop. +4. **Third-party cert story (Phase 2 cert).** Central cert mint + sync, or + sentinel TLS-terminate + plaintext forward. Decision gated on whether + third-party BYOC ingress is in scope. + +## Validation target + +A throwaway box on the lab BYOC host, deployed from the console, reachable at +its `-.example.com` URL from a browser (the workspace iframe embeds), +with TLS valid and the in-box auth gate intact. Tear down after. + +## Open questions + +- ~~Default `pool join` forwarded-port set — does it already include `:443`?~~ + **Resolved: yes** (`22,8080,443`). +- Phase-1 vs Phase-2 cert: is third-party BYOC public ingress actually in scope, + or is operator-owned BYOC (lab/GPU hosts) enough for now? That decides whether + we need the central-cert/plaintext-forward work at all. +- Wildcard scope: one `*.example.com` cert on each BYOC host vs per-name certs. From 5f12d5adeae3b4a97ee02450e011bb99f5b590a5 Mon Sep 17 00:00:00 2001 From: hsinhoyeh Date: Fri, 19 Jun 2026 14:18:39 +0800 Subject: [PATCH 2/2] =?UTF-8?q?docs:=20Phase=201=20findings=20=E2=80=94=20?= =?UTF-8?q?cert=20wall=20flips=20BYOC=20ingress=20to=20sentinel-terminate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Walking the path on the lab BYOC host confirmed transport is fine (tunnel carries :443; host:443→box-Caddy is a trivial incus proxy device) but the box host's Caddy has no cert for the managed wildcard (no DNS-01 token, no public :80) → public TLS aborts. Terminating TLS on the host needs a cert/token per BYOC host — a credential leak for third-party hosts. So the primary model flips to: sentinel terminates TLS (holds the wildcard) + plaintext-forwards to the box port over the tunnel (8080 already forwarded); host stays cert-free. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/BYOC-PUBLIC-INGRESS-DESIGN.md | 45 ++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/docs/BYOC-PUBLIC-INGRESS-DESIGN.md b/docs/BYOC-PUBLIC-INGRESS-DESIGN.md index 2390478f..3e523116 100644 --- a/docs/BYOC-PUBLIC-INGRESS-DESIGN.md +++ b/docs/BYOC-PUBLIC-INGRESS-DESIGN.md @@ -133,6 +133,51 @@ ensuring the workspace subdomain flows through the same path. - **Isolation unchanged.** The box's own in-box auth proxy (basic-auth + cookie) still gates access; ingress only carries traffic to it. +## Phase 1 findings (validated on the lab BYOC host) + +Walking the path on a real BYOC host confirmed the transport and isolated the +one hard problem to **certs**: + +- **host `:443` → box Caddy**: trivial. The host-side box Caddy + (`core-caddy`) listens on an internal bridge IP, not the host loopback the + tunnel forwards; an `incus` proxy device (`listen 127.0.0.1:443 → core-caddy`) + bridges it. Reversible, no other-tenant impact. +- **tunnel carries `:443`**: already true (same mechanism as the working + peer-proxy on `:8080`). +- **TLS on the host: FAILS.** `core-caddy` has the box's route but **no cert** + for `*.example.com` — a standalone BYOC host has no DNS-01 token and no + public `:80` for HTTP-01, so the TLS handshake aborts (`tlsv1 alert internal + error`). The box's own HTTP works locally; only the public TLS terminator is + missing. + +**Consequence — the cert decision is the design, and it flips the model.** +Terminating TLS *on the host* would require shipping a `*.example.com` cert (or +the DNS provider token) to every BYOC host — fine for an operator-owned host, +a **credential leak for a true third-party host**. So: + +> **Primary model: the sentinel terminates TLS** (it already holds the wildcard) +> **and forwards plaintext to the box's port over the tunnel.** The BYOC host +> needs **no cert**. (`8080` is already tunnel-forwarded, so the plaintext hop +> is available today.) + +The host-terminate / SNI-passthrough variant remains valid **only** for +operator-owned hosts (deliver the cert) and is the quickest way to finish an +end-to-end demo on the lab host — but it is **not** the product model. The real +work item is the sentinel HTTP layer below. + +### Revised primary data path + +``` +client ──TLS──> sentinel:443 (terminates with the *.example.com wildcard it holds) + │ route by HTTP Host = -.example.com + │ lookup → BYOC host spotID (cloud-pushed map) + ▼ + DialTunnel(spotID, 8080-or-box-port) ──yamux──> box wsauth:8080 (plaintext) +``` + +The sentinel gaining an HTTP reverse-proxy for tenant subdomains (it does raw +TCP/SNI today) is the main new piece; the host stays cert-free and unchanged. + ## Phases 1. **Path-only validation (operator host).** Phase-1 cert on the BYOC host +