Skip to content

Commit ff9522a

Browse files
authored
Merge pull request #1 from TinyCloudLabs/feat/tunnel-relay
TC-85: tunnel relay MVP -- remote reachability for local nodes (+ TC-244 batch)
2 parents edde906 + 7d7834d commit ff9522a

20 files changed

Lines changed: 1663 additions & 38 deletions

.env.example

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,3 +21,15 @@ ACME_PROPAGATION_TIMEOUT_MS=
2121

2222
# TEE attestation document (Phala dstack), optional outside of TEE deployments
2323
ATTESTATION_DOCUMENT=
24+
25+
# The control-plane API's own hostname. Requests whose Host header is anything
26+
# else under *.tinycloud.link are routed through the matching tunnel (TC-85)
27+
# instead of the /v1 API. Defaults to api.tinycloud.link.
28+
API_HOSTNAME=api.tinycloud.link
29+
30+
# Max bytes accepted for a single proxied tunnel request body. Defaults to
31+
# 26214400 (25MB).
32+
TUNNEL_MAX_BODY_BYTES=
33+
# Max concurrently registered (authenticated) tunnels; further connection
34+
# attempts are dropped once at the cap. Defaults to 1000.
35+
TUNNEL_MAX_CONCURRENT=

README.md

Lines changed: 84 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,8 @@ PUT /v1/names/:name
5252
DELETE /v1/names/:name
5353
5454
POST /v1/certs/:name
55+
56+
WS /v1/tunnel/:name (see "Remote reachability: the tunnel relay" below)
5557
```
5658

5759
### `PUT /v1/names/:name`
@@ -97,11 +99,91 @@ Public, unsigned. Returns `{ name, subject, lanIps, updatedAt }` or `404`.
9799
```
98100

99101
- Verifies the requester owns `name` and the signature is valid.
100-
- The CSR's CN and every SAN entry must be **exactly** `<name>.local.tinycloud.link` -- no wildcards, no extra names.
102+
- The CSR's CN and every SAN entry must be **exactly** `<name>.local.tinycloud.link` -- no wildcards, no extra names. The CSR's key may be RSA or ECDSA (P-256); both are validated identically (`src/names.ts`'s `assertCsrMatchesDomain` parses with `@peculiar/x509`, not node-forge, specifically so ECDSA node keys aren't rejected).
101103
- Rate-limited to 5 issuances/day per name to protect Let's Encrypt's rate limits.
104+
- **Sequence is bumped before the ACME round trip, not after.** Once the request passes ownership/signature/CSR checks, the handler immediately persists the new `sequence` on the name record and only then calls the ACME issuer. This is deliberate: an ACME DNS-01 order is slow (DNS propagation + CA validation, seconds to tens of seconds) and not idempotent to retry blindly, so the sequence bump has to happen *before* it, not after it succeeds. That way a replayed or concurrent request carrying the same (now-stale) sequence number is rejected with `409` immediately instead of racing a second ACME order for the same name while the first is still in flight. The tradeoff: if the ACME call itself fails after the bump (rate limit, DNS-01 timeout, CA outage), the sequence has still moved forward and the node must retry with a strictly higher sequence, not the same one -- there is no "undo" of the bump on failure.
102105
- Opens an ACME order against `ACME_DIRECTORY` (Let's Encrypt staging by default), completes it via DNS-01 through the configured `DnsProvider`, finalizes with the node's CSR, and returns `{ certChainPem, notAfter }`.
103106
- The private key backing the CSR is generated and held by the node. It is never transmitted to or stored by this service.
104107

108+
## Remote reachability: the tunnel relay
109+
110+
`*.local.tinycloud.link` only works on the node's own LAN. The tunnel relay (TC-85) adds a second, **remote** namespace for the same names: `https://<name>.tinycloud.link` (the apex zone, no `.local.`) is reachable from anywhere on the internet and is served by this process relaying each HTTPS request down a WebSocket the node itself opened. The node never needs an inbound port, a public IP, or NAT configuration -- it dials **out** to the relay and keeps that one socket alive.
111+
112+
There is one name registry with two surfaces: a name claimed via `PUT /v1/names/:name` gets LAN A/AAAA records (`<name>.local.tinycloud.link`) *and* the right to open a tunnel for `<name>.tinycloud.link`. Same owner, same signing key, same sequence counter.
113+
114+
**Sequence coordination is per-name, not per-operation**: `claim`/`delete` (`PUT`/`DELETE /v1/names/:name`), `cert` (`POST /v1/certs/:name`), and `tunnel` (the WS auth frame) all read and bump the *same* `sequence` column on the name's row in `names`. There is exactly one counter per name, shared across every action type -- a node must treat it as a single source of truth (e.g. keep one in-memory counter per name and increment it before every signed write, regardless of which endpoint), not maintain a separate counter per action. Reusing a sequence across two different action types for the same name is exactly as stale/rejected as reusing it for the same action twice.
115+
116+
### Lifecycle
117+
118+
```
119+
node relay (this service)
120+
| |
121+
| 1. WebSocket connect |
122+
|----- wss://api.tinycloud.link/v1/tunnel/<name> ---->|
123+
| |
124+
| 2. auth frame (first WS message, JSON) |
125+
|-------------------------------------------->| verifies: name is claimed,
126+
| {version, action:"tunnel", name, | subject owns it, sequence >
127+
| subject, sequence, signature} | stored, signature valid;
128+
| | then persists the new sequence
129+
| 3. ack frame |
130+
|<--------------------------------------------| {"type":"ack"} tunnel is live
131+
| |
132+
| ... for each HTTPS request to |
133+
| https://<name>.tinycloud.link ... |
134+
| |
135+
| 4. request + requestBody frames |
136+
|<--------------------------------------------|
137+
| 5. response + responseBody frames |
138+
|-------------------------------------------->|
139+
| |
140+
| (relay pings every 30s; the node's WS |
141+
| stack must answer pongs or the socket |
142+
| is terminated as dead) |
143+
```
144+
145+
- **Auth** (`src/names.ts` `TunnelAuthRecord`, enforced in `src/tunnel/upgrade.ts`): the first WS message must be a JSON record signed exactly like every other write -- canonical payload `{"version":1,"action":"tunnel","name":...,"subject":...,"sequence":...}` (fixed key order, `signature` field excluded from the signed bytes), same did:pkh (EIP-191) / did:key (Ed25519, base64url signature) schemes, `sequence` strictly greater than the name record's stored sequence. On success the stored sequence is bumped (same bump-before-side-effect pattern as the cert flow) and the relay sends `{"type":"ack"}`. The node has 5 seconds from socket open to deliver the auth frame.
146+
- **Close codes on rejection** (RFC 6455 private-use range): `4400` malformed auth frame or frame `name` != URL name, `4401` invalid signature, `4403` subject does not own the name, `4404` name not claimed, `4408` auth timeout, `4409` stale sequence, `4410` superseded by a newer connection for the same name.
147+
- **One socket per name, newest wins** (`src/tunnel/registry.ts`): a second authenticated connection for the same name evicts the first (close `4410`) rather than being rejected -- a node reconnecting after a network blip must be able to take over from its own half-dead previous socket. (Each reconnect needs a fresh auth frame with a higher sequence.)
148+
- **Keep-alive**: the relay pings every 30s and terminates the socket if the previous ping got no pong. Standard WS libraries answer pings automatically; a Rust client must confirm its library does (tungstenite does).
149+
- **Connection-attempt limits** (`src/tunnel/upgrade.ts`, checked in the raw HTTP `upgrade` handler, before the WS handshake and before any Postgres read): at most 30 upgrade attempts per remote IP per minute, at most 10 upgrade attempts per tunnel name per minute (bounds reconnect/eviction churn on one name), and a global cap of 1000 concurrently registered tunnels (env-tunable via `TUNNEL_MAX_CONCURRENT`, see below). An attempt over any of these limits gets the raw TCP socket destroyed -- no WS handshake, no close code, nothing for the node to parse.
150+
151+
### Frame protocol
152+
153+
Defined in `src/tunnel/protocol.ts` -- that file is the source of truth for the Rust node client. Every frame is one JSON **text** message (binary WS messages are never used); body bytes travel base64-encoded inside the JSON. After the ack, the frames are:
154+
155+
| frame | direction | fields |
156+
|---|---|---|
157+
| `request` | relay → node | `id` (UUID string), `method`, `path` (path + query, always starts with `/`), `headers` (ordered `[name, value]` pairs; `Host` and hop-by-hop headers stripped) |
158+
| `requestBody` | relay → node | `id`, `chunk` (base64, may be `""`), `done` (bool) |
159+
| `response` | node → relay | `id`, `status` (int), `headers` (ordered `[name, value]` pairs) |
160+
| `responseBody` | node → relay | `id`, `chunk` (base64, may be `""`), `done` (bool) |
161+
| `error` | either direction | `message`, optional `id` (fails just that request without closing the socket) |
162+
163+
Rules a node client must follow:
164+
165+
- Requests are multiplexed: frames for different `id`s interleave freely on the one socket. Echo the request's `id` on every frame you send for it. A frame carrying an `id` the relay/node doesn't recognize (e.g. arriving after that request already failed or timed out) is silently ignored, never treated as a protocol error.
166+
- **Headers are ordered `[name, value]` pairs, not an object** -- so a header repeated on the wire (most importantly `Set-Cookie`, but any header) survives as multiple array entries instead of colliding on one object key. A client must be able to both emit and parse duplicate entries for the same header name.
167+
- The relay sends `request` first, then one or more `requestBody` frames ending with `done: true`; the node replies with exactly one `response` frame, then one or more `responseBody` frames ending with `done: true`. An empty body is still exactly one frame: `{"...Body","id":...,"chunk":"","done":true}`.
168+
- **Size limits**: no single WS frame may exceed `MAX_FRAME_PAYLOAD_BYTES` (1MB) -- the relay's `WebSocketServer` is configured with this as `maxPayload` and will close a connection that sends a larger one (WS close code `1009`). A body larger than `BODY_CHUNK_BYTES` (256KB, chosen to stay well under 1MB once base64-inflated) must be split across multiple body frames; only the last carries `done: true`. The relay itself follows this when sending `requestBody` frames and expects a node to do the same for `responseBody`. Independent of frame size, the full reassembled body (request or response) is capped at `DEFAULT_MAX_BODY_BYTES` (25MB, overridable via the `TUNNEL_MAX_BODY_BYTES` env var, see below): a request over the cap gets `413` directly from the relay before it ever reaches the tunnel; a response over the cap is aborted with `502` to the HTTP caller and an `{"type":"error","id":...}` frame is sent back to the node so it knows to stop sending.
169+
- If the node cannot serve a request it sends `{"type":"error","id":...,"message":...}`; the relay answers the HTTP caller with `502`. The relay applies a 30s per-request timeout (caller gets `504`).
170+
- HTTP callers with no live tunnel for their Host get `502` directly from the relay.
171+
172+
### Ingress and TLS for tunnels
173+
174+
TLS for `https://<name>.tinycloud.link` terminates **at the relay's front door, with one wildcard certificate** -- not with per-name certificates, and not inside this Node process.
175+
176+
The constraint that forces this: on Phala, this service runs behind `dstack-ingress` (see `docker-compose.phala.yml`), an HAProxy-based L4 TCP proxy that terminates TLS itself using its own certbot/DNS-01 machinery and forwards the decrypted stream to `TARGET_ENDPOINT`. Everything arriving on the CVM's public `:443` goes through it. From inside the CVM there is no per-SNI certificate hook: this repo's process never sees the TLS handshake, and `dstack-ingress` serves the certificates *it* obtained at bootstrap, not ones handed to it at runtime by another container. So the "extend the ACME issuer to obtain a cert for each `<name>.tinycloud.link` on tunnel registration" design is a dead end in this deployment model -- this process could *obtain* those certs (it already has the DNS-01 machinery), but nothing here could ever *serve* them. Design honesty: we don't ship per-name relay certs.
177+
178+
What ships instead:
179+
180+
- `dstack-ingress` runs with `DOMAIN=*.tinycloud.link` -- it supports wildcard domains natively (wildcard DNS-01 order, `issuewild` CAA). One certificate covers `api.tinycloud.link` and every tunnel name.
181+
- DNS needs one wildcard record: `*.tinycloud.link` → the same dstack gateway target `api.tinycloud.link` already points at (human step, same as the original `api` wiring). Tunnels create **no** per-name DNS records; the per-name records this service writes remain what they always were -- LAN A/AAAA under `*.local.tinycloud.link`.
182+
- Behind the ingress, this process routes by `Host` header (`src/tunnel/host-router.ts`): `API_HOSTNAME` (default `api.tinycloud.link`) gets the normal `/v1` API; any other single-label `<name>.tinycloud.link` host is looked up in the tunnel registry and proxied.
183+
- **Open verification item**: dstack-ingress documents that wildcard `DOMAIN` requires dstack-gateway wildcard TXT resolution support ([dstack#545](https://github.com/Dstack-TEE/dstack/pull/545)). Whether the production gateway has it can't be verified from this repo. If it doesn't, the wildcard ACME order fails at ingress bootstrap; the fallback is `DOMAIN=api.tinycloud.link` (control-plane API keeps working exactly as today, tunnels stay dark) until the gateway supports it. Must be checked on a staging CVM before the tunnel-enabled image is deployed.
184+
185+
Trade-offs accepted: all tunnel names share one certificate (one CT-log entry for `*.tinycloud.link` rather than a per-name entry -- strictly *less* name disclosure than the LAN cert flow, which CT-logs every name); and tunnel TLS terminates in the ingress container rather than end-to-end at the node, meaning plaintext HTTP crosses the CVM-internal hop between ingress and this process -- the same boundary the control-plane API traffic has always crossed.
186+
105187
## DNS provider
106188

107189
`src/dns/provider.ts` defines the `DnsProvider` interface (`upsertAddressRecords`, `deleteAddressRecords`, `createTxtRecord`, `deleteTxtRecord`). `src/dns/cloudflare.ts` is the production implementation against the Cloudflare API (zone id + token from env). `src/dns/memory.ts` is an in-memory fake used by tests -- no test ever makes a live DNS or ACME call.
@@ -152,7 +234,7 @@ docker run -p 3000:3000 --env-file .env tinycloud/tinycloud-link
152234
- `ACME_EMAIL`
153235
- `CERTBOT_EMAIL` (used by `dstack-ingress` for the API's own front-door cert)
154236
- `DSTACK_GATEWAY_DOMAIN`
155-
4. Point `api.tinycloud.link` at the deployment's dstack gateway (human DNS step, same pattern as `registry.tinycloud.xyz`).
237+
4. Point `api.tinycloud.link` at the deployment's dstack gateway (human DNS step, same pattern as `registry.tinycloud.xyz`). For tunnels (TC-85), also point the wildcard `*.tinycloud.link` at the same gateway target and confirm the gateway supports wildcard TXT resolution for the ingress's wildcard cert order -- see "Ingress and TLS for tunnels" above.
156238
5. Verify:
157239
```bash
158240
curl https://api.tinycloud.link/health

docker-compose.phala.yml

Lines changed: 30 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,38 @@
11
services:
2-
# Public HTTPS ingress for the control-plane API itself. This is separate
3-
# from the *.local.tinycloud.link zone the service manages for nodes: the
4-
# API is reachable at a normal public hostname (api.tinycloud.link), not a
5-
# LAN-only name. Wiring api.tinycloud.link's DNS to this deployment's
6-
# dstack gateway is a human step -- see README.md "Deploy to Phala".
2+
# Public HTTPS ingress, for BOTH the control-plane API itself
3+
# (api.tinycloud.link) and every tunnel name (<name>.tinycloud.link, the
4+
# TC-85 remote-reachability namespace -- distinct from the LAN-only
5+
# *.local.tinycloud.link zone the service also manages via DNS, which never
6+
# goes through this ingress).
7+
#
8+
# dstack-ingress is an HAProxy-based L4 (TCP) proxy: it terminates TLS with
9+
# its own ACME DNS-01 flow and forwards the decrypted stream to
10+
# TARGET_ENDPOINT untouched, Host header and all. It natively supports a
11+
# WILDCARD `DOMAIN` (one ACME order for *.tinycloud.link instead of one
12+
# order per name), which is why DOMAIN is a wildcard below rather than the
13+
# single api.tinycloud.link host used before TC-85: any single-label
14+
# subdomain of tinycloud.link -- api.tinycloud.link, mynode.tinycloud.link,
15+
# anything a node claims -- is covered by the same cert and forwarded to
16+
# the same tinycloud-link process, which then routes by Host header
17+
# (api.tinycloud.link -> the normal /v1 API; anything else -> the matching
18+
# tunnel, see src/tunnel/host-router.ts). No per-name ACME order, no TLS
19+
# termination inside this repo's process at all.
20+
#
21+
# CONSTRAINT (unverified, must be confirmed before this is deployed):
22+
# dstack-ingress's own docs say wildcard DOMAIN "requires dstack-gateway
23+
# with wildcard TXT resolution support" (Dstack-TEE/dstack#545). Whether
24+
# this deployment's dstack-gateway includes that support cannot be checked
25+
# from this repo/sandbox. If it doesn't, the ACME order for *.tinycloud.link
26+
# will fail at bootstrap and this config falls back to the single-domain
27+
# form (`DOMAIN=api.tinycloud.link`, i.e. no change from before TC-85) --
28+
# see README.md "Ingress and TLS for tunnels" for that fallback and why a
29+
# per-name-cert-at-the-relay design was considered and rejected.
730
dstack-ingress:
831
image: dstacktee/dstack-ingress:20250924@sha256:40429d78060ef3066b5f93676bf3ba7c2e9ac47d4648440febfdda558aed4b32
932
ports:
1033
- "443:443"
1134
environment:
12-
- DOMAIN=api.tinycloud.link
35+
- DOMAIN=*.tinycloud.link
1336
- TARGET_ENDPOINT=http://tinycloud-link:3000
1437
- CLOUDFLARE_API_TOKEN=${CLOUDFLARE_API_TOKEN}
1538
- GATEWAY_DOMAIN=_.${DSTACK_GATEWAY_DOMAIN}
@@ -30,6 +53,7 @@ services:
3053
- ACME_EMAIL=${ACME_EMAIL}
3154
- ACME_DIRECTORY=${ACME_DIRECTORY:-https://acme-staging-v02.api.letsencrypt.org/directory}
3255
- ATTESTATION_DOCUMENT=${ATTESTATION_DOCUMENT:-}
56+
- API_HOSTNAME=${API_HOSTNAME:-api.tinycloud.link}
3357
depends_on:
3458
postgres:
3559
condition: service_healthy

0 commit comments

Comments
 (0)