Skip to content

TC-85: tunnel relay MVP -- remote reachability for local nodes (+ TC-244 batch) - #1

Merged
samgbafa merged 7 commits into
mainfrom
feat/tunnel-relay
Jul 20, 2026
Merged

TC-85: tunnel relay MVP -- remote reachability for local nodes (+ TC-244 batch)#1
samgbafa merged 7 commits into
mainfrom
feat/tunnel-relay

Conversation

@samgbafa

Copy link
Copy Markdown
Contributor

What

TC-85 (tunnel relay MVP): nodes open one outbound WebSocket to wss://api.tinycloud.link/v1/tunnel/<name>, authenticate with the same signed-payload scheme as every other write (action: "tunnel", subject must own the name, strictly-increasing sequence), and the relay proxies HTTPS requests for https://<name>.tinycloud.link down the socket as JSON request/response frames.

  • src/tunnel/protocol.ts — HTTP-over-WS framing (request/requestBody/response/responseBody/error, base64 bodies, multiplexed by request id). Source of truth for the Rust node client; the README documents the full contract (frames, auth, close codes, lifecycle).
  • src/tunnel/upgrade.ts — WS handshake: 5s auth timeout, ownership + signature verification, sequence bump before going live (same bump-before-side-effect pattern as the cert flow), ping/pong heartbeat with dead-peer termination.
  • src/tunnel/registry.ts — one live socket per name, newest wins (a reconnecting node evicts its own stale socket, close code 4410).
  • src/tunnel/host-router.ts — Host-header routing middleware: API_HOSTNAME (default api.tinycloud.link) gets the normal /v1 API, any other <name>.tinycloud.link host is proxied through its tunnel (502 no tunnel / 504 timeout). Opt-in via ServerConfig.tunnelRegistry; existing behavior is unchanged when unset.

TC-244 batch:

  • ECDSA P-256 CSRs are now accepted: assertCsrMatchesDomain parses with @peculiar/x509 instead of node-forge (forge cannot parse EC CSRs at all — "OID is not RSA"). RSA still works; both are tested end-to-end through the fake ACME client.
  • README documents the bump-before-ACME sequence semantics (why the bump precedes the slow, non-idempotent ACME round trip, and the retry-with-higher-sequence consequence).

Ingress/TLS finding (design honesty)

Per-name relay certificates are a dead end in the Phala deployment model: dstack-ingress (HAProxy L4) terminates all of the CVM's :443 with certificates it obtained itself at bootstrap — this process never sees the TLS handshake and has no way to serve certs it might obtain. Shipped design instead: one wildcard cert via DOMAIN=*.tinycloud.link on dstack-ingress (native wildcard DNS-01 support) + Host-header routing behind it. Open item before deploy: dstack-ingress's wildcard mode requires dstack-gateway wildcard TXT resolution (dstack#545) — must be verified on a staging CVM; fallback is DOMAIN=api.tinycloud.link (API unaffected, tunnels dark). Details in README "Ingress and TLS for tunnels".

Not in this PR

  • No deploy. The running api.tinycloud.link service is untouched; deploying the tunnel-enabled image (and the wildcard DNS record + gateway verification) is a separate reviewed step.
  • Node-side (Rust) client — separate ticket, consumes the documented contract.

Validation

  • tsc --noEmit clean
  • 66/66 tests pass (was 51; +9 tunnel tests: auth ownership/signature/sequence, framing roundtrip vs a mock ws node, newest-socket-wins; +3 tunnel-auth unit tests; +3 ECDSA CSR tests)
  • No live network in tests (in-memory DNS/ACME/name stores, real WS over loopback)

samgbafa added 7 commits July 19, 2026 15:06
ws: WebSocket server/client for the tunnel relay (TC-85).
@peculiar/x509: replaces node-forge for CSR parsing so ECDSA P-256 CSRs
are accepted (TC-244); promoted from a transitive (via acme-client) to
a direct dependency.
assertCsrMatchesDomain (names.ts) now parses CSRs with @peculiar/x509
instead of node-forge. Forge cannot parse a CSR with an EC public key
at all ("OID is not RSA"), so every ECDSA node CSR was being rejected
before it even reached the CN/SAN check. @peculiar/x509 reads the
subject/SAN independently of the key algorithm, so RSA and ECDSA CSRs
are now validated identically.

Also updates the test-only CSR/fake-ACME helpers (test-support/csr.ts,
test-support/fake-acme-client.ts) to use @peculiar/x509 so the full
claim -> CSR -> DNS-01 -> certificate round trip is exercised for both
key types in tests, not just the CN/SAN validation in isolation.
Explains why POST /v1/certs/:name persists the bumped sequence before
calling the (slow, non-idempotent) ACME issuer rather than after, and
the resulting tradeoff (a failed ACME call still consumes the sequence
bump; the node must retry with a higher sequence, not the same one).
…uting

Adds the remote-reachability tunnel relay: a node opens one outbound
WebSocket to wss://<host>/v1/tunnel/<name>, authenticates with the same
signed-payload scheme as claim/delete/cert (action: "tunnel", subject
must own <name>, sequence anti-replay -- src/names.ts), and the relay
then proxies inbound HTTPS requests for <name>.tinycloud.link down that
socket as simple JSON request/response frames (src/tunnel/protocol.ts).

- src/tunnel/registry.ts: one live socket per name, newest-wins eviction
  (a fresh registration closes the prior socket with a distinct close
  code rather than rejecting the new connection).
- src/tunnel/proxy.ts: sends request/requestBody frames, demuxes
  response/responseBody/error frames by request id (concurrency-safe,
  many in-flight requests can share one socket), with a timeout.
- src/tunnel/upgrade.ts: the WS handshake -- auth-frame timeout, name
  ownership + signature verification, sequence bump (same
  bump-before-side-effect pattern as the cert flow), ws ping/pong
  heartbeat with dead-peer termination.
- src/tunnel/host-router.ts: Hono middleware that proxies any request
  whose Host header names a claimed tunnel through its socket, and
  leaves the control-plane API's own hostname (api.tinycloud.link)
  untouched. Wired into ServerConfig as an opt-in `tunnelRegistry`
  field -- omitting it (as all existing callers/tests do) leaves /v1
  routing byte-for-byte unchanged.
- src/index.ts: constructs the registry and attaches the WS upgrade
  handler to the same http.Server @hono/node-server returns.

Node-side (Rust client) implementation is out of scope; the wire
protocol in src/tunnel/protocol.ts and the auth/lifecycle rules in
src/tunnel/upgrade.ts are the contract for it (README documents both
precisely). Ingress/TLS wiring for *.tinycloud.link traffic is a
separate, deliberately unresolved question here -- see the README's
"Ingress and TLS for tunnels" section and the follow-up docker-compose
change, neither of which is deployed by this PR.
README: full tunnel relay section -- lifecycle diagram, auth record and
close codes, the frame protocol table with the exact rules a Rust node
client must follow (src/tunnel/protocol.ts is the source of truth), and
the ingress/TLS finding: dstack-ingress terminates all of :443 inside
the CVM with certs it obtained itself, so per-name relay certificates
obtained by this process could never be served -- the shipped design is
one wildcard cert (DOMAIN=*.tinycloud.link on dstack-ingress, native
wildcard DNS-01 support) plus Host-header routing in this process, with
the dstack-gateway wildcard-TXT support flagged as the open item to
verify on a staging CVM before deploying.

docker-compose.phala.yml: switch dstack-ingress DOMAIN to the wildcard
(constraint + fallback documented inline; NOT deployed by this change)
and pass API_HOSTNAME through. .env.example documents API_HOSTNAME.
…te limiting, ordered-pair headers

Addresses the round-1 review of the tunnel relay (PR #1):

1. Body/frame size limits: proxied request bodies are now read with a
   streaming cap (host-router.ts's readLimitedBody, 413 past the cap) and
   response bodies accumulated in proxy.ts are capped too (502 + an `error`
   frame back to the node past the cap). Default 25MB, overridable via
   TUNNEL_MAX_BODY_BYTES. The relay's WebSocketServer now sets `maxPayload`
   (1MB, protocol.ts's MAX_FRAME_PAYLOAD_BYTES) so no single WS frame can be
   unbounded; request bodies are chunked into <=256KB requestBody frames
   (protocol.ts's BODY_CHUNK_BYTES) to respect it. Also adds a 'ws' `error`
   listener in upgrade.ts -- without one, a maxPayload violation's unhandled
   'error' event would crash the whole process, not just the one tunnel.

2. Rate/connection limiting on the WS upgrade path (upgrade.ts, new
   rate-limit.ts): a per-IP connection-attempt limiter (30/min), a per-name
   churn limiter (10/min), and a global concurrent-tunnel cap (1000, env
   TUNNEL_MAX_CONCURRENT) all run in the raw HTTP `upgrade` handler, before
   the WS handshake and before authenticate()'s nameStore.get Postgres read.
   Deliberately in-memory (not the Postgres-backed cert rate limiter): these
   are short, per-minute windows checked on every upgrade attempt.

3. Frame headers are now ordered `[name, value]` pairs (protocol.ts's
   TunnelRequestFrame/TunnelResponseFrame) instead of a Record, so duplicate
   header names -- most importantly Set-Cookie -- survive the roundtrip
   instead of colliding on one object key. host-router.ts rebuilds the final
   client-facing Response via repeated Headers.append() calls so multiple
   Set-Cookie entries are preserved end to end.

Tests: 9 new (66 -> 75) covering each fix -- request/response body caps,
multi-chunk body reassembly, WS maxPayload enforcement, per-IP/per-name/
global connection limits, header-pair passthrough, and a two-Set-Cookie
roundtrip.
- Sequence coordination is per-name, not per-operation: claim/delete/cert/
  tunnel-auth all share one counter, single source of truth on the node.
- Max frame (1MB) and body (25MB, TUNNEL_MAX_BODY_BYTES) size limits, and
  the requestBody/responseBody chunking contract this implies.
- Frames carrying an unrecognized request id are silently ignored.
- Headers are ordered [name, value] pairs so duplicate header names (e.g.
  Set-Cookie) survive.
- Connection-attempt limits enforced before the WS handshake (per-IP,
  per-name churn, global concurrent-tunnel cap via TUNNEL_MAX_CONCURRENT).
@samgbafa

Copy link
Copy Markdown
Contributor Author

Review round complete: security-review fixes applied

Addressed all four items from the round-1 security review, pushed as two commits (6e45067, 7d7834d) on feat/tunnel-relay.

1. Body/frame size limitssrc/tunnel/host-router.ts's readLimitedBody streams and caps proxied request bodies (413 past the cap, no unbounded buffering); src/tunnel/proxy.ts now tracks accumulated response-body bytes and aborts with 502 + an error frame to the node past the cap. Default 25MB both directions, overridable via TUNNEL_MAX_BODY_BYTES. The relay's WebSocketServer now sets maxPayload (MAX_FRAME_PAYLOAD_BYTES, 1MB); request bodies are chunked into <=256KB requestBody frames (BODY_CHUNK_BYTES) so the relay's own sends respect it. Also added a ws 'error' listener in upgrade.ts — without one, a maxPayload violation's unhandled error event crashes the whole process (found this the hard way: it was an uncaught exception in the new maxPayload test before the fix).

2. Rate/connection limitingsrc/tunnel/upgrade.ts (new src/tunnel/rate-limit.ts) now enforces a per-IP connection-attempt limiter (30/min), a per-name churn limiter (10/min), and a global concurrent-tunnel cap (1000, env TUNNEL_MAX_CONCURRENT), all checked in the raw HTTP upgrade handler — before the WS handshake and before authenticate()'s nameStore.get Postgres read. Deliberately in-memory rather than reusing the Postgres-backed cert rate limiter: these are short per-minute windows checked on every single upgrade attempt, and adding a Postgres round trip there would defeat the point.

3. Headers as ordered pairsTunnelRequestFrame/TunnelResponseFrame.headers changed from Record<string,string> to Array<[string,string]> (breaking protocol change; no version marker exists to bump, and nothing consumes the protocol yet per the PR description). host-router.ts rebuilds the outgoing client Response via repeated Headers.append() calls so multiple Set-Cookie entries survive end to end. Covered by an explicit two-Set-Cookie roundtrip test.

4. README — documented shared per-name sequence coordination across claim/cert/delete/tunnel, max frame/body sizes and the chunking contract, unknown-id frames being ignored, and the pair-array header format.

Validation: tsc --noEmit clean; 75/75 tests pass (66 + 9 new covering each fix: request/response body caps, multi-chunk body reassembly, WS maxPayload enforcement, per-IP/per-name/global connection limits, header-pair passthrough, duplicate-Set-Cookie roundtrip).

@samgbafa
samgbafa marked this pull request as ready for review July 19, 2026 19:59
@samgbafa
samgbafa merged commit ff9522a into main Jul 20, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant