Skip to content

README.md

Latest commit

 

History

History
417 lines (339 loc) · 18.7 KB

README.md

File metadata and controls

417 lines (339 loc) · 18.7 KB
Raw

dstack-ingress

TCP proxy with automatic TLS termination for dstack applications.

Overview

dstack-ingress is a HAProxy-based L4 (TCP) proxy that provides:

  • Automatic SSL certificate provisioning and renewal via Let's Encrypt
  • Multi-provider DNS support (Cloudflare, Linode DNS, Namecheap)
  • Pure TCP proxying — all protocols (HTTP, WebSocket, gRPC, arbitrary TCP) work transparently
  • Wildcard domain support
  • SNI-based multi-domain routing
  • Certificate evidence generation for TEE attestation verification
  • Strong TLS configuration (TLS 1.2+, AES-GCM, ChaCha20-Poly1305)

How It Works

  1. Bootstrap: On first start, obtains SSL certificates from Let's Encrypt using DNS-01 validation and configures DNS records (CNAME, TXT, optional CAA).

  2. TLS Termination: HAProxy terminates TLS and forwards the decrypted TCP stream to your backend. No HTTP inspection — the proxy operates entirely at L4.

  3. Certificate Renewal: A background daemon checks for renewal every 12 hours. On renewal, HAProxy is gracefully reloaded with zero downtime.

  4. Evidence Generation: Generates cryptographically linked attestation evidence (ACME account, certificates, TDX quote) for TEE verification.

Wildcard Domain Support

You can use a wildcard domain (e.g. *.myapp.com) to route all subdomains to a single dstack application:

  • The TXT record is automatically set as _dstack-app-address-wildcard.myapp.com (instead of _dstack-app-address.*.myapp.com)
  • CAA records use the issuewild tag on the base domain
  • Requires dstack-gateway with wildcard TXT resolution support (dstack#545)
services:
  dstack-ingress:
    image: dstacktee/dstack-ingress:2.2@sha256:d05a7b343c37c1cca1bba8dbf7e8f3c6d2118158af2d41c455103796db4f67f0
    ports:
      - "443:443"
    environment:
      - CLOUDFLARE_API_TOKEN=${CLOUDFLARE_API_TOKEN}
      - DOMAIN=*.myapp.com
      - GATEWAY_DOMAIN=_.dstack-prod5.phala.network
      - CERTBOT_EMAIL=${CERTBOT_EMAIL}
      - SET_CAA=true
      - TARGET_ENDPOINT=http://app:80
    volumes:
      - /var/run/dstack.sock:/var/run/dstack.sock
      - /var/run/tappd.sock:/var/run/tappd.sock
      - cert-data:/etc/letsencrypt
    restart: unless-stopped
  app:
    image: nginx
    restart: unless-stopped
volumes:
  cert-data:

Usage

Single Domain

services:
  dstack-ingress:
    image: dstacktee/dstack-ingress:2.2@sha256:d05a7b343c37c1cca1bba8dbf7e8f3c6d2118158af2d41c455103796db4f67f0
    ports:
      - "443:443"
    environment:
      - DNS_PROVIDER=cloudflare
      - CLOUDFLARE_API_TOKEN=${CLOUDFLARE_API_TOKEN}
      - DOMAIN=${DOMAIN}
      - GATEWAY_DOMAIN=${GATEWAY_DOMAIN}
      - CERTBOT_EMAIL=${CERTBOT_EMAIL}
      - SET_CAA=true
      - TARGET_ENDPOINT=app:80
    volumes:
      - /var/run/dstack.sock:/var/run/dstack.sock
      - /var/run/tappd.sock:/var/run/tappd.sock
      - cert-data:/etc/letsencrypt
      - evidences:/evidences
    restart: unless-stopped

  app:
    image: your-app
    volumes:
      - evidences:/evidences:ro
    restart: unless-stopped

volumes:
  cert-data:
  evidences:

TARGET_ENDPOINT accepts bare host:port (preferred) or with protocol prefix (http://app:80, grpc://app:50051). The protocol prefix is stripped — HAProxy forwards raw TCP regardless of protocol.

Multi-Domain with Routing

Use ROUTING_MAP to route different domains to different backends via SNI:

services:
  ingress:
    image: dstacktee/dstack-ingress:2.2@sha256:d05a7b343c37c1cca1bba8dbf7e8f3c6d2118158af2d41c455103796db4f67f0
    ports:
      - "443:443"
    environment:
      DNS_PROVIDER: cloudflare
      CLOUDFLARE_API_TOKEN: ${CLOUDFLARE_API_TOKEN}
      CERTBOT_EMAIL: ${CERTBOT_EMAIL}
      GATEWAY_DOMAIN: _.dstack-prod5.phala.network
      SET_CAA: true
      DOMAINS: |
        app.example.com
        api.example.com
      ROUTING_MAP: |
        app.example.com=app-main:80
        api.example.com=app-api:8080
    volumes:
      - /var/run/tappd.sock:/var/run/tappd.sock
      - letsencrypt:/etc/letsencrypt
      - evidences:/evidences
    restart: unless-stopped

  app-main:
    image: nginx
    volumes:
      - evidences:/evidences:ro
    restart: unless-stopped

  app-api:
    image: your-api
    volumes:
      - evidences:/evidences:ro
    restart: unless-stopped

volumes:
  letsencrypt:
  evidences:

Wildcard Domains

Wildcard certificates work out of the box with DNS-01 validation:

environment:
  - DOMAIN=*.example.com
  - TARGET_ENDPOINT=app:80

Environment Variables

Required

Variable Description
DOMAIN Your domain (single-domain mode). Supports wildcards (*.example.com)
TARGET_ENDPOINT Backend address, e.g. app:80 or http://app:80
GATEWAY_DOMAIN dstack gateway domain (e.g. _.dstack-prod5.phala.network)
ACME_EMAIL (optional) ACME contact address, in either mode. CERTBOT_EMAIL is the historical name and still works. See below — it is optional, and published
DNS_PROVIDER DNS provider (cloudflare, linode, namecheap)

Optional

Variable Default Description
PORT 443 HAProxy listen port
DOMAINS Multiple domains, one per line
ROUTING_MAP Multi-domain routing: domain=host:port per line
SET_CAA false Enable CAA DNS record (dns-01 only; tls-alpn-01 cannot write DNS)
TXT_PREFIX _dstack-app-address DNS TXT record prefix
ACME_STAGING false Use Let's Encrypt staging, in either mode. CERTBOT_STAGING is the historical name and still works
CHALLENGE_TYPE dns-01 dns-01 (certbot + DNS credentials) or tls-alpn-01 (lego, no DNS credentials)
DNS_SETUP_MODE wait tls-alpn-01 only: wait, print or webhook — see below
DNS_SETUP_TIMEOUT 1800 tls-alpn-01 only: seconds to wait for the records to appear
DNS_SETUP_INTERVAL 15 tls-alpn-01 only: seconds between DNS checks
DNS_WEBHOOK_URL tls-alpn-01 + DNS_SETUP_MODE=webhook: endpoint to notify
DNS_WEBHOOK_TOKEN Shared secret; the payload is HMAC-SHA256 signed with it
DOH_RESOLVERS Google + Cloudflare Comma-separated DoH endpoints used to verify records
TLSALPN_PORT 9443 Loopback port lego's ACME responder binds to
TLS_TERMINATE_PORT 9444 Loopback port the TLS frontend moves to in tls-alpn-01 mode
RENEW_DAYS_BEFORE client default Days of remaining lifetime that trigger renewal. Applies to both modes: passed to lego as --renew-days, and written to certbot's renew_before_expiry
RENEW_INTERVAL 43200 Seconds between successful certificate passes
DNS_SETTLE_SECONDS 30 Wait after DNS verifies, so the gateway's own TXT cache expires
MAXCONN 4096 HAProxy max connections
TIMEOUT_CONNECT 10s Backend connect timeout
TIMEOUT_CLIENT 86400s Client-side timeout (24h for long-lived connections)
TIMEOUT_SERVER 86400s Server-side timeout
EVIDENCE_SERVER true Serve evidence files at /evidences/ on the TLS port
EVIDENCE_PORT 80 Internal port for evidence HTTP server
ALPN TLS ALPN protocols (e.g. h2,http/1.1). Only set if backends support h2c
ACME_CHALLENGE_ALIAS Delegate the ACME DNS-01 challenge to this zone (see below) so the DNS token needs no access to the served domain's own zone
ACME_CHALLENGE_PROPAGATION_SECONDS 30 Wait after writing the delegated challenge TXT before validation (only with ACME_CHALLENGE_ALIAS). Keep well under ~250s — certbot is killed after a 300s per-run timeout
ALLOW_MISSING_CAA false In delegation mode, continue even if the required accounturi CAA cannot be confirmed. Default fails closed (see below)

For DNS provider credentials, see DNS_PROVIDERS.md.

ACME challenge delegation (least-privilege DNS token)

By default the DNS token must be able to edit the served domain's own zone (to write _acme-challenge, and — with SET_CAA — the CAA record). If the served name lives under a shared production zone (e.g. svc.example.com under example.com), that token can edit every record in the zone, which may be more privilege than you want.

Set ACME_CHALLENGE_ALIAS=<delegation-zone> to answer the DNS-01 challenge in a separate zone that your token controls, so the token never touches the served domain's zone. In this mode dstack-ingress only manages the challenge TXT in the delegation zone; you set the following records once, statically, in the served domain's production zone (the container prints the exact values on start):

svc.example.com                       CNAME  <GATEWAY_DOMAIN>
_dstack-app-address.svc.example.com   TXT    "<app-id>:<port>"
_acme-challenge.svc.example.com       CNAME  _acme-challenge.svc.example.com.<delegation-zone>
svc.example.com                       CAA    0 issue "letsencrypt.org;validationmethods=dns-01;accounturi=<account-uri>"

Security note. The accounturi CAA restricts issuance to this enclave's ACME account and is the control that prevents forged certificates. In delegation mode dstack-ingress cannot set it for you (no token for the served zone), so it prints the record and verifies its presence via DoH. If the CAA is confirmed absent the container fails to start (set ALLOW_MISSING_CAA=true to override; a transient DoH failure only warns, it does not block). Without this CAA, anyone who can satisfy the delegated challenge could obtain a certificate for the domain.

Use a dedicated delegation zone and scope the token to only that zone — a zone shared with other tenants lets anyone with write access to it complete the challenge. SET_CAA does not affect delegation mode (this CAA path always runs). If a certificate was previously issued with the standard DNS-plugin authenticator, delete /etc/letsencrypt/renewal/<domain>.conf before enabling delegation, otherwise certbot renew reuses the old plugin (which needs the production-zone token this mode avoids).

Evidence & Attestation

Evidence files are served at https://your-domain.com/evidences/ by default (via payload inspection in HAProxy's TCP mode). They can also be accessed by the backend application through the shared /evidences volume.

To disable the built-in evidence endpoint and serve evidence files only through your backend, set EVIDENCE_SERVER=false.

Evidence Files

File Description
acme-account.json ACME account used to request certificates
cert-{domain}.pem Let's Encrypt certificate for each domain
sha256sum.txt SHA-256 checksums of all evidence files
quote.json TDX quote with sha256sum.txt digest in report_data

Verification Chain

  1. Verify the TDX quote in quote.json
  2. Extract report_data — it contains the SHA-256 of sha256sum.txt
  3. Verify checksums in sha256sum.txt against acme-account.json and cert-*.pem
  4. This proves the certificates were obtained within the TEE

Building

./build-image.sh
# Or push directly:
./build-image.sh --push yourusername/dstack-ingress:tag

The build script ensures reproducibility via pinned packages, deterministic timestamps, and specific buildkit version.

License

MIT License

Copyright (c) 2025

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Certificates without DNS credentials (tls-alpn-01)

The default dns-01 flow needs a DNS API token inside the CVM: the container creates the CNAME, TXT and CAA records itself. CHALLENGE_TYPE=tls-alpn-01 removes that requirement — nothing in the container can touch your DNS zone — at the cost of you creating three records by hand (or via a webhook).

services:
  dstack-ingress:
    image: dstacktee/dstack-ingress:<tag>
    environment:
      - CHALLENGE_TYPE=tls-alpn-01
      - DOMAIN=app.example.com
      - TARGET_ENDPOINT=http://app:80
      # Printed as the CNAME target, and used to verify that the hostname
      # really resolves to the gateway before issuance starts.
      - GATEWAY_DOMAIN=_.dstack-prod5.phala.network
      # - ACME_EMAIL=you@example.com   # optional, and published (see below)
      # - DNS_SETUP_MODE=wait          # default; blocks until the records exist
    ports:
      - "443:443"
    volumes:
      - /var/run/dstack.sock:/var/run/dstack.sock
      - cert-data:/etc/letsencrypt
      - evidences:/evidences

On first start the container prints the exact records to create, then polls public DNS until they are visible:

==========================================================================
  DNS records required for app.example.com
==========================================================================
  CNAME  app.example.com
         -> _.dstack-prod5.phala.network
  TXT    _dstack-app-address.app.example.com
         -> b1ea785543bbbb19ce9de33744321360992bf63b:443
  CAA    app.example.com
         -> 0 issue "letsencrypt.org;validationmethods=tls-alpn-01;accounturi=https://..."
==========================================================================

DNS_SETUP_MODE picks what happens after printing: wait (default) blocks until the records resolve or DNS_SETUP_TIMEOUT elapses; print continues immediately; webhook POSTs the records to DNS_WEBHOOK_URL first and then waits, so a service of yours can create them automatically.

The TXT record names an instance, not an app

Under dns-01 the TXT record carries the app ID and the gateway load-balances across every instance of the app. tls-alpn-01 cannot work that way. The challenge is answered by whichever instance holds the ACME order, and Let's Encrypt validates from several vantage points at once (5 distinct source IPs within ~2 seconds, measured), while the gateway races connections across the app's instances. The challenge would land on the wrong replica almost every time. So this mode publishes the instance ID instead, pinning the hostname to one instance.

Two consequences:

  • tls-alpn-01 mode is effectively single-instance. All traffic for the hostname goes to the pinned instance; you lose the gateway's failover.
  • The TXT record changes when the CVM instance is replaced. Redeploying means updating DNS. DNS_SETUP_MODE=webhook exists so this can be automated; doing it by hand means downtime on every redeploy.

The ACME contact address is optional, and public

ACME_EMAIL (or CERTBOT_EMAIL) may be left unset in either mode. RFC 8555 makes the ACME contact field optional, and Let's Encrypt stopped sending expiry notification mail in 2025, so setting one buys little.

It also does not stay private. The ACME account document is published as attestation evidence at /evidences/acme-account.json, so an address set here is readable by anyone who fetches the evidence endpoint. Leave it unset unless you specifically want a contact on the account.

Under the hood the two clients differ: lego simply omits the flag, while certbot needs --register-unsafely-without-email — its "unsafely" naming predates Let's Encrypt dropping expiry mail, and the container passes it for you.

Limitations

  • No wildcards. RFC 8737 forbids tls-alpn-01 for wildcard identifiers, and the CA will not offer the challenge. The container refuses to start rather than serve a name it can never obtain a certificate for, so a wildcard anywhere in DOMAIN/DOMAINS is a startup error even if the other names are fine. Use dns-01 for *.example.com.
  • The gateway must be reachable on port 443. The CA connects to port 443 of whatever the CNAME resolves to; the port is fixed by the protocol.
  • CAA must permit tls-alpn-01. A record left over from a dns-01 deployment says validationmethods=dns-01 and will make the CA refuse. The container checks this before asking for a certificate, so you get a clear message instead of a failed validation.
  • Losing the cert-data volume changes the account URI. With dns-01 the container just rewrites the CAA record; here it cannot, so a pinned accounturi would start rejecting issuance until you update it by hand.

Webhook payload

DNS_SETUP_MODE=webhook POSTs this envelope to DNS_WEBHOOK_URL:

{
  "payload": "{\"app_id\":\"…\",\"challenge\":\"tls-alpn-01\",\"domain\":\"app.example.com\",\"instance_id\":\"…\",\"records\":[…],\"timestamp\":1234567890,\"version\":1}",
  "hmac_sha256": "…",
  "attestation": { "quote": "…", "report_data": "…" }
}

payload is a string so you sign and hash exactly the bytes you received. Verify hmac_sha256 with DNS_WEBHOOK_TOKEN, and — since this request asks you to point a hostname at the instance it names — verify attestation too: the quote's report_data is sha256(payload), so it proves which enclave produced those records. Check the app ID and measurements in it before changing DNS.

Why lego and not certbot

certbot cannot do tls-alpn-01. Its standalone plugin is HTTP-01 only, the maintainers declined to implement the challenge (certbot#6724), and the acme library removed it outright in 4.2.0 — the last release carrying acme.standalone.TLSALPN01Server is 4.1.1, where it is already marked deprecated. This path therefore runs lego, pinned by checksum in the Dockerfile. The dns-01 path still uses certbot and is untouched.

Because haproxy owns the public port, the proxy peeks at each ClientHello and forwards only connections advertising the acme-tls/1 ALPN protocol to lego's responder on loopback; everything else goes to the normal TLS frontend. Issuance and renewal therefore never interrupt serving traffic. In this mode haproxy starts on a self-signed placeholder certificate — it has to be listening before the first certificate can be issued — and reloads onto the real one as soon as it arrives.