feat(dstack-ingress): delegate every record, not just the challenge

Challenge delegation moved the ACME challenge out of the served domain's
zone, but left three records behind that the operator still had to create
by hand, and two reasons to come back to them.

The accounturi CAA needs a second visit: the container can compute it but
not write it, and it names an ACME account that does not exist until the
first issuance has succeeded.

The app-address TXT carries the app id, which by default is the hash of
app-compose.json and so moves when the compose does -- add an environment
variable and the gateway stops routing until someone updates the record.
It can be pinned with `vmm-cli deploy --app-id`, which is how an app is
upgraded in place, so this is avoidable rather than inevitable. But it is
the default, and it fails silently: the certificate keeps renewing, only
traffic stops.

Alias all three names into the delegation zone instead. The operator
creates three CNAMEs before deploying and is then done, permanently:

    svc.example.com                      CNAME  svc.example.com.deleg.net
    _dstack-app-address.svc.example.com  CNAME  _dstack-app-address.svc.example.com.deleg.net
    _acme-challenge.svc.example.com      CNAME  _acme-challenge.svc.example.com.deleg.net

The container publishes what they point at, including the CAA.

The gateway pointer and the CAA end up on one name, which RFC 1034 does
not allow -- a CNAME excludes every other type at its name. Cloudflare
allows the pair anyway and Let's Encrypt honours the CAA it finds there,
verified against the staging CA: a CAA that forbids the CA stops issuance
with `While processing CAA for svc.example.com: CAA record for
svc.example.com prevents issuance`. The resolver stops at the first name
in the chain carrying a CAA rather than following to the end, so a CAA at
the delegation zone apex is *not* consulted -- also verified.

Providers that enforce the standard reject the pair, so the record type
follows the provider: Cloudflare gets a CNAME and keeps DNS following a
gateway that moves; everything else gets an address record, which can
carry the CAA legally, at the cost of re-resolving GATEWAY_DOMAIN once a
pass. Only Cloudflare has been tested, so only Cloudflare gets the
optimistic default; DELEGATION_GATEWAY_RECORD overrides either way.

The variable is now DELEGATION_ZONE. ACME_CHALLENGE_ALIAS was accurate
when the challenge was the only thing delegated and is not any more; it
is removed rather than deprecated, since the feature is days old and the
published image does not carry it.

Renewal passes verify only the challenge alias. The other two are about
serving, and dns-01 never connects here, so blocking a renewal on them
would turn a routing problem into an expired certificate. The first pass
checks all three, because that is setup.

The delegated challenge's propagation wait also had to go from 30s to
120s: it must outlast the record's own 60s TTL, or the second of the two
issuance attempts fails Let's Encrypt's multi-perspective check with
"During secondary validation: Incorrect TXT record found".

Author: kvinwang

custom-domain/dstack-ingress/README.md

@@ -191,8 +191,9 @@ environment:
 | `EVIDENCE_SERVER` | `true` | Serve evidence files at `/evidences/` on the TLS port |
 | `EVIDENCE_PORT` | `80` | Internal port for evidence HTTP server |
 | `ALPN` | | TLS ALPN protocols (e.g. `h2,http/1.1`). Only set if backends support h2c |
-| `ACME_CHALLENGE_ALIAS` | | Delegate the ACME DNS-01 challenge to this zone (see below) so the DNS token needs no access to the served domain's own zone |
-| `ACME_CHALLENGE_PROPAGATION_SECONDS` | `30` | Wait after writing the delegated challenge TXT before validation (only with `ACME_CHALLENGE_ALIAS`). Keep well under ~250s — certbot is killed after a 300s per-run timeout |
+| `DELEGATION_ZONE` | | Zone this container writes into, so the DNS token needs no access to the served domain's own zone (see below). `ACME_CHALLENGE_ALIAS` is the original name and still works |
+| `ACME_CHALLENGE_PROPAGATION_SECONDS` | `120` | Wait after writing the delegated challenge TXT before validation. Must outlast the record TTL (60s), or a resolver still serving the previous attempt's value fails validation. Keep well under ~250s — certbot is killed after a 300s per-run timeout |
+| `DELEGATION_GATEWAY_RECORD` | per provider | How the delegated name points at the gateway: `cname` (Cloudflare default; follows a gateway that moves) or `a` (default elsewhere, for providers that refuse a CAA beside a CNAME) |
 | `ALLOW_MISSING_CAA` | `false` | In delegation mode, treat an unconfirmed `accounturi` CAA as a warning instead of a blocker. Default fails closed (see below) |
 
 For DNS provider credentials, see [DNS_PROVIDERS.md](DNS_PROVIDERS.md).
@@ -205,19 +206,55 @@ served name lives under a shared production zone (e.g. `svc.example.com` under
 `example.com`), that token can edit every record in the zone, which may be more
 privilege than you want.
 
-Set `ACME_CHALLENGE_ALIAS=<delegation-zone>` to answer the DNS-01 challenge in a
-separate zone that your token controls, so the token never touches the served
-domain's zone. In this mode dstack-ingress **only** manages the challenge TXT in
-the delegation zone; you set the following records **once, statically**, in the
-served domain's production zone (the container prints the exact values on start):
+Set `DELEGATION_ZONE=<delegation-zone>` to move every name this deployment
+needs into a zone your token controls. You create three CNAMEs in the served
+domain's zone **once, before deploying**, and then never touch DNS again — not
+when the app id changes with the compose, not when the ACME account is
+recreated, not when the gateway moves:
 
 ```
-svc.example.com                       CNAME  <GATEWAY_DOMAIN>
-_dstack-app-address.svc.example.com   TXT    "<app-id>:<port>"
-_acme-challenge.svc.example.com       CNAME  _acme-challenge.svc.example.com.<delegation-zone>
-svc.example.com                       CAA    0 issue "letsencrypt.org;validationmethods=dns-01;accounturi=<account-uri>"
+svc.example.com                      CNAME  svc.example.com.deleg.example.net
+_dstack-app-address.svc.example.com  CNAME  _dstack-app-address.svc.example.com.deleg.example.net
+_acme-challenge.svc.example.com      CNAME  _acme-challenge.svc.example.com.deleg.example.net
 ```
 
+dstack-ingress publishes what they point at, in the delegation zone:
+
+```
+svc.example.com.deleg.example.net                      CNAME  <GATEWAY_DOMAIN>
+svc.example.com.deleg.example.net                      CAA    0 issue "letsencrypt.org;validationmethods=dns-01;accounturi=…"
+_dstack-app-address.svc.example.com.deleg.example.net  TXT    "<app-id>:<port>"
+_acme-challenge.svc.example.com.deleg.example.net      TXT    <challenge>   (transient)
+```
+
+The gateway pointer and the CAA share a name, which RFC 1034 does not allow —
+a CNAME excludes every other type at its name. **Cloudflare allows the pair
+anyway**, and Let's Encrypt honours the CAA it finds there: with a CAA that
+forbids it, the staging CA refuses with `While processing CAA for
+svc.example.com: CAA record for svc.example.com prevents issuance`.
+
+Providers that enforce the standard will reject the CAA beside the CNAME, so
+they get an address record instead — `A` and `CAA` coexist legally. The default
+follows the provider:
+
+| Provider | Gateway pointer | Gateway that moves |
+|---|---|---|
+| `cloudflare` | `CNAME` to `GATEWAY_DOMAIN` | followed by DNS, immediately |
+| everything else | `A`, re-resolved each pass | picked up within `RENEW_INTERVAL` |
+
+`DELEGATION_GATEWAY_RECORD=cname\|a` overrides it. Only Cloudflare has been
+tested; if another provider accepts a CAA beside a CNAME, setting `cname` there
+gets the better behaviour.
+
+A wildcard and its base name share a delegated target: both `*.example.com` and
+`example.com` map to `example.com.<delegation-zone>`. One deployment serving both
+is fine, since it publishes the same values twice. Two deployments pointing at
+different gateways are not — give them separate delegation zones.
+
+**Wildcards keep an operator-managed CAA.** RFC 8659 evaluates `*.example.com`
+at `example.com`, which is your own name and cannot be aliased away, so for a
+wildcard the container prints the CAA and verifies it rather than publishing it.
+
 > **Security note.** The `accounturi` CAA restricts issuance to this enclave's
 > ACME account and is the control that prevents forged certificates. In
 > delegation mode dstack-ingress cannot set it for you (no token for the served
@@ -236,7 +273,14 @@ svc.example.com                       CAA    0 issue "letsencrypt.org;validation
 
 The records above are checked the same way tls-alpn-01 checks its own: two DoH
 resolvers, both CAA wire formats, and `DNS_SETUP_MODE` deciding what happens
-while they are missing. `wait` (the default) blocks until they appear, so you can
+while they are missing.
+
+Only the `_acme-challenge` CNAME is checked on later passes. Under dns-01 the CA
+reads a TXT record and never connects here, so the other two records are about
+serving rather than issuance, and a renewal is not blocked on them — a routing
+problem can be fixed at any time, an expired certificate cannot. They are still
+checked on the first pass, where the point is to tell you whether you created
+them correctly. `wait` (the default) blocks until they appear, so you can
 start the container and create the records afterwards; `print` lists them and
 continues without checking; `webhook` POSTs them to `DNS_WEBHOOK_URL` for an
 operator service to create automatically.

custom-domain/dstack-ingress/TESTING.md

@@ -201,25 +201,28 @@ first issuance.
 
 ### Challenge delegation (dns-01)
 
-`ACME_CHALLENGE_ALIAS` answers the DNS-01 challenge in a zone the container's
+`DELEGATION_ZONE` answers the DNS-01 challenge in a zone the container's
 token controls, so the token needs no access to the served domain's own zone.
 Testing it looks like it needs a second registrar account. It does not.
 
 Any name under a zone you already control works as the delegation zone, because
 the provider resolves a zone by longest-suffix match on the record name. With
-`ACME_CHALLENGE_ALIAS=deleg.example.com` and a served domain of
+`DELEGATION_ZONE=deleg.example.com` and a served domain of
 `svc.example.com`, the challenge record is
 `_acme-challenge.svc.example.com.deleg.example.com`, which lands in
 `example.com` — the same zone, but through the delegation code path.
 
-Play the operator and create the three static records the container prints:
+Play the operator and create the three CNAMEs the container prints:
 
 ```
-svc.example.com                    CNAME  <GATEWAY_DOMAIN>
-_dstack-app-address.svc.example.com  TXT  "<app-id>:443"
-_acme-challenge.svc.example.com    CNAME  _acme-challenge.svc.example.com.deleg.example.com
+svc.example.com                      CNAME  svc.example.com.deleg.example.com
+_dstack-app-address.svc.example.com  CNAME  _dstack-app-address.svc.example.com.deleg.example.com
+_acme-challenge.svc.example.com      CNAME  _acme-challenge.svc.example.com.deleg.example.com
 ```
 
+Everything they point at is published by the container, so this is the whole of
+the operator's job — the app id can change afterwards and no DNS edit follows.
+
 Then start the container with a real provider token and `ACME_STAGING=true`, and
 watch for: the three records verifying, `Executing (challenge-delegation):` with
 `--manual` and both hooks, the certificate arriving, and the challenge TXT being
@@ -251,7 +254,9 @@ These fail fast and are cheap, so run them on every change:
 | Scenario | Expected |
 |---|---|
 | tls-alpn-01 + a wildcard domain | Refused before any ACME call, citing RFC 8737 |
+| Delegation, CAA in the delegated zone changed to forbid the CA | Blocked before any ACME attempt — proves the published CAA is reachable through the CNAME |
 | Delegation with no CAA record at all | Blocked — unlike normal issuance, where "no CAA" means unrestricted and passes |
+| Delegation, later pass, gateway CNAME broken | Renewal proceeds — that record is for serving, and dns-01 does not use it |
 | A CAA record with `validationmethods=dns-01`, mode tls-alpn-01 | Blocked with the restriction quoted back |
 | TXT holding the wrong value | Reported as `want <x>, saw <y>`, not "missing" |
 | An instance ID the gateway does not know | The CA reports a connection error — the gateway will not route to it |

custom-domain/dstack-ingress/scripts/acme-dns-alias-hook.sh

@@ -3,10 +3,10 @@
 #
 # Instead of writing the `_acme-challenge` TXT into the served domain's own
 # zone (which requires a DNS token for that zone), this writes it into a
-# delegated zone that our token controls (ACME_CHALLENGE_ALIAS). The served
+# delegated zone that our token controls (DELEGATION_ZONE). The served
 # domain's zone only needs one static, operator-managed CNAME:
 #
-#   _acme-challenge.<domain>  CNAME  _acme-challenge.<domain>.<ACME_CHALLENGE_ALIAS>
+#   _acme-challenge.<domain>  CNAME  _acme-challenge.<domain>.<DELEGATION_ZONE>
 #
 # Let's Encrypt follows that CNAME during validation and reads the TXT from the
 # delegated zone, so the enclave's token never needs access to the served
@@ -18,8 +18,9 @@ set -euo pipefail
 
 action="${1:-}"
 
-if [ -z "${ACME_CHALLENGE_ALIAS:-}" ]; then
-    echo "acme-dns-alias-hook: ACME_CHALLENGE_ALIAS is not set" >&2
+zone="${DELEGATION_ZONE:-}"
+if [ -z "$zone" ]; then
+    echo "acme-dns-alias-hook: DELEGATION_ZONE is not set" >&2
     exit 1
 fi
 if [ -z "${CERTBOT_DOMAIN:-}" ]; then
@@ -28,17 +29,24 @@ if [ -z "${CERTBOT_DOMAIN:-}" ]; then
 fi
 
 # certbot always passes the base domain (no leading "*.") in CERTBOT_DOMAIN.
-record="_acme-challenge.${CERTBOT_DOMAIN}.${ACME_CHALLENGE_ALIAS}"
+record="_acme-challenge.${CERTBOT_DOMAIN}.${zone}"
 
 case "$action" in
     auth)
         : "${CERTBOT_VALIDATION:?CERTBOT_VALIDATION not set}"
         echo "acme-dns-alias-hook: writing challenge TXT to delegated record $record"
         dnsman.py set_txt --domain "$record" --content "$CERTBOT_VALIDATION"
         # certbot asks Let's Encrypt to validate immediately after this hook
-        # returns, so wait for the record to propagate on the delegated zone's
-        # authoritative servers before returning.
-        sleep "${ACME_CHALLENGE_PROPAGATION_SECONDS:-30}"
+        # returns, so wait for the record to propagate before returning.
+        #
+        # This has to outlast the record's own TTL, not just the time the write
+        # takes. dnsman.py publishes TXT with a 60s TTL, so a resolver that saw
+        # the *previous* challenge value keeps serving it for up to that long --
+        # which is exactly the situation on the second of the two issuance
+        # attempts. At 30s Let's Encrypt's multi-perspective check fails with
+        # "During secondary validation: Incorrect TXT record found". The dns-01
+        # plugin path waits 120s for the same reason.
+        sleep "${ACME_CHALLENGE_PROPAGATION_SECONDS:-120}"
         ;;
     cleanup)
         echo "acme-dns-alias-hook: removing challenge TXT $record"

custom-domain/dstack-ingress/scripts/certman.py

@@ -279,11 +279,11 @@ def _build_certbot_command(self, action: str, domain: str, email: str) -> List[s
         """Build certbot command using provider configuration."""
         certbot_cmd = self._get_certbot_command()
 
-        # Challenge-delegation mode: when ACME_CHALLENGE_ALIAS is set, answer the
+        # Challenge-delegation mode: when DELEGATION_ZONE is set, answer the
         # DNS-01 challenge in a delegated zone via a manual hook instead of the
         # provider's certbot plugin, so our DNS token never needs access to the
         # served domain's own zone. See scripts/acme-dns-alias-hook.sh.
-        if os.environ.get("ACME_CHALLENGE_ALIAS", "").strip():
+        if os.environ.get("DELEGATION_ZONE", "").strip():
             hook = "/scripts/acme-dns-alias-hook.sh"
             base_cmd = certbot_cmd + [action, "--non-interactive", "-v"]
             if action == "certonly":