feat(dstack-ingress): delegate every record, so DNS is configured once

custom-domain/dstack-ingress/README.md

@@ -191,9 +191,8 @@ environment:
 | `EVIDENCE_SERVER` | `true` | Serve evidence files at `/evidences/` on the TLS port |
 | `EVIDENCE_PORT` | `80` | Internal port for evidence HTTP server |
 | `ALPN` | | TLS ALPN protocols (e.g. `h2,http/1.1`). Only set if backends support h2c |
-| `ACME_CHALLENGE_ALIAS` | | Delegate the ACME DNS-01 challenge to this zone (see below) so the DNS token needs no access to the served domain's own zone |
-| `ACME_CHALLENGE_PROPAGATION_SECONDS` | `30` | Wait after writing the delegated challenge TXT before validation (only with `ACME_CHALLENGE_ALIAS`). Keep well under ~250s — certbot is killed after a 300s per-run timeout |
-| `ALLOW_MISSING_CAA` | `false` | In delegation mode, treat an unconfirmed `accounturi` CAA as a warning instead of a blocker. Default fails closed (see below) |
+| `DELEGATION_ZONE` | | Zone this container writes into, so the DNS token needs no access to the served domain's own zone (see below) |
+| `DELEGATION_PROPAGATION_SECONDS` | `120` | Wait after writing the delegated challenge TXT before validation. Must outlast the record TTL (60s), or a resolver still serving the previous attempt's value fails validation. Keep well under ~250s — certbot is killed after a 300s per-run timeout |
 
 For DNS provider credentials, see [DNS_PROVIDERS.md](DNS_PROVIDERS.md).
 
@@ -205,38 +204,66 @@ served name lives under a shared production zone (e.g. `svc.example.com` under
 `example.com`), that token can edit every record in the zone, which may be more
 privilege than you want.
 
-Set `ACME_CHALLENGE_ALIAS=<delegation-zone>` to answer the DNS-01 challenge in a
-separate zone that your token controls, so the token never touches the served
-domain's zone. In this mode dstack-ingress **only** manages the challenge TXT in
-the delegation zone; you set the following records **once, statically**, in the
-served domain's production zone (the container prints the exact values on start):
+> **Renamed.** This was `ACME_CHALLENGE_ALIAS`, from when the challenge was the
+> only thing delegated. The old name is **not** accepted — a deployment still
+> setting it silently leaves delegation mode and starts writing to the served
+> domain's zone, which is what delegation exists to avoid. Rename it when
+> upgrading.
+
+Set `DELEGATION_ZONE=<delegation-zone>` to move every name this deployment
+needs into a zone your token controls. You create three CNAMEs in the served
+domain's zone **once, before deploying**, and then never touch DNS again — not
+when the app id moves, not when the ACME account is recreated, not when the
+gateway moves:
+
+```
+svc.example.com                      CNAME  svc.example.com.deleg.example.net
+_dstack-app-address.svc.example.com  CNAME  _dstack-app-address.svc.example.com.deleg.example.net
+_acme-challenge.svc.example.com      CNAME  _acme-challenge.svc.example.com.deleg.example.net
+```
+
+dstack-ingress publishes what they point at, in the delegation zone:
+
+```
+svc.example.com.deleg.example.net                      CNAME  <GATEWAY_DOMAIN>
+svc.example.com.deleg.example.net                      CAA    0 issue "letsencrypt.org;validationmethods=dns-01;accounturi=…"
+_dstack-app-address.svc.example.com.deleg.example.net  TXT    "<app-id>:<port>"
+_acme-challenge.svc.example.com.deleg.example.net      TXT    <challenge>   (transient)
+```
+
+The gateway pointer and the CAA share a name, which DNS does not allow for a
+CNAME. Which form works is a property of the provider — Cloudflare tolerates the
+pair, Linode publishes an address record instead — and `set_alias_record` in the
+provider layer is where that choice is made. Delegation asks for an alias and
+takes what it gets. A provider that needs a form other than CNAME overrides that
+method, as Linode does.
+
+**A wildcard needs a fourth CNAME.** RFC 8659 evaluates `*.app.example.com` at
+`app.example.com`, so the base gets its own alias and the container publishes an
+`issuewild` CAA behind it:
 
 ```
-svc.example.com                       CNAME  <GATEWAY_DOMAIN>
-_dstack-app-address.svc.example.com   TXT    "<app-id>:<port>"
-_acme-challenge.svc.example.com       CNAME  _acme-challenge.svc.example.com.<delegation-zone>
-svc.example.com                       CAA    0 issue "letsencrypt.org;validationmethods=dns-01;accounturi=<account-uri>"
+*.app.example.com                            CNAME  app.example.com.deleg.example.net
+app.example.com                              CNAME  app.example.com.deleg.example.net
+_dstack-app-address-wildcard.app.example.com CNAME  _dstack-app-address-wildcard.app.example.com.deleg.example.net
+_acme-challenge.app.example.com              CNAME  _acme-challenge.app.example.com.deleg.example.net
 ```
 
-> **Security note.** The `accounturi` CAA restricts issuance to this enclave's
-> ACME account and is the control that prevents forged certificates. In
-> delegation mode dstack-ingress cannot set it for you (no token for the served
-> zone), so it prints the record and verifies it. **A CAA that is confirmed
-> absent stops issuance** — set `ALLOW_MISSING_CAA=true` to downgrade that to a
-> warning. Without this CAA, anyone who can satisfy the delegated challenge
-> could obtain a certificate for the domain.
->
-> Use a **dedicated** delegation zone and scope the token to only that zone — a
-> zone shared with other tenants lets anyone with write access to it complete
-> the challenge. `SET_CAA` does not affect delegation mode (this CAA path always
-> runs). If a certificate was previously issued with the standard DNS-plugin
-> authenticator, delete `/etc/letsencrypt/renewal/<domain>.conf` before enabling
-> delegation, otherwise `certbot renew` reuses the old plugin (which needs the
-> production-zone token this mode avoids).
+The base has to be a name you can alias, which rules out a zone apex — an apex
+carries SOA and NS records and a CNAME excludes them. `*.example.com` served
+straight off the `example.com` zone is therefore not a fit for delegation; a
+label down, `*.app.example.com`, is.
 
 The records above are checked the same way tls-alpn-01 checks its own: two DoH
 resolvers, both CAA wire formats, and `DNS_SETUP_MODE` deciding what happens
-while they are missing. `wait` (the default) blocks until they appear, so you can
+while they are missing.
+
+Only the `_acme-challenge` CNAME is checked on later passes. Under dns-01 the CA
+reads a TXT record and never connects here, so the other two records are about
+serving rather than issuance, and a renewal is not blocked on them — a routing
+problem can be fixed at any time, an expired certificate cannot. They are still
+checked on the first pass, where the point is to tell you whether you created
+them correctly. `wait` (the default) blocks until they appear, so you can
 start the container and create the records afterwards; `print` lists them and
 continues without checking; `webhook` POSTs them to `DNS_WEBHOOK_URL` for an
 operator service to create automatically.

custom-domain/dstack-ingress/TESTING.md

@@ -201,25 +201,28 @@ first issuance.
 
 ### Challenge delegation (dns-01)
 
-`ACME_CHALLENGE_ALIAS` answers the DNS-01 challenge in a zone the container's
+`DELEGATION_ZONE` answers the DNS-01 challenge in a zone the container's
 token controls, so the token needs no access to the served domain's own zone.
 Testing it looks like it needs a second registrar account. It does not.
 
 Any name under a zone you already control works as the delegation zone, because
 the provider resolves a zone by longest-suffix match on the record name. With
-`ACME_CHALLENGE_ALIAS=deleg.example.com` and a served domain of
+`DELEGATION_ZONE=deleg.example.com` and a served domain of
 `svc.example.com`, the challenge record is
 `_acme-challenge.svc.example.com.deleg.example.com`, which lands in
 `example.com` — the same zone, but through the delegation code path.
 
-Play the operator and create the three static records the container prints:
+Play the operator and create the three CNAMEs the container prints:
 
 ```
-svc.example.com                    CNAME  <GATEWAY_DOMAIN>
-_dstack-app-address.svc.example.com  TXT  "<app-id>:443"
-_acme-challenge.svc.example.com    CNAME  _acme-challenge.svc.example.com.deleg.example.com
+svc.example.com                      CNAME  svc.example.com.deleg.example.com
+_dstack-app-address.svc.example.com  CNAME  _dstack-app-address.svc.example.com.deleg.example.com
+_acme-challenge.svc.example.com      CNAME  _acme-challenge.svc.example.com.deleg.example.com
 ```
 
+Everything they point at is published by the container, so this is the whole of
+the operator's job — the app id can change afterwards and no DNS edit follows.
+
 Then start the container with a real provider token and `ACME_STAGING=true`, and
 watch for: the three records verifying, `Executing (challenge-delegation):` with
 `--manual` and both hooks, the certificate arriving, and the challenge TXT being
@@ -251,7 +254,9 @@ These fail fast and are cheap, so run them on every change:
 | Scenario | Expected |
 |---|---|
 | tls-alpn-01 + a wildcard domain | Refused before any ACME call, citing RFC 8737 |
+| Delegation, CAA in the delegated zone changed to forbid the CA | Blocked before any ACME attempt — proves the published CAA is reachable through the CNAME |
 | Delegation with no CAA record at all | Blocked — unlike normal issuance, where "no CAA" means unrestricted and passes |
+| Delegation, later pass, gateway CNAME broken | Renewal proceeds — that record is for serving, and dns-01 does not use it |
 | A CAA record with `validationmethods=dns-01`, mode tls-alpn-01 | Blocked with the restriction quoted back |
 | TXT holding the wrong value | Reported as `want <x>, saw <y>`, not "missing" |
 | An instance ID the gateway does not know | The CA reports a connection error — the gateway will not route to it |

custom-domain/dstack-ingress/scripts/acme-dns-alias-hook.sh

@@ -3,10 +3,10 @@
 #
 # Instead of writing the `_acme-challenge` TXT into the served domain's own
 # zone (which requires a DNS token for that zone), this writes it into a
-# delegated zone that our token controls (ACME_CHALLENGE_ALIAS). The served
+# delegated zone that our token controls (DELEGATION_ZONE). The served
 # domain's zone only needs one static, operator-managed CNAME:
 #
-#   _acme-challenge.<domain>  CNAME  _acme-challenge.<domain>.<ACME_CHALLENGE_ALIAS>
+#   _acme-challenge.<domain>  CNAME  _acme-challenge.<domain>.<DELEGATION_ZONE>
 #
 # Let's Encrypt follows that CNAME during validation and reads the TXT from the
 # delegated zone, so the enclave's token never needs access to the served
@@ -18,8 +18,9 @@ set -euo pipefail
 
 action="${1:-}"
 
-if [ -z "${ACME_CHALLENGE_ALIAS:-}" ]; then
-    echo "acme-dns-alias-hook: ACME_CHALLENGE_ALIAS is not set" >&2
+zone="${DELEGATION_ZONE:-}"
+if [ -z "$zone" ]; then
+    echo "acme-dns-alias-hook: DELEGATION_ZONE is not set" >&2
     exit 1
 fi
 if [ -z "${CERTBOT_DOMAIN:-}" ]; then
@@ -28,17 +29,24 @@ if [ -z "${CERTBOT_DOMAIN:-}" ]; then
 fi
 
 # certbot always passes the base domain (no leading "*.") in CERTBOT_DOMAIN.
-record="_acme-challenge.${CERTBOT_DOMAIN}.${ACME_CHALLENGE_ALIAS}"
+record="_acme-challenge.${CERTBOT_DOMAIN}.${zone}"
 
 case "$action" in
     auth)
         : "${CERTBOT_VALIDATION:?CERTBOT_VALIDATION not set}"
         echo "acme-dns-alias-hook: writing challenge TXT to delegated record $record"
         dnsman.py set_txt --domain "$record" --content "$CERTBOT_VALIDATION"
         # certbot asks Let's Encrypt to validate immediately after this hook
-        # returns, so wait for the record to propagate on the delegated zone's
-        # authoritative servers before returning.
-        sleep "${ACME_CHALLENGE_PROPAGATION_SECONDS:-30}"
+        # returns, so wait for the record to propagate before returning.
+        #
+        # This has to outlast the record's own TTL, not just the time the write
+        # takes. dnsman.py publishes TXT with a 60s TTL, so a resolver that saw
+        # the *previous* challenge value keeps serving it for up to that long --
+        # which is exactly the situation on the second of the two issuance
+        # attempts. At 30s Let's Encrypt's multi-perspective check fails with
+        # "During secondary validation: Incorrect TXT record found". The dns-01
+        # plugin path waits 120s for the same reason.
+        sleep "${DELEGATION_PROPAGATION_SECONDS:-120}"
         ;;
     cleanup)
         echo "acme-dns-alias-hook: removing challenge TXT $record"

custom-domain/dstack-ingress/scripts/certman.py

@@ -279,11 +279,11 @@ def _build_certbot_command(self, action: str, domain: str, email: str) -> List[s
         """Build certbot command using provider configuration."""
         certbot_cmd = self._get_certbot_command()
 
-        # Challenge-delegation mode: when ACME_CHALLENGE_ALIAS is set, answer the
+        # Challenge-delegation mode: when DELEGATION_ZONE is set, answer the
         # DNS-01 challenge in a delegated zone via a manual hook instead of the
         # provider's certbot plugin, so our DNS token never needs access to the
         # served domain's own zone. See scripts/acme-dns-alias-hook.sh.
-        if os.environ.get("ACME_CHALLENGE_ALIAS", "").strip():
+        if os.environ.get("DELEGATION_ZONE", "").strip():
             hook = "/scripts/acme-dns-alias-hook.sh"
             base_cmd = certbot_cmd + [action, "--non-interactive", "-v"]
             if action == "certonly":