fix(dstack-ingress): keep the delegated gateway pointer a CNAME

The A record was my mistake. I reasoned from RFC 1034 -- a CNAME excludes
every other type at its name, so the delegated name could not hold both
the gateway pointer and the CAA -- and forgot that this repo's own
testing had already disproved it: Cloudflare allows the two together, and
Let's Encrypt refuses to issue when that CAA forbids it, which means it
reads it.

Confirmed again for the delegated shape specifically: with
`svc.example.com` aliased to a name that carries both a CNAME to the
gateway and a CAA, the staging CA reported `While processing CAA for
svc.example.com: CAA record for svc.example.com prevents issuance`. The
resolver stops at the first name in the chain that has a CAA rather than
following to the end.

So the gateway pointer stays a CNAME and a gateway that moves is followed
automatically, with no address tracking. DELEGATION_GATEWAY_RECORD=a
remains for providers that enforce the standard and refuse the CAA.

Also raise the delegated challenge's propagation wait from 30s to 120s.
It has to outlast the record's own 60s TTL, not just the write: on the
second of the two issuance attempts a resolver still serving the previous
challenge value fails Let's Encrypt's multi-perspective check with
"During secondary validation: Incorrect TXT record found". The dns-01
plugin path already waits 120s for the same reason.

Author: kvinwang

custom-domain/dstack-ingress/README.md

@@ -192,7 +192,8 @@ environment:
 | `EVIDENCE_PORT` | `80` | Internal port for evidence HTTP server |
 | `ALPN` | | TLS ALPN protocols (e.g. `h2,http/1.1`). Only set if backends support h2c |
 | `ACME_CHALLENGE_ALIAS` | | Delegate the ACME DNS-01 challenge to this zone (see below) so the DNS token needs no access to the served domain's own zone |
-| `ACME_CHALLENGE_PROPAGATION_SECONDS` | `30` | Wait after writing the delegated challenge TXT before validation (only with `ACME_CHALLENGE_ALIAS`). Keep well under ~250s — certbot is killed after a 300s per-run timeout |
+| `ACME_CHALLENGE_PROPAGATION_SECONDS` | `120` | Wait after writing the delegated challenge TXT before validation. Must outlast the record TTL (60s), or a resolver still serving the previous attempt's value fails validation. Keep well under ~250s — certbot is killed after a 300s per-run timeout |
+| `DELEGATION_GATEWAY_RECORD` | `cname` | How the delegated name points at the gateway: `cname` (follows a gateway that moves) or `a` (for providers that refuse a CAA beside a CNAME) |
 | `ALLOW_MISSING_CAA` | `false` | In delegation mode, treat an unconfirmed `accounturi` CAA as a warning instead of a blocker. Default fails closed (see below) |
 
 For DNS provider credentials, see [DNS_PROVIDERS.md](DNS_PROVIDERS.md).
@@ -220,22 +221,24 @@ _acme-challenge.svc.example.com      CNAME  _acme-challenge.svc.example.com.dele
 dstack-ingress publishes what they point at, in the delegation zone:
 
 ```
-svc.example.com.deleg.example.net                      A    <gateway address>
-svc.example.com.deleg.example.net                      CAA  0 issue "letsencrypt.org;validationmethods=dns-01;accounturi=…"
-_dstack-app-address.svc.example.com.deleg.example.net  TXT  "<app-id>:<port>"
-_acme-challenge.svc.example.com.deleg.example.net      TXT  <challenge>   (transient)
+svc.example.com.deleg.example.net                      CNAME  <GATEWAY_DOMAIN>
+svc.example.com.deleg.example.net                      CAA    0 issue "letsencrypt.org;validationmethods=dns-01;accounturi=…"
+_dstack-app-address.svc.example.com.deleg.example.net  TXT    "<app-id>:<port>"
+_acme-challenge.svc.example.com.deleg.example.net      TXT    <challenge>   (transient)
 ```
 
-The gateway pointer is an address record rather than a CNAME on purpose. Let's
-Encrypt follows a CNAME when it looks up CAA and honours what it finds at the
-target, but a CNAME excludes every other type at that name, so a CNAME there
-would leave nowhere to put the CAA. It does not climb past the target either —
-a CAA at the delegation zone apex is ignored. An address record is what lets one
-name carry both.
-
-The cost is that the container now tracks the gateway's address itself, where a
-CNAME used to follow it automatically. It re-resolves `GATEWAY_DOMAIN` on every
-pass, so a gateway that moves is picked up within `RENEW_INTERVAL`.
+A CNAME and a CAA on the same name is a Cloudflare allowance, not a standard
+one — RFC 1034 says a CNAME excludes every other type at its name. It works
+because the resolver stops at the first name in the chain that carries a CAA,
+and Let's Encrypt honours what it finds there: with a CAA that forbids it,
+the staging CA refuses with `While processing CAA for svc.example.com: CAA
+record for svc.example.com prevents issuance`.
+
+Providers that enforce the standard will reject the CAA beside the CNAME. Set
+`DELEGATION_GATEWAY_RECORD=a` there: the container resolves `GATEWAY_DOMAIN`
+itself and publishes an address record, which can carry the CAA. The cost is
+that a gateway which moves is then only picked up on the next pass, where a
+CNAME follows it immediately.
 
 **Wildcards keep an operator-managed CAA.** RFC 8659 evaluates `*.example.com`
 at `example.com`, which is your own name and cannot be aliased away, so for a

custom-domain/dstack-ingress/scripts/acme-dns-alias-hook.sh

@@ -36,9 +36,16 @@ case "$action" in
         echo "acme-dns-alias-hook: writing challenge TXT to delegated record $record"
         dnsman.py set_txt --domain "$record" --content "$CERTBOT_VALIDATION"
         # certbot asks Let's Encrypt to validate immediately after this hook
-        # returns, so wait for the record to propagate on the delegated zone's
-        # authoritative servers before returning.
-        sleep "${ACME_CHALLENGE_PROPAGATION_SECONDS:-30}"
+        # returns, so wait for the record to propagate before returning.
+        #
+        # This has to outlast the record's own TTL, not just the time the write
+        # takes. dnsman.py publishes TXT with a 60s TTL, so a resolver that saw
+        # the *previous* challenge value keeps serving it for up to that long --
+        # which is exactly the situation on the second of the two issuance
+        # attempts. At 30s Let's Encrypt's multi-perspective check fails with
+        # "During secondary validation: Incorrect TXT record found". The dns-01
+        # plugin path waits 120s for the same reason.
+        sleep "${ACME_CHALLENGE_PROPAGATION_SECONDS:-120}"
         ;;
     cleanup)
         echo "acme-dns-alias-hook: removing challenge TXT $record"

custom-domain/dstack-ingress/scripts/dns01.sh

@@ -112,37 +112,44 @@ set_txt_record() {
 # never needs touching again -- not when the app id changes with the compose,
 # not when the ACME account is recreated, not when the gateway moves.
 #
-# The gateway pointer has to be an A record rather than a CNAME. Let's Encrypt
-# follows a CNAME when it looks up CAA and honours what it finds at the target
-# (verified against the staging CA), but a CNAME excludes every other type at
-# that name, so a CNAME there would leave nowhere to put the CAA. It does not
-# climb past the target either: a CAA at the delegation zone apex is ignored.
-# An address record is what lets one name carry both.
+# The gateway pointer stays a CNAME, so a gateway that moves is followed
+# automatically. That the same name can also carry the CAA is a Cloudflare
+# allowance rather than a standard one -- RFC 1034 says a CNAME excludes every
+# other type at its name -- but Let's Encrypt honours it: the resolver stops at
+# the first name in the chain that has a CAA, and the staging CA refuses to
+# issue when that record forbids it. Providers that enforce the standard will
+# reject the CAA here; see DELEGATION_GATEWAY_RECORD.
 delegated_name() {
     local domain="${1#\*.}"
     echo "${domain}.${ACME_CHALLENGE_ALIAS}"
 }
 
-# Resolve GATEWAY_DOMAIN to addresses. Cheap and re-run every pass, since this
-# is now the only thing tracking the gateway; a CNAME used to do it for us.
-gateway_addresses() {
-    dnsguide.py --resolve "$GATEWAY_DOMAIN" 2>/dev/null
-}
-
 delegation_publish() {
-    local domain="$1" target addrs addr txt_name
+    local domain="$1" target txt_name
     target=$(delegated_name "$domain")
 
-    addrs=$(gateway_addresses)
-    if [ -z "$addrs" ]; then
-        echo "Error: could not resolve $GATEWAY_DOMAIN to any address" >&2
-        return 1
-    fi
-    while IFS= read -r addr; do
-        [ -n "$addr" ] || continue
-        dnsman.py set_a --domain "$target" --content "$addr" || return 1
-        break
-    done <<<"$addrs"
+    case "${DELEGATION_GATEWAY_RECORD:-cname}" in
+        cname)
+            dnsman.py set_alias --domain "$target" --content "$GATEWAY_DOMAIN" || return 1
+            ;;
+        a)
+            # For providers that refuse a CAA beside a CNAME. Costs the
+            # automatic following of a gateway that moves: the address is
+            # re-resolved once per pass instead.
+            local addrs addr
+            addrs=$(dnsguide.py --resolve "$GATEWAY_DOMAIN" 2>/dev/null)
+            if [ -z "$addrs" ]; then
+                echo "Error: could not resolve $GATEWAY_DOMAIN to any address" >&2
+                return 1
+            fi
+            addr=$(echo "$addrs" | head -1)
+            dnsman.py set_a --domain "$target" --content "$addr" || return 1
+            ;;
+        *)
+            echo "Error: invalid DELEGATION_GATEWAY_RECORD (expected cname or a)" >&2
+            return 1
+            ;;
+    esac
 
     txt_name="$(txt_record_name "$domain").${ACME_CHALLENGE_ALIAS}"
     dnsman.py set_txt --domain "$txt_name" --content "$(txt_record_value)" || return 1