feat(dstack-ingress): delegate every record, not just the challenge

Delegation moved the ACME challenge out of the served domain's zone, but
left three records behind that the operator still had to create by hand --
and one of them, the app-address TXT, carries the app id, which changes
whenever the compose changes. So "set it up once" was never true: edit the
compose, and someone has to go back to the production zone.

Alias all three names into the delegation zone instead. The operator
creates three CNAMEs before deploying and is then done, permanently:

    svc.example.com                      CNAME  svc.example.com.deleg.net
    _dstack-app-address.svc.example.com  CNAME  _dstack-app-address.svc.example.com.deleg.net
    _acme-challenge.svc.example.com      CNAME  _acme-challenge.svc.example.com.deleg.net

The container publishes what they point at, including the accounturi CAA
it previously could only ask for.

The gateway pointer becomes an address record rather than a CNAME, which
is the one real cost here. Let's Encrypt follows a CNAME when it looks up
CAA and honours what it finds at the target -- verified against the
staging CA -- but a CNAME excludes every other type at that name, so a
CNAME there would leave nowhere to put the CAA. It does not climb past
the target either: a CAA at the delegation zone apex is ignored, also
verified. An address record is what lets one name carry both. The
container re-resolves GATEWAY_DOMAIN every pass, so a gateway that moves
is picked up within RENEW_INTERVAL, where a CNAME used to follow it
immediately.

Wildcards keep the operator-managed CAA. RFC 8659 evaluates
*.example.com at example.com, which is the operator's own name and cannot
be aliased away.

Renewal passes still verify only the challenge alias. The other two are
about serving, and dns-01 never connects here, so blocking a renewal on
them would turn a routing problem into an expired certificate. The first
pass checks all three, because that is setup.

Author: kvinwang

custom-domain/dstack-ingress/README.md

@@ -205,19 +205,42 @@ served name lives under a shared production zone (e.g. `svc.example.com` under
 `example.com`), that token can edit every record in the zone, which may be more
 privilege than you want.
 
-Set `ACME_CHALLENGE_ALIAS=<delegation-zone>` to answer the DNS-01 challenge in a
-separate zone that your token controls, so the token never touches the served
-domain's zone. In this mode dstack-ingress **only** manages the challenge TXT in
-the delegation zone; you set the following records **once, statically**, in the
-served domain's production zone (the container prints the exact values on start):
+Set `ACME_CHALLENGE_ALIAS=<delegation-zone>` to move every name this deployment
+needs into a zone your token controls. You create three CNAMEs in the served
+domain's zone **once, before deploying**, and then never touch DNS again — not
+when the app id changes with the compose, not when the ACME account is
+recreated, not when the gateway moves:
 
 ```
-svc.example.com                       CNAME  <GATEWAY_DOMAIN>
-_dstack-app-address.svc.example.com   TXT    "<app-id>:<port>"
-_acme-challenge.svc.example.com       CNAME  _acme-challenge.svc.example.com.<delegation-zone>
-svc.example.com                       CAA    0 issue "letsencrypt.org;validationmethods=dns-01;accounturi=<account-uri>"
+svc.example.com                      CNAME  svc.example.com.deleg.example.net
+_dstack-app-address.svc.example.com  CNAME  _dstack-app-address.svc.example.com.deleg.example.net
+_acme-challenge.svc.example.com      CNAME  _acme-challenge.svc.example.com.deleg.example.net
 ```
 
+dstack-ingress publishes what they point at, in the delegation zone:
+
+```
+svc.example.com.deleg.example.net                      A    <gateway address>
+svc.example.com.deleg.example.net                      CAA  0 issue "letsencrypt.org;validationmethods=dns-01;accounturi=…"
+_dstack-app-address.svc.example.com.deleg.example.net  TXT  "<app-id>:<port>"
+_acme-challenge.svc.example.com.deleg.example.net      TXT  <challenge>   (transient)
+```
+
+The gateway pointer is an address record rather than a CNAME on purpose. Let's
+Encrypt follows a CNAME when it looks up CAA and honours what it finds at the
+target, but a CNAME excludes every other type at that name, so a CNAME there
+would leave nowhere to put the CAA. It does not climb past the target either —
+a CAA at the delegation zone apex is ignored. An address record is what lets one
+name carry both.
+
+The cost is that the container now tracks the gateway's address itself, where a
+CNAME used to follow it automatically. It re-resolves `GATEWAY_DOMAIN` on every
+pass, so a gateway that moves is picked up within `RENEW_INTERVAL`.
+
+**Wildcards keep an operator-managed CAA.** RFC 8659 evaluates `*.example.com`
+at `example.com`, which is your own name and cannot be aliased away, so for a
+wildcard the container prints the CAA and verifies it rather than publishing it.
+
 > **Security note.** The `accounturi` CAA restricts issuance to this enclave's
 > ACME account and is the control that prevents forged certificates. In
 > delegation mode dstack-ingress cannot set it for you (no token for the served
@@ -236,7 +259,14 @@ svc.example.com                       CAA    0 issue "letsencrypt.org;validation
 
 The records above are checked the same way tls-alpn-01 checks its own: two DoH
 resolvers, both CAA wire formats, and `DNS_SETUP_MODE` deciding what happens
-while they are missing. `wait` (the default) blocks until they appear, so you can
+while they are missing.
+
+Only the `_acme-challenge` CNAME is checked on later passes. Under dns-01 the CA
+reads a TXT record and never connects here, so the other two records are about
+serving rather than issuance, and a renewal is not blocked on them — a routing
+problem can be fixed at any time, an expired certificate cannot. They are still
+checked on the first pass, where the point is to tell you whether you created
+them correctly. `wait` (the default) blocks until they appear, so you can
 start the container and create the records afterwards; `print` lists them and
 continues without checking; `webhook` POSTs them to `DNS_WEBHOOK_URL` for an
 operator service to create automatically.

custom-domain/dstack-ingress/TESTING.md

@@ -212,14 +212,17 @@ the provider resolves a zone by longest-suffix match on the record name. With
 `_acme-challenge.svc.example.com.deleg.example.com`, which lands in
 `example.com` — the same zone, but through the delegation code path.
 
-Play the operator and create the three static records the container prints:
+Play the operator and create the three CNAMEs the container prints:
 
 ```
-svc.example.com                    CNAME  <GATEWAY_DOMAIN>
-_dstack-app-address.svc.example.com  TXT  "<app-id>:443"
-_acme-challenge.svc.example.com    CNAME  _acme-challenge.svc.example.com.deleg.example.com
+svc.example.com                      CNAME  svc.example.com.deleg.example.com
+_dstack-app-address.svc.example.com  CNAME  _dstack-app-address.svc.example.com.deleg.example.com
+_acme-challenge.svc.example.com      CNAME  _acme-challenge.svc.example.com.deleg.example.com
 ```
 
+Everything they point at is published by the container, so this is the whole of
+the operator's job — the app id can change afterwards and no DNS edit follows.
+
 Then start the container with a real provider token and `ACME_STAGING=true`, and
 watch for: the three records verifying, `Executing (challenge-delegation):` with
 `--manual` and both hooks, the certificate arriving, and the challenge TXT being
@@ -251,7 +254,9 @@ These fail fast and are cheap, so run them on every change:
 | Scenario | Expected |
 |---|---|
 | tls-alpn-01 + a wildcard domain | Refused before any ACME call, citing RFC 8737 |
+| Delegation, CAA in the delegated zone changed to forbid the CA | Blocked before any ACME attempt — proves the published CAA is reachable through the CNAME |
 | Delegation with no CAA record at all | Blocked — unlike normal issuance, where "no CAA" means unrestricted and passes |
+| Delegation, later pass, gateway CNAME broken | Renewal proceeds — that record is for serving, and dns-01 does not use it |
 | A CAA record with `validationmethods=dns-01`, mode tls-alpn-01 | Blocked with the restriction quoted back |
 | TXT holding the wrong value | Reported as `want <x>, saw <y>`, not "missing" |
 | An instance ID the gateway does not know | The CA reports a connection error — the gateway will not route to it |

custom-domain/dstack-ingress/scripts/dns01.sh

@@ -105,6 +105,67 @@ set_txt_record() {
     fi
 }
 
+# --- Full delegation --------------------------------------------------------
+#
+# The operator aliases three names into a zone this container can write, once,
+# before deploying. Everything those names resolve to is published here, so DNS
+# never needs touching again -- not when the app id changes with the compose,
+# not when the ACME account is recreated, not when the gateway moves.
+#
+# The gateway pointer has to be an A record rather than a CNAME. Let's Encrypt
+# follows a CNAME when it looks up CAA and honours what it finds at the target
+# (verified against the staging CA), but a CNAME excludes every other type at
+# that name, so a CNAME there would leave nowhere to put the CAA. It does not
+# climb past the target either: a CAA at the delegation zone apex is ignored.
+# An address record is what lets one name carry both.
+delegated_name() {
+    local domain="${1#\*.}"
+    echo "${domain}.${ACME_CHALLENGE_ALIAS}"
+}
+
+# Resolve GATEWAY_DOMAIN to addresses. Cheap and re-run every pass, since this
+# is now the only thing tracking the gateway; a CNAME used to do it for us.
+gateway_addresses() {
+    dnsguide.py --resolve "$GATEWAY_DOMAIN" 2>/dev/null
+}
+
+delegation_publish() {
+    local domain="$1" target addrs addr txt_name
+    target=$(delegated_name "$domain")
+
+    addrs=$(gateway_addresses)
+    if [ -z "$addrs" ]; then
+        echo "Error: could not resolve $GATEWAY_DOMAIN to any address" >&2
+        return 1
+    fi
+    while IFS= read -r addr; do
+        [ -n "$addr" ] || continue
+        dnsman.py set_a --domain "$target" --content "$addr" || return 1
+        break
+    done <<<"$addrs"
+
+    txt_name="$(txt_record_name "$domain").${ACME_CHALLENGE_ALIAS}"
+    dnsman.py set_txt --domain "$txt_name" --content "$(txt_record_value)" || return 1
+}
+
+# The accounturi CAA goes on the delegated name, beside the address record.
+#
+# Not for a wildcard: RFC 8659 evaluates `*.example.com` at `example.com`, which
+# is the operator's own name and cannot be aliased away. Wildcards keep the
+# operator-managed CAA.
+delegation_publish_caa() {
+    local domain="$1" account_file account_uri
+    if [[ "$domain" == \*.* ]]; then
+        return 2
+    fi
+    account_file=$(get_letsencrypt_account_file) || return 1
+    account_uri=$(jq -j '.uri' "$account_file")
+    dnsman.py set_caa \
+        --domain "$(delegated_name "$domain")" \
+        --caa-tag issue \
+        --caa-value "letsencrypt.org;validationmethods=dns-01;accounturi=$account_uri"
+}
+
 # Challenge delegation: this container holds no token for the served domain's
 # zone, so the operator creates those records. dnsguide.py is what prints and
 # verifies operator-managed records everywhere else in this image, so use it
@@ -160,12 +221,21 @@ delegation_verify_caa() {
 set_caa_record() {
     local domain="$1"
 
-    # Delegation is handled separately and is deliberately NOT gated by
-    # SET_CAA: there we cannot write the record, so verifying it is the only
-    # protection left.
+    # Delegation is deliberately NOT gated by SET_CAA: this record is the only
+    # thing stopping anyone else who can satisfy the delegated challenge, so it
+    # is not optional here.
     if [ -n "${ACME_CHALLENGE_ALIAS:-}" ]; then
-        delegation_verify_caa "$domain" || exit 1
-        return
+        delegation_publish_caa "$domain"
+        case $? in
+            0) return ;;
+            2) # Wildcard: RFC 8659 evaluates *.example.com at example.com,
+               # which is the operator's own name and cannot be aliased away,
+               # so ask for it instead of publishing it.
+               delegation_verify_caa "$domain" || exit 1
+               return ;;
+            *) echo "Error: could not publish the CAA record for $domain" >&2
+               exit 1 ;;
+        esac
     fi
 
     if [ "$SET_CAA" != "true" ]; then
@@ -204,10 +274,30 @@ process_domain() {
     local domain="$1" first status
 
     if [ -n "${ACME_CHALLENGE_ALIAS:-}" ]; then
-        # Delegation: the operator owns these. Verify them before spending an
-        # ACME attempt that would fail on records nobody has created yet. CAA
-        # is checked after the first issuance, once the account URI exists.
-        if ! delegation_guide "$domain" cname,txt,challenge-cname; then
+        # Delegation: the operator created three CNAMEs once, before deploying.
+        # Everything they point at is published here. Which of them block
+        # depends on the pass.
+        #
+        # Only the _acme-challenge alias is needed to issue: under dns-01 the CA
+        # reads a TXT record and never connects here, so the gateway CNAME and
+        # the app-address TXT are about serving, not about issuance. Blocking a
+        # renewal on them would turn a routing problem -- fixable at any time --
+        # into an expired certificate, which is not. Renewal wins.
+        #
+        # The first pass still checks all three, because that is setup: the
+        # operator has just been handed a list of records to create and wants to
+        # know whether they got them right.
+        # Publish first: the aliases the operator created point at names that
+        # have to exist before checking them proves anything.
+        if ! delegation_publish "$domain"; then
+            echo "Error: could not publish $domain into $ACME_CHALLENGE_ALIAS" >&2
+            return 1
+        fi
+        local want=challenge-cname
+        if [ "$FIRST_PASS" = "true" ]; then
+            want=delegated
+        fi
+        if ! delegation_guide "$domain" "$want"; then
             echo "Error: required DNS records for $domain are not in place" >&2
             return 1
         fi
@@ -278,6 +368,11 @@ collect_evidence() {
     evidence_finalize
 }
 
+# True until the first pass finishes. The first pass is setup and checks more
+# than issuance strictly needs; later passes check only what would stop a
+# renewal.
+FIRST_PASS=true
+
 # Set when certificates have been renewed but not yet applied to haproxy.
 # Survives across passes, so a failed apply is retried on the next one.
 APPLY_PENDING=false
@@ -319,6 +414,7 @@ run_pass() {
             echo "[$(date)] Certificate apply incomplete; retrying on the next pass" >&2
         fi
     fi
+    FIRST_PASS=false
     [ "$failed" -eq 0 ]
 }
 

custom-domain/dstack-ingress/scripts/dnsguide.py

@@ -402,6 +402,31 @@ def build_records(args: argparse.Namespace) -> List[Record]:
                 note="tells the gateway which instance to route this hostname to",
             )
         )
+    if "delegated" in include and args.challenge_alias:
+        # Full delegation: every name this deployment needs is aliased into a
+        # zone the container can write, so the operator creates these three once
+        # and never touches DNS again -- not when the app id changes, not when
+        # the ACME account changes, not when the gateway moves.
+        #
+        # Each is exact: the target holds only what we put there, so there is no
+        # address to fall back to and a wrong target must be reported as wrong
+        # rather than as "does not resolve".
+        base = args.domain[2:] if args.domain.startswith("*.") else args.domain
+        alias = args.challenge_alias
+        for name, note in (
+            (args.domain, "routes traffic for this hostname into the delegated zone"),
+            (args.txt_name, "lets the container publish the gateway routing target"),
+            (f"_acme-challenge.{base}", "delegates the ACME challenge"),
+        ):
+            records.append(
+                Record(
+                    type="CNAME",
+                    name=name,
+                    value=f"{name}.{alias}" if name != args.domain else f"{base}.{alias}",
+                    note=note,
+                    exact=True,
+                )
+            )
     if "challenge-cname" in include and args.challenge_alias:
         # Challenge delegation: the CA follows this CNAME when it looks for
         # _acme-challenge, so the TXT can live in a zone whose token we hold and
@@ -434,10 +459,10 @@ def build_records(args: argparse.Namespace) -> List[Record]:
 
 def main() -> int:
     parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument("--domain", required=True)
-    parser.add_argument("--alias-target", required=True, help="gateway domain")
-    parser.add_argument("--txt-name", required=True)
-    parser.add_argument("--txt-value", required=True)
+    parser.add_argument("--domain", default="")
+    parser.add_argument("--alias-target", default="", help="gateway domain")
+    parser.add_argument("--txt-name", default="")
+    parser.add_argument("--txt-value", default="")
     parser.add_argument("--caa-name", default="")
     parser.add_argument("--caa-tag", default="issue", choices=["issue", "issuewild"])
     parser.add_argument("--caa-value", default="")
@@ -471,15 +496,36 @@ def main() -> int:
         default=os.environ.get("DOH_RESOLVERS", os.environ.get("DOH_RESOLVER", DEFAULT_DOH)),
         help="comma-separated DoH endpoints",
     )
+    parser.add_argument(
+        "--resolve",
+        default="",
+        help="print the A records for a name and exit (used to publish the "
+             "gateway address into a delegated zone)",
+    )
     parser.add_argument("--json", action="store_true", help="also emit the records as JSON")
     parser.add_argument(
         "--include",
         default="cname,txt,caa",
         help="comma-separated subset of records to handle "
-             "(cname,txt,caa,challenge-cname)",
+             "(cname,txt,caa,challenge-cname,delegated)",
     )
     args = parser.parse_args()
 
+    if not args.resolve:
+        missing = [n for n, v in (("--domain", args.domain),
+                                  ("--alias-target", args.alias_target),
+                                  ("--txt-name", args.txt_name),
+                                  ("--txt-value", args.txt_value)) if not v]
+        if missing:
+            parser.error(f"{', '.join(missing)} required unless --resolve is given")
+
+    if args.resolve:
+        # Address lookup for callers that need to publish an A record rather
+        # than a CNAME. Same resolvers, same union semantics as everything else.
+        for addr in query_union(args.resolve, RR_A, args.resolver):
+            print(addr)
+        return 0
+
     if not args.caa_name:
         args.caa_name = args.domain.lstrip("*.")
 

custom-domain/dstack-ingress/scripts/dnsman.py

@@ -14,7 +14,7 @@ def main():
     )
     parser.add_argument(
         "action",
-        choices=["set_cname", "set_alias", "set_txt", "unset_txt", "set_caa"],
+        choices=["set_cname", "set_alias", "set_a", "set_txt", "unset_txt", "set_caa"],
         help="Action to perform",
     )
     parser.add_argument("--domain", required=True, help="Domain name")
@@ -67,6 +67,13 @@ def main():
                 sys.exit(1)
             print(f"Successfully set TXT record for {args.domain}")
 
+        elif args.action == "set_a":
+            success = provider.set_a_record(args.domain, args.content)
+            if not success:
+                print(f"Failed to set A record for {args.domain}", file=sys.stderr)
+                sys.exit(1)
+            print(f"Successfully set A record for {args.domain}")
+
         elif args.action == "unset_txt":
             success = provider.unset_txt_record(args.domain)
             if not success: