fix(dstack-ingress): address review — reachable & fail-closed CAA, wildcard CNAME, robust DoH verify

- entrypoint.sh: the delegation-mode accounturi-CAA check was unreachable in the
  default config (it lived after the `SET_CAA != true` early return, and SET_CAA
  defaults to false), so the forge-prevention control silently never ran. Move
  it above the SET_CAA gate (delegation CAA is independent of SET_CAA) and make
  it fail closed: if the required CAA is confirmed absent the container exits,
  with ALLOW_MISSING_CAA=true to override. A transient DoH failure only warns.
- entrypoint.sh: fix the wildcard `_acme-challenge` CNAME guidance — strip `*.`
  so it matches the base name certbot/LE validate and the hook writes to.
- entrypoint.sh: parse the DoH response with jq and match with `grep -F`
  (the account URI contains `/` and `.`, which must not be regex), and
  distinguish "confirmed absent" (fail) from "could not verify" (warn).
- base.py: unset_txt_record no longer reports success for a record it could not
  address (no id) — counts it as failure and warns.
- README: document fail-closed CAA + ALLOW_MISSING_CAA, the propagation ceiling
  vs certbot's 300s timeout, using a dedicated delegation zone, and deleting a
  stale plugin renewal conf before switching to delegation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xa1zFJs3FVP8UTsSWuS3Yj

Author: Ravenyjh

custom-domain/dstack-ingress/README.md

@@ -180,7 +180,8 @@ environment:
 | `EVIDENCE_PORT` | `80` | Internal port for evidence HTTP server |
 | `ALPN` | | TLS ALPN protocols (e.g. `h2,http/1.1`). Only set if backends support h2c |
 | `ACME_CHALLENGE_ALIAS` | | Delegate the ACME DNS-01 challenge to this zone (see below) so the DNS token needs no access to the served domain's own zone |
-| `ACME_CHALLENGE_PROPAGATION_SECONDS` | `30` | Wait after writing the delegated challenge TXT before validation (only used with `ACME_CHALLENGE_ALIAS`) |
+| `ACME_CHALLENGE_PROPAGATION_SECONDS` | `30` | Wait after writing the delegated challenge TXT before validation (only with `ACME_CHALLENGE_ALIAS`). Keep well under ~250s — certbot is killed after a 300s per-run timeout |
+| `ALLOW_MISSING_CAA` | `false` | In delegation mode, continue even if the required `accounturi` CAA cannot be confirmed. Default fails closed (see below) |
 
 For DNS provider credentials, see [DNS_PROVIDERS.md](DNS_PROVIDERS.md).
 
@@ -206,11 +207,21 @@ svc.example.com                       CAA    0 issue "letsencrypt.org;validation
 ```
 
 > **Security note.** The `accounturi` CAA restricts issuance to this enclave's
-> ACME account. In delegation mode dstack-ingress cannot set it for you (no
-> token for the served zone), so it prints the record and verifies (via DoH)
-> that it is present, warning loudly if not. Without this CAA, anyone who can
-> satisfy the delegated challenge could obtain a certificate for the domain.
-> Provide the token scoped only to `<delegation-zone>`.
+> ACME account and is the control that prevents forged certificates. In
+> delegation mode dstack-ingress cannot set it for you (no token for the served
+> zone), so it prints the record and verifies its presence via DoH. **If the
+> CAA is confirmed absent the container fails to start** (set `ALLOW_MISSING_CAA=true`
+> to override; a transient DoH failure only warns, it does not block). Without
+> this CAA, anyone who can satisfy the delegated challenge could obtain a
+> certificate for the domain.
+>
+> Use a **dedicated** delegation zone and scope the token to only that zone — a
+> zone shared with other tenants lets anyone with write access to it complete
+> the challenge. `SET_CAA` does not affect delegation mode (this CAA path always
+> runs). If a certificate was previously issued with the standard DNS-plugin
+> authenticator, delete `/etc/letsencrypt/renewal/<domain>.conf` before enabling
+> delegation, otherwise `certbot renew` reuses the old plugin (which needs the
+> production-zone token this mode avoids).
 
 ## Evidence & Attestation
 

custom-domain/dstack-ingress/scripts/dns_providers/base.py

@@ -261,7 +261,11 @@ def unset_txt_record(self, name: str) -> bool:
         """
         ok = True
         for record in self.get_dns_records(name, RecordType.TXT):
-            if record.id and not self.delete_dns_record(record.id, name):
+            if not record.id:
+                print(f"Warning: TXT record for {name} has no id; cannot delete")
+                ok = False
+                continue
+            if not self.delete_dns_record(record.id, name):
                 ok = False
         return ok
 

custom-domain/dstack-ingress/scripts/entrypoint.sh

@@ -269,10 +269,13 @@ backend ${be_name}
 set_alias_record() {
     local domain="$1"
     if [ -n "${ACME_CHALLENGE_ALIAS:-}" ]; then
+        # certbot validates the challenge at the base name even for a wildcard,
+        # so the _acme-challenge CNAME must use the base domain (strip "*.").
+        local base="${domain#\*.}"
         echo "[challenge-delegation] Not touching ${domain}'s own zone (token is scoped to the delegated zone)."
         echo "[challenge-delegation] Set these in your production zone yourself (static, one-time):"
         echo "    ${domain}  CNAME  ${GATEWAY_DOMAIN}"
-        echo "    _acme-challenge.${domain}  CNAME  _acme-challenge.${domain}.${ACME_CHALLENGE_ALIAS}"
+        echo "    _acme-challenge.${base}  CNAME  _acme-challenge.${base}.${ACME_CHALLENGE_ALIAS}"
         return
     fi
     echo "Setting alias record for $domain"
@@ -324,61 +327,101 @@ set_txt_record() {
     fi
 }
 
-set_caa_record() {
+# caa_domain_and_tag DOMAIN -> prints "caa_domain caa_tag" (strips a wildcard).
+caa_domain_and_tag() {
     local domain="$1"
-    if [ "$SET_CAA" != "true" ]; then
-        echo "Skipping CAA record setup"
-        return
+    if [[ "$domain" == \*.* ]]; then
+        echo "${domain#\*.} issuewild"
+    else
+        echo "$domain issue"
     fi
+}
 
-    local ACCOUNT_URI
-    local account_file
+# In delegation mode we have NO token for the served domain's zone, so we cannot
+# set the accounturi CAA ourselves. That CAA is the forge-prevention (only this
+# enclave's ACME account may issue), so this path is deliberately NOT gated by
+# SET_CAA and fails closed if the record is confirmed absent — otherwise anyone
+# who can satisfy the delegated DNS-01 challenge could obtain a cert for the
+# domain. Set ALLOW_MISSING_CAA=true to override (accept the risk).
+verify_delegation_caa() {
+    local domain="$1"
+    local account_file account_uri caa_domain caa_tag caa_value resp status found
 
     if ! account_file=$(get_letsencrypt_account_file); then
-        echo "Warning: Cannot set CAA record - account file not found"
-        echo "This is not critical - certificates can still be issued without CAA records"
+        echo "ERROR: cannot read the Let's Encrypt account file to determine the required accounturi CAA" >&2
+        caa_fail_or_allow "$domain"
         return
     fi
+    read -r caa_domain caa_tag < <(caa_domain_and_tag "$domain")
+    account_uri=$(jq -j '.uri' "$account_file")
+    caa_value="letsencrypt.org;validationmethods=dns-01;accounturi=$account_uri"
 
-    local caa_domain caa_tag
-    if [[ "$domain" == \*.* ]]; then
-        caa_domain="${domain#\*.}"
-        caa_tag="issuewild"
-    else
-        caa_domain="$domain"
-        caa_tag="issue"
+    echo "[challenge-delegation] Set this CAA in your production zone (static, one-time):"
+    echo "    ${caa_domain}  CAA  0 ${caa_tag} \"${caa_value}\""
+
+    # Verify via DNS-over-HTTPS (dig is not installed in the image; curl+jq are).
+    resp=$(curl -s --max-time 10 "https://dns.google/resolve?name=${caa_domain}&type=257" 2>/dev/null || true)
+    if [ -z "$resp" ]; then
+        echo "WARNING: could not reach dns.google to verify the CAA (network issue) — NOT confirmed; continuing"
+        return
+    fi
+    status=$(echo "$resp" | jq -r '.Status // empty' 2>/dev/null || true)
+    if [ "$status" != "0" ]; then
+        echo "WARNING: CAA DoH query for ${caa_domain} returned status ${status:-unknown} — NOT confirmed; continuing"
+        return
+    fi
+    # Literal (grep -F) match on the CAA data fields — the account URI contains
+    # '/' and '.', which must not be treated as regex.
+    found=$(echo "$resp" | jq -r '.Answer // [] | .[] | .data' 2>/dev/null | grep -F "accounturi=$account_uri" || true)
+    if [ -n "$found" ]; then
+        echo "[challenge-delegation] Verified: accounturi CAA is present for $caa_domain"
+        return
+    fi
+    echo "ERROR: the accounturi CAA is NOT present for $caa_domain." >&2
+    echo "ERROR: without it, anyone who can satisfy the delegated DNS-01 challenge could obtain a" >&2
+    echo "ERROR: publicly-trusted certificate for this domain (forged TLS termination)." >&2
+    caa_fail_or_allow "$domain"
+}
+
+caa_fail_or_allow() {
+    local domain="$1"
+    if [ "${ALLOW_MISSING_CAA:-false}" = "true" ]; then
+        echo "WARNING: ALLOW_MISSING_CAA=true — continuing without a verified accounturi CAA for $domain"
+        return 0
     fi
+    echo "ERROR: refusing to continue without the accounturi CAA for $domain." >&2
+    echo "ERROR: set the CAA record shown above, or ALLOW_MISSING_CAA=true to override." >&2
+    exit 1
+}
 
-    ACCOUNT_URI=$(jq -j '.uri' "$account_file")
+set_caa_record() {
+    local domain="$1"
 
+    # Delegation mode is handled separately and is NOT gated by SET_CAA.
     if [ -n "${ACME_CHALLENGE_ALIAS:-}" ]; then
-        # Delegation mode: our token has no access to the served domain's zone,
-        # so we cannot set the accounturi CAA ourselves. This CAA is the
-        # forge-prevention (only the enclave's ACME account may issue), so we do
-        # NOT silently skip it — we print the exact record the operator must set,
-        # then best-effort verify it and warn loudly if it is missing.
-        local caa_value="letsencrypt.org;validationmethods=dns-01;accounturi=$ACCOUNT_URI"
-        echo "[challenge-delegation] Set this CAA in your production zone yourself (static):"
-        echo "    ${caa_domain}  CAA  0 ${caa_tag} \"${caa_value}\""
-        # Verify via DNS-over-HTTPS (dig is not installed; curl is).
-        local caa_answer
-        caa_answer=$(curl -s --max-time 10 "https://dns.google/resolve?name=${caa_domain}&type=257" 2>/dev/null || true)
-        if echo "$caa_answer" | grep -q "accounturi=$ACCOUNT_URI"; then
-            echo "[challenge-delegation] Verified: accounturi CAA is present for $caa_domain"
-        else
-            echo "WARNING: accounturi CAA is NOT present for $caa_domain."
-            echo "WARNING: Without it, anyone who can satisfy the DNS-01 challenge could obtain a"
-            echo "WARNING: publicly-trusted certificate for this domain (forged TLS termination)."
-            echo "WARNING: Set the CAA record shown above before relying on this endpoint."
-        fi
+        verify_delegation_caa "$domain"
+        return
+    fi
+
+    if [ "$SET_CAA" != "true" ]; then
+        echo "Skipping CAA record setup"
+        return
+    fi
+
+    local account_file account_uri caa_domain caa_tag
+    if ! account_file=$(get_letsencrypt_account_file); then
+        echo "Warning: Cannot set CAA record - account file not found"
+        echo "This is not critical - certificates can still be issued without CAA records"
         return
     fi
+    read -r caa_domain caa_tag < <(caa_domain_and_tag "$domain")
+    account_uri=$(jq -j '.uri' "$account_file")
 
-    echo "Adding CAA record ($caa_tag) for $caa_domain, accounturi=$ACCOUNT_URI"
+    echo "Adding CAA record ($caa_tag) for $caa_domain, accounturi=$account_uri"
     dnsman.py set_caa \
         --domain "$caa_domain" \
         --caa-tag "$caa_tag" \
-        --caa-value "letsencrypt.org;validationmethods=dns-01;accounturi=$ACCOUNT_URI"
+        --caa-value "letsencrypt.org;validationmethods=dns-01;accounturi=$account_uri"
 
     if [ $? -ne 0 ]; then
         echo "Warning: Failed to set CAA record for $domain"