added DOMAIN_ALIAS concept with weighted records; still need to test extensively and add double TXT support

Author: wwwehr

custom-domain/dstack-ingress/DNS_PROVIDERS.md

@@ -13,17 +13,18 @@ This guide explains how to configure dstack-ingress to work with different DNS p
 
 ### Common Variables (Required for all providers)
 
-- `DOMAIN` - Your custom domain (e.g., `app.example.com`)
+- `DOMAIN` - Your node-specific domain (e.g., `node1.app.example.com`)
 - `GATEWAY_DOMAIN` - dstack gateway domain (e.g., `_.dstack-prod5.phala.network`)
 - `CERTBOT_EMAIL` - Email for Let's Encrypt registration
 - `TARGET_ENDPOINT` - Backend application endpoint to proxy to
-- `DNS_PROVIDER` - DNS provider to use (`cloudflare`, `linode`, `namecheap`)
+- `DNS_PROVIDER` - DNS provider to use (`cloudflare`, `linode`, `namecheap`, `route53`)
 
 ### Optional Variables
 
 - `SET_CAA` - Enable CAA record setup (default: false)
 - `PORT` - HTTPS port (default: 443)
 - `TXT_PREFIX` - Prefix for TXT records (default: "_tapp-address")
+- `ALIAS_DOMAIN` - Public-facing domain shared across multiple nodes (e.g., `app.example.com`). Added as a SAN on the TLS certificate and to nginx `server_name`. When set alongside `ROUTE53_INITIAL_WEIGHT` (Route53 only), also creates a weight-0 weighted CNAME `ALIAS_DOMAIN → DOMAIN` to register this node in the pool without routing traffic to it. See [Weighted Routing with ALIAS_DOMAIN](#weighted-routing-with-alias_domain-route53).
 
 ## Provider-Specific Configuration
 
@@ -104,11 +105,12 @@ PolicyDocument:
 ```
 
 **Optional Variables for Route53:**
-- `ROUTE53_INITIAL_WEIGHT` - Enables [Weighted Routing](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy-weighted.html) on the CNAME record created for your domain. Set to an integer (e.g., `100`) to assign that weight to the record. A unique `SetIdentifier` is generated automatically. TXT records (used for Certbot DNS challenges) are never weighted. Omit this variable to create a standard non-weighted CNAME record.
+- `ROUTE53_INITIAL_WEIGHT` - Enables [Weighted Routing](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy-weighted.html) on the CNAME record created for `DOMAIN` (e.g. `node1.app.example.com → phala gateway`). Set to an integer (e.g., `100`) to assign that weight to the record. A unique `SetIdentifier` is generated automatically. TXT records are never weighted. Omit to create a standard non-weighted CNAME. When `ALIAS_DOMAIN` is also set, this variable additionally triggers creation of a weight-0 weighted CNAME `ALIAS_DOMAIN → DOMAIN` — see below.
+- `ALIAS_DOMAIN` - See [Weighted Routing with ALIAS_DOMAIN](#weighted-routing-with-alias_domain-route53).
 
 **Important Notes for Route53:**
-- The certbot plugin uses the format `certbot-dns-route53` package
-- CAA will merge AWS & Let's Encrypt CA domains to existing records if they exist
+- The certbot plugin uses the `certbot-dns-route53` package
+- CAA will merge AWS & Let's Encrypt CA domains into existing records if they exist
 - It is essential that the AWS service account used can only assume the limited role. See cloudformation example.
 
 ## Docker Compose Examples
@@ -165,22 +167,22 @@ services:
       - ./evidences:/evidences
 ```
 
-### Route53 Example
+### Route53 Example (single node, no weighted routing)
 
 ```yaml
 services:
   dstack-ingress:
-    image: dstack-ingress:latest
+    image: dstacktee/dstack-ingress:20250929@sha256:2b47b3e538df0b3e7724255b89369194c8c83a7cfba64d2faf0115ad0a586458
     restart: unless-stopped
     volumes:
-    - /var/run/dstack.sock:/var/run/dstack.sock
-    - cert-data:/etc/letsencrypt
+      - /var/run/tappd.sock:/var/run/tappd.sock
+      - cert-data:/etc/letsencrypt
     ports:
-    - 443:443
+      - "443:443"
     environment:
       DNS_PROVIDER: route53
       DOMAIN: app.example.com
-      GATEWAY_DOMAIN: _.${DSTACK_GATEWAY_DOMAIN}
+      GATEWAY_DOMAIN: _.dstack-prod5.phala.network
 
       AWS_REGION: ${AWS_REGION}
       AWS_ROLE_ARN: ${AWS_ROLE_ARN}
@@ -190,9 +192,99 @@ services:
       CERTBOT_EMAIL: ${CERTBOT_EMAIL}
       TARGET_ENDPOINT: http://backend:8080
       SET_CAA: 'true'
+volumes:
+  cert-data:
+```
+
+For multi-node weighted routing, see [Weighted Routing with ALIAS_DOMAIN](#weighted-routing-with-alias_domain-route53).
+
+## Weighted Routing with ALIAS_DOMAIN (Route53)
+
+This pattern lets you run multiple independent TEE nodes behind a single public domain using Route53 weighted routing. Each node manages its own Phala identity (TXT record, CNAME to gateway) while also being registered as a weighted target for the shared public domain.
+
+### DNS Record Layout
+
+```
+Public domain (shared across nodes)
+────────────────────────────────────────────────────────────────
+  app.example.com  CNAME  node1.app.example.com  weight=100  id=node1.app.example.com
+  app.example.com  CNAME  node2.app.example.com  weight=0    id=node2.app.example.com  ← new, dark
+
+Per-node records (managed by each node's dstack-ingress)
+────────────────────────────────────────────────────────────────
+  node1.app.example.com   CNAME  <appid1>.dstack-prod5.phala.network  weight=100
+  node2.app.example.com   CNAME  <appid2>.dstack-prod5.phala.network  weight=100
+
+  _dstack-app-address.node1.app.example.com  TXT  <appid1>:443
+  _dstack-app-address.node2.app.example.com  TXT  <appid2>:443
+
+TLS certificate (on each node)
+────────────────────────────────────────────────────────────────
+  Subject:  node1.app.example.com
+  SAN:      app.example.com
+```
 
+### Variable Interactions
+
+| `ALIAS_DOMAIN` | `ROUTE53_INITIAL_WEIGHT` | Cert SAN | Nginx server_name | Weight-0 CNAME for ALIAS_DOMAIN |
+|---|---|---|---|---|
+| not set | any | no | no | no |
+| set | not set | yes | yes | no |
+| set | set | yes | yes | yes |
+
+### Node Startup Sequence
+
+When a new node starts with both variables set, dstack-ingress performs these steps automatically:
+
+```
+1. CNAME  node2.app.example.com → <appid2>.dstack.phala.network  (weighted, ROUTE53_INITIAL_WEIGHT)
+2. TXT    _dstack-app-address.node2.app.example.com → <appid2>:443
+3. CNAME  app.example.com → node2.app.example.com                (weight=0, SetIdentifier=node2.app.example.com)
+4. Obtain TLS cert for node2.app.example.com + app.example.com (SAN)
 ```
 
+The node is fully verified and ready before step 3 introduces it to the pool. Traffic only starts flowing after an operator explicitly sets the weight above 0 in Route53.
+
+### Docker Compose Example (weighted, two-node setup)
+
+Node 1 (`node1.app.example.com`):
+
+```yaml
+services:
+  dstack-ingress:
+    image: dstacktee/dstack-ingress:20250929@sha256:2b47b3e538df0b3e7724255b89369194c8c83a7cfba64d2faf0115ad0a586458
+    restart: unless-stopped
+    ports:
+      - "443:443"
+    environment:
+      DNS_PROVIDER: route53
+      DOMAIN: node1.app.example.com
+      ALIAS_DOMAIN: app.example.com
+      GATEWAY_DOMAIN: _.dstack-prod5.phala.network
+      CERTBOT_EMAIL: ${CERTBOT_EMAIL}
+      TARGET_ENDPOINT: http://app:80
+      SET_CAA: 'true'
+      ROUTE53_INITIAL_WEIGHT: '100'
+
+      AWS_REGION: ${AWS_REGION}
+      AWS_ROLE_ARN: ${AWS_ROLE_ARN}
+      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
+      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
+    volumes:
+      - /var/run/tappd.sock:/var/run/tappd.sock
+      - cert-data:/etc/letsencrypt
+volumes:
+  cert-data:
+```
+
+Node 2 (`node2.app.example.com`) uses identical config with `DOMAIN: node2.app.example.com`. On first boot it registers itself at weight 0. To promote it, update the record weight in Route53.
+
+### Important Notes
+
+- The weight-0 CNAME for `ALIAS_DOMAIN` uses `DOMAIN` as the `SetIdentifier`, so each node has a stable, unique slot in the weighted record set that survives restarts without creating duplicates.
+- For non-Route53 providers (Cloudflare, Linode, Namecheap), `ALIAS_DOMAIN` still adds the SAN and updates nginx, but those providers do not support weighted CNAME records at the DNS level. You would need to manage traffic distribution through the provider's own load balancing features.
+- `ALIAS_DOMAIN` must be in the same Route53 hosted zone as `DOMAIN`, or at least a zone your credentials can write to.
+
 ## Migration from Cloudflare-only Setup
 
 If you're currently using the Cloudflare-only version:

custom-domain/dstack-ingress/README.md

@@ -184,6 +184,7 @@ configs:
 - `PROXY_BUFFERS`: Optional value for nginx `proxy_buffers` (format: `number size`, e.g. `4 256k`) in single-domain mode
 - `PROXY_BUSY_BUFFERS_SIZE`: Optional value for nginx `proxy_busy_buffers_size` (numeric with optional `k|m` suffix, e.g. `256k`) in single-domain mode
 - `CERTBOT_STAGING`: Optional; set this value to the string `true` to set the `--staging` server option on the [`certbot` cli](https://eff-certbot.readthedocs.io/en/stable/using.html#certbot-command-line-options)
+- `ALIAS_DOMAIN`: Optional; a single public-facing domain shared across multiple nodes (e.g. `app.example.com`). Added as a SAN on the TLS certificate and to nginx `server_name`. When combined with `ROUTE53_INITIAL_WEIGHT` (Route53 only), also registers a weight-0 weighted CNAME `ALIAS_DOMAIN → DOMAIN` so the node is in the pool but dark until promoted. See [Multi-Node Weighted Routing](#multi-node-weighted-routing-with-alias_domain).
 
 **Backward Compatibility:**
 
@@ -252,6 +253,68 @@ volumes:
   cert-data:
 ```
 
+## Multi-Node Weighted Routing with ALIAS_DOMAIN
+
+`ALIAS_DOMAIN` enables a pattern where multiple independent TEE nodes share a single public-facing domain via DNS weighted routing, while each node maintains its own Phala-verified identity.
+
+### How It Works
+
+Each node has a **node domain** (`DOMAIN`, e.g. `node1.app.example.com`) that is the primary identity used for Phala's TXT-based app routing. A single **public domain** (`ALIAS_DOMAIN`, e.g. `app.example.com`) is shared across all nodes and used as the user-facing address.
+
+```
+                         ┌─────────────────────────────────────────┐
+Users                    │         Route53 Weighted CNAMEs          │
+  │                      │                                          │
+  └─► app.example.com ───┼──► node1.app.example.com  (weight=100) ─┼──► <appid1>.dstack.phala.network
+                         │                                          │
+                         └──► node2.app.example.com  (weight=0)  ──┼──► <appid2>.dstack.phala.network
+                                                                    │
+                         ┌─────────────────────────────────────────┘
+                         │         Phala Gateway TXT Routing
+                         │
+                         │  _dstack-app-address.node1.app.example.com → <appid1>:443
+                         │  _dstack-app-address.node2.app.example.com → <appid2>:443
+                         │
+                         │         TLS Certificate (on each node)
+                         │
+                         │  node1.app.example.com  ← primary
+                         │  app.example.com         ← SAN
+                         └─────────────────────────────────────────
+```
+
+### What dstack-ingress Does Automatically
+
+When `ALIAS_DOMAIN` is set:
+
+1. **Certificate** — issues a SAN cert covering both `DOMAIN` and `ALIAS_DOMAIN`, so nginx can serve TLS regardless of which hostname the client connected through.
+2. **Nginx** — adds `ALIAS_DOMAIN` to `server_name` so requests arriving via the public domain are accepted.
+3. **Weighted CNAME** *(Route53 only, requires `ROUTE53_INITIAL_WEIGHT`)* — creates a weighted CNAME record `ALIAS_DOMAIN → DOMAIN` at **weight 0**, registering this node in the pool without routing any traffic to it yet. The `SetIdentifier` is set to `DOMAIN` so each node occupies a unique, stable slot.
+
+### Lifecycle
+
+```
+Node starts
+    │
+    ├── CNAME:   node1.app.example.com → <appid>.dstack.phala.network  (weight=ROUTE53_INITIAL_WEIGHT)
+    ├── TXT:     _dstack-app-address.node1.app.example.com → <appid>:443
+    ├── CNAME:   app.example.com → node1.app.example.com               (weight=0)  ← new node, dark
+    └── CERT:    node1.app.example.com + app.example.com (SAN)
+
+Operator promotes node
+    └── Update Route53: app.example.com → node1.app.example.com weight 0 → desired weight
+```
+
+The node is fully provisioned and verified before receiving any user traffic. Traffic is enabled by a deliberate operator action (bumping the weight in Route53), not automatically.
+
+### Configuration
+
+| Variable | Required | Description |
+|---|---|---|
+| `ALIAS_DOMAIN` | No | Public-facing domain shared across nodes (e.g. `app.example.com`) |
+| `ROUTE53_INITIAL_WEIGHT` | No | Weight for this node's primary CNAME. When combined with `ALIAS_DOMAIN`, also triggers creation of the weight-0 CNAME for the public domain. |
+
+See [DNS Provider Configuration](DNS_PROVIDERS.md#weighted-routing-with-alias_domain-route53) for a full example.
+
 ## Domain Attestation and Verification
 
 The dstack-ingress system provides mechanisms to verify and attest that your custom domain endpoint is secure and properly configured. This comprehensive verification approach ensures the integrity and authenticity of your application.

custom-domain/dstack-ingress/scripts/certman.py

@@ -287,7 +287,10 @@ def _build_certbot_command(self, action: str, domain: str, email: str) -> List[s
 
         if action == "certonly":
             base_cmd.extend(["--agree-tos", "--no-eff-email",
-                            "--email", email, "-d", domain])
+                            "--email", email, "--cert-name", domain, "-d", domain])
+            alias_domain = os.environ.get("ALIAS_DOMAIN", "").strip()
+            if alias_domain:
+                base_cmd.extend(["--expand", "-d", alias_domain])
         if os.environ.get("CERTBOT_STAGING", "false") == "true":
             base_cmd.extend(["--staging"])
 

custom-domain/dstack-ingress/scripts/dns_providers/base.py

@@ -247,6 +247,36 @@ def set_txt_record(self, name: str, content: str, ttl: int = 60) -> bool:
         )
         return self.create_dns_record(new_record)
 
+    def set_weighted_cname_record(
+        self,
+        name: str,
+        content: str,
+        weight: int,
+        set_identifier: str,
+        ttl: int = 60,
+    ) -> bool:
+        """Set a weighted CNAME record.
+
+        Default implementation ignores weight/set_identifier and creates a plain
+        CNAME. Providers that support weighted routing (e.g. Route53) should
+        override this method.
+
+        Args:
+            name: The record name (ALIAS_DOMAIN)
+            content: The CNAME target (primary DOMAIN)
+            weight: Routing weight (0 = registered but receives no traffic)
+            set_identifier: Unique identifier for this node in the weighted pool
+            ttl: Time to live
+
+        Returns:
+            True if successful, False otherwise
+        """
+        print(
+            f"Note: provider does not support weighted CNAMEs; "
+            f"creating plain CNAME for {name}"
+        )
+        return self.set_cname_record(name, content, ttl)
+
     def set_caa_record(
         self,
         name: str,

custom-domain/dstack-ingress/scripts/dns_providers/route53.py

@@ -224,6 +224,7 @@ def set_alias_record(self, name: str, content: str, ttl: int = 60, proxied: bool
         want_weight = int(env_weight) if env_weight and env_weight.isdigit() else None
 
         existing_records = self.get_dns_records(name, RecordType.CNAME)
+        record_to_replace = None
         for record in existing_records:
             if record.content == content:
                 has_weight = record.data and "weight" in record.data
@@ -234,15 +235,17 @@ def set_alias_record(self, name: str, content: str, ttl: int = 60, proxied: bool
                     print("Weighted CNAME record with the same content and weight already exists")
                     return True
                 # Weight config mismatch — delete existing record before upserting,
-        # because Route53 forbids mixing weighted and non-weighted RRSets
-        # with the same name and type.
-        has_weight = record.data and "weight" in record.data
-        if (want_weight is not None and not has_weight) or (want_weight is None and has_weight):
+                # because Route53 forbids mixing weighted and non-weighted RRSets
+                # with the same name and type.
+                record_to_replace = record
+                break
+        if record_to_replace is not None:
+            has_weight = record_to_replace.data and "weight" in record_to_replace.data
             print(
                 f"Deleting existing {'non-weighted' if not has_weight else 'weighted'} "
                 f"CNAME before creating {'weighted' if want_weight is not None else 'non-weighted'} one"
             )
-            if not self.delete_dns_record(record.id, name):
+            if not self.delete_dns_record(record_to_replace.id, name):
                 print(f"Error: Failed to delete existing CNAME record", file=sys.stderr)
                 return False
 
@@ -256,6 +259,52 @@ def set_alias_record(self, name: str, content: str, ttl: int = 60, proxied: bool
         )
         return self.create_dns_record(new_record)
 
+    def set_weighted_cname_record(
+        self,
+        name: str,
+        content: str,
+        weight: int,
+        set_identifier: str,
+        ttl: int = 60,
+    ) -> bool:
+        """Create or update a weighted CNAME record with an explicit weight.
+
+        Unlike set_alias_record, this bypasses ROUTE53_INITIAL_WEIGHT and uses
+        the provided weight directly. set_identifier should be the primary node
+        domain so each node occupies a unique slot in the weighted pool.
+        """
+        existing_records = self.get_dns_records(name, RecordType.CNAME)
+        for record in existing_records:
+            if record.data and record.data.get("set_identifier") == set_identifier:
+                if record.content == content and record.data.get("weight") == weight:
+                    print(
+                        f"Weighted CNAME for {name} "
+                        f"(id={set_identifier}, weight={weight}) already exists"
+                    )
+                    return True
+                # Same identifier, different weight or content — delete and recreate
+                print(
+                    f"Updating weighted CNAME for {name} "
+                    f"(id={set_identifier}) to weight={weight}"
+                )
+                if record.id and not self.delete_dns_record(record.id, name):
+                    print(
+                        f"Error: Failed to delete existing weighted CNAME",
+                        file=sys.stderr,
+                    )
+                    return False
+                break
+
+        new_record = DNSRecord(
+            id=None,
+            name=name,
+            type=RecordType.CNAME,
+            content=content,
+            ttl=ttl,
+            data={"weight": weight, "set_identifier": set_identifier},
+        )
+        return self.create_dns_record(new_record)
+
     def create_dns_record(self, record: DNSRecord) -> bool:
         """
         Create a DNS record.