AI Edge: eliminate slow/failed TLS cert provisioning at scale via Envoy Gateway extension server

[scotwells]

Problem

AI Edge sites — especially those with custom hostnames — had slow or failed HTTPS provisioning: certificates stuck "pending," ordinary sites taking 10+ minutes instead of seconds.

Every configuration change rebuilt the full Envoy config for all gateways and re-applied each gateway's customizations (WAF, connectors) via EnvoyPatchPolicy. With WAF + connectors past ~140 gateways, programming took 30+ seconds — long enough to miss the time-limited Let's Encrypt challenge window, so certificates never issued. Latent since inception; surfaced past a scale we'd never load-tested.

Direction

Retire per-gateway EnvoyPatchPolicy and apply WAF/connector customizations inline during config build via an Envoy Gateway extension server.

Result: ~30 s → ~50 ms at 250 gateways. Shipped in NSO v0.22.0; provisioning restored and faster than before. Parent incident: datum-cloud/engineering#317.

Work

Direction & core migration

• Enhancement proposal: Envoy Gateway extension server #187 Enhancement proposal: Envoy Gateway extension server
• Modernize the gateway stack: Envoy Gateway 1.8 & Gateway API 1.5 #189 Modernize the gateway stack: Envoy Gateway 1.8 & Gateway API 1.5
• Envoy Gateway extension server #192 Envoy Gateway extension server (implementation)
• fix: gate connector EnvoyPatchPolicy per HTTPS listener cert readiness #185 Interim mitigation: gate connector EnvoyPatchPolicy per HTTPS-listener cert readiness

Provisioning fix & cluster-naming follow-ups (post-upgrade)

• fix: AI Edge cert/hostname provisioning broken after v0.22.0 upgrade #196 Fix AI Edge cert/hostname provisioning broken after v0.22.0 upgrade
• fix(replicator): fall back to storage version for downstream watch #197 Replicator: fall back to storage version for downstream watch
• fix(gateway): tolerate legacy slash-encoded upstream cluster names #199 Tolerate legacy slash-encoded upstream cluster names
• fix(gateway,iroh-dns): tolerate legacy slash-encoded cluster names in remaining decoders #200 Tolerate legacy slash-encoded cluster names in remaining decoders
• ci(publish): publish kustomize bundle only after the image build #201 Publish kustomize bundle only after the image build

Behavior restored/improved under the extension server

• fix(httpproxy): keep offline connector backendRef in extension-server mode #202 Keep offline connector backendRef in extension-server mode
• fix(extension-server): deterministic tunnel-offline 503 on the user path for offline connectors #207 Deterministic tunnel-offline 503 on the user path for offline connectors
• fix(connector): propagate liveness to the edge so online connectors aren't treated as offline #208 Propagate connector liveness to the edge
• Fix tunnels staying offline at the edge after a connector connects #211 Fix tunnels staying offline at the edge after a connector connects
• fix(extension-server): unique connector domain to stop Envoy config NACK #214 Unique connector domain to stop Envoy config NACK
• docs: branded data-plane error pages enhancement proposal #204 Branded data-plane error pages — enhancement proposal
• feat(extension-server): branded data-plane error pages #205 Branded data-plane error pages
• docs(extension-server): document edge re-translation for connector liveness #213 Docs: edge re-translation for connector liveness

Observability, CI & tests

• CI maintenance: golangci-lint for Go 1.26 + datum-cloud/actions v1.15.0 #191 golangci-lint for Go 1.26 + actions v1.15.0
• fix(e2e): Install missing NSO CRDs on the downstream  #198 Install missing NSO CRDs on the downstream (e2e)
• chore: add last transition time metrics #203 Add last-transition-time metrics
• chore: add last transition resource metrics #210 Add last-transition resource metrics
• test(e2e): fix flaky CA-secret race in gateway-accepted #215 Fix flaky CA-secret race in gateway-accepted (e2e)

Safety / rollback

• fix(envoy-gateway): roll back EG control plane to v1.7.4 #193 Roll back EG control plane to v1.7.4 (mitigated an EG 1.8.0 OIDC regression hit during rollout)

Related hardening (tracked separately under #317)

• Expired/abandoned customer certs can freeze an entire edge — health-gate cert propagation + isolate per-cert failures #212 Expired/abandoned customer certs can freeze an entire edge
• Connector upstream-status annotation not re-mirrored after Ready condition flips True #209 Connector upstream-status annotation not re-mirrored after Ready flips True