Edge gateway HTTP-01 challenges stuck at ACME self-check (DNS/404/500/503) — NSO solver is live; control-plane vs member Challenge split

[ecv]

Important

2026-07-17 status — open; the control-plane↔member Challenge split is unresolved. No ACME cert has issued on datum-gateway-http fleet-wide since ~2026-07-09. The infra-side count improvement (datum-cloud/infra#3419) is a deletion artifact: ~87 edge Certificates were deleted ~2026-07-15 19:00Z on a stably-running NSO v0.24.2 (no restart, no roll) while Ready=True stayed flat — no v0.24.x release fixed this. Evidence: #261 (comment). Definition-of-done: the infra gateway-custom-hostname-cert e2e goes green on a scheduled prod run — not a lower Ready=False count.

(OP rewritten 2026-07-13 after live investigation disproved the original "stalled solver" diagnosis; the two minimized comments hold that outdated history.)

Summary

Edge gateway custom-hostname certificates are not issuing. But the GatewayDownstreamCertificateSolver is not the cause — it is running, holds leadership, and reconciles the Challenges it can see, creating the expected HTTP01 solver HTTPRoute/HTTPRouteFilter. The Challenges are stuck at cert-manager's ACME HTTP-01 self-check, which is failing for heterogeneous reasons (DNS resolution failures, HTTP 404/500/503). Separately, the control-plane Challenge set NSO watches has no object newer than 2026-06-18, while edge member clusters carry fresh Challenges — a topology split that needs explaining.

Verified topology (reproduced NSO's client)

• The solver watches the Karmada control-plane apiserver (karmada-apiserver.karmada-system.svc.cluster.local:5443), reached via the pod's federated-kubeconfig + a projected SA token (network-services-operator-controller-manager, audience = that apiserver). Reproduced this exact client from the hub and queried it directly.
• As of 2026-07-13, that apiserver served 16 gateway acme.cert-manager.io/v1 Challenges (issuerRef=ClusterIssuer/datum-gateway-http, type: HTTP-01), all created ≤ 2026-06-18 (two older, from March/May), all pending, and actively updated by cert-manager (managedFields: cert-manager-challenges@2026-07-13T05:23Z).
• The solver's surviving http01-solver HTTPRoute/HTTPRouteFilter set on the edge maps 1:1 to these 16 control-plane Challenges.

What is actually happening (verified 2026-07-13)

The solver is live, not stalled. Leader pod network-services-operator-controller-manager-849f57c79d-xwp54 (fresh, restartCount 0) logs the full happy path with 0 errors, and re-reconciles on cert-manager's live status updates:

Reconciling ACME challenge solver
Processing challenge for Gateway-owned certificate
Successfully retrieved key and token from cert-manager Challenge
HTTPRouteFilter reconciled
HTTPRoute reconciled

The Challenges are stuck at the HTTP-01 self-check, for mixed reasons (live status on the 16):

Failure reason (from status.reason)	Notes
dial tcp: lookup <domain> on 10.2.0.10:53: server misbehaving	cluster-DNS resolution failing for the self-check (e.g. app.datum.design)
wrong status code '404', expected '200'	e.g. javalords-ink…, swirls-k8s…
wrong status code '500'	e.g. pcep-lol…, tunnel-dmsrd…
wrong status code '503'	seen intermittently on the same objects
failed to perform self check	generic self-check failure

So even objects for which the solver has created a route still fail the self-check (404/500/503), and some never get that far (DNS). This is a data-path / self-check problem, not a "solver isn't creating routes" problem.

The real open question — control-plane vs member Challenge split

• The control-plane apiserver NSO watches has no Challenge newer than 2026-06-18. That, not a dead watch, is why no new solver routes appear after 06-18 — NSO faithfully serves what's there, and there's nothing newer.
• Meanwhile the edge member us-central-1-alice (a registered Karmada member) has 96 Challenges created 2026-07-13 02:43, which do not appear in the control-plane view. The 06-18 control-plane object javalords-ink-https-hostname-0-1-2137251722-2111718441 is even NotFound on that member.
• So there appear to be two Challenge populations (control-plane vs member) that don't reconcile. Where are new tenant gateway Challenges being created, and why don't they land in the apiserver NSO watches?

What to investigate next (owner)

1. Why the HTTP-01 self-check fails per-domain: cluster-DNS server misbehaving for control-plane cert-manager, and edge Envoy returning 404/500/503 on /.well-known/acme-challenge/<token> even where a solver route exists.
2. The control-plane ↔ member Challenge topology: is new tenant cert issuance supposed to create Challenges in the control-plane apiserver (where NSO + datum-gateway-http live), and if so why has nothing appeared since 06-18? Note the prod propagation policy is now mirrored in-repo (config/federation/clusterpropagationpolicy.yaml, Carry edge policy from the control plane out to the edges #227): it propagates the http01-solver HTTPRoute/HTTPRouteFilter and issuer Secrets to members but not Certificates/Orders/Challenges — under that intended topology, member-local Challenges shouldn't exist at all.
3. Whether the 06-18 ceiling correlates with the v0.22.0 (EG 1.8 / Gateway API 1.5, Modernize the gateway stack: Envoy Gateway 1.8 & Gateway API 1.5 #189/Envoy Gateway extension server #192, released 2026-06-18T01:07Z) change to the federated/aggregated wiring — but note this is about what gets into the apiserver, not the solver's watch, which is demonstrably live.

Refs

• datum-cloud/infra#3419 (field symptom tracker)
• AI Edge: eliminate slow/failed TLS cert provisioning at scale via Envoy Gateway extension server #216 (prior TLS-at-scale incident — different, already-fixed cause)
• feat: add automatic re-issuance of failed custom hostname certificates #259 / Unify or delineate cert-failure recovery: re-issuance (#259) vs health-gating (#212) #260 (re-issuance and its interaction — still a churn hazard against any stuck cert)

[ecv]

Live prosecution (prod hub, 2026-07-13 ~03:47Z)

Ran the confirm steps against prod. All three network-services-operator-controller-manager pods on v0.23.10 (started 2026-07-13 03:00–03:02, --cluster-sharding-enabled=true, lease prefix nso in kube-system).

Symptom confirmed live. Gateway path healthy; solver produces nothing despite active failing Challenges.

Check	Result
Gateway controller alive	✅ 90 / 105 / 113 gateway reconciles in the recent window across the 3 pods (plus a few connector)
Solver controller activity	❌ zero solver / challenge / acme / Reconciling ACME lines in any pod
Live cert-failure impact	✅ 16–17 "We couldn't issue a TLS certificate" messages per pod — real domains incl. swirls.ai, app.datum.design, agent.lkms.app
Singleton lease location	❌ no nso-prefixed Lease in hub kube-system — confirms the singleton leader lease lives on the Karmada aggregated apiserver, as described above

This isolates the fault to the singleton solver, not the gateway controller — matching the report.

What this run could NOT settle (hub visibility limits)

• Only the recent steady-state log window was readable; startup lines (leader election, Starting workers … downstream-certificate-solver, the initial Challenge LIST) had already scrolled off.
• The hub flux-MCP can't reach the Karmada aggregated apiserver, so confirm steps 1 (controller started + holds leadership) and 2 (Challenge watch established vs. dead-after-initial-LIST) remain open. Need the leader pod's startup logs or Karmada kubeconfig access.

Two possible causes remain, and this run can't tell them apart:

1. the solver controller never started on the leader, or
2. it started but its Challenge watch died after the initial LIST.

Next diagnostic is confirm step 1 — is the controller started and holding leadership. Cheapest check, and the one hub visibility can't answer.

[ecv]

Deeper investigation — the "does nothing / maybe not started" framing is wrong. It's a frozen Challenge informer serving phantom objects. (prod, 2026-07-13 ~05:14Z)

Got kubectl to the hub GKE cluster (gke_datum-cloud-prod…infrastructure-control-plane-prod), where the three v0.23.10 pods actually run, so their full logs are readable (the flux-MCP tail caps at ~750 lines ≈ 30s given these pods emit ~20k lines/hr).

Step 1 — is the singleton solver started + holding leadership? YES.

Leader is pod …-849f57c79d-xwp54 (fresh: created 2026-07-13T03:02, restartCount: 0). Its log shows the downstream-certificate-solver controller actively reconciling — full happy path, no errors:

Reconciling ACME challenge solver
Processing challenge for Gateway-owned certificate
Successfully retrieved key and token from cert-manager Challenge
HTTPRouteFilter reconciled
HTTPRoute reconciled

So the earlier "not started / not acting" hypothesis is wrong — it is started, holds leadership, and its reconcile loop runs. This also rules out the sub-hypothesis "never started on the leader."

Step 2 — is the Challenge watch alive? NO — the informer is frozen at the 2026-06-18 LIST snapshot.

Over a 47-minute window (04:27–05:14Z) the solver logged 59 Reconciling ACME challenge solver — every single one for the same object:
ns-4250da12…/javalords-ink-https-hostname-0-1-2137251722-2111718441. 59 reconciles, 59 HTTPRoute reconciled, 0 errors, 0 other challenges touched.

Then the killer cross-check against the edge (datum-us-central-1-alice):

Check	Result
The hot-looped challenge on the edge	NotFound — it was deleted (it's a phantom the solver still thinks exists)
Its HTTPRoute creationTimestamp	2026-06-18T03:06:27Z — never moves; the reconcile is idempotent no-op on a pre-existing route, which is why HTTPRoute reconciled logs but nothing changes
Live challenges in that same namespace	14 real ones, all created 2026-07-13T02:43, pending — the solver reconciled none of them
Newest http01-solver HTTPRoute edge-wide	still 2026-06-18T03:08:51Z, unchanged across two snapshots 1.5h apart

What this means

The singleton solver's Challenge informer is stuck on the ~2026-06-18 LIST. It keeps re-delivering cached objects from that snapshot — including at least one already deleted — so it busy-loops reconciling a phantom, while receiving no ADD events for the 95+ challenges created since and no DELETE for the ones removed. Routes therefore haven't advanced past 2026-06-18, and every new edge challenge 404s. This is the concrete form of "watch died after the initial LIST."

The sharper clue for the owner

The leader pod is fresh (03:02 roll today), yet its informer is serving 25-day-old data including deleted objects. A fresh pod's informer can't hold stale in-process state — so the frozen snapshot is coming from what the federated/aggregated apiserver serves to DownstreamCluster.GetCache()'s LIST/watch, not from NSO's in-memory cache. That also explains why plain restarts (incl. the v0.23.10 roll) don't recover it: a new informer re-attaches to the same frozen upstream stream. Points at the Karmada aggregation / federated-client path (consistent with the v0.22.0 EG 1.8 + Gateway API 1.5 modernization touching that wiring) rather than the solver controller code itself, which is behaving correctly given the objects it's handed.

Recommend: check the federated/aggregated apiserver's watch-cache for acme.cert-manager.io/v1 Challenge (resourceVersion it serves on LIST vs. current), and whether the aggregation layer is pinning an old RV / dropping the watch silently.

[ecv]

Prod update: the count "recovery" was mass deletion; no issuance since ~07-09 — the split is still unresolved (2026-07-17)

Re the "control-plane vs member Challenge split" open question. New prod evidence says the split is not resolved, and the apparent improvement on the infra side (datum-cloud/infra#3419) is a deletion artifact.

• On every *-alice edge, Ready=False custom-hostname certs dropped (42 → 13 on us-central-1-alice) at ~2026-07-15 19:00Z. But total Certificates fell 300 → 213 (−87) at the same instant, while Ready=True stayed 58, unchanged. The failing certs were deleted, not issued.
• Newest ACME cert on datum-gateway-http across all 16 edges has an identical notAfter ≈ 2026-10-07 (issued ~07-09). Zero issued fleet-wide since 07-13. The 07-13 Challenge burst yielded no certs.
• The deletion happened in-process on a stably-running NSO v0.24.2 (13:00–19:47Z): no restart, no leader failover, no version roll, no infra propagation-path commit. No v0.24.x release fixed this — consistent with the solver being live but starved of fresh Challenges (the split), and the starved certs then being GC'd.

Net: the solver still isn't producing certs for post-06-18 Challenges; the member↔control-plane Challenge topology gap remains the live blocker.

Two things worth checking on your side:

1. Whether NSO#259 reissuance is the deleter (it deletes failed certs to retry) and, if so, why recreation didn't follow — total held at 213 for four days, so nothing was recreated.
2. The federated/aggregated apiserver's Challenge population: are member-created Challenges (07-13 onward) expected to land in the control-plane apiserver the solver watches, and why they still don't.

Hub now on v0.24.4.

[ecv]

Coupling: this issue's definition-of-done is an infra e2e test

Flagging for whoever picks up the fix — there's a Chainsaw e2e that is the end-to-end acceptance gate for this issue: gateway-custom-hostname-cert (https://github.com/datum-cloud/infra/pull/3415, part of the construct suite in datum-cloud/infra#3346 / https://github.com/datum-cloud/infra/issues/3347).

It drives a verified custom hostname all the way to a publicly-trusted TLS handshake — curl without -k, want 200. No mock, no -k, no canonical-listener shortcut. That assertion can only pass when the edge HTTP01 solve completes: exactly when the control-plane↔member Challenge split tracked here is closed and the solver route actually serves /.well-known/acme-challenge/. It fails today for the same 404 this issue is about.

Status of that test:

• Held spec.skip: true in infra until this lands (mirrors the existing waf-mode-flip skip-with-reason precedent), so it does not red the scheduled prod suite while Edge gateway HTTP-01 challenges stuck at ACME self-check (DNS/404/500/503) — NSO solver is live; control-plane vs member Challenge split #261 is open.
• The moment the fix ships and rolls to the gateway-enabled edge members, it flips live. A green scheduled-prod run is the proof that edge custom-hostname issuance works end-to-end — and the signal to close the infra-side symptom tracker, datum-cloud/infra#3419.

Net: treat this test as the definition-of-done for #261. The fix isn't proven until that no--k 200 lands on a scheduled prod run — not by a lower Ready=False count (that was a deletion artifact) and not by a green unit test.

[ecv]

ok engaging frontier AI