Skip to content

Commit fb0a3ff

Browse files
scotwellsclaude
andcommitted
docs(extension-server): record shared-config uniqueness invariant, NACK-freeze observability, and known gaps
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent de0dc70 commit fb0a3ff

1 file changed

Lines changed: 51 additions & 0 deletions

File tree

  • docs/enhancements/envoy-gateway-extension-server

docs/enhancements/envoy-gateway-extension-server/README.md

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -251,6 +251,22 @@ computed from the snapshot Envoy Gateway passes in plus policy read from
251251
Kubernetes (cached; see [Sourcing
252252
Policy](#sourcing-policy-how-the-extension-server-knows-what-to-inject)).
253253

254+
#### Invariant: mutations to shared resources must be globally unique
255+
256+
Because Envoy Gateway merges all customer gateways into one shared
257+
configuration, several resources the Extension Server mutates are themselves
258+
shared — most notably the HTTP listener's route configuration, whose
259+
virtual-host domains form a single global namespace. Envoy enforces that these
260+
domains are unique and rejects the entire xDS snapshot if any two collide. Any
261+
identifier the Extension Server adds to a shared resource — a virtual-host
262+
domain above all — must therefore be globally unique across the whole fleet, not
263+
merely unique within one connector or gateway. The Connector satisfies this by
264+
deriving a synthetic per-connector domain rather than reusing the backend host
265+
(which is frequently a non-unique value such as `localhost`). This invariant is
266+
load-bearing under the fail-closed delivery posture: a single duplicate value
267+
does not degrade one gateway, it NACKs the snapshot and freezes configuration
268+
delivery for every gateway (see [High Availability](#high-availability)).
269+
254270
### Extension Mechanism: Which Hook and Why
255271

256272
Envoy Gateway's
@@ -592,6 +608,14 @@ Gateway share a namespace, and the Gateway is named after the `HTTPProxy`. The
592608
controller needs `get;patch` on `gateways.gateway.networking.k8s.io` in addition
593609
to the extension server's read-only policy access.
594610

611+
One coverage edge remains: the controller watches Connectors, but the
612+
connector→Gateway association is resolved through the `HTTPProxy`. A change that
613+
newly links an existing Gateway to an already-online Connector — an `HTTPProxy`
614+
created or repointed after the Connector's liveness has settled — produces no
615+
Connector event, so the Gateway is not stamped until an unrelated rebuild fires.
616+
Closing this fully requires the controller to additionally watch `HTTPProxy` and
617+
map back to the affected Gateway.
618+
595619
The alternative considered was having the replicator write a monotonic nonce into
596620
the downstream Connector's `spec` when the `Ready` condition flips, which would
597621
increment `metadata.generation` and satisfy the `GenerationChangedPredicate`. This
@@ -671,10 +695,30 @@ Envoy Gateway is configured with:
671695
Server recovers, which is exactly why the two-replica, probe-gated,
672696
retry-backed posture above is mandatory rather than optional.
673697

698+
Fail-closed protects against the hook *erroring*, but not against the hook
699+
returning a response Envoy Gateway accepts and pushes yet Envoy itself then
700+
rejects (for example, a malformed or colliding resource — see the uniqueness
701+
invariant above). Envoy applies each xDS snapshot atomically, so one rejected
702+
resource discards the whole update and freezes configuration for every gateway
703+
at once. This failure is invisible to Kubernetes state: Gateway and Route status
704+
stay `Programmed=True` because translation succeeded. The only signals are
705+
Envoy's `*.update_rejected` counters (LDS/RDS/CDS) and Envoy Gateway's
706+
translation-error logs. Alerting on these xDS rejection metrics — RDS and CDS
707+
rejections as well as LDS — is required; without it a fleet-wide config freeze is
708+
silent.
709+
674710
Latency must be monitored: per-build hook latency and error rate are
675711
platform-health metrics. Policy reads stay off the synchronous build path via
676712
the informer cache described above.
677713

714+
Operability note: because the Extension Server embeds controller-runtime (the
715+
policy cache and the re-translation controller), it must install a logger at
716+
process startup. controller-runtime suppresses all of its own and its
717+
controllers' log output until a logger is set, so an Extension Server that skips
718+
this step runs without controller or hook logs — removing the second of the only
719+
two diagnostic signals (logs and xDS metrics) precisely when an incident needs
720+
them.
721+
678722
### Reference Implementation
679723

680724
A working prototype of the Extension Server has been built and validated
@@ -897,6 +941,13 @@ configuration updates simultaneously; an EPP misconfiguration affects only one
897941
gateway's EPP object. This is the primary operational tradeoff (see
898942
[High Availability](#high-availability) for mitigations).
899943

944+
**Fleet-wide config freeze is invisible to Kubernetes status.** A single
945+
malformed or colliding resource in a shared config makes Envoy reject the whole
946+
atomic snapshot, freezing updates for every gateway while Gateway/Route status
947+
still reads `Programmed=True`. Detection depends on Envoy xDS-rejection metrics
948+
and EG translation logs rather than Kubernetes conditions (see
949+
[High Availability](#high-availability)).
950+
900951
**Coupling to EG's extension hook API.** A supported, upstream-recommended
901952
mechanism, but an additional API surface to track across EG version upgrades.
902953

0 commit comments

Comments
 (0)