fix: stale HTTPRoute/GRPCRoute/TLSRoute/TCPRoute/UDPRoute status after gateway-only spec changes - #8876
fix: stale HTTPRoute/GRPCRoute/TLSRoute/TCPRoute/UDPRoute status after gateway-only spec changes#8876yuehaii wants to merge 14 commits into
Conversation
✅ Deploy Preview for cerulean-figolla-1f9435 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9f6c48c111
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
hi @zirain @zhaohuabing @rudrakhp . Good day. I am not sure who is the maintainer of the related code in this pr. Can you please help review this pr when you are convenient? Thanks in advance. |
| // For gateway-driven conditions, use the Gateway's generation as observedGeneration | ||
| // Gateway-only edit produces a condition value that differs | ||
| // from the stale stored one, unblocking the watchable DeepEqual gate. | ||
| observedGeneration := httpRoute.GetGeneration() |
There was a problem hiding this comment.
a route may have multiple parents, what will happen then?
There was a problem hiding this comment.
each parent's generation will be updated into the store's watchable map. so if there is one parent gateway generation change, it will trigger the route status update event.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #8876 +/- ##
==========================================
+ Coverage 75.20% 75.64% +0.44%
==========================================
Files 252 252
Lines 41049 41483 +434
==========================================
+ Hits 30871 31380 +509
+ Misses 8086 7945 -141
- Partials 2092 2158 +66 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
working on the conformance test failure |
|
I have fixed the conformance test failure, by embed the gateway's generation into the 'Message' field. @zirain , can you please help review again? this guarantees that any gateway update produces a different 'Message' string, which breaks the DeepEqual gate at |
|
can you fix the conlict and add a e2e for this? |
|
this is a bug, but the fix doesnt look right if a Gateway does not attach to a Route anymore, Envoy Gateway should remove any status created by it
but we dont update status for it this is something we should do in the provider layer |
thanks for the input. when the gateway's host is updated, the subscription will receives an update with update.Delete = false. so the 'update.Delete == true' seems unrelated with normal reconcile issue for any normal gateway update. the 'update.Delete' is set to true only when an entry is explicitly removed from the map, e.g. route deleted from cluster. as you indicated, this happens only in deleteAllKeys() when 'r.ProviderResources.ListenerSetStatuses.Delete(key)' been called. please let me know if I miss anything. |
|
I added an e2e test. But it always PASS even without the fix. go test -v -tags e2e -timeout 10m ./e2e --gateway-class=envoy-gateway -run-test HTTPRouteStaleStatus I launched the integration test again with envoy-gateway v1.7.0. But I can't reproduce the issue. |
|
This pull request has been automatically marked as stale because it has not had activity in the last 30 days. Please feel free to give a status update now, ping for review, when it's ready. Thank you for your contributions! |
|
@yuehaii can you rebase this one with main? |
|
can you rebase with main correctly, I think that would also make the DCO happy. |
…r gateway-only spec changes Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
…edGeneration Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
…th sequential chaining (envoyproxy#8458) * feat: add support for multiple ExtensionManagers with sequential chaining Add a new `extensionManagers` plural field to `EnvoyGatewaySpec` that allows registering multiple extension managers with sequential chaining semantics. Each extension's output becomes the next extension's input. Key changes: - Add `Name` field to `ExtensionManager` and `ExtensionManagers` list to `EnvoyGatewaySpec` - Add `GetExtensionManagers()` helper to normalize singular/plural fields - Add mutual exclusivity validation between singular and plural fields - Implement `CompositeManager` wrapping multiple managers behind the `Manager` interface - Implement `compositeXDSHookClient` with per-extension policy filtering and per-extension resource-type gating in `PostTranslateModifyHook` - Merge `TranslationConfig` using OR semantics across all managers - Add `CleanupHookConns()` to the `Manager` interface - Unify `NewManager` factory to handle 0, 1, and N extensions Signed-off-by: Marc Navarro Sonnenfeld <marcnavarro@tetrate.io> * Fix lint issues Signed-off-by: Marc Navarro Sonnenfeld <marcnavarro@tetrate.io> * Add more test cases Signed-off-by: Marc Navarro Sonnenfeld <marcnavarro@tetrate.io> * Per-extension manager filter Resources, BackendResources, and PolicyResources accordingly Signed-off-by: Marc Navarro Sonnenfeld <marcnavarro@tetrate.io> * Implement new hook method PostEndpointsModifyHook Signed-off-by: Marc Navarro Sonnenfeld <marcnavarro@tetrate.io> * Respect failOpen when GetPre/PostXDSHookClient errors in CompositeManager Signed-off-by: Marc Navarro Sonnenfeld <marcnavarro@tetrate.io> * Simplified and unified client getter and tests Signed-off-by: Marc Navarro Sonnenfeld <marcnavarro@tetrate.io> * Add resilience tests for multiple ExtensionManagers chaining and resource isolation Enhance the simple-extension-server with a configurable --suffix flag and resource-aware PostRouteModify to support testing multiple extension managers. Add resilience tests that verify sequential chaining of VirtualHost mutations and per-extension resource isolation via extensionRef filters. Signed-off-by: Marc Navarro Sonnenfeld <marcnavarro@tetrate.io> * Add CompositeManager coverage to xDS translator tests and document plural ExtensionManagers Mirror the existing extension-manager translator error-handling tests through CompositeManager via a new NewInMemoryCompositeManager helper: a 1-entry composite verifies child errors are swallowed when failOpen is true, and a 2-entry composite verifies errors are propagated with the 'extension "<name>":' prefix when failOpen is false. Extract a shared buildManagerGVKSets helper to avoid duplication between NewManager and the new in-memory constructor. Also extend the extension-server docs with a "Multiple extension servers" subsection covering sequential chaining, per-server resource isolation, per-server failOpen, the unique name requirement, and mutual exclusivity with the singular extensionManager field. Signed-off-by: Marc Navarro Sonnenfeld <marcnavarro@tetrate.io> * Fix lint errors Signed-off-by: Marc Navarro Sonnenfeld <marcnavarro@tetrate.io> * Match extension resources by group/kind, and wire cleanup for in-memory composite Two fixes in the composite-extension path: 1. CompositeManager filter now matches by group/kind only, aligning with runner.ExtensionGroupKinds, Manager.HasExtension, and Gateway API extensionRef. Exact-GVK matching could silently drop resources whose served version differed from the one declared in ExtensionManager.Resources, diverging from single-manager behavior when CRDs serve multiple versions. 2. NewInMemoryCompositeManager wires a sync.Once-guarded cleanup into every entry's cleanupHookConn, so CompositeManager.CleanupHookConns() (from the Manager interface) tears down the shared bufconn/server. The separate cleanup func return is removed; callers use CleanupHookConns() on the returned Manager. Signed-off-by: Marc Navarro Sonnenfeld <marcnavarro@tetrate.io> * Add MinItems=1 and mutual-exclusion validation on extensionManagers * +kubebuilder:validation:MinItems=1 on EnvoyGatewaySpec.ExtensionManagers, mirrored by a runtime check that rejects an explicitly-set-but-empty list (a nil slice still means "omitted"). * +kubebuilder:validation:XValidation on EnvoyGatewaySpec declaring that extensionManager and extensionManagers are mutually exclusive. The same constraint is already enforced at runtime in validateEnvoyGatewayExtensionManagers; the marker documents the schema and is wired up for consumers that validate against the generated OpenAPI. Signed-off-by: Marc Navarro Sonnenfeld <marcnavarro@tetrate.io> * Fix MultiextensionManagers tests Signed-off-by: Marc Navarro Sonnenfeld <marcnavarro@tetrate.io> --------- Signed-off-by: Marc Navarro Sonnenfeld <marcnavarro@tetrate.io> Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
* feat: bandwidth limit Signed-off-by: kkk777-7 <kota.kimura0725@gmail.com> * add: crd validation Signed-off-by: kkk777-7 <kota.kimura0725@gmail.com> --------- Signed-off-by: kkk777-7 <kota.kimura0725@gmail.com> Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com> Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
…uld check the 'no ready listeners' status. Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
…est case with NoReadyListeners reason Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
|
error: Get "http://172.18.0.205:8080/bar": dial tcp 172.18.0.205:8080: connect: connection refused, not counting the round trip /retest |
hi @zirain , good morning. I have rebased the code and fixed the CI failures. could you please review this pr again when it is convenient? thanks. |
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
Signed-off-by: hai.yue <20416005+yuehaii@users.noreply.github.com>
thanks for the correction, but this is the area of the code that needs to be fixed |
The stale status problem is that when a gateway-only change happens, the watchable DeepEqual gate suppresses the update before it even reaches updateStatusFromSubscriptions. So updateStatusFromSubscriptions in status.go never gets called. |
What type of PR is this?
fix: stale HTTPRoute/GRPCRoute/TLSRoute/TCPRoute/UDPRoute/ status after gateway-only spec changes
Issue description
When a Gateway resource is updated without touching any Route resource (e.g. a listener hostname or allowedRoutes change), the Route's status.parents[*].conditions become stale, the observedGeneration field continues pointing at the old gateway generation instead of the new one.
Root cause
The bug only manifests as False→False (same outcome, only ObservedGeneration should advance). If the gateway update trigger a new route reconcile, GetGeneration() hasn't changed because the route was not modified. So the newly computed condition is byte-for-byte identical to what's already in the store and never trigger the status update.
Fix
Fixed the conformance test failure, by embed the gateway's generation into the 'Message' field. This guarantees that any gateway update produces a different 'Message' string, which breaks the DeepEqual gate
Verification
Which issue(s) this PR fixes:
Fixes #8861
Release Notes: Yes