feat(xds): raise xDS gRPC server default receive limit and make it configurable - #9564
Conversation
✅ Deploy Preview for cerulean-figolla-1f9435 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a2ef866102
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
|
||
| if r.EnvoyGateway.XDSServer != nil && r.EnvoyGateway.XDSServer.MaxRecvMsgSize != nil { | ||
| maxRecvMsgSize, _ := r.EnvoyGateway.XDSServer.MaxRecvMsgSize.AsInt64() | ||
| baseKeepaliveOptions = append(baseKeepaliveOptions, grpc.MaxRecvMsgSize(int(maxRecvMsgSize))) |
There was a problem hiding this comment.
For the documented scale case where the xDS snapshot/DiscoveryResponse is over 4MiB, Envoy Gateway is the sender and the Envoy proxy is the receiver; grpc.MaxRecvMsgSize here only raises the Go server's inbound DiscoveryRequest limit. Setting xdsServer.maxRecvMsgSize therefore leaves the advertised received message larger than max failure on Envoy's receive path unchanged, so large snapshots will still fail unless the Envoy bootstrap/client receive limit is configured instead.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
DeltaDiscoveryRequest.resource_names_subscribe and DeltaDiscoveryRequest.initial_resource_versions from the XDS Client may grow large when the snapshot contains many resources. This is the concern addressed here. WDYT @codex ?
There was a problem hiding this comment.
Codex Review: Didn't find any major issues. What shall we delve into next?
Reviewed commit: 4f97097f92
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
/retest |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9564 +/- ##
=======================================
Coverage 75.67% 75.67%
=======================================
Files 254 254
Lines 41933 41942 +9
=======================================
+ Hits 31731 31740 +9
- Misses 8071 8072 +1
+ Partials 2131 2130 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
/retest |
1 similar comment
|
/retest |
| // If unspecified, the default gRPC receive limit applies (4MiB). | ||
| // | ||
| // +optional | ||
| MaxRecvMsgSize *resource.Quantity `json:"maxRecvMsgSize,omitempty"` |
There was a problem hiding this comment.
nit: in extension manager we use MaxMessageSize, so maybe MaxReceiveMessageSize
494fa76 to
054fa05
Compare
|
cc @zhaohuabing @arkodg - in case of very large snapshots, EG can reject XDS subscribe messages. |
|
@linmosko could you please modify the the PR description? It's the xDS request from the Envoy to EG that caused the issue, not the snapshot. |
| @@ -0,0 +1 @@ | |||
| Added `xdsServer.maxReceiveMessageSize` field to the EnvoyGateway API, allowing operators to increase the xDS gRPC server's maximum receive message size when large xDS snapshots at scale exceed the default 4MiB limit, which previously caused "received message larger than max" stream errors. | |||
There was a problem hiding this comment.
| Added `xdsServer.maxReceiveMessageSize` field to the EnvoyGateway API, allowing operators to increase the xDS gRPC server's maximum receive message size when large xDS snapshots at scale exceed the default 4MiB limit, which previously caused "received message larger than max" stream errors. | |
| Added `xdsServer.maxReceiveMessageSize` to the EnvoyGateway API, allowing operators to raise the xDS gRPC server's receive limit. At large resource counts, an Envoy proxy's delta xDS request on stream reconnect can exceed the 4MiB gRPC default, which previously broke the stream with "received message larger than max" errors and left the proxy on its last known-good configuration. |
Signed-off-by: Lin Moskovitch <lin.moskovitch@sap.com>
Signed-off-by: Lin Moskovitch <lin.moskovitch@sap.com>
Signed-off-by: Lin Moskovitch <lin.moskovitch@sap.com>
Signed-off-by: Lin Moskovitch <lin.moskovitch@sap.com>
Signed-off-by: Lin Moskovitch <lin.moskovitch@sap.com>
93f4915 to
ef7fcbe
Compare
|
/retest |
Signed-off-by: Lin Moskovitch <lin.moskovitch@sap.com>
linmosko
changed the title
|
/retest |
|
/retest |
|
Hi @zhaohuabing ! |
|
/retest |
2 similar comments
|
/retest |
|
/retest |
…nfigurable (envoyproxy#9564) * feat(xds): add configurable max receive message size for xDS gRPC server Signed-off-by: Lin Moskovitch <lin.moskovitch@sap.com> * update release notes Signed-off-by: Lin Moskovitch <lin.moskovitch@sap.com> * test(xds): add coverage for MaxRecvMsgSize runner option Signed-off-by: Lin Moskovitch <lin.moskovitch@sap.com> * lint Signed-off-by: Lin Moskovitch <lin.moskovitch@sap.com> * refactor: rename MaxRecvMsgSize to MaxReceiveMessageSize for consistency Signed-off-by: Lin Moskovitch <lin.moskovitch@sap.com> * docs: improve MaxReceiveMessageSize comment and release note Signed-off-by: Lin Moskovitch <lin.moskovitch@sap.com> * feat(xds): set default max receive message size to 32MiB Signed-off-by: Lin Moskovitch <lin.moskovitch@sap.com> --------- Signed-off-by: Lin Moskovitch <lin.moskovitch@sap.com> Co-authored-by: Guy Daich <guy.daich@sap.com>
4 participants
Raises the xDS gRPC server's default receive limit from 4MiB to 32MiB,
and adds
xdsServer.maxReceiveMessageSizeto the EnvoyGateway API, allowing to raise the xDS gRPC server's receive limit.At large resource counts, an Envoy proxy's delta xDS request on stream reconnect
echoes back the name and version of every resource it holds. This request can exceed
the 4MiB gRPC default, breaking the stream immediately with:
This leaves the proxy stuck on its last known-good configuration. Setting
maxReceiveMessageSizein the EnvoyGateway config resolves this.Note: this limit applies only to what Envoy Gateway receives; the configuration
it sends to Envoy is not bounded by it.
PR Checklist
git commit -s). See DCO: Sign your work./api), the API was discussed and agreed before the implementation. The API change can be in a separate PR, or in the same PR, but the API must be agreed before implementation. N/A if this PR does not contain API changes.make generate gen-check,make lint, and the unit-test/coverage build pass. (Flaky e2e failures are not considered breakages, butgen-check,lint, and coverage MUST pass.)release-notes/current/<section>/<pr-number>-<slug>.md(seerelease-notes/current/README.mdfor sections and naming). N/A if this PR does not contain non-trivial changes.make gen-checkand committed the result if API/helm charts/modules changed.release-notes/current/breaking_changes/.