Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
- op: add
path: /dynamic_resources/ads_config/grpc_services/0/envoy_grpc/max_receive_message_length
value:
value: 8388608
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
admin:
access_log:
- name: envoy.access_loggers.file
typed_config:
'@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/null
address:
socket_address:
address: 127.0.0.1
port_value: 19000
cluster_manager:
local_cluster_name: local_cluster
deferredStatOptions:
enableDeferredCreationStats: true
dynamic_resources:
ads_config:
api_type: DELTA_GRPC
grpc_services:
- envoy_grpc:
cluster_name: xds_cluster
max_receive_message_length:
value: 8388608
set_node_on_first_message_only: true
transport_api_version: V3
cds_config:
ads: {}
initial_fetch_timeout: 0s
resource_api_version: V3
lds_config:
ads: {}
initial_fetch_timeout: 0s
resource_api_version: V3
layered_runtime:
layers:
- name: global_config
static_layer:
envoy.restart_features.use_eds_cache_for_ads: true
re2.max_program_size.error_level: 4294967295
re2.max_program_size.warn_level: 1000
node:
locality:
zone: $(ENVOY_SERVICE_ZONE)
overload_manager:
refresh_interval: 0.25s
resource_monitors:
- name: envoy.resource_monitors.global_downstream_max_connections
typed_config:
'@type': type.googleapis.com/envoy.extensions.resource_monitors.downstream_connections.v3.DownstreamConnectionsConfig
max_active_downstream_connections: 50000
static_resources:
clusters:
- connect_timeout: 0.250s
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: prometheus_stats
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 19000
name: prometheus_stats
type: STATIC
- connect_timeout: 10s
eds_cluster_config:
eds_config:
ads: {}
resource_api_version: V3
service_name: local_cluster
load_balancing_policy:
policies:
- typed_extension_config:
name: envoy.load_balancing_policies.least_request
typed_config:
'@type': type.googleapis.com/envoy.extensions.load_balancing_policies.least_request.v3.LeastRequest
locality_lb_config:
zone_aware_lb_config:
min_cluster_size: "1"
name: local_cluster
type: EDS
- connect_timeout: 10s
load_assignment:
cluster_name: xds_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: envoy-gateway
port_value: 18000
load_balancing_weight: 1
load_balancing_weight: 1
name: xds_cluster
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
'@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
tls_certificate_sds_secret_configs:
- name: xds_certificate
sds_config:
path_config_source:
path: /sds/xds-certificate.json
resource_api_version: V3
tls_params:
tls_maximum_protocol_version: TLSv1_3
validation_context_sds_secret_config:
name: xds_trusted_ca
sds_config:
path_config_source:
path: /sds/xds-trusted-ca.json
resource_api_version: V3
type: STRICT_DNS
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
'@type': type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
explicit_http_config:
http2_protocol_options:
connection_keepalive:
interval: 30s
timeout: 5s
listeners:
- address:
socket_address:
address: 0.0.0.0
port_value: 19001
protocol: TCP
bypass_overload_manager: true
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
http_filters:
- name: envoy.filters.http.router
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
normalize_path: true
route_config:
name: local_route
virtual_hosts:
- domains:
- '*'
name: prometheus_stats
routes:
- match:
headers:
- name: :method
string_match:
exact: GET
path: /stats/prometheus
route:
cluster: prometheus_stats
stat_prefix: eg-stats-http
name: envoy-gateway-proxy-stats-0.0.0.0-19001
stats_config:
stats_tags:
- regex: \.zone(\.(([^\.]+)\.))
tag_name: from_zone
- regex: \.zone\.[^\.]+\.(([^\.]+)\.)
tag_name: to_zone
- regex: ^cluster(\..+\.(.+))\.total_match_count$
tag_name: socket_match_name
- regex: circuit_breakers\.((.+?)\.).+
tag_name: priority
use_all_default_tags: true
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ dynamicResources:
grpcServices:
- envoyGrpc:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you also add this to one of the in/out merge tests? Demonstrate that the merge is not block and it works as expected, patching the relevant fields.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, added

clusterName: xds_cluster
maxReceiveMessageLength: 8388608

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there an equivalent parameter on the Envoy Gateway side that also needs tuning to allow sending large messages? I think grpc doesn't limit sent message sizes by default, but EG/go-control-plane may.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

internal/xds/runner/runner.go:212 uses grpc.NewServer() with no send size limit - Go gRPC default is unlimited for sends.

setNodeOnFirstMessageOnly: true
transportApiVersion: V3
ldsConfig:
Expand Down
7 changes: 7 additions & 0 deletions internal/xds/bootstrap/util_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,13 @@ func TestApplyBootstrapConfig(t *testing.T) {
},
defaultBootstrap: str,
},
{
name: "merge-max-receive-message-length",
boostrapConfig: &egv1a1.ProxyBootstrap{
Type: new(egv1a1.BootstrapTypeJSONPatch),
},
defaultBootstrap: str,
},
{
name: "patch-global-config",
boostrapConfig: &egv1a1.ProxyBootstrap{
Expand Down
5 changes: 4 additions & 1 deletion internal/xds/bootstrap/validate.go
Original file line number Diff line number Diff line change
Expand Up @@ -61,9 +61,12 @@ func Validate(boostrapConfig *egv1a1.ProxyBootstrap) error {
}

// Ensure dynamic resources config is same except for initial_fetch_timeout
// and max_receive_message_length on the xDS gRPC service (allows tuning the
// gRPC receive buffer when the xDS snapshot grows large at scale).
if userBootstrap.DynamicResources == nil ||
cmp.Diff(userBootstrap.DynamicResources, defaultBootstrap.DynamicResources, protocmp.Transform(),
protocmp.IgnoreFields(&corev3.ConfigSource{}, "initial_fetch_timeout")) != "" {
protocmp.IgnoreFields(&corev3.ConfigSource{}, "initial_fetch_timeout"),
protocmp.IgnoreFields(&corev3.GrpcService_EnvoyGrpc{}, "max_receive_message_length")) != "" {
return fmt.Errorf("dynamic_resources cannot be modified")
}

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Added support for updating `max_receive_message_length` on the xDS gRPC service via bootstrap override, allowing operators to tune the gRPC receive buffer when the xDS snapshot grows large at scale.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use a numeric release-note fragment filename

When make lint runs in CI, its lint.release-notes-filenames target invokes tools/hack/check-release-notes-filenames.sh, which requires current fragments to match ^[0-9]+-[a-z0-9-]+\.md$; this XXXXX-... placeholder filename fails that check and will block the lint job until it is renamed with a numeric PR/issue prefix.

Useful? React with 👍 / 👎.