Add max connection age settings to gRPC servers for improved load balancing

[dsmith3197]

Clients often use indefinitely persistent connections. Given that we recommend network-level load balancers, this can result in sub-par balancing as the worker pool grows/shrinks due to a lack of rebalancing.

Similar to the work done for HTTP servers in #19141, we should add the following config options to sources that run gRPC servers (following the gRPC standards).

keepalive.max_connection_age
keepalive.max_connection_age_grace

This requires upstream work to be done first grpc/grpc-rust#1428

Related: #10728

[lewo-hkcm]

I would also have liked to be able configure a grpc connection max age in order to rebalance client connections. The feature is now supported by grpc-rust since grpc/grpc-rust#1865 has been merged.

Do you plan to add this parameter to the otel source?

[pront]

I would also have liked to be able configure a grpc connection max age in order to rebalance client connections. The feature is now supported by grpc-rust since grpc/grpc-rust#1865 has been merged.

Do you plan to add this parameter to the otel source?

Hi @lewo-hkcm, we are not actively working on this feature at the moment. But we would warmly welcome a contribution.

[fpytloun]

Implemented #25660 and #25662, which finally make it possible to properly load-balance Vector sources and sinks.

Before:

After:

The last screenshot shows two deployments:

• On the left: Vector aggregators running on EKS, using load_balance mode from downstream hosts together with the keepalive timeout. The load is now distributed very evenly.
• On the right: clients shipping data to srv01 and srv02 using the fallback_primary strategy with idempotently shuffled remote hosts in the client Vector config. Clients prefer the primary host and use the second host as failover. After recovery, Vector on srv01/srv02 forces connection rebalancing via the keepalive timeout.