Client
v1.57.0
Environment
go 1.26.2
Expected behavior
Client uses a reasonable amount of memory and CPU.
Actual behavior
Client uses an insane amount of CPU and memory.
Screenshots
I filed an issue about this a long time ago: #13092. It turned out that I had set the connection pool size too large, but even with a connection pool size of 1 the client uses a significant amount of memory but it was tolerable. However, sometime in the last few months GCS has started rolling out some kind of backend change that massively inflates the number of connections that the storage client makes. The issue started manifesting in different <regions,az> pairs at different times and was not caused by any code or deployment change. Software that had been running just fine for weeks or months suddenly started using way more memory and when we checked heap profiles it was all caused by gRPC connections.
There was a thread about this in the grpc-go library: grpc/grpc-go#8888 (comment) but that was closed as this is clearly a GCP/GCS related issue (although arguably the grpc-go library should have a knob that limits the maximum numbers of connections that will be established).
Google has admitted that this problem is caused by a backend change and that're investigating in direct support tickets with multiple of my customers, but we haven't received any official acknowledgement in public channels that there is something wrong or that the SDKs are being modified or fixed. We've had to disable direct connectivity in our software for all our customers and that results in significantly higher latency for them.
Client
v1.57.0
Environment
go 1.26.2
Expected behavior
Client uses a reasonable amount of memory and CPU.
Actual behavior
Client uses an insane amount of CPU and memory.
Screenshots
I filed an issue about this a long time ago: #13092. It turned out that I had set the connection pool size too large, but even with a connection pool size of 1 the client uses a significant amount of memory but it was tolerable. However, sometime in the last few months GCS has started rolling out some kind of backend change that massively inflates the number of connections that the storage client makes. The issue started manifesting in different <regions,az> pairs at different times and was not caused by any code or deployment change. Software that had been running just fine for weeks or months suddenly started using way more memory and when we checked heap profiles it was all caused by gRPC connections.
There was a thread about this in the grpc-go library: grpc/grpc-go#8888 (comment) but that was closed as this is clearly a GCP/GCS related issue (although arguably the grpc-go library should have a knob that limits the maximum numbers of connections that will be established).
Google has admitted that this problem is caused by a backend change and that're investigating in direct support tickets with multiple of my customers, but we haven't received any official acknowledgement in public channels that there is something wrong or that the SDKs are being modified or fixed. We've had to disable direct connectivity in our software for all our customers and that results in significantly higher latency for them.