bigtable: potential memory leak from xDS runner goroutine when DirectPath is unavailable, introduced in v1.46.0

[mnjnwang]

Client

v1.46.0+

Environment

go 1.26.3

Code and Dependencies

// initialize bigtable client
bigtableClientClientConf := gcpBigtable.ClientConfig{
		AppProfile:      appProfile,
		MetricsProvider: gcpBigtable.NoopMetricsProvider{},
	}
bigtableClient, btErr := gcpBigtable.NewClientWithConfig(ctx, projectID, instanceID, bigtableClientClientConf, option.WithGRPCConnectionPool(cfg.BigtableConnectionPoolSize))
...

go.mod

module modname

go 1.25.0

require (
   cloud.google.com/go/bigtable v1.47.0
)

Expected behavior

Memory usage is stable, consistent with v1.43.0–v1.45.0 behavior

Actual behavior

Heap grows continuously after upgrading to v1.46.0. Heap profile shows (*adsStreamImpl).runner from google.golang.org/grpc@v1.80.0/internal/xds/clients/xdsclient/ads_stream.go accumulating memory. Which caused OOM in one of our pods.

Screenshots

Memory Usage Trends

Heap Live Size

Additional context

We are suspecting the bug is the unconditional DirectPath probe spawning xDS goroutines that can't clean up on non-DirectPath infrastructure.
The trigger is v1.46.0 enabling BigtableChannelPool by default, which exposed this to all users.

Our temporary workaround :
After setting CBT_ENABLE_DIRECTPATH=false, the memory usage returns to stable:

[bhshkh]

Reassigning to @sushanb who works on the data client.

[sushanb]

@mnjnwang to clarify, this is about the step increase in the overall memory utilisation when upgrading to 1.46.0 client, right?

I just want to make sure you are seeing the memory increase and stay constant but the constant memory accumulation through the lifetime of the pod.

[mnjnwang]

@mnjnwang to clarify, this is about the step increase in the overall memory utilisation when upgrading to 1.46.0 client, right?

I just want to make sure you are seeing the memory increase and stay constant but the constant memory accumulation through the lifetime of the pod.

It is not a stable step increase. We are seeing constant memory accumulation over the lifetime of the pod. It steadily climbs and looks likely a memory leak introduced in 1.46.0.

[sushanb]

Thanks. We are looking into it.
One more question, do you set the connection pool size or use the default one?

[mnjnwang]

@sushanb Yes we set the pool size to 2 by adding option.WithGRPCConnectionPool(2)) in the opts during client initialization.