[spanner]: MultiplexedSessionDatabaseClient.CHANNEL_USAGE static HashMap leaks SpannerImpl instances on close

[cgati]

Environment details

1. API: Cloud Spanner
2. OS: Linux (GKE, debian-based container)
3. Java version: OpenJDK 17.0.15+6-LTS
4. google-cloud-spanner: 6.111.1 (via google-libraries-bom 26.77.0). Regression introduced in 6.108.0 by googleapis/java-spanner#4191.

Steps to reproduce

1. Create a Spanner client via SpannerOptions.newBuilder().build().getService()
2. Call getDatabaseClient(databaseId) on it
3. Close the client via spanner.close()
4. Repeat over time

Each cycle permanently leaks the SpannerImpl and all its retained objects (GapicSpannerRpc, gRPC channels, Netty SSL contexts) into a static HashMap.

Code example

import com.google.cloud.NoCredentials;
import com.google.cloud.spanner.DatabaseClient;
import com.google.cloud.spanner.DatabaseId;
import com.google.cloud.spanner.Spanner;
import com.google.cloud.spanner.SpannerOptions;

import java.lang.reflect.Field;
import java.util.Map;

/**
 * Demonstrates that MultiplexedSessionDatabaseClient.CHANNEL_USAGE
 * leaks SpannerImpl instances on close().
 *
 * No emulator or credentials required — the leak occurs during
 * client construction, before any RPCs are made.
 */
public class SpannerChannelUsageLeak {
    public static void main(String[] args) throws Exception {
        Class<?> clazz = Class.forName(
            "com.google.cloud.spanner.MultiplexedSessionDatabaseClient");
        Field field = clazz.getDeclaredField("CHANNEL_USAGE");
        field.setAccessible(true);
        Map<?, ?> channelUsage = (Map<?, ?>) field.get(null);

        DatabaseId dbId = DatabaseId.of("test-project", "test-instance", "test-db");

        System.out.println("Creating and closing 100 Spanner clients...\n");

        for (int i = 1; i <= 100; i++) {
            Spanner spanner = SpannerOptions.newBuilder()
                .setProjectId("test-project")
                .setCredentials(NoCredentials.getInstance())
                .build()
                .getService();

            DatabaseClient client = spanner.getDatabaseClient(dbId);
            spanner.close();

            if (i % 10 == 0) {
                System.gc();
                long usedMb = (Runtime.getRuntime().totalMemory()
                    - Runtime.getRuntime().freeMemory()) / (1024 * 1024);
                System.out.printf(
                    "Iteration %3d: CHANNEL_USAGE size = %d, heap used = %d MB%n",
                    i, channelUsage.size(), usedMb);
            }
        }

        System.out.println("\nExpected CHANNEL_USAGE size after 100 create/close cycles: 0");
        System.out.println("Actual CHANNEL_USAGE size: " + channelUsage.size());
    }
}

$ gradle run 2>&1 | grep Iteration
Iteration  10: CHANNEL_USAGE size = 10, heap used = 15 MB
Iteration  20: CHANNEL_USAGE size = 20, heap used = 17 MB
Iteration  30: CHANNEL_USAGE size = 30, heap used = 18 MB
Iteration  40: CHANNEL_USAGE size = 40, heap used = 21 MB
Iteration  50: CHANNEL_USAGE size = 50, heap used = 22 MB
Iteration  60: CHANNEL_USAGE size = 60, heap used = 24 MB
Iteration  70: CHANNEL_USAGE size = 70, heap used = 26 MB
Iteration  80: CHANNEL_USAGE size = 80, heap used = 28 MB
Iteration  90: CHANNEL_USAGE size = 90, heap used = 29 MB
Iteration 100: CHANNEL_USAGE size = 100, heap used = 31 MB

Stack trace

None.

External references such as API reference guides

None.

Any additional information below

MultiplexedSessionDatabaseClient's constructor adds the SpannerImpl to a static HashMap:

// MultiplexedSessionDatabaseClient.java
private static final Map<SpannerImpl, BitSet> CHANNEL_USAGE = new HashMap<>();

// In constructor:
synchronized (CHANNEL_USAGE) {
    CHANNEL_USAGE.putIfAbsent(sessionClient.getSpanner(), new BitSet(numChannels));
    this.channelUsage = CHANNEL_USAGE.get(sessionClient.getSpanner());
}

close() never removes the entry:

void close() {
    synchronized (this) {
        if (!this.isClosed) {
            this.isClosed = true;
            this.maintainer.stop();
        }
    }
    // CHANNEL_USAGE entry is never removed
  }

Prior to #4191 (shipped in 6.108.0), MultiplexedSessionDatabaseClient was only created when SessionPoolOptions.useMultiplexedSession was true (default: false), so this code path was not exercised.

After #4191, creation is unconditional in SpannerImpl.getDatabaseClient(), triggering the leak for any application that creates and closes Spanner instances.

[glasser]

Perhaps CHANNEL_USAGE should be a WeakHashMap?