Skip to content

Spanner: client-side metrics export uses wrong project #13240

@RRap0so

Description

@RRap0so

Bug

When SpannerOptions.newBuilder().build() is used without an explicit .setProjectId(), the project ID defaults to the GCP metadata server value — which on GKE with shared VPC is the host project (e.g., a-gke-project-unrelated), not the application project.

DatabaseId.of("my-app-project", instance, db) correctly routes the database connection to the application project. However, the client-side built-in metrics export uses SpannerOptions.getProjectId() to determine the Cloud Monitoring target project. This causes metrics to be exported to the host project, which the service account typically lacks monitoring.metricWriter permission on.

Environment

  • GKE with shared VPC (host project != application project)
  • Spanner Java client with built-in metrics enabled (default in recent versions)
  • Service account has Spanner + Monitoring permissions on the application project only

Error

createServiceTimeSeries request failed for spanner metrics.
Need monitoring metric writer permission on project=a-gke-project-unrelated

Impact

  • Silent metric data loss
  • Repeated permission errors in logs
  • Potential memory pressure from accumulated retry buffers
  • Affects any Java Spanner client on GKE shared VPC without explicit .setProjectId()

Workaround

Explicitly set .setProjectId() on SpannerOptions.newBuilder():

SpannerOptions options = SpannerOptions.newBuilder()
    .setProjectId("my-app-project")
    .build();

Suggested fix

Add a setMetricsProjectId(String) to SpannerOptions.Builder that allows configuring the metrics export project independently. When not set, fall back to getProjectId(). Additionally, log a warning in getDatabaseClient() when the DatabaseId project differs from the metrics project.

Reference implementation: https://github.com/RRap0so/google-cloud-java/tree/fix/metrics-project-default

Metadata

Metadata

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions