Skip to content

grpc-js-xds: WeightedCluster with omitted total_weight can select no cluster and crash in resolver #3069

Description

@shadialtarsha

Problem description

@grpc/grpc-js-xds can crash when an xDS route uses weighted_clusters, omits the deprecated total_weight field, and the cluster weights do not add up to 100.

For example, a route containing one cluster with weight 1 fails with:

TypeError: Cannot read properties of undefined (reading 'ref')
    at Object.invoke (.../node_modules/@grpc/grpc-js-xds/build/src/resolver-xds.js:343:36)

The same xDS route works with grpcurl/gRPC Go.

The resolver currently defaults an absent total_weight to 100:

routeAction = new WeightedClusterRouteAction(
  weightedClusters,
  route.route!.weighted_clusters!.total_weight?.value ?? 100,
  // ...
);

WeightedClusterRouteAction therefore generates a random value in [0, 100). With one cluster of weight 1, only values in [0, 1) select that cluster. The remaining ~99% return an empty cluster name.

The resolver then looks up that empty name and dereferences undefined:

const clusterResult = action.getCluster();
const clusterRef = this.clusterRefs.get(clusterResult.name)!;
clusterRef.ref();

The generated Envoy API documentation says that total_weight is deprecated and clients should use the sum of all cluster weights.

Relevant code:

Reproduction steps

  1. Configure an RDS route similar to:
route:
  weighted_clusters:
    clusters:
      - name: test-cluster
        weight:
          value: 1
    # total_weight intentionally omitted
  1. Register @grpc/grpc-js-xds and create an xDS client:
const grpc = require('@grpc/grpc-js');
require('@grpc/grpc-js-xds').register();

const client = new grpc.Client(
  'xds:///test-service',
  grpc.credentials.createInsecure()
);
  1. Make repeated RPCs through the route.

  2. Most calls fail with:

TypeError: Cannot read properties of undefined (reading 'ref')

Setting the cluster weight to 100, explicitly setting total_weight: 1, or using a non-weighted single-cluster route avoids the failure.

Expected behavior

When total_weight is omitted, the client should use the sum of the supplied cluster weights.

For one cluster with weight 1, that cluster should receive 100% of requests.

The resolver should also return a controlled error instead of throwing if cluster selection somehow produces an unknown or empty cluster name.

Environment

  • OS: Linux x86_64, Kubernetes
  • Node version: [fill in]
  • @grpc/grpc-js version: [fill in]
  • @grpc/grpc-js-xds version: [fill in]
  • Installation method: npm

Suggested fix

Calculate the denominator from the cluster weights:

const totalWeight = weightedClusters.reduce(
  (sum, cluster) => sum + cluster.weight,
  0
);

Use that value when creating WeightedClusterRouteAction, rather than defaulting an omitted total_weight to 100.

It would also be helpful to guard the clusterRefs.get() result so an invalid selection produces a gRPC status error instead of an uncaught TypeError.

Additional context

A configuration with weights 1 and 99 works because the weights happen to total 100. The failure occurs when the supplied weights have a nonzero sum other than 100 and total_weight is omitted.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions