Skip to content

client routes: add exclusive-proxy config to control fallback behavior#881

Open
nikagra wants to merge 2 commits intoscylladb:scylla-4.xfrom
nikagra:fix/client-routes-disable-fallback
Open

client routes: add exclusive-proxy config to control fallback behavior#881
nikagra wants to merge 2 commits intoscylladb:scylla-4.xfrom
nikagra:fix/client-routes-disable-fallback

Conversation

@nikagra
Copy link
Copy Markdown

@nikagra nikagra commented Apr 28, 2026

Summary

  • Adds advanced.client-routes.exclusive-proxy (boolean, default false) to control whether the driver falls back to a node's broadcast address when no route entry exists in system.client_routes
  • When false (default): nodes without a route entry are reached directly via their broadcast address — backward-compatible, supports mixed proxy/direct topologies
  • When true: any node without a route is treated as unreachable; resolve() throws IllegalStateException with a WARN log, the node stays DOWN, and the reconnection loop retries until CLIENT_ROUTES_CHANGE populates the route

Motivation

Two requirements needed to coexist:

  1. Mixed proxy/direct topologies (original design): some nodes are behind the private endpoint, others are not — the driver connects directly to unrouted nodes. The default exclusive-proxy = false preserves this behavior.

  2. Strict proxy enforcement (new): when a new node is added there is a race window before its system.client_routes entry is published. With exclusive-proxy = true the driver refuses to bypass the proxy during this window, matching the behavior of the Rust driver fix (failing address translation on missing entries). The node goes DOWN and reconnects once the route is available.

Changes

  • DefaultDriverOption: new CLIENT_ROUTES_EXCLUSIVE_PROXY enum constant
  • TypedDriverOption: new typed Boolean constant
  • reference.conf: advanced.client-routes.exclusive-proxy = false with documentation
  • ClientRoutesEndPoint: restored fallbackEndPoint field; resolve() uses the fallback when exclusive-proxy = false, throws IllegalStateException + WARN when exclusive-proxy = true
  • ClientRoutesTopologyMonitor: reads config flag in constructor; passes fallback endpoint (null when exclusive-proxy is true) and flag to ClientRoutesEndPoint
  • Tests: constructor calls updated; separate test cases for exclusive and non-exclusive modes; createHandlerWithExclusiveProxy() helper in the topology monitor test

Not in this PR

  • Graceful migration (soft-drop of existing direct connections when a route is added for a node) — tracked as a follow-up

@nikagra nikagra force-pushed the fix/client-routes-disable-fallback branch from b870f1d to 90b8684 Compare April 28, 2026 14:52
@dkropachev
Copy link
Copy Markdown

@nikagra , I would prefere to have an option to enable it and handle case when host switches from direct connection to client routes, you can make host connection pool either softly drop all the connections that used to directly, check if it won't blow the PR.

@nikagra nikagra changed the title client routes: disable fallback to broadcast address when route is missing client routes: add exclusive-proxy config to control fallback behavior Apr 28, 2026
@nikagra nikagra force-pushed the fix/client-routes-disable-fallback branch from 1d9717b to 805ec7f Compare April 28, 2026 18:24
@nikagra nikagra marked this pull request as ready for review April 28, 2026 20:16
@nikagra nikagra requested review from Copilot and dkropachev April 28, 2026 20:16
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an advanced.client-routes.exclusive-proxy configuration flag to control whether the driver may fall back to a node’s broadcast address when no entry exists in system.client_routes.

Changes:

  • Introduces CLIENT_ROUTES_EXCLUSIVE_PROXY as a new built-in driver option (typed + defaulted).
  • Updates client-routes endpoint resolution to either fall back (default) or throw when a route is missing (exclusive mode).
  • Updates reference configuration documentation and expands tests to cover both exclusive/non-exclusive behaviors.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
core/src/main/java/com/datastax/oss/driver/internal/core/metadata/ClientRoutesTopologyMonitor.java Reads the new flag and passes fallback endpoint as null in exclusive mode; adjusts endpoint construction behavior.
core/src/main/java/com/datastax/oss/driver/internal/core/metadata/ClientRoutesEndPoint.java Adds exclusive-proxy behavior in resolve() (fallback vs WARN+throw) and updates constructor contract.
core/src/main/java/com/datastax/oss/driver/api/core/config/DefaultDriverOption.java Adds CLIENT_ROUTES_EXCLUSIVE_PROXY option path.
core/src/main/java/com/datastax/oss/driver/api/core/config/TypedDriverOption.java Adds typed Boolean option constant for exclusive-proxy.
core/src/main/java/com/datastax/oss/driver/api/core/config/OptionsMap.java Registers default value (false) for the new typed option.
core/src/main/resources/reference.conf Documents advanced.client-routes.exclusive-proxy with default false.
core/src/test/java/com/datastax/oss/driver/internal/core/metadata/ClientRoutesTopologyMonitorTest.java Adds/updates tests for exclusive vs non-exclusive fallback behavior and host_id-null handling.
core/src/test/java/com/datastax/oss/driver/internal/core/metadata/ClientRoutesEndPointTest.java Updates constructor calls and adds tests for exclusive mode throwing vs fallback mode.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

nikagra added 2 commits April 29, 2026 23:52
…ssing

When a node has client-routes configured but no matching entry is found,
fall back to the broadcast address by default (backward-compatible).

When advanced.client-routes.exclusive-proxy=true, throw instead of
falling back so the node stays DOWN until a CLIENT_ROUTES_CHANGE event
provides the route.

The constructor of ClientRoutesEndPoint now enforces that fallbackEndPoint
is non-null when exclusiveProxy=false, making the invariant explicit at
construction time rather than silently failing at resolve() time.

The WARN log on null host_id no longer mentions exclusive-proxy behaviour
since that path fires regardless of the flag; the message now focuses on
the actual cause (corrupted system tables).
…cement

Introduces advanced.client-routes.exclusive-proxy (default: false).

When false (default), the driver falls back to the node broadcast
address when no client-route entry exists — preserving backward
compatibility with mixed proxy/direct topologies.

When true, no fallback is attempted: the node stays DOWN until a
CLIENT_ROUTES_CHANGE event publishes the route. This prevents silent
bypass of the proxy infrastructure during node additions.
@nikagra nikagra force-pushed the fix/client-routes-disable-fallback branch from 805ec7f to d341fef Compare April 29, 2026 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants