Skip to content

Commit 61574e5

Browse files
gopalldbclaude
gopalldb
and
claude
authored
Add UseQueryForMetadata connection property for Thrift metadata operations (#1242)
## Summary - Add opt-in connection property `UseQueryForMetadata` that makes the Thrift path use SQL SHOW commands (via `DatabricksMetadataQueryClient`) instead of native Thrift RPCs for metadata operations - Rename `DatabricksMetadataSdkClient` → `DatabricksMetadataQueryClient` and internal field `sdkClient` → `queryExecutionClient` since the class now serves both SEA and Thrift paths ## Problem Thrift metadata operations (`GetTables`, `GetSchemas`, `GetColumns`, etc.) pass filter patterns directly to the server, which treats `_` as a single-character wildcard. This causes incorrect results — e.g., querying for catalog `a_b` also returns `axb`, `a1b`, etc. The SEA metadata path already solved this by using SQL SHOW commands with `LIKE` patterns (via `CommandBuilder` + `WildcardUtil`), which handle `_` correctly. ## Solution When `UseQueryForMetadata=1` and the client type is THRIFT, `DatabricksSession` creates a `DatabricksMetadataQueryClient` wrapping the Thrift `IDatabricksClient`. This reuses the entire SEA metadata implementation — no code duplication needed — because `DatabricksMetadataQueryClient` only calls `executeStatement()` and `getConnectionContext()` on the underlying client, both of which `DatabricksThriftServiceClient` already implements. ### How it works **`DatabricksSession.getDatabricksMetadataClient()`:** - **THRIFT + `UseQueryForMetadata=0` (default):** returns the Thrift client cast to `IDatabricksMetadataClient` (unchanged behavior) - **THRIFT + `UseQueryForMetadata=1`:** returns a `DatabricksMetadataQueryClient` wrapping the Thrift client, which executes SHOW SQL commands for metadata - **SEA:** returns `DatabricksMetadataQueryClient` as before The feature is also correctly wired in all SEA→Thrift fallback paths (temporary redirect and rate limit). ## Changes | File | Change | |------|--------| | `DatabricksJdbcUrlParams.java` | Add `USE_QUERY_FOR_METADATA` enum constant | | `IDatabricksConnectionContext.java` | Add `useQueryForMetadata()` interface method | | `DatabricksConnectionContext.java` | Implement `useQueryForMetadata()` accessor | | `DatabricksSession.java` | Create `DatabricksMetadataQueryClient` for Thrift when enabled; update `getDatabricksMetadataClient()` dispatch; handle SEA→Thrift fallback paths | | `DatabricksMetadataSdkClient.java` → `DatabricksMetadataQueryClient.java` | Rename class + rename `sdkClient` field → `queryExecutionClient` | | `DatabricksSessionTest.java` | Add tests for enabled/disabled dispatch behavior | | `DatabricksMetadataSdkClientTest.java` → `DatabricksMetadataQueryClientTest.java` | Rename test class | | `NEXT_CHANGELOG.md` | Add changelog entry | ## Backward Compatibility - **Opt-in only:** `UseQueryForMetadata=0` by default — existing Thrift behavior is completely unchanged - **Independent of `EnableShowCommandForGetFunctions`:** that property continues to work separately - **SEA path unchanged:** `DatabricksMetadataQueryClient` is still used for SEA as before ## Test plan - [x] `DatabricksSessionTest` — 16/16 pass (includes 2 new tests verifying dispatch with `UseQueryForMetadata=1` and default) - [x] `DatabricksMetadataQueryClientTest` — 44/44 pass (all existing metadata tests pass with renamed class) - [x] `mvn spotless:check` — clean - [x] `mvn clean install -DskipTests` — builds successfully 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Gopal Lal <gopal.lal@databricks.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent a4eb66d commit 61574e5

8 files changed

Lines changed: 158 additions & 49 deletions

File tree

‎NEXT_CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44

55
### Added
66
- Added connection property `OAuthWebServerTimeout` to configure the OAuth browser authentication timeout for U2M (user-to-machine) flows, and also updated hardcoded 1-hour timeout to default 120 seconds timeout.
7+
- Added connection property `UseQueryForMetadata` to use SQL SHOW commands instead of Thrift RPCs for metadata operations (getCatalogs, getSchemas, getTables, getColumns, getFunctions). This fixes incorrect wildcard matching where `_` was treated as a single-character wildcard in Thrift metadata pattern filters.
78

89
### Updated
910
- Bumped `com.fasterxml.jackson.core:jackson-core` from 2.18.3 to 2.18.6.

‎src/main/java/com/databricks/jdbc/api/impl/DatabricksConnectionContext.java‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1098,6 +1098,11 @@ public boolean enableShowCommandsForGetFunctions() {
10981098
return getParameter(DatabricksJdbcUrlParams.ENABLE_SHOW_COMMAND_FOR_GET_FUNCTIONS).equals("1");
10991099
}
11001100

1101+
@Override
1102+
public boolean useQueryForMetadata() {
1103+
return getParameter(DatabricksJdbcUrlParams.USE_QUERY_FOR_METADATA).equals("1");
1104+
}
1105+
11011106
@Override
11021107
public boolean getEnableMetricViewMetadata() {
11031108
return getParameter(DatabricksJdbcUrlParams.ENABLE_METRIC_VIEW_METADATA).equals("1");

‎src/main/java/com/databricks/jdbc/api/impl/DatabricksSession.java‎

Lines changed: 26 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
import com.databricks.jdbc.dbclient.IDatabricksClient;
1515
import com.databricks.jdbc.dbclient.IDatabricksMetadataClient;
1616
import com.databricks.jdbc.dbclient.impl.sqlexec.DatabricksEmptyMetadataClient;
17-
import com.databricks.jdbc.dbclient.impl.sqlexec.DatabricksMetadataSdkClient;
17+
import com.databricks.jdbc.dbclient.impl.sqlexec.DatabricksMetadataQueryClient;
1818
import com.databricks.jdbc.dbclient.impl.sqlexec.DatabricksSdkClient;
1919
import com.databricks.jdbc.dbclient.impl.thrift.DatabricksThriftServiceClient;
2020
import com.databricks.jdbc.exception.DatabricksHttpException;
@@ -84,8 +84,9 @@ public DatabricksSession(IDatabricksConnectionContext connectionContext)
8484
public DatabricksSession(
8585
IDatabricksConnectionContext connectionContext, IDatabricksClient testDatabricksClient) {
8686
this.databricksClient = testDatabricksClient;
87-
if (databricksClient instanceof DatabricksSdkClient) {
88-
this.databricksMetadataClient = new DatabricksMetadataSdkClient(databricksClient);
87+
if (databricksClient instanceof DatabricksSdkClient
88+
|| connectionContext.useQueryForMetadata()) {
89+
this.databricksMetadataClient = new DatabricksMetadataQueryClient(databricksClient);
8990
}
9091
this.isSessionOpen = false;
9192
this.sessionInfo = null;
@@ -141,12 +142,17 @@ public void open() throws SQLException {
141142
this.databricksClient =
142143
DatabricksMetricsTimedProcessor.createProxy(
143144
new DatabricksThriftServiceClient(connectionContext));
145+
if (connectionContext.useQueryForMetadata()) {
146+
this.databricksMetadataClient =
147+
DatabricksMetricsTimedProcessor.createProxy(
148+
new DatabricksMetadataQueryClient(databricksClient));
149+
}
144150
} else {
145151
this.databricksClient =
146152
DatabricksMetricsTimedProcessor.createProxy(new DatabricksSdkClient(connectionContext));
147153
this.databricksMetadataClient =
148154
DatabricksMetricsTimedProcessor.createProxy(
149-
new DatabricksMetadataSdkClient(databricksClient));
155+
new DatabricksMetadataQueryClient(databricksClient));
150156
}
151157
}
152158

@@ -161,6 +167,13 @@ public void open() throws SQLException {
161167
this.databricksClient =
162168
DatabricksMetricsTimedProcessor.createProxy(
163169
new DatabricksThriftServiceClient(connectionContext));
170+
if (connectionContext.useQueryForMetadata()) {
171+
this.databricksMetadataClient =
172+
DatabricksMetricsTimedProcessor.createProxy(
173+
new DatabricksMetadataQueryClient(databricksClient));
174+
} else {
175+
this.databricksMetadataClient = null;
176+
}
164177
this.sessionInfo =
165178
this.databricksClient.createSession(
166179
this.computeResource, this.catalog, this.schema, this.sessionConfigs);
@@ -180,7 +193,13 @@ public void open() throws SQLException {
180193
this.databricksClient =
181194
DatabricksMetricsTimedProcessor.createProxy(
182195
new DatabricksThriftServiceClient(connectionContext));
183-
this.databricksMetadataClient = null;
196+
if (connectionContext.useQueryForMetadata()) {
197+
this.databricksMetadataClient =
198+
DatabricksMetricsTimedProcessor.createProxy(
199+
new DatabricksMetadataQueryClient(databricksClient));
200+
} else {
201+
this.databricksMetadataClient = null;
202+
}
184203
try {
185204
this.sessionInfo =
186205
this.databricksClient.createSession(
@@ -239,7 +258,8 @@ public IDatabricksClient getDatabricksClient() {
239258
@Override
240259
public IDatabricksMetadataClient getDatabricksMetadataClient() {
241260
LOGGER.debug("public IDatabricksClient getDatabricksMetadataClient()");
242-
if (this.connectionContext.getClientType() == DatabricksClientType.THRIFT) {
261+
if (this.connectionContext.getClientType() == DatabricksClientType.THRIFT
262+
&& !this.connectionContext.useQueryForMetadata()) {
243263
return (IDatabricksMetadataClient) databricksClient;
244264
}
245265
return databricksMetadataClient;

‎src/main/java/com/databricks/jdbc/api/internal/IDatabricksConnectionContext.java‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -405,6 +405,8 @@ public interface IDatabricksConnectionContext {
405405

406406
boolean enableShowCommandsForGetFunctions();
407407

408+
boolean useQueryForMetadata();
409+
408410
/** Returns whether batched INSERT optimization is enabled */
409411
boolean isBatchedInsertsEnabled();
410412

‎src/main/java/com/databricks/jdbc/common/DatabricksJdbcUrlParams.java‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -168,6 +168,10 @@ public enum DatabricksJdbcUrlParams {
168168
"CloudFetchSpeedThreshold", "Minimum expected download speed in MB/s", "0.1"),
169169
ENABLE_SHOW_COMMAND_FOR_GET_FUNCTIONS(
170170
"EnableShowCommandForGetFunctions", "Use SQL command to fetch function list", "0"),
171+
USE_QUERY_FOR_METADATA(
172+
"UseQueryForMetadata",
173+
"Use SQL SHOW commands instead of Thrift RPCs for metadata operations. When enabled, EnableShowCommandForGetFunctions is redundant",
174+
"0"),
171175
ENABLE_BATCHED_INSERTS("EnableBatchedInserts", "Enable batched INSERT optimization", "0"),
172176
ENABLE_SQL_VALIDATION_FOR_IS_VALID(
173177
"EnableSQLValidationForIsValid",

src/main/java/com/databricks/jdbc/dbclient/impl/sqlexec/DatabricksMetadataSdkClient.java renamed to src/main/java/com/databricks/jdbc/dbclient/impl/sqlexec/DatabricksMetadataQueryClient.java

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -25,20 +25,21 @@
2525
import java.util.concurrent.Executors;
2626

2727
/** Implementation for {@link IDatabricksMetadataClient} using {@link IDatabricksClient}. */
28-
public class DatabricksMetadataSdkClient implements IDatabricksMetadataClient {
28+
public class DatabricksMetadataQueryClient implements IDatabricksMetadataClient {
2929

3030
private static final JdbcLogger LOGGER =
31-
JdbcLoggerFactory.getLogger(DatabricksMetadataSdkClient.class);
31+
JdbcLoggerFactory.getLogger(DatabricksMetadataQueryClient.class);
3232
private static final int DEFAULT_MAX_THREADS_METADATA_FETCH = 10;
3333
private static final int TASK_TIMEOUT_METADATA_FETCH_SEC = 90;
3434
private static final Object THREAD_POOL_LOCK = new Object();
3535
private static ExecutorService metadataThreadPool = null;
36-
private final IDatabricksClient sdkClient;
36+
private final IDatabricksClient queryExecutionClient;
3737
private final MetadataResultSetBuilder metadataResultSetBuilder;
3838

39-
public DatabricksMetadataSdkClient(IDatabricksClient sdkClient) {
40-
this.sdkClient = sdkClient;
41-
this.metadataResultSetBuilder = new MetadataResultSetBuilder(sdkClient.getConnectionContext());
39+
public DatabricksMetadataQueryClient(IDatabricksClient queryExecutionClient) {
40+
this.queryExecutionClient = queryExecutionClient;
41+
this.metadataResultSetBuilder =
42+
new MetadataResultSetBuilder(queryExecutionClient.getConnectionContext());
4243
}
4344

4445
@Override
@@ -381,8 +382,8 @@ public DatabricksResultSet listCrossReferences(
381382
}
382383

383384
private boolean isMultipleCatalogSupportDisabled() {
384-
return sdkClient.getConnectionContext() != null
385-
&& !sdkClient.getConnectionContext().getEnableMultipleCatalogSupport();
385+
return queryExecutionClient.getConnectionContext() != null
386+
&& !queryExecutionClient.getConnectionContext().getEnableMultipleCatalogSupport();
386387
}
387388

388389
/**
@@ -406,7 +407,7 @@ private String autoFillCatalog(String catalog, String currentCatalog) {
406407
private DatabricksResultSet getResultSet(
407408
String SQL, IDatabricksSession session, MetadataOperationType metadataOperationType)
408409
throws SQLException {
409-
return sdkClient.executeStatement(
410+
return queryExecutionClient.executeStatement(
410411
SQL,
411412
session.getComputeResource(),
412413
new HashMap<>(),

‎src/test/java/com/databricks/jdbc/api/impl/DatabricksSessionTest.java‎

Lines changed: 78 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313
import com.databricks.jdbc.api.internal.IDatabricksConnectionContext;
1414
import com.databricks.jdbc.common.DatabricksClientType;
1515
import com.databricks.jdbc.common.DatabricksJdbcUrlParams;
16-
import com.databricks.jdbc.dbclient.impl.sqlexec.DatabricksMetadataSdkClient;
16+
import com.databricks.jdbc.dbclient.impl.sqlexec.DatabricksMetadataQueryClient;
1717
import com.databricks.jdbc.dbclient.impl.sqlexec.DatabricksSdkClient;
1818
import com.databricks.jdbc.dbclient.impl.thrift.DatabricksThriftServiceClient;
1919
import com.databricks.jdbc.exception.DatabricksParsingException;
@@ -101,7 +101,7 @@ public void testOpenRedirectedThriftSession() throws SQLException {
101101

102102
DatabricksSession session = new DatabricksSession(connectionContext, sdkClient);
103103
assertEquals(DatabricksClientType.SEA, connectionContext.getClientType());
104-
assertInstanceOf(DatabricksMetadataSdkClient.class, session.getDatabricksMetadataClient());
104+
assertInstanceOf(DatabricksMetadataQueryClient.class, session.getDatabricksMetadataClient());
105105
assertFalse(session.isOpen());
106106

107107
session.open();
@@ -294,4 +294,80 @@ public void testSessionOpensWithLazyClientTypeForCluster() throws SQLException {
294294
// Client type should be THRIFT for all-purpose cluster
295295
assertEquals(DatabricksClientType.THRIFT, connectionContext.getClientType());
296296
}
297+
298+
@Test
299+
public void testUseQueryForMetadataEnabled() throws SQLException {
300+
setupWarehouseWithQueryMetadata();
301+
DatabricksSession session = new DatabricksSession(connectionContext, thriftClient);
302+
assertEquals(DatabricksClientType.THRIFT, connectionContext.getClientType());
303+
assertTrue(connectionContext.useQueryForMetadata());
304+
assertInstanceOf(
305+
DatabricksMetadataQueryClient.class,
306+
session.getDatabricksMetadataClient(),
307+
"When UseQueryForMetadata=1, metadata client should be DatabricksMetadataQueryClient");
308+
}
309+
310+
static void setupWarehouseWithQueryMetadata() throws SQLException {
311+
String url =
312+
"jdbc:databricks://sample-host.18.azuredatabricks.net:9999/default;transportMode=http;ssl=1;"
313+
+ "AuthMech=3;httpPath=/sql/1.0/warehouses/warehouse_id;UseQueryForMetadata=1";
314+
connectionContext = DatabricksConnectionContext.parse(url, new Properties());
315+
}
316+
317+
@Test
318+
public void testUseQueryForMetadataDisabledByDefault() throws SQLException {
319+
setupWarehouse(true /* useThrift */);
320+
DatabricksSession session = new DatabricksSession(connectionContext, thriftClient);
321+
assertFalse(connectionContext.useQueryForMetadata());
322+
assertInstanceOf(
323+
DatabricksThriftServiceClient.class,
324+
session.getDatabricksMetadataClient(),
325+
"When UseQueryForMetadata is default (0), metadata client should be the Thrift client");
326+
}
327+
328+
@Test
329+
public void testUseQueryForMetadataWithRedirectFallback() throws SQLException {
330+
// SEA connection with UseQueryForMetadata=1
331+
String url =
332+
"jdbc:databricks://sample-host.18.azuredatabricks.net:9999/default;transportMode=http;ssl=1;"
333+
+ "AuthMech=3;httpPath=/sql/1.0/warehouses/warehouse_id;UseThriftClient=0;UseQueryForMetadata=1";
334+
connectionContext = DatabricksConnectionContext.parse(url, new Properties());
335+
336+
ImmutableSessionInfo sessionInfo =
337+
ImmutableSessionInfo.builder()
338+
.sessionId(SESSION_ID)
339+
.computeResource(WAREHOUSE_COMPUTE)
340+
.build();
341+
when(sdkClient.createSession(eq(WAREHOUSE_COMPUTE), any(), any(), any()))
342+
.thenThrow(new DatabricksTemporaryRedirectException(TEMPORARY_REDIRECT_EXCEPTION));
343+
when(thriftClient.createSession(any(), any(), any(), any())).thenReturn(sessionInfo);
344+
try (MockedStatic<DatabricksMetricsTimedProcessor> proxyMock =
345+
Mockito.mockStatic(DatabricksMetricsTimedProcessor.class)) {
346+
proxyMock
347+
.when(() -> DatabricksMetricsTimedProcessor.createProxy(any()))
348+
.thenAnswer(
349+
invocation -> {
350+
Object arg = invocation.getArgument(0);
351+
if (arg instanceof DatabricksMetadataQueryClient) {
352+
return arg;
353+
}
354+
return thriftClient;
355+
});
356+
357+
DatabricksSession session = new DatabricksSession(connectionContext, sdkClient);
358+
assertEquals(DatabricksClientType.SEA, connectionContext.getClientType());
359+
// Before redirect: SEA path sets DatabricksMetadataQueryClient via test constructor
360+
assertInstanceOf(DatabricksMetadataQueryClient.class, session.getDatabricksMetadataClient());
361+
362+
session.open();
363+
364+
assertTrue(session.isOpen());
365+
assertEquals(DatabricksClientType.THRIFT, connectionContext.getClientType());
366+
// After redirect: UseQueryForMetadata=1 should preserve DatabricksMetadataQueryClient
367+
assertInstanceOf(
368+
DatabricksMetadataQueryClient.class,
369+
session.getDatabricksMetadataClient(),
370+
"After SEA→Thrift redirect with UseQueryForMetadata=1, metadata client should be DatabricksMetadataQueryClient");
371+
}
372+
}
297373
}

0 commit comments

Comments
 (0)