Commit 9770ecf
authored
[refactor](jdbc) Unify JDBC scanning into FileQueryScanNode/JniReader framework (#61141)
### What problem does this PR solve?
Refactor the JDBC data source scanning architecture to integrate with
the
unified FileScanner/JniReader framework, replacing the standalone
ExternalScanNode-based JDBC scan path.
#### Motivation
The JDBC scan path was independently implemented with its own operator
(JDBCScanLocalState), scanner (JdbcScanner), and JNI connector
(JniConnector
→ BaseJdbcExecutor hierarchy), while other JNI-based data sources
(Paimon,
Hudi, MaxCompute, TrinoConnector) already use the unified FileScanner →
JniReader → JniScanner path. This caused code duplication, maintenance
burden,
and architectural inconsistency.
#### Changes
1. BE: Split JniConnector into two focused classes:
- JniReader: JNI lifecycle management (open/read/close), base class for
all JNI readers (PaimonJniReader, HudiJniReader, JdbcJniReader, etc.)
- JniDataBridge: Stateless data exchange between C++ Blocks and Java
shared memory via JNI
2. Java: Introduce Strategy Pattern for database-specific type handling:
- JdbcTypeHandler interface with DefaultTypeHandler base implementation
- Per-database handlers: MySQLTypeHandler, OracleTypeHandler,
PostgreSQLTypeHandler, ClickHouseTypeHandler, SQLServerTypeHandler,
DB2TypeHandler, SapHanaTypeHandler, TrinoTypeHandler, GbaseTypeHandler
- JdbcTypeHandlerFactory for handler selection
- JdbcJniScanner (extends JniScanner) for read path
- JdbcJniWriter (extends JniWriter) for write path
- Old BaseJdbcExecutor subclasses marked @deprecated but preserved
3. FE: Migrate JdbcScanNode from ExternalScanNode to FileQueryScanNode:
- JdbcScanNode now extends FileQueryScanNode
- Introduces JdbcSplit (extends FileSplit) to carry JDBC params
- Uses TFileScanRange with FORMAT_JNI and table_format_type="jdbc"
- Adds jdbc_params field to TTableFormatFileDesc in Thrift
- Adapts PhysicalPlanTranslator.visitPhysicalJdbcScan() accordingly
4. BE: Add JdbcJniReader and integrate into FileScanner:
- JdbcJniReader handles special types (bitmap, HLL, quantile_state,
JSONB)
via string-based intermediary and CAST
- FileScanner._get_next_reader() adds "jdbc" table_format_type branch
- JdbcUtils utility for JDBC driver URL resolution
- Existing JdbcScanner preserved as transitional (deprecated)
All predicate push-down logic (createJdbcFilters, getJdbcQueryStr, etc.)
is preserved from the original JdbcScanNode implementation.1 parent 2205eff commit 9770ecf
File tree
81 files changed
+5426
-2817
lines changed- be
- src
- core/data_type_serde
- exec
- connector
- scan
- sink/writer
- exprs
- aggregate
- function
- cast
- table_function
- format
- avro
- jni
- table
- transformer
- service
- util
- test/exec/connector
- fe
- be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc
- fe-core/src/main/java/org/apache/doris
- datasource/jdbc
- client
- source
- nereids/glue/translator
- gensrc/thrift
- regression-test
- data/external_table_p0/jdbc
- suites/external_table_p0/jdbc
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
81 files changed
+5426
-2817
lines changedLines changed: 6 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
67 | 67 | | |
68 | 68 | | |
69 | 69 | | |
70 | | - | |
71 | | - | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
72 | 74 | | |
73 | 75 | | |
74 | 76 | | |
75 | 77 | | |
76 | 78 | | |
77 | | - | |
78 | | - | |
| 79 | + | |
| 80 | + | |
79 | 81 | | |
80 | 82 | | |
81 | 83 | | |
| |||
0 commit comments