[FLINK-38733] Add new SplitterEnumerator on JdbcSource by eskabetxe · Pull Request #180 · apache/flink-connector-jdbc

eskabetxe · 2025-11-25T15:35:42Z

This pull request refactors the JDBC source split enumeration logic in the Flink connector, replacing the previous SqlTemplateSplitEnumerator and related classes with a new abstraction called SplitterEnumerator. The builder and enumerator classes are updated to use this new abstraction, simplifying split management and improving extensibility. The changes also update lineage handling and testing methods to use the new splitter abstraction.

Split Enumeration Refactoring

Replaced SqlTemplateSplitEnumerator and JdbcSqlSplitEnumeratorBase with the new SplitterEnumerator abstraction across JdbcSource, JdbcSourceBuilder, and JdbcSourceEnumerator, updating constructors, fields, and method calls accordingly. [1] [2] [3] [4]
Updated the builder to support setting a custom SplitterEnumerator, and added logic to select the appropriate splitter implementation based on provided parameters. [1] [2] [3]
Refactored enumerator startup and split assignment logic to use the new splitter, improving asynchronous split enumeration and handling both bounded and unbounded modes.

Lineage and Testing Updates

Changed lineage vertex extraction to use the new splitter abstraction for query information, removing direct calls to SqlTemplateSplitEnumerator. [1] [2]
Updated visible-for-testing methods to expose the new splitter abstraction instead of the old provider.

Miscellaneous

Updated equality checks and state snapshot logic to use the new splitter abstraction and its serializable state. [1] [2]
Removed obsolete annotations for the old split enumerator methods in the architecture violation files.

eskabetxe · 2025-11-25T17:34:24Z

@ferenc-csaky can you have a look when have free time..

ferenc-csaky

Did not check the implementation details yet, added some interface-related comments.

RocMarshal

Thanks @eskabetxe .

It looks better now.
Left a few of comments, PTAL.

RocMarshal · 2026-01-09T14:33:55Z

Thx for the update. LGTM on the whole, just left a few of minor comments.

eskabetxe · 2026-02-24T19:07:30Z

@RocMarshal I have rebase and tests are passing now.. if you could recheck..

RocMarshal · 2026-02-25T01:26:19Z

Hi, @eskabetxe Thanks for the ping.
As mentioned in https://apache-flink.slack.com/archives/C03GV7L3G2C/p1771922394391139
After the version tags ready, I'll advance the next steps of some pending PRs(included this one) .

RocMarshal

Hi, @eskabetxe Sorry for the late response.

LGTM +1 now,

Hi, @snuyanzin @ferenc-csaky Could you help take a look ? thx

eskabetxe · 2026-04-09T10:07:00Z

@RocMarshal @ferenc-csaky any news on this?

eskabetxe · 2026-05-08T12:02:31Z

Hi @ferenc-csaky can you take a look here?

eskabetxe · 2026-05-09T12:01:27Z

@ruanhang1993 thanks for your review..
fixed all comments, if you can review again..

eskabetxe · 2026-05-13T20:43:44Z

@snuyanzin @ruanhang1993 can you recheck?
I solve all comments

ruanhang1993 · 2026-05-14T06:19:59Z

+        int paramLength = params.length;
+        List<JdbcSourceSplit> splitList = new ArrayList<>(paramLength);
+        if (paramLength == 0) {
+            splitList.add(createSplit(null));


There is a different behavior in the new enumerator.

For the old enumerator SqlTemplateSplitEnumerator:

When the paramLength is 0 in SqlTemplateSplitEnumerator#enumerateSplits, it will return an empty list.

When the params is null in SqlTemplateSplitEnumerator#enumerateSplits, it will return a jdbc split with the null param.

For the new enumerator SqlSplitterEnumerator:

When the paramLength is 0 in SqlSplitterEnumerator#enumerateSplits, it will return a jdbc split with the null param.

When the next split is not available,SlideTimingSplitterEnumerator#getSqlParameters will return an empty params , which will cause that it will return a jdbc split with the null param. The jdbc split with a null param will throw an exception in prepareStatement. So the job may fail for this.

WDYT?

@ruanhang1993 you are right..
fixed and add a test to avoid future regression

if you can check again I appreciate

eskabetxe · 2026-06-04T07:34:00Z

@ruanhang1993 can you recheck please

eskabetxe · 2026-06-09T08:57:11Z

@snuyanzin @ruanhang1993 can you recheck?

RocMarshal · 2026-06-17T10:47:40Z

Hi, @eskabetxe @snuyanzin @ruanhang1993
May I know : Is this feature planned to be completed before v5.0?
Thx
CC @och5351

eskabetxe · 2026-06-17T10:49:01Z

Is completed, need only finish the revision..

snuyanzin · 2026-06-17T10:51:01Z

+1 to merge from my side once ci is green

RocMarshal · 2026-06-18T05:09:11Z

Thanks all involved in the improvement.
The CI was successful.
So merging...

boring-cyborg Bot added the component=JDBC/Core label Nov 25, 2025

eskabetxe force-pushed the source_dinamically branch 6 times, most recently from 35399ed to 59dea79 Compare November 25, 2025 17:12

ferenc-csaky requested changes Nov 27, 2025

View reviewed changes

eskabetxe force-pushed the source_dinamically branch 2 times, most recently from cdbee4f to 6964598 Compare November 28, 2025 11:16

eskabetxe requested a review from ferenc-csaky December 4, 2025 20:33

eskabetxe force-pushed the source_dinamically branch from 6964598 to 63ad89f Compare December 15, 2025 15:55

eskabetxe requested a review from RocMarshal December 15, 2025 15:58

eskabetxe force-pushed the source_dinamically branch 3 times, most recently from 63a2301 to 4b0c038 Compare December 15, 2025 16:33

RocMarshal reviewed Jan 8, 2026

View reviewed changes

Comment thread ...ink/connector/jdbc/core/datastream/source/enumerator/splitter/JdbcSqlSplitterEnumerator.java

Comment thread ...bc-core/src/main/java/org/apache/flink/connector/jdbc/core/datastream/source/JdbcSource.java

RocMarshal reviewed Jan 9, 2026

View reviewed changes

eskabetxe force-pushed the source_dinamically branch 2 times, most recently from 087a11a to 89b104a Compare January 19, 2026 17:41

boring-cyborg Bot added the component=JDBC/OceanBase label Jan 19, 2026

eskabetxe force-pushed the source_dinamically branch 2 times, most recently from 2688172 to 4f5aae9 Compare January 20, 2026 09:12

eskabetxe force-pushed the source_dinamically branch from 4f5aae9 to d5b14dd Compare February 24, 2026 12:55

RocMarshal approved these changes Mar 21, 2026

View reviewed changes

ruanhang1993 reviewed May 9, 2026

View reviewed changes

eskabetxe force-pushed the source_dinamically branch 2 times, most recently from 97ae616 to c4e8376 Compare May 9, 2026 12:00

snuyanzin reviewed May 9, 2026

View reviewed changes

Comment thread ...va/org/apache/flink/connector/jdbc/core/datastream/source/JdbcSourceStreamRelatedITCase.java Outdated

snuyanzin reviewed May 9, 2026

View reviewed changes

Comment thread ...ector/jdbc/core/datastream/source/enumerator/splitter/SlideTimingSplitterEnumeratorTest.java Outdated

ruanhang1993 reviewed May 13, 2026

View reviewed changes

Comment thread ...ink/connector/jdbc/core/datastream/source/enumerator/splitter/JdbcSqlSplitterEnumerator.java Outdated

eskabetxe added 6 commits May 13, 2026 22:43

[FLINK-38733] Add new SplitterEnumerator on JdbcSource

a6f915d

[FLINK-38733] Add new SplitterEnumerator on JdbcSource

5b5f47c

[FLINK-38733] Add new SplitterEnumerator on JdbcSource

c7c296e

[FLINK-38733] Add new SplitterEnumerator on JdbcSource

34602bf

[FLINK-38733] Add new SplitterEnumerator on JdbcSource

f4db074

[FLINK-38733] Add new SplitterEnumerator on JdbcSource

8f4cdeb

eskabetxe force-pushed the source_dinamically branch from a8f2a99 to 8f4cdeb Compare May 13, 2026 20:45

ruanhang1993 reviewed May 14, 2026

View reviewed changes

[FLINK-38733] Add new SplitterEnumerator on JdbcSource

44ca564

eskabetxe requested review from ruanhang1993 and snuyanzin May 26, 2026 21:42

eskabetxe force-pushed the source_dinamically branch from e939e22 to b818753 Compare June 9, 2026 08:30

[FLINK-38733] Add new SplitterEnumerator on JdbcSource

dd28adf

eskabetxe force-pushed the source_dinamically branch from b818753 to dd28adf Compare June 9, 2026 12:21

RocMarshal merged commit af99465 into apache:main Jun 18, 2026
7 of 13 checks passed

Uh oh!

Conversation

eskabetxe commented Nov 25, 2025

Split Enumeration Refactoring

Lineage and Testing Updates

Miscellaneous

Uh oh!

eskabetxe commented Nov 25, 2025

Uh oh!

ferenc-csaky left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RocMarshal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RocMarshal commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eskabetxe commented Feb 24, 2026

Uh oh!

RocMarshal commented Feb 25, 2026

Uh oh!

RocMarshal left a comment

Choose a reason for hiding this comment

Uh oh!

eskabetxe commented Apr 9, 2026

Uh oh!

eskabetxe commented May 8, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eskabetxe commented May 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eskabetxe commented May 13, 2026

Uh oh!

ruanhang1993 May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eskabetxe May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eskabetxe commented Jun 4, 2026

Uh oh!

eskabetxe commented Jun 9, 2026

Uh oh!

RocMarshal commented Jun 17, 2026

Uh oh!

eskabetxe commented Jun 17, 2026

Uh oh!

snuyanzin commented Jun 17, 2026

Uh oh!

RocMarshal commented Jun 18, 2026

Uh oh!

Uh oh!

RocMarshal commented Jan 9, 2026 •

edited

Loading

ruanhang1993 May 14, 2026 •

edited

Loading

eskabetxe May 26, 2026 •

edited

Loading