[FLINK-37834] Flink JDBC support ClickHouse Dialect by Biasqo · Pull Request #171 · apache/flink-connector-jdbc

Biasqo · 2025-06-02T20:10:42Z

🔌 ClickHouse Dialect for Flink Link JDBC Connector

This project provides a ClickHouse SQL dialect implementation for the Apache Flink JDBC Connector.

🚀 Purpose
Apache Flink’s Link JDBC connector offers a flexible way to integrate with relational databases through dialects. However, native support for ClickHouse’s SQL features (e.g. merge trees, upserts, batch writes) is limited in SQL CLI.

✨ Features
✅ Custom ClickHouseDialect integration
✅ Column quoting and identifier case sensitivity fixes
✅ Insert/update query support tailored for ClickHouse (pkFields removed from update statement)

TODO:
Refactor Tests: Some methods are private (like getStreamFields() that's why I had to recreate some test classes)

Jira Issue:
https://issues.apache.org/jira/projects/FLINK/issues/FLINK-37834

Biasqo · 2025-06-07T12:54:18Z

@eskabetxe could you please start CI or review PR?

eskabetxe

Thanks for your contribution, I can't start CI/CD sorry. @1996fanrui can you start it?

I review the main code and left minor comments.
I dont review the tests as you have a TODO to some refactoring there.

You also need to add this module on flink-connector-jdbc-architecture.

eskabetxe · 2025-06-11T20:28:04Z

@@ -1,24 +1,24 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <project version="4">


the changes in this file should be reverted

eskabetxe · 2025-06-11T20:29:13Z

+    <packaging>jar</packaging>
+
+    <properties>
+        <clickhouse.version>0.6.2</clickhouse.version>


Why not use the last version available 0.8.6?

Biasqo · 2025-06-12T05:10:17Z

@eskabetxe
Thanks for getting back to me

I’m using version 0.6.2 since 0.8+ has some dependency issues with httpcore, and I didn’t want to change the existing httpcore version. If it’s okay to upgrade some of the libraries already included in the distribution, I can go ahead and fix it.
I will revert .vcs
About the tests: I’m not sure if I can override some private methods that are part of abstract test classes (for example, the abstract class TableBase, method getStreamFields). If I’m allowed to make them protected, the number of test classes could be significantly reduced. Clickhouse has its unique create statement - that's why it was a problem

Biasqo · 2025-06-14T12:26:32Z

Thanks for your contribution, I can't start CI/CD sorry. @1996fanrui can you start it?

I review the main code and left minor comments.

I dont review the tests as you have a TODO to some refactoring there.

You also need to add this module on flink-connector-jdbc-architecture.

Is it okay if I refactor some of the core classes in the tests, like making them protected?

eskabetxe · 2025-06-19T09:50:31Z

@eskabetxe Thanks for getting back to me

I’m using version 0.6.2 since 0.8+ has some dependency issues with httpcore, and I didn’t want to change the existing httpcore version. If it’s okay to upgrade some of the libraries already included in the distribution, I can go ahead and fix it.

I will revert .vcs

About the tests: I’m not sure if I can override some private methods that are part of abstract test classes (for example, the abstract class TableBase, method getStreamFields). If I’m allowed to make them protected, the number of test classes could be significantly reduced. Clickhouse has its unique create statement - that's why it was a problem

Its ok to bump some versions and is ok also change some test methods from private to protected if you need to override them..

… archunit, 3. Driver version for tests = 0.6.2 (ClickHouse testcontainers support that version) 4. Refactored UPDATE statement

Biasqo · 2025-06-20T20:38:17Z

Ive refactored tests but cannot use click house driver version 0.7+ since click house testcontainers support 0.6.2.
Clickhouse driver dependency is only needed for tests

Biasqo · 2025-09-02T09:57:38Z

@eskabetxe
Whenever you get a chance, could you please take a look again?

Biasqo · 2025-11-29T19:12:53Z

@davidradl maybe you can review?

gaaraG · 2025-12-21T08:28:37Z

Hi @Biasqo,

Thank you for working on the ClickHouse dialect support!

I noticed that this PR hasn't been updated for a while. Are you still planning to continue working on this? If you are busy or no longer interested, please let us know so that others in the community might pick it up.

If you intend to drop this, I would be happy to take over and address the review comments.

Looking forward to your reply.

Biasqo · 2025-12-21T10:22:03Z

Hi @gaaraG!

Ive passed the tests and got everything working, the only issue is driver version.
I stuck to the 0.6.2 version which is suitable for most cases

If jdbc version is a major issue i am ready to update it
If there are any major problems with connector, I'll fix it

Overall
I am ready to continue working on it

gaaraG · 2025-12-22T00:23:50Z

That sounds great! Thanks for getting back to us.

Looking forward to seeing your updates!

Biasqo · 2025-12-22T11:00:42Z

I'll check the driver version and get back to you
If everything will be ok we can merge

1. refactored some tests and CI fixes

Biasqo · 2025-12-31T18:30:31Z

Hi @gaaraG!

This branch is ready to merge
If there are any comments on the code please let me know

Changelog:

Fixed CI and changed some tests
Added timestampFilterExpression method to be compatible with ClickHouse

Lineage:

ClickHouse is not included among the built‑in extractors

Driver:

Version 0.9.5 is dependent on commons io version 2.16, so I stuck to the working 0.6.2 classic clickhouse jdbc

github-actions · 2026-04-01T06:52:10Z

This PR is being marked as stale since it has not had any activity in the last 90 days.
If you would like to keep this PR alive, please leave a comment asking for a review.
If the PR has merge conflicts, update it with the latest from the base branch.

If you are having difficulty finding a reviewer, please reach out to the
community, contact details can be found here: https://flink.apache.org/what-is-flink/community/

If this PR is no longer valid or desired, please feel free to close it.
If no activity occurs in the next 30 days, it will be automatically closed.

och5351 · 2026-04-23T01:23:39Z

Hi, @Biasqo !
Are you still looking at this PR?

Biasqo · 2026-04-23T04:57:57Z

Hi, @och5351
The code was ready for its time.
I've waited for so long and got no answer.
I don't know if anyone actually needs a clickhouse connector, so it's up to you.

och5351 · 2026-04-23T06:08:26Z

@Biasqo
I think the ClickHouse connector will be really useful.
I will review and support this PR as soon as I have time (as soon as possible).
Of course, as long as you're okay with it.

Thank you for your contribution.

Biasqo · 2026-04-23T06:11:31Z

@och5351
Alright,
If there are any issues with the pr leave a comment and I'll do my best to fix them.

Biasqo · 2026-04-24T17:59:45Z

@och5351
I've made some changes you mentioned above

I've updated clickhouse server to 25.8
I've updated clickhouse driver to 0.9.7
I've added lineage to clickhouse (I think its better to add that clickhouse support into openlineage)
I've also changed deprecated class com.clickhouse.jdbc.ClickHouseDriver to com.clickhouse.jdbc.Driver class

I've passed the tests in my forked repo, so its ready to merge after your review

och5351

Thanks @Biasqo!
I've submitted an OpenLineage PR for this (OpenLineage/OpenLineage#4495), but it will take time to merge.

@RocMarshal @eskabetxe Should we use this custom extractor for now and migrate later, or would you prefer to wait for the OpenLineage release?

och5351 · 2026-04-26T11:44:35Z

+public class ClickHouseFactory implements JdbcFactory {
+    @Override
+    public boolean acceptsURL(String url) {
+        return url.startsWith("jdbc:clickhouse:");


Should we support the jdbc:ch: alias as well?

Suggested change

return url.startsWith("jdbc:clickhouse:");

return url.startsWith("jdbc:clickhouse:") || url.startsWith("jdbc:ch:");

That's a good idea
I will add it

…tory and JdbcExtractor, 2. Refactored ClickHouseJdbcExtractor (#4)

Biasqo · 2026-04-26T17:58:54Z

@och5351
Thank you for your review
Added ch prefix + suggested your changes in latest commit

Upd:
Should i also add some documentation on clickhouse connector?
I've seen some documentation in docs folder

eskabetxe · 2026-05-05T07:39:27Z

+
+    @Override
+    public String timestampFilterExpression() {
+        return "timestamp6_col > TO_TIMESTAMP_LTZ('2020-01-01 15:35:00') "


Why we change this to a zoned timestamp??

That's an old typo
I will refresh the tests, there are not enough tests for clickhouse

davidradl

sorry updated by mistake

och5351 · 2026-05-08T01:49:05Z

Hi @RocMarshal, @davidradl, @eskabetxe, @Biasqo!

Could you please share your thoughts on this plan?

Flink Clickhouse JDBC Connector needs OpenLineage support.

Wait for the PR in OpenLineage to be merged and update the version in this repo. (flink: add ClickHouse JDBC support OpenLineage/OpenLineage#4495)
First merge the custom lineage implementation, then update the version and apply it.

That's the plan I came up with. If you have any better ideas or advice,
I'd really appreciate it if you could share them!

1. Refactored ClickHouseDynamicTableSourceITCase (adding more columns to source table) 2. Added MAP type support to JdbcTypeUtil (since MAP type is actively used in ClickHouse DB) 3. Refactored ClickHouseDialectConverter: added MAP and ARRAY support to internal converter and did some minor fixes

Biasqo · 2026-05-08T01:53:35Z

@eskabetxe @och5351
I refactored ClickHouse tests and they look cleaner now.
Also fixed minor bugs in ClickHouse converter.

Biasqo · 2026-05-08T01:56:34Z

Hi @RocMarshal, @davidradl, @eskabetxe, @Biasqo!

Could you please share your thoughts on this plan?

Flink Clickhouse JDBC Connector needs OpenLineage support.

Wait for the PR in OpenLineage to be merged and update the version in this repo. (flink: add ClickHouse JDBC support OpenLineage/OpenLineage#4495)

First merge the custom lineage implementation, then update the version and apply it.

That's the plan I came up with. If you have any better ideas or advice, I'd really appreciate it if you could share them!

I think we should wait for the PR in OpenLineage, it will keep the codebase clean

* Adding openlineage for ClickHouse * Minor compatibility tweaks

Biasqo · 2026-05-25T15:39:27Z

Hi @och5351, @davidradl, @eskabetxe!
Ive added OpenLineage native support for ClickHouse DB and made some minor fixes.
Looking forward to seeing your comments!

Biasqo · 2026-06-18T19:35:55Z

Hi @och5351, @davidradl, @eskabetxe!
Could you please rerun checks?
The error seems to be in core module, I've passed all checks in my forked main branch

Biasqo · 2026-06-19T13:57:31Z

Hi @och5351, @davidradl, @eskabetxe!

Could you please rerun checks?

The error seems to be in core module, I've passed all checks in my forked main branch

CI seems to be unstable once again
Now the error is in Db2 module

och5351 · 2026-06-20T09:48:54Z

By the way, just to confirm — was this Jira ticket
(FLINK-37834) created by @Biasqo, and is the account name Arsene?

https://issues.apache.org/jira/browse/FLINK-37834

If that's correct, would it be okay to ask @RocMarshal
to assign it?

Biasqo · 2026-06-20T12:03:53Z

By the way, just to confirm — was this Jira ticket (FLINK-37834) created by @Biasqo, and is the account name Arsene?

https://issues.apache.org/jira/browse/FLINK-37834

If that's correct, would it be okay to ask @RocMarshal to assign it?

Yes thats correct, I created FLINK-37834, Arsene is my name)))
Thx for checking

och5351 · 2026-06-20T10:13:53Z

+                        "BINARY/VARBINARY types are not supported by ClickHouse dialect. "
+                                + "Use STRING instead.");
+            case CHAR:
+            case VARCHAR:


This is already handled identically in AbstractDialectConverter. Consider removing this case and delegating via a default fallthrough to avoid duplication.

och5351 · 2026-06-20T10:15:19Z

+        switch (type.getTypeRoot()) {
+            case NULL:
+                return null;
+            case BOOLEAN:


This is already handled identically in AbstractDialectConverter. Consider removing this case and delegating via a default fallthrough to avoid duplication.

och5351 · 2026-06-20T10:15:58Z

+    @Override
+    public JdbcDeserializationConverter createInternalConverter(LogicalType type) {
+        switch (type.getTypeRoot()) {
+            case NULL:


This is already handled identically in AbstractDialectConverter. Consider removing this case and delegating via a default fallthrough to avoid duplication.

och5351 · 2026-06-20T10:16:43Z

+                                : val instanceof Timestamp
+                                        ? TimestampData.fromTimestamp((Timestamp) val)
+                                        : val;
+            case DECIMAL:


This is already handled identically in AbstractDialectConverter. Consider removing this case and delegating via a default fallthrough to avoid duplication.

och5351 · 2026-06-20T12:23:14Z

+    }
+
+    // adding support to MAP and ARRAY types
+    public static Object toExternalSerializer(Object value, LogicalType type) {


How about changing it to private?

och5351 · 2026-06-20T12:43:27Z

+                                                () ->
+                                                        new RuntimeException(
+                                                                "Unknown array element type"));
+                final JdbcDeserializationConverter elementConverter =


Consider using createNullableInternalConverter instead of createInternalConverter for the element converter. PostgresDialectConverter follows this pattern, which delegates null handling to the framework and removes the need for the manual null check on the next line.

och5351 · 2026-06-20T12:46:03Z

+                    }
+                    return new GenericArrayData(converted);
+                };
+            case MAP:


It would be cleaner to extract the ARRAY and MAP logic into separate private methods (e.g., createArrayConverter, createMapConverter), following the pattern in PostgresDialectConverter.createPostgresArrayConverter.

This keeps the switch cases concise and separates traversal logic from null handling.
What do you think about handling it this way?

och5351 · 2026-06-20T13:07:47Z

Hi, @Biasqo !
I apologize for the delay.

PTAL !

och5351 · 2026-06-20T13:12:13Z

+            case TIME_WITHOUT_TIME_ZONE:
+                return Time.valueOf(LocalTime.ofNanoOfDay((int) value * 1_000_000L));
+            case TIMESTAMP_WITH_TIME_ZONE:
+            case TIMESTAMP_WITHOUT_TIME_ZONE:


Suggested change

case TIMESTAMP_WITHOUT_TIME_ZONE:

case TIMESTAMP_WITH_TIME_ZONE:

case TIMESTAMP_WITHOUT_TIME_ZONE:

return ((TimestampData) value).toTimestamp();

The value here is TimestampData (Flink's internal type returned by ElementGetter), not LocalDateTime or Timestamp. This falls through to : value, returning TimestampData directly to the ClickHouse JDBC driver, which will cause a failure. Following the same approach as AbstractDialectConverter.createExternalConverter, which calls val.getTimestamp(...).toTimestamp()

Should I do same for DATE type?

No, DATE already returns java.sql.Date correctly.

och5351 · 2026-06-20T13:13:31Z

+                        : value instanceof Timestamp
+                                ? TimestampData.fromTimestamp((Timestamp) value)
+                                : value;
+            case DECIMAL:


Suggested change

case DECIMAL:

case DECIMAL:

return ((DecimalData) value).toBigDecimal();

Refactored code based on requested changes

Biasqo · 2026-06-21T01:13:50Z

Hi @och5351!
Ive refactored converter based on your changes and added MAP and ARRAY into tests (latest commit a1d4bfc)
Let me know if there are any remaining comments.

* Added test for MAP and ARRAY in JdbcSink and JdbcSource

Biasqo · 2026-06-22T10:42:35Z

CI checks are failing, but they're all passing on my branch.

RocMarshal · 2026-06-22T16:16:10Z

Thanks @Biasqo @och5351

Could you @eskabetxe help take a look ? thx

eskabetxe suggested changes Jun 11, 2025

View reviewed changes

Biasqo closed this Jun 20, 2025

Biasqo force-pushed the main branch from ff294d3 to f697986 Compare June 20, 2025 19:54

Biasqo added 2 commits June 20, 2025 23:05

Adding clickhouse dialect 1. Refactored tests. 2. Added clickhouse to…

5d98d62

… archunit, 3. Driver version for tests = 0.6.2 (ClickHouse testcontainers support that version) 4. Refactored UPDATE statement

Mini fix for methods accessibility

e6ac432

Biasqo reopened this Jun 20, 2025

boring-cyborg Bot added the component=JDBC/Core label Jun 20, 2025

Biasqo requested a review from eskabetxe June 20, 2025 20:38

Biasqo and others added 3 commits December 24, 2025 03:12

Merge branch 'apache:main' into main

2f916fa

1. refactored some tests and CI fixes

15bb059

Merge pull request #1 from Biasqo/prerelease

921e18f

1. refactored some tests and CI fixes

github-actions Bot added the stale label Apr 1, 2026

github-actions Bot removed the stale label Apr 23, 2026

och5351 mentioned this pull request Apr 26, 2026

[FEATURE] [FLINK] Add ClickHouse JDBC support for lineage tracking OpenLineage/OpenLineage#4494

Closed

1 task

och5351 suggested changes Apr 26, 2026

View reviewed changes

1. Minor fixes requested by och5351: added ch prefix to ClickHouseFac…

d95b299

…tory and JdbcExtractor, 2. Refactored ClickHouseJdbcExtractor (#4)

och5351 mentioned this pull request May 3, 2026

flink: add ClickHouse JDBC support OpenLineage/OpenLineage#4495

Merged

eskabetxe reviewed May 5, 2026

View reviewed changes

davidradl reviewed May 5, 2026

View reviewed changes

Biasqo added 2 commits May 24, 2026 19:10

Merge branch 'apache:main' into main

75f6bbb

Updating Openlineage version (1.47.1) to support ClickHouse DB

d43f80a

* Adding openlineage for ClickHouse * Minor compatibility tweaks

RocMarshal self-assigned this Jun 20, 2026

och5351 suggested changes Jun 20, 2026

View reviewed changes

Biasqo added 2 commits June 20, 2026 20:11

Merge branch 'apache:main' into main

aed7d79

Hotfix/FLINK-37834

84d17c4

Refactored code based on requested changes

Biasqo and others added 2 commits June 21, 2026 20:05

Minor fix for MAP type

91f2ec1

ClickHouse sink and source tests: Adding MAP and ARRAY data

a1d4bfc

* Added test for MAP and ARRAY in JdbcSink and JdbcSource

		@@ -1,24 +1,24 @@
		<?xml version="1.0" encoding="UTF-8"?>
		<project version="4">

	return url.startsWith("jdbc:clickhouse:");
	return url.startsWith("jdbc:clickhouse:") \|\| url.startsWith("jdbc:ch:");

	case DECIMAL:
	case DECIMAL:
	return ((DecimalData) value).toBigDecimal();

Uh oh!

Conversation

Biasqo commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Biasqo commented Jun 7, 2025

Uh oh!

eskabetxe left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Biasqo commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Biasqo commented Jun 14, 2025

Uh oh!

eskabetxe commented Jun 19, 2025

Uh oh!

Biasqo commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Biasqo commented Sep 2, 2025

Uh oh!

Biasqo commented Nov 29, 2025

Uh oh!

gaaraG commented Dec 21, 2025

Uh oh!

Biasqo commented Dec 21, 2025

Uh oh!

gaaraG commented Dec 22, 2025

Uh oh!

Biasqo commented Dec 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Biasqo commented Dec 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Apr 1, 2026

Uh oh!

och5351 commented Apr 23, 2026

Uh oh!

Biasqo commented Apr 23, 2026

Uh oh!

och5351 commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Biasqo commented Apr 23, 2026

Uh oh!

Biasqo commented Apr 24, 2026

Uh oh!

och5351 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Biasqo commented Apr 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidradl left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

och5351 commented May 8, 2026

Uh oh!

Biasqo commented May 8, 2026

Uh oh!

Biasqo commented May 8, 2026

Uh oh!

Biasqo commented May 25, 2026

Biasqo commented Jun 2, 2025 •

edited

Loading

Biasqo commented Jun 12, 2025 •

edited

Loading

Biasqo commented Jun 20, 2025 •

edited

Loading

Biasqo commented Dec 22, 2025 •

edited

Loading

Biasqo commented Dec 31, 2025 •

edited

Loading

och5351 commented Apr 23, 2026 •

edited

Loading

och5351 left a comment •

edited

Loading

Biasqo commented Apr 26, 2026 •

edited

Loading

davidradl left a comment •

edited

Loading

och5351 Jun 20, 2026 •

edited

Loading

Biasqo commented Jun 21, 2026 •

edited

Loading