Skip to content

Commit 9f85a5e

Browse files
ThorneANNThorneANN
authored
Merge branch 'apache:master' into FLINK-38911-binlog-added-new-table 
2 parents 4e83d27 + 283edb6 commit 9f85a5e

356 files changed

Lines changed: 11898 additions & 2562 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/flink_cdc_base.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -108,7 +108,7 @@ jobs:
108108
109109
build_maven_parameter="${build_maven_parameter:+$build_maven_parameter }${{ inputs.custom-maven-parameter }}"
110110
111-
mvn --no-snapshot-updates -B -DskipTests -pl $compile_modules -am install && mvn --no-snapshot-updates -B $build_maven_parameter -pl $modules -DspecifiedParallelism=${{ inputs.parallelism }} -Duser.timezone=$jvm_timezone verify
111+
mvn --no-snapshot-updates -B -DskipTests ${{ inputs.custom-maven-parameter }} -pl $compile_modules -am install && mvn --no-snapshot-updates -B $build_maven_parameter -pl $modules -DspecifiedParallelism=${{ inputs.parallelism }} -Duser.timezone=$jvm_timezone verify
112112
113113
- name: Print JVM thread dumps when cancelled
114114
if: ${{ failure() }}

‎.github/workflows/flink_cdc_ci.yml‎

Lines changed: 40 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -63,18 +63,42 @@ jobs:
6363
with:
6464
java-versions: "[11]"
6565
modules: "['core']"
66+
common_2_x:
67+
name: Common Unit Tests 2.x
68+
uses: ./.github/workflows/flink_cdc_base.yml
69+
with:
70+
java-versions: "[11]"
71+
flink-versions: "['2.2.0']"
72+
custom-maven-parameter: "-Pflink2"
73+
modules: "['core']"
6674
pipeline-ut:
6775
name: Pipeline Unit Tests
6876
uses: ./.github/workflows/flink_cdc_base.yml
6977
with:
7078
java-versions: "[11]"
7179
modules: "['doris, elasticsearch, fluss, hudi, iceberg, kafka, maxcompute, mysql-pipeline, oceanbase-pipeline, paimon, postgres-pipeline, starrocks, oracle-pipeline']"
80+
pipeline-connectors-ut-2-x:
81+
name: Pipeline Connectors Unit Tests 2.x
82+
uses: ./.github/workflows/flink_cdc_base.yml
83+
with:
84+
java-versions: "[11]"
85+
flink-versions: "['2.2.0']"
86+
custom-maven-parameter: "-Pflink2"
87+
modules: "['doris, elasticsearch, fluss, iceberg, kafka, maxcompute, mysql-pipeline, oceanbase-pipeline, paimon, postgres-pipeline, starrocks, oracle-pipeline']"
7288
source-ut:
7389
name: Source Unit Tests
7490
uses: ./.github/workflows/flink_cdc_base.yml
7591
with:
7692
java-versions: "[11]"
7793
modules: "['mysql-source', 'postgres-source, oceanbase-source, tidb, vitess', 'oracle, sqlserver', 'db2, mongodb']"
94+
source-ut-2-x:
95+
name: Source Unit Tests 2.x
96+
uses: ./.github/workflows/flink_cdc_base.yml
97+
with:
98+
java-versions: "[11]"
99+
flink-versions: "['2.2.0']"
100+
custom-maven-parameter: "-Pflink2"
101+
modules: "['mysql-source', 'postgres-source, oceanbase-source, tidb, vitess', 'oracle, sqlserver', 'db2, mongodb']"
78102
pipeline_e2e:
79103
strategy:
80104
max-parallel: 2
@@ -85,13 +109,27 @@ jobs:
85109
uses: ./.github/workflows/flink_cdc_base.yml
86110
with:
87111
java-versions: "[11]"
88-
flink-versions: "['1.19.3', '1.20.3']"
112+
flink-versions: "['1.20.3']"
113+
modules: "['pipeline_e2e']"
114+
parallelism: ${{ matrix.parallelism }}
115+
pipeline_e2e_2_x:
116+
strategy:
117+
max-parallel: 2
118+
fail-fast: false
119+
matrix:
120+
parallelism: [ 1, 4 ]
121+
name: Pipeline E2E Tests 2.x (${{ matrix.parallelism }}-Parallelism)
122+
uses: ./.github/workflows/flink_cdc_base.yml
123+
with:
124+
java-versions: "[11]"
125+
flink-versions: "['2.2.0']"
126+
custom-maven-parameter: "-Pflink2"
89127
modules: "['pipeline_e2e']"
90128
parallelism: ${{ matrix.parallelism }}
91129
source_e2e:
92130
name: Source E2E Tests
93131
uses: ./.github/workflows/flink_cdc_base.yml
94132
with:
95133
java-versions: "[11]"
96-
flink-versions: "['1.19.3', '1.20.3']"
134+
flink-versions: "['1.20.3']"
97135
modules: "['source_e2e']"

‎.github/workflows/flink_cdc_ci_nightly.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,15 @@ jobs:
5454
with:
5555
java-versions: "[17]"
5656
modules: "['core']"
57+
common_2_x:
58+
if: github.repository == 'apache/flink-cdc'
59+
name: Common Unit Tests 2.x
60+
uses: ./.github/workflows/flink_cdc_base.yml
61+
with:
62+
java-versions: "[17]"
63+
flink-versions: "['2.2.0']"
64+
custom-maven-parameter: "-Pflink2"
65+
modules: "['core_2.x']"
5766
pipeline-ut:
5867
if: github.repository == 'apache/flink-cdc'
5968
name: Pipeline Unit Tests
@@ -82,6 +91,21 @@ jobs:
8291
flink-versions: "['1.19.3', '1.20.3']"
8392
modules: "['pipeline_e2e']"
8493
parallelism: ${{ matrix.parallelism }}
94+
pipeline_e2e_2_x:
95+
if: github.repository == 'apache/flink-cdc'
96+
strategy:
97+
max-parallel: 2
98+
fail-fast: false
99+
matrix:
100+
parallelism: [ 1, 4 ]
101+
name: Pipeline E2E Tests 2.x (${{ matrix.parallelism }}-Parallelism)
102+
uses: ./.github/workflows/flink_cdc_base.yml
103+
with:
104+
java-versions: "[17]"
105+
custom-maven-parameter: "-Pflink2"
106+
flink-versions: "['2.2.0']"
107+
modules: "['pipeline_e2e_2.x']"
108+
parallelism: ${{ matrix.parallelism }}
85109
source_e2e:
86110
if: github.repository == 'apache/flink-cdc'
87111
name: Source E2E Tests

‎.github/workflows/modules.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -173,7 +173,7 @@
173173
compile_modules = set()
174174

175175
for module in INPUT_MODULES.split(', '):
176-
module_list = set(globals()['MODULES_' + module.upper().replace('-', '_')])
176+
module_list = set(globals()['MODULES_' + module.upper().replace('-', '_').replace('.', '_')])
177177
test_modules |= module_list
178178
if module == 'source_e2e' or module == 'pipeline_e2e':
179179
compile_modules |= ALL_MODULES

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,7 @@ SELECT * FROM test_table;
100100
pipeline:
101101
name: Sync MySQL Database to Doris
102102
parallelism: 2
103+
route-mode: ALL_MATCH
103104
user-defined-function:
104105
- name: addone
105106
classpath: com.example.functions.AddOneFunctionClass

‎docs/content.zh/docs/connectors/flink-sources/mysql-cdc.md‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -30,9 +30,9 @@ MySQL CDC 连接器允许从 MySQL 数据库读取快照数据和增量数据。
3030

3131
## 支持的数据库
3232

33-
| Connector | Database | Driver |
34-
|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------|
35-
| mysql-cdc | <li> [MySQL](https://dev.mysql.com/doc): 5.6, 5.7, 8.0.x <li> [RDS MySQL](https://www.aliyun.com/product/rds/mysql): 5.6, 5.7, 8.0.x <li> [PolarDB MySQL](https://www.aliyun.com/product/polardb): 5.6, 5.7, 8.0.x <li> [Aurora MySQL](https://aws.amazon.com/cn/rds/aurora): 5.6, 5.7, 8.0.x <li> [MariaDB](https://mariadb.org): 10.x <li> [PolarDB X](https://github.com/ApsaraDB/galaxysql): 2.0.1 | JDBC Driver: 8.0.27 |
33+
| Connector | Database | Driver |
34+
|-----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------|
35+
| mysql-cdc | <li> [MySQL](https://dev.mysql.com/doc): 5.7, 8.0.x, 8.4+ <li> [RDS MySQL](https://www.aliyun.com/product/rds/mysql): 5.6, 5.7, 8.0.x <li> [PolarDB MySQL](https://www.aliyun.com/product/polardb): 5.6, 5.7, 8.0.x <li> [Aurora MySQL](https://aws.amazon.com/cn/rds/aurora): 5.6, 5.7, 8.0.x <li> [MariaDB](https://mariadb.org): 10.x <li> [PolarDB X](https://github.com/ApsaraDB/galaxysql): 2.0.1 | JDBC Driver: 8.0.27 |
3636

3737
依赖
3838
------------
@@ -122,6 +122,16 @@ mysql> FLUSH PRIVILEGES;
122122
- `wait_timeout`: 服务器在关闭非交互连接之前等待活动的秒数。 更多信息请参考 [MySQL documentations](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_wait_timeout).
123123

124124

125+
### MySQL 8.4+ 兼容性
126+
127+
MySQL 8.4 引入了一些影响 CDC 连接器的重大变更：
128+
129+
- `SHOW MASTER STATUS` 已被弃用，替换为 `SHOW BINARY LOG STATUS`
130+
- 错误消息术语从 `slave/master` 变更为 `replica/source`
131+
132+
MySQL CDC 连接器会自动探测 MySQL 服务器版本，并根据服务器版本使用相应的 SQL 语句。这确保了与 MySQL 8.4+ 及更早版本的兼容性，无需额外配置。
133+
134+
125135
如何创建 MySQL CDC 表
126136
----------------
127137

0 commit comments

Comments
 (0)