[#11063] fix(lance): manage storage config in Gravitino#11061
Closed
FANNG1 wants to merge 2 commits into
Closed
Conversation
FANNG1
marked this pull request as draft
May 13, 2026 03:26
FANNG1
changed the title
Contributor
Author
|
Superseded by #11070 on a clean branch containing only the Lance storage-config fix. |
yuqi1129
pushed a commit
that referenced
this pull request
May 22, 2026
### What changes were proposed in this pull request? This PR adds Gravitino Flink connector support for Flink 1.19 and Flink 1.20 on top of the versioned-layout baseline established for Flink 1.18 (#10517). Main changes: - Add `flink-connector/v1.19/flink` + `flink-connector/v1.19/flink-runtime` modules. - Add `flink-connector/v1.20/flink` + `flink-connector/v1.20/flink-runtime` modules. - Add version-specific catalog/factory entry classes (`*Flink119` / `*Flink120`) for Hive, Iceberg, JDBC-MySQL, JDBC-Postgres, and Paimon. - Reuse `DefaultCatalogCompat` for Flink 1.19 and add `CatalogCompatFlink120` that uses `CatalogTable.newBuilder(...)` because `CatalogTable.of(...)` is deprecated in Flink 1.20. - The JDBC catalog for 1.19 / 1.20 uses the relocated `org.apache.flink.connector.jdbc.core.*` factory packages shipped in `flink-connector-jdbc 3.3.0-1.19` / `3.3.0-1.20`. - Add provider-specific integration-test entry classes for each new minor version. - Add Flink 1.20-specific overrides for missing required catalog options where Flink 1.20 reports `IllegalArgumentException` rather than the `ValidationException` thrown by older minor versions. - Extend the Flink CI job to run embedded and deploy mode integration tests across 1.18, 1.19, and 1.20. - Exclude the new Flink modules from unrelated Backend IT and Trino CI jobs. No `flink-common` production changes are required; the 1.18 baseline already exposes the necessary extension hooks (`BaseCatalog#catalogCompat()`, the protected `GravitinoJdbcCatalog(Context, ..., AbstractCatalog)` constructor, etc.). The shared Iceberg REST SQL integration test now lives in the common abstract IT base so the 1.18 / 1.19 / 1.20 entry classes can remain thin stubs. This PR supersedes #11061 (closed) and the now-stale #10501. ### Why are the changes needed? The Flink 1.18 baseline (#10517) was intentionally limited to one minor and deferred 1.19 / 1.20 to follow-up work under parent issue #9710. This PR is that follow-up, so users can target the two currently-maintained Flink minor versions. Fix: #9710 ### Does this PR introduce _any_ user-facing change? Yes. Two new connector artifacts are produced: - `gravitino-flink-connector-runtime-1.19_2.12` - `gravitino-flink-connector-runtime-1.20_2.12` The user-facing SQL syntax is unchanged. ### How was this patch tested? Local validations (all green): - `./gradlew spotlessApply` - `./gradlew :flink-connector:flink-1.19:test -PskipITs` - `./gradlew :flink-connector:flink-1.20:test -PskipITs` - `./gradlew :flink-connector:flink-runtime-1.19:test` - `./gradlew :flink-connector:flink-runtime-1.20:test` - `./gradlew :flink-connector:flink-runtime-1.19:shadowJar :flink-connector:flink-runtime-1.20:shadowJar` - `./gradlew :spotlessKotlinGradleCheck :flink-connector:flink-1.20:test -PskipITs` - `./gradlew :flink-connector:flink-common:compileTestJava :flink-connector:flink-1.19:compileJava :flink-connector:flink-1.19:compileTestJava :flink-connector:flink-1.20:compileTestJava :flink-connector:flink-common:spotlessJavaCheck :flink-connector:flink-1.19:spotlessJavaCheck :flink-connector:flink-1.20:spotlessJavaCheck -PskipITs` The runtime-jar tests verify that the version-specific SPI descriptors (`META-INF/services/org.apache.flink.table.factories.Factory`) are merged into the shaded uber-jar. The full Hive / Iceberg / JDBC / Paimon integration tests for both new minor versions will run in CI via the updated `.github/workflows/flink-integration-test-action.yml`. --------- Co-authored-by: fanng <“fanng@apache.org”>
danhuawang
pushed a commit
to danhuawang/gravitino
that referenced
this pull request
Jun 8, 2026
…pache#11071) ### What changes were proposed in this pull request? This PR adds Gravitino Flink connector support for Flink 1.19 and Flink 1.20 on top of the versioned-layout baseline established for Flink 1.18 (apache#10517). Main changes: - Add `flink-connector/v1.19/flink` + `flink-connector/v1.19/flink-runtime` modules. - Add `flink-connector/v1.20/flink` + `flink-connector/v1.20/flink-runtime` modules. - Add version-specific catalog/factory entry classes (`*Flink119` / `*Flink120`) for Hive, Iceberg, JDBC-MySQL, JDBC-Postgres, and Paimon. - Reuse `DefaultCatalogCompat` for Flink 1.19 and add `CatalogCompatFlink120` that uses `CatalogTable.newBuilder(...)` because `CatalogTable.of(...)` is deprecated in Flink 1.20. - The JDBC catalog for 1.19 / 1.20 uses the relocated `org.apache.flink.connector.jdbc.core.*` factory packages shipped in `flink-connector-jdbc 3.3.0-1.19` / `3.3.0-1.20`. - Add provider-specific integration-test entry classes for each new minor version. - Add Flink 1.20-specific overrides for missing required catalog options where Flink 1.20 reports `IllegalArgumentException` rather than the `ValidationException` thrown by older minor versions. - Extend the Flink CI job to run embedded and deploy mode integration tests across 1.18, 1.19, and 1.20. - Exclude the new Flink modules from unrelated Backend IT and Trino CI jobs. No `flink-common` production changes are required; the 1.18 baseline already exposes the necessary extension hooks (`BaseCatalog#catalogCompat()`, the protected `GravitinoJdbcCatalog(Context, ..., AbstractCatalog)` constructor, etc.). The shared Iceberg REST SQL integration test now lives in the common abstract IT base so the 1.18 / 1.19 / 1.20 entry classes can remain thin stubs. This PR supersedes apache#11061 (closed) and the now-stale apache#10501. ### Why are the changes needed? The Flink 1.18 baseline (apache#10517) was intentionally limited to one minor and deferred 1.19 / 1.20 to follow-up work under parent issue apache#9710. This PR is that follow-up, so users can target the two currently-maintained Flink minor versions. Fix: apache#9710 ### Does this PR introduce _any_ user-facing change? Yes. Two new connector artifacts are produced: - `gravitino-flink-connector-runtime-1.19_2.12` - `gravitino-flink-connector-runtime-1.20_2.12` The user-facing SQL syntax is unchanged. ### How was this patch tested? Local validations (all green): - `./gradlew spotlessApply` - `./gradlew :flink-connector:flink-1.19:test -PskipITs` - `./gradlew :flink-connector:flink-1.20:test -PskipITs` - `./gradlew :flink-connector:flink-runtime-1.19:test` - `./gradlew :flink-connector:flink-runtime-1.20:test` - `./gradlew :flink-connector:flink-runtime-1.19:shadowJar :flink-connector:flink-runtime-1.20:shadowJar` - `./gradlew :spotlessKotlinGradleCheck :flink-connector:flink-1.20:test -PskipITs` - `./gradlew :flink-connector:flink-common:compileTestJava :flink-connector:flink-1.19:compileJava :flink-connector:flink-1.19:compileTestJava :flink-connector:flink-1.20:compileTestJava :flink-connector:flink-common:spotlessJavaCheck :flink-connector:flink-1.19:spotlessJavaCheck :flink-connector:flink-1.20:spotlessJavaCheck -PskipITs` The runtime-jar tests verify that the version-specific SPI descriptors (`META-INF/services/org.apache.flink.table.factories.Factory`) are merged into the shaded uber-jar. The full Hive / Iceberg / JDBC / Paimon integration tests for both new minor versions will run in CI via the updated `.github/workflows/flink-integration-test-action.yml`. --------- Co-authored-by: fanng <“fanng@apache.org”>
danhuawang
pushed a commit
to danhuawang/gravitino
that referenced
this pull request
Jun 9, 2026
…pache#11071) ### What changes were proposed in this pull request? This PR adds Gravitino Flink connector support for Flink 1.19 and Flink 1.20 on top of the versioned-layout baseline established for Flink 1.18 (apache#10517). Main changes: - Add `flink-connector/v1.19/flink` + `flink-connector/v1.19/flink-runtime` modules. - Add `flink-connector/v1.20/flink` + `flink-connector/v1.20/flink-runtime` modules. - Add version-specific catalog/factory entry classes (`*Flink119` / `*Flink120`) for Hive, Iceberg, JDBC-MySQL, JDBC-Postgres, and Paimon. - Reuse `DefaultCatalogCompat` for Flink 1.19 and add `CatalogCompatFlink120` that uses `CatalogTable.newBuilder(...)` because `CatalogTable.of(...)` is deprecated in Flink 1.20. - The JDBC catalog for 1.19 / 1.20 uses the relocated `org.apache.flink.connector.jdbc.core.*` factory packages shipped in `flink-connector-jdbc 3.3.0-1.19` / `3.3.0-1.20`. - Add provider-specific integration-test entry classes for each new minor version. - Add Flink 1.20-specific overrides for missing required catalog options where Flink 1.20 reports `IllegalArgumentException` rather than the `ValidationException` thrown by older minor versions. - Extend the Flink CI job to run embedded and deploy mode integration tests across 1.18, 1.19, and 1.20. - Exclude the new Flink modules from unrelated Backend IT and Trino CI jobs. No `flink-common` production changes are required; the 1.18 baseline already exposes the necessary extension hooks (`BaseCatalog#catalogCompat()`, the protected `GravitinoJdbcCatalog(Context, ..., AbstractCatalog)` constructor, etc.). The shared Iceberg REST SQL integration test now lives in the common abstract IT base so the 1.18 / 1.19 / 1.20 entry classes can remain thin stubs. This PR supersedes apache#11061 (closed) and the now-stale apache#10501. ### Why are the changes needed? The Flink 1.18 baseline (apache#10517) was intentionally limited to one minor and deferred 1.19 / 1.20 to follow-up work under parent issue apache#9710. This PR is that follow-up, so users can target the two currently-maintained Flink minor versions. Fix: apache#9710 ### Does this PR introduce _any_ user-facing change? Yes. Two new connector artifacts are produced: - `gravitino-flink-connector-runtime-1.19_2.12` - `gravitino-flink-connector-runtime-1.20_2.12` The user-facing SQL syntax is unchanged. ### How was this patch tested? Local validations (all green): - `./gradlew spotlessApply` - `./gradlew :flink-connector:flink-1.19:test -PskipITs` - `./gradlew :flink-connector:flink-1.20:test -PskipITs` - `./gradlew :flink-connector:flink-runtime-1.19:test` - `./gradlew :flink-connector:flink-runtime-1.20:test` - `./gradlew :flink-connector:flink-runtime-1.19:shadowJar :flink-connector:flink-runtime-1.20:shadowJar` - `./gradlew :spotlessKotlinGradleCheck :flink-connector:flink-1.20:test -PskipITs` - `./gradlew :flink-connector:flink-common:compileTestJava :flink-connector:flink-1.19:compileJava :flink-connector:flink-1.19:compileTestJava :flink-connector:flink-1.20:compileTestJava :flink-connector:flink-common:spotlessJavaCheck :flink-connector:flink-1.19:spotlessJavaCheck :flink-connector:flink-1.20:spotlessJavaCheck -PskipITs` The runtime-jar tests verify that the version-specific SPI descriptors (`META-INF/services/org.apache.flink.table.factories.Factory`) are merged into the shaded uber-jar. The full Hive / Iceberg / JDBC / Paimon integration tests for both new minor versions will run in CI via the updated `.github/workflows/flink-integration-test-action.yml`. --------- Co-authored-by: fanng <“fanng@apache.org”>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR moves Lance storage configuration into Gravitino-managed catalog properties and teaches Lance REST to use those catalog defaults when serving Spark clients. It removes the need to pass spark.sql.catalog.lance.storage.* in the demo path, while keeping table-level lance.storage.* overrides supported.
Why are the changes needed?
Spark currently has to repeat MinIO/S3 settings that are already known to Gravitino, which makes Lance integration verbose and error-prone. This change keeps Gravitino as the source of truth for Lance storage configuration and simplifies Spark usage for Gravitino-managed Lance catalogs.
Fix: #11063
Does this PR introduce any user-facing change?
Yes.
How was this patch tested?