[#11063] fix(lance): manage lance storage config in Gravitino by FANNG1 · Pull Request #11070 · apache/gravitino

FANNG1 · 2026-05-13T06:30:14Z

What changes were proposed in this pull request?

This PR moves Lance storage configuration into Gravitino-managed catalog properties and teaches Lance REST to use those catalog defaults when serving Spark clients. It removes the need to pass spark.sql.catalog.lance.storage.* in the demo path, while keeping table-level lance.storage.* overrides supported.

Why are the changes needed?

Spark currently has to repeat MinIO/S3 settings that are already known to Gravitino, which makes Lance integration verbose and error-prone. This change keeps Gravitino as the source of truth for Lance storage configuration and simplifies Spark usage for Gravitino-managed Lance catalogs.

Fix: #11063

Does this PR introduce any user-facing change?

Yes.

lance.storage.* is now supported as a catalog-level property family for generic lakehouse catalogs.
Lance REST responses now resolve storage options from catalog defaults when table-level values are absent.
The Spark demo no longer requires explicit Lance storage conf entries.
Table-level lance.storage.* remains supported as an override.

How was this patch tested?

./gradlew :lance:lance-common:test --tests org.apache.gravitino.lance.common.utils.TestLancePropertiesUtils -PskipITs -PskipDockerTests=true
./gradlew :catalogs:catalog-lakehouse-generic:test --tests org.apache.gravitino.catalog.lakehouse.generic.TestPropertiesMetadata -PskipITs -PskipDockerTests=true
./gradlew :lance:lance-rest-server:compileTestJava -PskipITs -PskipDockerTests=true

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR centralizes Lance storage configuration in Gravitino catalog properties and updates Lance REST behavior to resolve storage options from catalog defaults when table-level values are absent, simplifying Spark/Lance integration while preserving per-table overrides.

Changes:

Add Lance storage (lance.storage.*) as a supported catalog-level property family in the generic lakehouse catalog.
Update Lance REST create/describe responses to resolve effective storage options from catalog + table properties (table overrides catalog).
Update integration tests and docs to reflect catalog-managed storage configuration and reduced Spark-side config.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
lance/lance-rest-server/src/test/java/org/apache/gravitino/lance/integration/test/LanceSparkRESTServiceIT.java	Sets MinIO `lance.storage.*` on catalog creation in Spark REST IT flow.
lance/lance-rest-server/src/test/java/org/apache/gravitino/lance/integration/test/LanceRESTServiceIT.java	Adds IT coverage asserting REST responses include catalog-default storage options.
lance/lance-common/src/test/java/org/apache/gravitino/lance/common/utils/TestLancePropertiesUtils.java	Adds unit tests for extracting/resolving storage options from properties.
lance/lance-common/src/main/java/org/apache/gravitino/lance/common/utils/LancePropertiesUtils.java	Adds `resolveLanceStorageOptions` and improves storage-option extraction.
lance/lance-common/src/main/java/org/apache/gravitino/lance/common/ops/gravitino/GravitinoLanceTableOperations.java	Updates REST responses to return resolved (catalog+table) storage options.
docs/lance-rest-integration.md	Updates docs to recommend catalog-level storage config and keep per-table overrides.
docs/lakehouse-generic-lance-table.md	Updates MinIO section to describe catalog-level storage config; removes table-level example.
catalogs/catalog-lakehouse-generic/src/test/java/org/apache/gravitino/catalog/lakehouse/generic/TestPropertiesMetadata.java	Adds test asserting `lance.storage.*` is supported in catalog properties metadata.
catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/generic/GenericCatalogPropertiesMetadata.java	Registers `lance.storage.*` prefix as a supported catalog property family.
catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/generic/GenericCatalogOperations.java	Copies catalog-level Lance storage options into Lance table properties when absent.

github-actions · 2026-05-13T08:00:42Z

Code Coverage Report

Overall Project	66.08% `-0.12%`	🟢
Files changed	58.54%	🔴

Module	Coverage
aliyun	1.72%	🔴
api	47.13%	🟢
authorization-common	85.96%	🟢
aws	1.08%	🔴
azure	2.47%	🔴
catalog-common	10.2%	🔴
catalog-fileset	80.02%	🟢
catalog-glue	83.41%	🟢
catalog-hive	81.83%	🟢
catalog-jdbc-clickhouse	79.18% `+9.18%`	🟢
catalog-jdbc-common	43.93%	🟢
catalog-jdbc-doris	80.28%	🟢
catalog-jdbc-hologres	54.03%	🟢
catalog-jdbc-mysql	79.23%	🟢
catalog-jdbc-oceanbase	78.38%	🟢
catalog-jdbc-postgresql	82.05%	🟢
catalog-jdbc-starrocks	78.27%	🟢
catalog-kafka	77.01%	🟢
catalog-lakehouse-generic	44.89% `-44.22%`	🟢
catalog-lakehouse-hudi	79.1%	🟢
catalog-lakehouse-iceberg	86.98%	🟢
catalog-lakehouse-paimon	76.85%	🟢
catalog-model	77.72%	🟢
cli	44.51%	🟢
client-java	77.96%	🟢
common	50.0%	🟢
core	82.29%	🟢
filesystem-hadoop3	76.97%	🟢
flink	0.0%	🔴
flink-common	43.17%	🟢
flink-runtime	0.0%	🔴
gcp	14.12%	🔴
hadoop-common	10.39%	🔴
hive-metastore-common	46.83%	🟢
iceberg-common	55.46%	🟢
iceberg-rest-server	69.7%	🟢
idp-basic	94.68%	🟢
integration-test-common	0.0%	🔴
jobs	66.17%	🟢
lance-common	20.9% `-5.14%`	🔴
lance-rest-server	62.78%	🟢
lineage	53.02%	🟢
optimizer	82.95%	🟢
optimizer-api	21.95%	🔴
server	85.96%	🟢
server-common	71.23%	🟢
spark	32.79%	🔴
spark-common	39.09%	🔴
trino-connector	35.14%	🔴

Files

Module	File	Coverage
catalog-jdbc-clickhouse	ClickHouseTableSqlUtils.java	96.64%	🟢
	ClickHouseTableOperations.java	82.16%	🟢
catalog-lakehouse-generic	GenericCatalogPropertiesMetadata.java	100.0%	🟢
	LanceTableOperations.java	28.24%	🔴
	GenericCatalogOperations.java	27.87%	🔴
lance-common	LancePropertiesUtils.java	83.33%	🟢
	GravitinoLanceTableOperations.java	0.0%	🔴

…h normalization

FANNG1 · 2026-05-13T23:04:04Z

@yuqi1129 PTAL, thx

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

+                "The Lance storage options managed by the catalog.",
+                false /* immutable */,
+                null, /* defaultValue */
+                false /* hidden */,


+  public void setCatalogProperties(Map<String, String> catalogProperties) {
+    this.catalogProperties =
+        catalogProperties == null ? Map.of() : ImmutableMap.copyOf(catalogProperties);
+  }


+  private ManagedTableOperations configureTableOps(ManagedTableOperations ops) {
+    if (ops instanceof LanceTableOperations) {
+      ((LanceTableOperations) ops).setCatalogProperties(catalogProperties);
+    }


+    Map<String, String> storageProps =
+        LancePropertiesUtils.resolveLanceStorageOptions(catalogProperties, properties);


FANNG1 · 2026-05-14T12:38:40Z

Updated in the latest commit:

handleLanceTableChange now resolves lance.storage.* from catalog + table properties and passes the merged storage map into openDataset, so alter/index/drop operations work without Spark-side storage config.
Added a focused unit test in TestLanceTableOperations to cover the catalog-default storage path.

I am leaving the secret/public property split for a follow-up because that is a broader catalog-property cleanup across the codebase, not specific to this Lance fix. WDYT？@yuqi

yuqi1129 · 2026-05-15T03:47:34Z

Updated in the latest commit:

handleLanceTableChange now resolves lance.storage.* from catalog + table properties and passes the merged storage map into openDataset, so alter/index/drop operations work without Spark-side storage config.

Added a focused unit test in TestLanceTableOperations to cover the catalog-default storage path.

I am leaving the secret/public property split for a follow-up because that is a broader catalog-property cleanup across the codebase, not specific to this Lance fix. WDYT？@yuqi

It looks good to me.

…pache#11070) ### What changes were proposed in this pull request? This PR moves Lance storage configuration into Gravitino-managed catalog properties and teaches Lance REST to use those catalog defaults when serving Spark clients. It removes the need to pass spark.sql.catalog.lance.storage.* in the demo path, while keeping table-level lance.storage.* overrides supported. ### Why are the changes needed? Spark currently has to repeat MinIO/S3 settings that are already known to Gravitino, which makes Lance integration verbose and error-prone. This change keeps Gravitino as the source of truth for Lance storage configuration and simplifies Spark usage for Gravitino-managed Lance catalogs. Fix: apache#11063 ### Does this PR introduce _any_ user-facing change? Yes. - lance.storage.* is now supported as a catalog-level property family for generic lakehouse catalogs. - Lance REST responses now resolve storage options from catalog defaults when table-level values are absent. - The Spark demo no longer requires explicit Lance storage conf entries. - Table-level lance.storage.* remains supported as an override. ### How was this patch tested? - ./gradlew :lance:lance-common:test --tests org.apache.gravitino.lance.common.utils.TestLancePropertiesUtils -PskipITs -PskipDockerTests=true - ./gradlew :catalogs:catalog-lakehouse-generic:test --tests org.apache.gravitino.catalog.lakehouse.generic.TestPropertiesMetadata -PskipITs -PskipDockerTests=true - ./gradlew :lance:lance-rest-server:compileTestJava -PskipITs -PskipDockerTests=true --------- Co-authored-by: fanng <“fanng@apache.org”>

FANNG1 mentioned this pull request May 13, 2026

[#11063] fix(lance): manage storage config in Gravitino #11061

Closed

[apache#11063] fix(lance): manage storage config in Gravitino

fa33047

FANNG1 force-pushed the lance-11063-clean branch from 8f9585a to fa33047 Compare May 13, 2026 06:49

FANNG1 marked this pull request as draft May 13, 2026 07:06

FANNG1 changed the title ~~[#11063] fix(lance): manage storage config in Gravitino~~ [#11063] fix(lance): manage lance storage config in Gravitino May 13, 2026

FANNG1 requested a review from Copilot May 13, 2026 07:38

Copilot AI reviewed May 13, 2026

View reviewed changes

Copilot started reviewing on behalf of FANNG1 May 13, 2026 07:50 View session

fanng added 5 commits May 13, 2026 19:44

[apache#11063] fix(lance): avoid persisting catalog storage defaults

ce3eb3e

[apache#11063] fix(lance): return effective storage for exist_ok

8036427

[apache#11063] test(lance): narrow REST integration coverage

22f24ea

[apache#11063] test(lance): keep REST IT location behavior stable

c461fd4

[apache#11063] test(lance): align location assertion with catalog pat…

c7c848d

…h normalization

FANNG1 marked this pull request as ready for review May 13, 2026 23:04

yuqi1129 requested a review from Copilot May 14, 2026 03:26

Copilot started reviewing on behalf of yuqi1129 May 14, 2026 03:26 View session

Copilot AI reviewed May 14, 2026

View reviewed changes

[apache#11063] fix(lance): resolve storage defaults in alter path

87d0427

yuqi1129 approved these changes May 15, 2026

View reviewed changes

yuqi1129 merged commit fb902cc into apache:main May 15, 2026
32 of 33 checks passed

yuqi1129 assigned FANNG1 May 15, 2026

FANNG1 deleted the lance-11063-clean branch June 10, 2026 11:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[#11063] fix(lance): manage lance storage config in Gravitino #11070

[#11063] fix(lance): manage lance storage config in Gravitino #11070
yuqi1129 merged 7 commits into
apache:mainfrom
FANNG1:lance-11063-clean

FANNG1 commented May 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 13, 2026 •

edited

Loading

Uh oh!

FANNG1 commented May 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

FANNG1 commented May 14, 2026 •

edited

Loading

Uh oh!

yuqi1129 commented May 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		Map<String, String> storageProps =
		LancePropertiesUtils.resolveLanceStorageOptions(catalogProperties, properties);

Uh oh!

Conversation

FANNG1 commented May 13, 2026

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage Report

Uh oh!

FANNG1 commented May 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

FANNG1 commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yuqi1129 commented May 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented May 13, 2026 •

edited

Loading

FANNG1 commented May 14, 2026 •

edited

Loading