[VL][DELTA] Support Delta CDF scan offload by malinjawi · Pull Request #12218 · apache/gluten

malinjawi · 2026-06-01T14:12:32Z

What changes are proposed in this pull request?

Addresses #12195.

Delta CDF reads enter Spark as CDCReader.DeltaCDFRelation, so they do not initially have the normal FileSourceScanExec + DeltaParquetFileFormat shape that Gluten's existing Delta scan offload rule recognizes.

This PR adds a Gluten Delta planner strategy, wired from the Velox Delta component, that recognizes DeltaCDFRelation, expands it through Delta's own CDF batch planning path, and rewrites the original projection/filter attributes onto the expanded logical plan. After that, the existing Delta scan offload path can plan the underlying CDF file scans as DeltaScanTransformer.

The change is intentionally scoped to Gluten's Delta/Spark planning layer rather than Velox C++:

Add DeltaCDFScanStrategy for table_changes(...) and DataFrame readChangeFeed scans.
Add Delta-version helper shims for Delta 2.3, 2.4, 3.3, and 4.x API differences.
Register the planner strategy from VeloxDeltaComponent.
Add Delta regression coverage for insert/update/delete CDF rows, filter/projection handling, bounded version reads, DataFrame readChangeFeed, column mapping, and a startingVersion = 0 case.

One planner/test nuance: Delta CDF expansion can keep a Spark-side ExistingRDD branch for synthesized change rows, including on the tested update/delete CDF paths. The regression suite therefore compares results against vanilla Spark and asserts that the expanded CDF file scans are transformed to DeltaScanTransformer, rather than requiring the entire expanded CDF union to be globally fallback-free.

How was this patch tested?

Local checks used JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home.

git diff --check
./dev/format-scala-code.sh check
./build/mvn -pl gluten-delta -am -Pspark-3.5 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests test-compile
./build/mvn -pl gluten-delta -am -Pspark-4.0 -Pscala-2.13 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests test-compile
./build/mvn -pl gluten-delta -am -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests test-compile

Earlier cross-version compile checks also passed before the test-expectation-only update:

./build/mvn -pl gluten-delta -am -Pspark-3.3 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests -Dcheckstyle.skip=true -Dscalastyle.skip=true -Dspotless.check.skip=true test-compile
./build/mvn -pl gluten-delta -am -Pspark-3.4 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests -Dcheckstyle.skip=true -Dscalastyle.skip=true -Dspotless.check.skip=true test-compile
./build/mvn -pl gluten-delta -am -Pspark-3.5 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests -Dcheckstyle.skip=true -Dscalastyle.skip=true -Dspotless.check.skip=true test-compile
./build/mvn -pl gluten-delta -am -Pspark-4.0 -Pscala-2.13 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests -Dcheckstyle.skip=true -Dscalastyle.skip=true -Dspotless.check.skip=true test-compile
./build/mvn -pl gluten-delta -am -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pbackends-velox -Pdelta -DskipTests -Dcheckstyle.skip=true -Dscalastyle.skip=true -Dspotless.check.skip=true test-compile

Full native Velox runtime and benchmarking are left to CI / a native Gluten-optimized environment; this local checkout does not have cpp/build/releases/libgluten.so and the Velox external project build available.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: IBM BOB

github-actions · 2026-06-01T14:13:06Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-01T14:59:43Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-01T15:08:37Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-01T15:52:58Z

Run Gluten Clickhouse CI on x86

felipepessoto · 2026-06-02T19:53:56Z

Could you add a DV-enabled CDF regression test to confirm the behavior? Something like:

  CREATE TABLE t (id INT, name STRING) USING delta
  TBLPROPERTIES ('delta.enableChangeDataFeed'='true', 'delta.enableDeletionVectors'='true');
  INSERT INTO t VALUES (1,'v1'),(2,'v2'),(3,'v3');
  DELETE FROM t WHERE id = 2;
  SELECT id, name, _change_type FROM table_changes('t', 0);

compared against vanilla Spark (checkAnswer)

felipepessoto · 2026-06-02T19:55:27Z

+import org.apache.spark.sql.delta.BatchCDFSchemaEndVersion
+import org.apache.spark.sql.delta.commands.cdc.CDCReader
+
+object DeltaCDFRelationHelper {


Do you know what happens in Delta 4.1/4.2/4.3? Would it take the code from 4.0 folder?

Yes @felipepessoto I checked the current Maven profile/source wiring:

spark-4.0 sets delta.version=4.0.1 and delta.binary.version=40

spark-4.1 sets delta.version=4.1.0 and delta.binary.version=40

the Delta profile adds sources from src-delta${delta.binary.version}/main/scala

So with the current profile setup, both Spark 4.0 / Delta 4.0.x and Spark 4.1 / Delta 4.1.x use gluten-delta/src-delta40/....

For future Delta 4.2 / 4.3 support, it depends on how those profiles are introduced. If they keep delta.binary.version=40, they will continue to use the same src-delta40 helper. If Delta changes the relevant CDF APIs and Gluten introduces a new binary bucket, then we should add a new version-specific helper.

This also relates to the earlier folder-activation discussion in #11924: supporting family-level folders like src-spark4 / src-delta4 would make this cleaner and avoid copying code across Spark/Delta 4.x profiles when the APIs stay compatible. I think that belongs in a separate Maven/source-layout refactor rather than in this CDF PR.

malinjawi · 2026-06-08T08:26:26Z

Could you add a DV-enabled CDF regression test to confirm the behavior? Something like:
  CREATE TABLE t (id INT, name STRING) USING delta
  TBLPROPERTIES ('delta.enableChangeDataFeed'='true', 'delta.enableDeletionVectors'='true');
  INSERT INTO t VALUES (1,'v1'),(2,'v2'),(3,'v3');
  DELETE FROM t WHERE id = 2;
  SELECT id, name, _change_type FROM table_changes('t', 0);
compared against vanilla Spark (checkAnswer)

Thanks @felipepessoto good point! I agree this is worth covering explicitly.

I’ll add a focused DV-enabled CDF regression with delta.enableChangeDataFeed=true and delta.enableDeletionVectors=true, comparing table_changes(...) against vanilla Spark via the existing runAndCompare / checkAnswer path.

One important nuance: this PR is not intended to claim full native DV support. DV-backed Delta scan execution is still a separate active area in Gluten/Velox and may continue to fall back where native DV handling is not available. The goal of the regression here is to make sure enabling DV does not break the CDF planning path or result correctness.

github-actions · 2026-06-08T08:46:33Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-08T11:13:19Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-16T09:51:26Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-06-16T16:35:53Z

Run Gluten Clickhouse CI on x86

felipepessoto · 2026-06-16T16:44:04Z

+  // DeltaScanTransformer).
+  private def touchesDeletionVectors(cdfRelation: CDCReader.DeltaCDFRelation): Boolean =
+    cdfRelation.snapshotWithSchemaMode.snapshot.metadata.configuration
+      .get("delta.enableDeletionVectors")


This table property says if new writes can write DV. But I think you want to find if table has any DV

Good catch; you were right that the table property only controls future writes. This is now addressed in two layers:

3b462034a makes DeltaCDFScanStrategy inspect AddFile/RemoveFile actions only in the requested CDF version range.

Spark 3.4 CI then exposed an older physical fallback that still treated a DV-capable TahoeFileIndex as if every CDF range contained a DV. 049e608a3 now special-cases CdcAddFileIndex and TahoeRemoveFileIndex there as well, using their exact files/actions before the generic table-level fallback.

The regression covers both directions: a DV-enabled table's insert-only 0..1 range must offload, while a range containing a DV-backed delete must fall back even after delta.enableDeletionVectors is set back to false, with the CDF rows checked for correctness. Formatting and test compilation pass for Spark 3.3, 3.4, 3.5, 4.0, and 4.1.

Delta change-data-feed reads enter Spark as a CDCReader.DeltaCDFRelation, which does not have the FileSourceScanExec + DeltaParquetFileFormat shape that Gluten's Delta scan offload recognizes. Add a Gluten Delta planner strategy (wired from VeloxDeltaComponent) that recognizes DeltaCDFRelation, expands it through Delta's own CDF batch planning, and rewrites the projection/filter attributes onto the expanded plan, so the existing Delta scan offload path can plan the underlying CDF file scans as DeltaScanTransformer. CDF over a deletion-vector-enabled table is kept on Spark: native CDF lacks the DV-aware row-level reconciliation, so an offloaded CDF scan would emit still-live rows as `delete` change rows. Re-planning Delta's analyzed CDF batch plan also drops that reconciliation on some Delta versions (Delta 2.4 / Spark 3.4), where the remove side would surface every row of a logically removed file as a `delete` change row. The planner strategy therefore declines to intercept CDF reads whose table has deletion vectors enabled and leaves them to Delta's own DeltaCDFRelation scan, which reconciles DVs correctly on every supported Delta version; DeltaScanTransformer additionally guards both CDF scan sides -- the add side (CdcAddFileIndex) and the remove side (TahoeRemoveFileIndex) -- as a backstop. Normal (non-CDF) DV scans are unaffected and continue to apply the DV natively. Addresses apache#12195.

github-actions · 2026-06-16T19:36:45Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-07-19T09:39:38Z

Run Gluten Clickhouse CI on x86

Inspect AddFile and RemoveFile actions in the requested CDF version range instead of relying on the table property, which only controls future DV writes.\n\nCover both DV-enabled ranges without DV actions and existing DV actions after the property is disabled.

github-actions · 2026-07-19T10:49:09Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-07-19T12:01:16Z

Run Gluten Clickhouse CI on x86

github-actions Bot added VELOX DATA_LAKE labels Jun 1, 2026

malinjawi marked this pull request as ready for review June 1, 2026 18:50

felipepessoto reviewed Jun 2, 2026

View reviewed changes

malinjawi force-pushed the codex/delta-cdf-offload branch from 2cd31c0 to bca92ba Compare June 16, 2026 09:46

malinjawi force-pushed the codex/delta-cdf-offload branch from bca92ba to b14394d Compare June 16, 2026 16:30

felipepessoto reviewed Jun 16, 2026

View reviewed changes

malinjawi force-pushed the codex/delta-cdf-offload branch from b14394d to 582641a Compare June 16, 2026 19:36

Merge branch 'main' into codex/delta-cdf-offload

e0d9027

[VL][Delta] Check CDF indexes for deletion vectors

049e608

malinjawi requested a review from felipepessoto July 19, 2026 18:00

Uh oh!

Conversation

malinjawi commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes are proposed in this pull request?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

github-actions Bot commented Jun 1, 2026

Uh oh!

github-actions Bot commented Jun 1, 2026

Uh oh!

github-actions Bot commented Jun 1, 2026

Uh oh!

github-actions Bot commented Jun 1, 2026

Uh oh!

felipepessoto commented Jun 2, 2026

Uh oh!

felipepessoto Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

malinjawi Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

malinjawi commented Jun 8, 2026

Uh oh!

github-actions Bot commented Jun 8, 2026

Uh oh!

github-actions Bot commented Jun 8, 2026

Uh oh!

github-actions Bot commented Jun 16, 2026

Uh oh!

github-actions Bot commented Jun 16, 2026

Uh oh!

felipepessoto Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

malinjawi Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 16, 2026

Uh oh!

github-actions Bot commented Jul 19, 2026

Uh oh!

github-actions Bot commented Jul 19, 2026

Uh oh!

github-actions Bot commented Jul 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

malinjawi commented Jun 1, 2026 •

edited

Loading

malinjawi Jul 19, 2026 •

edited

Loading