Skip to content

Spark: Add support for 4.2.0#14984

Draft
manuzhang wants to merge 3 commits into
apache:mainfrom
manuzhang:spark4.2-preview
Draft

Spark: Add support for 4.2.0#14984
manuzhang wants to merge 3 commits into
apache:mainfrom
manuzhang:spark4.2-preview

Conversation

@manuzhang

@manuzhang manuzhang commented Jan 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

This PR adds support for Apache Spark 4.2.0.

The branch starts with two history-preserving commits that rename the Spark 4.1 module to 4.2 and then restore 4.1 so Git can retain file history. The subsequent commits contain the substantive Spark 4.2 compatibility changes.

Changes

  • Adds and registers the Spark 4.2 modules in Gradle, settings, packaging, benchmarks, and CI.
  • Adapts Spark 4.2 view handling to the ViewInfo-based APIs and native catalog interfaces.
  • Removes the SupportsReplaceView adapter and routes view creation and replacement through Spark 4.2 catalog APIs.
  • Adds Iceberg-specific view property commands so ALTER VIEW SET TBLPROPERTIES uses view.updateProperties().set(...).commit() for Iceberg-backed catalogs instead of replacing the entire view.
  • Reuses Spark 4.2 native view validation for temporary views, functions, variables, and generated aliases while retaining Iceberg's V2 cycle validation.
  • Updates Spark 4.2 tests and benchmarks for API and error-message changes.
  • Retains coverage for fully qualified function identifiers that must not be rewritten, scoped to catalogs where Spark 4.2 still fails as expected.
  • Improves SparkCatalog errors for impossible checked-exception branches with operation-specific messages.
  • Resolves the released Spark 4.2.0 artifacts from Maven Central without an RC staging repository.

Verification

  • Full Spark 4.2 TestViews suite.
  • Spark 4.2 core, extensions, and runtime assemblies.
  • Forced dependency refresh against the standard repository configuration.

AI Disclosure

  • Model: GPT-5 Codex
  • Platform/Tool: Codex
  • Human Oversight: partially reviewed
  • Prompt Summary: Add Spark 4.2 support, address compatibility and native view-validation gaps, and switch from RC staging artifacts to the final Spark 4.2.0 release.

@manuzhang manuzhang force-pushed the spark4.2-preview branch 7 times, most recently from 0d5d05d to 330955b Compare January 8, 2026 15:39
@manuzhang manuzhang force-pushed the spark4.2-preview branch 2 times, most recently from bd2bff7 to af86915 Compare February 7, 2026 15:06
@manuzhang

manuzhang commented Feb 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

This failure from testing Spark 4.2.0-preview2 is caused by apache/spark#53788, after which an AnalysisException would be thrown on Iceberg metadata tables like default.table.partitions.

TestAddFilesProcedure > addPartitionsWithNullValueShouldAddFilesToNullPartition() > catalogName = spark_catalog, implementation = org.apache.iceberg.spark.SparkSessionCatalog, config = {type=hive, default-namespace=default, parquet-enabled=true, cache-enabled=false}, formatVersion = 2 FAILED
    org.apache.spark.sql.AnalysisException: [REQUIRES_SINGLE_PART_NAMESPACE] spark_catalog requires a single-part namespace, but got `default`.`table`. SQLSTATE: 42K05

@manuzhang

manuzhang commented Mar 17, 2026

Copy link
Copy Markdown
Member Author

Failed tests after upgrading to Spark 4.2.0-preview3-rc1

  1. testJoinsHourToDays() in TestStoragePartitionedJoins.java:
Assertion failed: "SPJ should not change query output: number of results should match"
The actual and expected query result sizes differ, indicating that either the join logic or test data setup causes a mismatch.
  1. readFromViewReferencingTempFunction() in TestViews.java:
Assertion failed: Expected a specific routine not found error, but got an AnalysisException with different message details.

@manuzhang

manuzhang commented Mar 19, 2026

Copy link
Copy Markdown
Member Author

apache/spark#54884 has been opened to fix the first failure.

@manuzhang manuzhang mentioned this pull request Mar 19, 2026
Closed
@manuzhang manuzhang force-pushed the spark4.2-preview branch 3 times, most recently from cc08c55 to cbdcfc9 Compare March 19, 2026 14:35
@manuzhang

manuzhang commented Mar 20, 2026

Copy link
Copy Markdown
Member Author

I will update HourToDaysReducer following interface changes from apache/spark#54884 in next preview release. All other test failures have been fixed.

szehon-ho
szehon-ho reviewed Mar 20, 2026
View reviewed changes

@szehon-ho szehon-ho left a comment •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

leave a note here to implement the new method in the Reducer once apache/spark#54884 is in (next Spark 4.2 preview)

Sorry just saw, it is the same comment

@jackylee-ch jackylee-ch mentioned this pull request Mar 26, 2026
Merged
@manuzhang manuzhang force-pushed the spark4.2-preview branch 6 times, most recently from b19679d to d478651 Compare April 7, 2026 10:09
@manuzhang

manuzhang commented Apr 7, 2026

Copy link
Copy Markdown
Member Author

The failed tests in 4.2.0-preview3 have been fixed in 4.2.0-preview4.

@manuzhang manuzhang force-pushed the spark4.2-preview branch 4 times, most recently from 3759e1f to 1db4a96 Compare June 18, 2026 04:35
giggsoff pushed a commit to giggsoff/iceberg that referenced this pull request Jun 23, 2026
NGSOK-1746 Port Spark 4.2.0 view/geo API for 4.2.0.1-4.3.0-1
e3b25ea 
The Spark 4.2 version bump (4.2.0.1-4.3.0-0 -> -1) adopts vanilla Apache
Spark 4.2.0 APIs, breaking compilation of the v4.2 tree. Port the required
changes from apache#14984, scoped to v4.2 only:

- Views: migrate ViewCatalog + SupportsReplaceView to TableViewCatalog /
  ViewInfo across BaseCatalog, SparkCatalog, SparkSessionCatalog and SparkView
  (now a static ViewInfo converter); add loadTableOrView/commitView; remove
  SupportsReplaceView. Catalog edits are surgical so the fork's ADH purge
  directory-delete and existing time-travel code are preserved.
- spark-extensions: drop the old Create/Alter view execs, rename
  Describe/ShowCreate/ShowV2ViewProperties to Iceberg* variants, and update
  the view analysis/strategy rules to the native DSv2 view API.
- Geo: replace GeographyVal/GeometryVal with BinaryView + Geography/Geometry
  types and getBinaryView in StructInternalRow and SparkParquetReaders.
- Add reportDriverMetrics() to StagedSparkTable and RollbackStagedTable
  (StagedTable/TruncatableTable now declare it).
- Netty: Spark 4.2 calls PlatformDependent.hasDirectByteBufferAddress (added
  in Netty 4.2.12; SPARK-56817 ships 4.2.13) but iceberg-arrow pins 4.2.4.
  Force netty-buffer/netty-common to 4.2.13 in the Spark 4.2 build only,
  leaving the shared pin and other Spark versions untouched.
- Tests: update TestViews expectations for the new messages and SHOW CREATE
  output, and override loadTableOrView in TestSparkCatalog.
@aokolnychyi

aokolnychyi commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Did we convert this into a draft as we anticipate another RC with changes around view management?

@manuzhang manuzhang force-pushed the spark4.2-preview branch 2 times, most recently from d42a203 to e8d1500 Compare June 27, 2026 03:20
@manuzhang

manuzhang commented Jun 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

@aokolnychyi Nope, I've always kept it as a draft. My plan is to wait for everything being settled at Spark side. Nevertheless, your review and comments are always welcome.

@manuzhang manuzhang force-pushed the spark4.2-preview branch 2 times, most recently from 23b5607 to 7e62f3d Compare June 27, 2026 15:07
@manuzhang

manuzhang commented Jun 29, 2026

Copy link
Copy Markdown
Member Author

@aokolnychyi I've updated the last commit against latest Spark 4.2 changes. Can you help check whether everything is expected?

@manuzhang manuzhang force-pushed the spark4.2-preview branch 6 times, most recently from a81b329 to 44ab0cb Compare July 8, 2026 03:50
manuzhang and others added 2 commits July 12, 2026 23:20
@manuzhang @codex
Spark: Move 4.1 as 4.2
bc060a6 
Co-authored-by: Codex <codex@openai.com>
@manuzhang @codex
Spark: Copy back 4.2 as 4.1
04eee63 
Co-authored-by: Codex <codex@openai.com>
@manuzhang manuzhang force-pushed the spark4.2-preview branch 2 times, most recently from ea9a3c5 to 9457d06 Compare July 13, 2026 08:37
@manuzhang manuzhang force-pushed the spark4.2-preview branch 3 times, most recently from 9ab41be to e3bb43f Compare July 14, 2026 04:14
@manuzhang @codex
Spark: Initial support for 4.2.0
754d32d 
Add Spark 4.2 module support and include follow-up compatibility fixes for Spark 4.2 view command handling and test expectations.

Co-authored-by: Codex <codex@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@aokolnychyi aokolnychyi Awaiting requested review from aokolnychyi
@szehon-ho szehon-ho Awaiting requested review from szehon-ho
@pan3793 pan3793 Awaiting requested review from pan3793

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

build INFRA not-stale spark

Projects

None yet

Milestone

Iceberg 1.12.0

Development

Successfully merging this pull request may close these issues.

8 participants

@manuzhang @steveloughran @huaxingao @szehon-ho @anuragmantri @aokolnychyi @pan3793 @nssalian