Skip to content

Commit 91bb98d

Browse files
committed
[KYUUBI #7570] Support Spark 4.2
### Why are the changes needed? Spark 4.2.0 is released. This adds the `spark-4.2` profile and the `kyuubi-extension-spark-4-2` module, enabling Kyuubi to run on Spark 4.2. ### What changes are proposed? - Add `spark-4.2` Maven profile with the `kyuubi-extension-spark-4-2` module. Lakehouse engines (Delta/Hudi/Paimon/Iceberg) fall back to 4.1 artifacts until native 4.2 support lands. - Adapt the engine UI and servlet adapter for Spark 4.2's Jetty 12 (ee10) / Servlet 6.0 upgrade. - Re-fork `KyuubiEnsureRequirements` and update `MaxScanStrategy` for Spark 4.2 physical plan API changes. - Add spark-4.2 CI matrix entries (`normal` + `verify-on-spark-4.2-binary`). ### How was this patch tested? Pass GHA. ### Was this patch authored or co-authored using generative AI tooling? Assisted-by: GLM 5.2 Closes #7570 from pan3793/spark-4.2. Closes #7570 cbc1dd2 [Cheng Pan] Fix stale spark-4.1 references in kyuubi-extension-spark-4-2 edbdcbd [Cheng Pan] nit: style 7eb2fec [Cheng Pan] Apply suggestions from code review 562a43b [Cheng Pan] Support Spark 4.2 CatalogTable constructor in HiveTableCatalog 871db15 [Cheng Pan] Rename test helpers to avoid Spark 4.2 QueryTestBase conflicts f4c865d [Cheng Pan] Unwrap InvocationTargetException in setCurrentNamespace reflective call bc4174c [Cheng Pan] Fix setCurrentNamespace reflective call for Array[String] varargs 750d511 [Cheng Pan] Add spark-4.2 extension to release script eaf4a5d [Cheng Pan] Sync kyuubi-extension-spark-4-2 pom.xml with 4-1 sibling 20a4051 [Cheng Pan] Access CatalogManager reflectively for Spark 4.2 cross-version compat be7c994 [Cheng Pan] Reference SPARK JIRA tickets in Spark 4.x comments dfd3c1e [Cheng Pan] Remove redundant ANSI disable in authz SparkSessionProvider 9d04083 [Cheng Pan] Accept DAGScheduler.cleanupQueryJobs NPE in engine crash test 77f82e4 [Cheng Pan] Use Spark CurrentUserContext for session_user on Spark 4.0+ e68244c [Cheng Pan] update docs 71ea1e2 [Cheng Pan] Support Spark 4.2 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org>
1 parent 9afb1d8 commit 91bb98d

83 files changed

Lines changed: 8303 additions & 336 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/license.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ jobs:
4444
- run: |
4545
build/mvn org.apache.rat:apache-rat-plugin:check \
4646
-Ptpcds -Pkubernetes-it \
47-
-Pspark-3.3 -Pspark-3.4 -Pspark-3.5 -Pspark-4.0 -Pspark-4.1
47+
-Pspark-3.3 -Pspark-3.4 -Pspark-3.5 -Pspark-4.0 -Pspark-4.1 -Pspark-4.2
4848
- name: Upload rat report
4949
if: failure()
5050
uses: actions/upload-artifact@v7

.github/workflows/master.yml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,12 @@ jobs:
7474
spark-archive: '-Pscala-2.13'
7575
exclude-tags: ''
7676
comment: 'normal'
77+
- java: 21
78+
python: '3.11'
79+
spark: '4.2'
80+
spark-archive: '-Pscala-2.13'
81+
exclude-tags: '-Dmaven.plugin.scalatest.exclude.tags=org.scalatest.tags.Slow,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.IcebergTest,org.apache.kyuubi.tags.PaimonTest,org.apache.kyuubi.tags.HudiTest'
82+
comment: 'normal'
7783
- java: 8
7884
python: '3.9'
7985
spark: '3.5'
@@ -98,6 +104,12 @@ jobs:
98104
spark-archive: '-Pscala-2.13 -Dspark.archive.mirror=https://www.apache.org/dyn/closer.lua/spark/spark-4.1.2 -Dspark.archive.name=spark-4.1.2-bin-hadoop3.tgz'
99105
exclude-tags: '-Dmaven.plugin.scalatest.exclude.tags=org.scalatest.tags.Slow,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.IcebergTest,org.apache.kyuubi.tags.PaimonTest,org.apache.kyuubi.tags.HudiTest,org.apache.kyuubi.tags.SparkLocalClusterTest'
100106
comment: 'verify-on-spark-4.1-binary'
107+
- java: 17
108+
python: '3.11'
109+
spark: '3.5'
110+
spark-archive: '-Pscala-2.13 -Dspark.archive.mirror=https://www.apache.org/dyn/closer.lua/spark/spark-4.2.0 -Dspark.archive.name=spark-4.2.0-bin-hadoop3.tgz'
111+
exclude-tags: '-Dmaven.plugin.scalatest.exclude.tags=org.scalatest.tags.Slow,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.IcebergTest,org.apache.kyuubi.tags.PaimonTest,org.apache.kyuubi.tags.HudiTest,org.apache.kyuubi.tags.SparkLocalClusterTest'
112+
comment: 'verify-on-spark-4.2-binary'
101113
env:
102114
SPARK_LOCAL_IP: localhost
103115
steps:

.github/workflows/style.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,7 @@ jobs:
7171
build/mvn clean install -pl extensions/spark/kyuubi-extension-spark-3-5,extensions/spark/kyuubi-spark-connector-hive -am -Pspark-3.5
7272
build/mvn clean install -pl extensions/spark/kyuubi-extension-spark-4-0 -am -Pspark-4.0 -Pscala-2.13
7373
build/mvn clean install -pl extensions/spark/kyuubi-extension-spark-4-1 -am -Pspark-4.1 -Pscala-2.13
74+
build/mvn clean install -pl extensions/spark/kyuubi-extension-spark-4-2 -am -Pspark-4.2 -Pscala-2.13
7475
7576
- name: Scalastyle with maven
7677
id: scalastyle-check

build/release/release.sh

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,11 @@ upload_nexus_staging() {
130130
-s "${KYUUBI_DIR}/build/release/asf-settings.xml" \
131131
-pl extensions/spark/kyuubi-extension-spark-4-1 -am
132132

133+
# Spark Extension Plugin for Spark 4.2 and Scala 2.13
134+
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,flink-provided,spark-provided,hive-provided,spark-4.2,scala-2.13 \
135+
-s "${KYUUBI_DIR}/build/release/asf-settings.xml" \
136+
-pl extensions/spark/kyuubi-extension-spark-4-2 -am
137+
133138
# Spark Hive/TPC-DS/TPC-H Connector built with default Spark version (3.5) and Scala 2.13
134139
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,flink-provided,spark-provided,hive-provided,spark-3.5,scala-2.13 \
135140
-s "${KYUUBI_DIR}/build/release/asf-settings.xml" \

dev/kyuubi-codecov/pom.xml

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -280,6 +280,31 @@
280280
</dependency>
281281
</dependencies>
282282
</profile>
283+
<profile>
284+
<id>spark-4.2</id>
285+
<dependencies>
286+
<dependency>
287+
<groupId>org.apache.kyuubi</groupId>
288+
<artifactId>kyuubi-extension-spark-4-2_${scala.binary.version}</artifactId>
289+
<version>${project.version}</version>
290+
</dependency>
291+
<dependency>
292+
<groupId>org.apache.kyuubi</groupId>
293+
<artifactId>kyuubi-spark-connector-hive_${scala.binary.version}</artifactId>
294+
<version>${project.version}</version>
295+
</dependency>
296+
<dependency>
297+
<groupId>org.apache.kyuubi</groupId>
298+
<artifactId>kyuubi-spark-authz_${scala.binary.version}</artifactId>
299+
<version>${project.version}</version>
300+
</dependency>
301+
<dependency>
302+
<groupId>org.apache.kyuubi</groupId>
303+
<artifactId>kyuubi-spark-lineage_${scala.binary.version}</artifactId>
304+
<version>${project.version}</version>
305+
</dependency>
306+
</dependencies>
307+
</profile>
283308
<profile>
284309
<id>codecov</id>
285310
<build>

dev/reformat

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ set -x
2020

2121
KYUUBI_HOME="$(cd "`dirname "$0"`/.."; pwd)"
2222

23-
PROFILES="-Pflink-provided,hive-provided,spark-provided,spark-4.1,spark-4.0,spark-3.5,spark-3.4,spark-3.3,tpcds,kubernetes-it"
23+
PROFILES="-Pflink-provided,hive-provided,spark-provided,spark-4.2,spark-4.1,spark-4.0,spark-3.5,spark-3.4,spark-3.3,tpcds,kubernetes-it"
2424

2525
# python style checks rely on `black` in path
2626
if ! command -v black &> /dev/null

docs/extensions/engines/spark/rules.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,9 @@ And don't worry, Kyuubi will support the new Apache Spark version in the future.
6464
| kyuubi-extension-spark-3-3 | 3.3.x | 1.6.0-incubating | N/A | 1.6.0-incubating | spark-3.3 |
6565
| kyuubi-extension-spark-3-4 | 3.4.x | 1.8.0 | N/A | 1.8.0 | spark-3.4 |
6666
| kyuubi-extension-spark-3-5 | 3.5.x | 1.8.0 | N/A | 1.9.0 | spark-3.5 |
67+
| kyuubi-extension-spark-4-0 | 4.0.x | 1.10.0 | N/A | 1.10.0 | spark-4.0 |
68+
| kyuubi-extension-spark-4-1 | 4.1.x | 1.11.0 | N/A | 1.11.0 | spark-4.1 |
69+
| kyuubi-extension-spark-4-2 | 4.2.x | 1.12.0 | N/A | 1.12.0 | spark-4.2 |
6770

6871
1. Check the matrix that if you are using the supported Spark version, and find the corresponding Kyuubi Spark SQL Extension jar
6972
2. Get the Kyuubi Spark SQL Extension jar

docs/quick_start/quick_start.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ pre-installed and the ``JAVA_HOME`` is correctly set to each component.
4343
**Kyuubi** Gateway \ |release| \ - Kyuubi Server
4444
Engine lib - Kyuubi Engine
4545
Beeline - Kyuubi Beeline
46-
**Spark** Engine 3.3 to 3.5, 4.0 to 4.1 A Spark distribution
46+
**Spark** Engine 3.3 to 3.5, 4.0 to 4.2 A Spark distribution
4747
**Flink** Engine 1.17 to 1.20 A Flink distribution
4848
**Trino** Engine N/A A Trino cluster allows to access via trino-client v411
4949
**Doris** Engine N/A A Doris cluster
Lines changed: 222 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,222 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
~ Licensed to the Apache Software Foundation (ASF) under one or more
4+
~ contributor license agreements. See the NOTICE file distributed with
5+
~ this work for additional information regarding copyright ownership.
6+
~ The ASF licenses this file to You under the Apache License, Version 2.0
7+
~ (the "License"); you may not use this file except in compliance with
8+
~ the License. You may obtain a copy of the License at
9+
~
10+
~ http://www.apache.org/licenses/LICENSE-2.0
11+
~
12+
~ Unless required by applicable law or agreed to in writing, software
13+
~ distributed under the License is distributed on an "AS IS" BASIS,
14+
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
~ See the License for the specific language governing permissions and
16+
~ limitations under the License.
17+
-->
18+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
19+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
20+
<modelVersion>4.0.0</modelVersion>
21+
<parent>
22+
<groupId>org.apache.kyuubi</groupId>
23+
<artifactId>kyuubi-parent</artifactId>
24+
<version>1.12.0-SNAPSHOT</version>
25+
<relativePath>../../../pom.xml</relativePath>
26+
</parent>
27+
28+
<artifactId>kyuubi-extension-spark-4-2_${scala.binary.version}</artifactId>
29+
<packaging>jar</packaging>
30+
<name>Kyuubi Dev Spark Extensions (for Spark 4.2)</name>
31+
<url>https://kyuubi.apache.org/</url>
32+
33+
<dependencies>
34+
<dependency>
35+
<groupId>org.scala-lang</groupId>
36+
<artifactId>scala-library</artifactId>
37+
<scope>provided</scope>
38+
</dependency>
39+
40+
<dependency>
41+
<groupId>org.apache.spark</groupId>
42+
<artifactId>spark-sql_${scala.binary.version}</artifactId>
43+
<scope>provided</scope>
44+
</dependency>
45+
46+
<dependency>
47+
<groupId>org.apache.spark</groupId>
48+
<artifactId>spark-hive_${scala.binary.version}</artifactId>
49+
<scope>provided</scope>
50+
</dependency>
51+
52+
<dependency>
53+
<groupId>org.apache.hadoop</groupId>
54+
<artifactId>hadoop-client-api</artifactId>
55+
<scope>provided</scope>
56+
</dependency>
57+
58+
<dependency>
59+
<groupId>org.apache.kyuubi</groupId>
60+
<artifactId>kyuubi-download</artifactId>
61+
<version>${project.version}</version>
62+
<type>pom</type>
63+
<scope>test</scope>
64+
</dependency>
65+
66+
<dependency>
67+
<groupId>org.apache.kyuubi</groupId>
68+
<artifactId>kyuubi-util-scala_${scala.binary.version}</artifactId>
69+
<version>${project.version}</version>
70+
<type>test-jar</type>
71+
<scope>test</scope>
72+
</dependency>
73+
74+
<dependency>
75+
<groupId>org.apache.spark</groupId>
76+
<artifactId>spark-core_${scala.binary.version}</artifactId>
77+
<type>test-jar</type>
78+
<scope>test</scope>
79+
</dependency>
80+
81+
<dependency>
82+
<groupId>org.apache.spark</groupId>
83+
<artifactId>spark-catalyst_${scala.binary.version}</artifactId>
84+
<type>test-jar</type>
85+
<scope>test</scope>
86+
</dependency>
87+
88+
<dependency>
89+
<groupId>org.scalatestplus</groupId>
90+
<artifactId>scalacheck-1-17_${scala.binary.version}</artifactId>
91+
<scope>test</scope>
92+
</dependency>
93+
94+
<dependency>
95+
<groupId>org.apache.spark</groupId>
96+
<artifactId>spark-sql_${scala.binary.version}</artifactId>
97+
<version>${spark.version}</version>
98+
<type>test-jar</type>
99+
<scope>test</scope>
100+
</dependency>
101+
102+
<dependency>
103+
<groupId>org.apache.hadoop</groupId>
104+
<artifactId>hadoop-client-runtime</artifactId>
105+
<scope>test</scope>
106+
</dependency>
107+
108+
<dependency>
109+
<groupId>javax.servlet</groupId>
110+
<artifactId>javax.servlet-api</artifactId>
111+
<scope>test</scope>
112+
</dependency>
113+
114+
<dependency>
115+
<groupId>jakarta.servlet</groupId>
116+
<artifactId>jakarta.servlet-api</artifactId>
117+
<scope>test</scope>
118+
</dependency>
119+
120+
<dependency>
121+
<groupId>org.apache.logging.log4j</groupId>
122+
<artifactId>log4j-slf4j-impl</artifactId>
123+
<scope>test</scope>
124+
</dependency>
125+
</dependencies>
126+
127+
<build>
128+
129+
<plugins>
130+
<plugin>
131+
<groupId>org.codehaus.mojo</groupId>
132+
<artifactId>build-helper-maven-plugin</artifactId>
133+
<executions>
134+
<execution>
135+
<id>regex-property</id>
136+
<goals>
137+
<goal>regex-property</goal>
138+
</goals>
139+
<configuration>
140+
<name>spark.home</name>
141+
<value>${project.basedir}/../../../externals/kyuubi-download/target/${spark.archive.name}</value>
142+
<regex>(.+)\.tgz</regex>
143+
<replacement>$1</replacement>
144+
</configuration>
145+
</execution>
146+
</executions>
147+
</plugin>
148+
<plugin>
149+
<groupId>org.antlr</groupId>
150+
<artifactId>antlr4-maven-plugin</artifactId>
151+
<configuration>
152+
<visitor>true</visitor>
153+
<sourceDirectory>${project.basedir}/src/main/antlr4</sourceDirectory>
154+
</configuration>
155+
</plugin>
156+
157+
<plugin>
158+
<groupId>org.apache.maven.plugins</groupId>
159+
<artifactId>maven-shade-plugin</artifactId>
160+
<configuration>
161+
<shadedArtifactAttached>false</shadedArtifactAttached>
162+
<artifactSet>
163+
<includes>
164+
<include>org.apache.kyuubi:*</include>
165+
</includes>
166+
</artifactSet>
167+
</configuration>
168+
<executions>
169+
<execution>
170+
<goals>
171+
<goal>shade</goal>
172+
</goals>
173+
<phase>package</phase>
174+
</execution>
175+
</executions>
176+
</plugin>
177+
<plugin>
178+
<groupId>org.scalatest</groupId>
179+
<artifactId>scalatest-maven-plugin</artifactId>
180+
<configuration>
181+
<environmentVariables>
182+
<!--
183+
Some tests run Spark in local-cluster mode, which uses SPARK_HOME and
184+
SPARK_SCALA_VERSION to build the command that launches a Spark Standalone
185+
cluster. SPARK_SCALA_VERSION is always derived from the build here;
186+
SPARK_HOME is provided by the spark-home-from-archive profile only when the
187+
environment does not already supply it.
188+
-->
189+
<SPARK_SCALA_VERSION>${scala.binary.version}</SPARK_SCALA_VERSION>
190+
</environmentVariables>
191+
</configuration>
192+
</plugin>
193+
</plugins>
194+
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
195+
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
196+
</build>
197+
<profiles>
198+
<profile>
199+
<!-- When SPARK_HOME is not provided by the environment, point it at the downloaded
200+
Spark archive for tests that launch a Spark Standalone cluster. -->
201+
<id>spark-home-from-archive</id>
202+
<activation>
203+
<property>
204+
<name>!env.SPARK_HOME</name>
205+
</property>
206+
</activation>
207+
<build>
208+
<plugins>
209+
<plugin>
210+
<groupId>org.scalatest</groupId>
211+
<artifactId>scalatest-maven-plugin</artifactId>
212+
<configuration>
213+
<environmentVariables>
214+
<SPARK_HOME>${spark.home}</SPARK_HOME>
215+
</environmentVariables>
216+
</configuration>
217+
</plugin>
218+
</plugins>
219+
</build>
220+
</profile>
221+
</profiles>
222+
</project>

0 commit comments

Comments
 (0)