feat: extend Spark dependency matrix with Cloudera support and task aliases

- Added Cloudera Spark/Hadoop dependency line to test matrix with resolvable runtime classpaths.
- Introduced compatibility task aliases for legacy HMS test tasks.
- Updated documentation to reflect matrix structure, task usage, and runtime configuration details.
- Enhanced test setups with uncompressed Parquet default and improved HiveMetastoreClient compatibility.

Author: CoderYellow

CONTRIBUTING.md

@@ -134,6 +134,8 @@ Run the full Spark matrix when touching runtime config composition, HMS selectio
 GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkBigDataMatrixTest
 ```
 
+The Spark matrix has two kinds of axes. Container/service axes belong in TOML and can be selected with `bigdata.test.config`. Dependency axes belong in Gradle tasks because they require different resolved test JVM classpaths. Add a new dependency line by creating a resolvable runtime classpath in `example/spark/build.gradle.kts`, then pair it with the existing TOML service variants.
+
 The Gradle configuration-cache warning from `net.researchgate.release` is currently expected and unrelated to test behavior.
 
 ## Troubleshooting During Development

README.md

@@ -79,19 +79,23 @@ Run the Spark smoke test:
 GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:test
 ```
 
-Run the Spark HMS/Kerberos matrix:
+Run the Spark dependency/HMS/Kerberos matrix:
 
 ```bash
 GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkBigDataMatrixTest
 ```
 
-Individual matrix cells are also available:
+Individual matrix cells are also available. The first axis selects the Spark/Hadoop dependency line, then HMS implementation, then Kerberos:
 
 ```bash
-GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkApacheHmsTest
-GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkApacheHmsKerberosTest
-GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkClouderaHmsTest
-GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkClouderaHmsKerberosTest
+GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkApacheDepsApacheHmsTest
+GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkApacheDepsApacheHmsKerberosTest
+GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkApacheDepsClouderaHmsTest
+GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkApacheDepsClouderaHmsKerberosTest
+GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkClouderaDepsApacheHmsTest
+GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkClouderaDepsApacheHmsKerberosTest
+GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkClouderaDepsClouderaHmsTest
+GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkClouderaDepsClouderaHmsKerberosTest
 ```
 
 ## Documentation

doc/user-guide.adoc

@@ -245,6 +245,8 @@ When `bigdata.test.config.replace=true`, task config replaces startup config fro
 
 Extension config has the same launcher hook: `bigdata.extensions.config` and `bigdata.extensions.config.replace`.
 
+Container data-driven tests can be TOML-driven because service images, ports, Kerberos flags, and extension inputs are runtime configuration. Dependency data-driven tests must be Gradle-task/classpath-driven because Spark, Hadoop, Iceberg, and vendor distributions are resolved before the test JVM starts. The Spark example uses separate resolvable runtime classpaths for Apache and Cloudera dependency lines, then pairs those classpaths with TOML service variants.
+
 `hiveMetastore` is the open-source HMS image and defaults to `ghcr.io/openprojectx/hive:3.1.3-hadoop-3.4.2-gcs-4.0.4-jdk17-0.1.4`. Hive 4 images can be tested by overriding this image key, but Spark 3.x brings a Hive 2.3 metastore client and should use the Hive 3 image unless that client stack is changed. The open-source HMS path starts an external PostgreSQL support container using `hiveMetastorePostgres`.
 
 `clouderaHms` is the embedded-Postgres Cloudera HMS image. Use the `clouderaHms = true` annotation flag when you want this implementation. A test class must not enable both `hiveMetastore` and `clouderaHms`.
@@ -801,18 +803,74 @@ Run it with:
 GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:test
 ----
 
-The Spark example also has Gradle tasks for the HMS/Kerberos matrix:
+The Spark example also has Gradle tasks for a dependency/HMS/Kerberos matrix:
+
+[source,bash]
+----
+GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkBigDataMatrixTest
+----
+
+The aggregate task runs these cells:
+
+|===
+|Task |Dependency line |HMS |Kerberos
+
+|`sparkApacheDepsApacheHmsTest`
+|Apache Spark/Hadoop
+|Open-source Hive 3 HMS
+|No
+
+|`sparkApacheDepsApacheHmsKerberosTest`
+|Apache Spark/Hadoop
+|Open-source Hive 3 HMS
+|Yes
+
+|`sparkApacheDepsClouderaHmsTest`
+|Apache Spark/Hadoop
+|Cloudera HMS
+|No
+
+|`sparkApacheDepsClouderaHmsKerberosTest`
+|Apache Spark/Hadoop
+|Cloudera HMS
+|Yes
+
+|`sparkClouderaDepsApacheHmsTest`
+|Cloudera Spark/Hadoop
+|Open-source Hive 3 HMS
+|No
+
+|`sparkClouderaDepsApacheHmsKerberosTest`
+|Cloudera Spark/Hadoop
+|Open-source Hive 3 HMS
+|Yes
+
+|`sparkClouderaDepsClouderaHmsTest`
+|Cloudera Spark/Hadoop
+|Cloudera HMS
+|No
+
+|`sparkClouderaDepsClouderaHmsKerberosTest`
+|Cloudera Spark/Hadoop
+|Cloudera HMS
+|Yes
+|===
+
+The Kerberos axis currently enables the shared KDC and Kafka Kerberos path used by the Spark Kafka source. The HMS axis switches between the open-source Hive 3 HMS image and the Cloudera HMS image. The dependency axis switches the test JVM runtime classpath; it is intentionally implemented in Gradle rather than TOML.
+
+Legacy HMS-only task aliases are still present:
 
 [source,bash]
 ----
 GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkApacheHmsTest
 GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkApacheHmsKerberosTest
 GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkClouderaHmsTest
 GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkClouderaHmsKerberosTest
-GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkBigDataMatrixTest
 ----
 
-The Kerberos axis currently enables the shared KDC and Kafka Kerberos path used by the Spark Kafka source. The HMS axis switches between the open-source Hive 3 HMS image and the Cloudera HMS image.
+The `sparkClouderaHmsTest` and `sparkClouderaHmsKerberosTest` aliases select the Cloudera dependency line and Cloudera HMS. Use `sparkApacheDepsClouderaHmsTest` when you specifically want Apache Spark/Hadoop dependencies against Cloudera HMS.
+
+Spark example test tasks always execute instead of reusing previous test outputs, because these are container smoke tests. The shared Spark TOML writes service container logs to `example/spark/build/bigdata-test-container-logs`; Gradle's test output contains the test JVM logs and points to the HTML test report on failure.
 
 == Troubleshooting
 

example/spark/build.gradle.kts

@@ -1,4 +1,8 @@
-import org.gradle.api.artifacts.component.ModuleComponentIdentifier
+import org.gradle.api.artifacts.Configuration
+import org.gradle.api.attributes.Bundling
+import org.gradle.api.attributes.Category
+import org.gradle.api.attributes.LibraryElements
+import org.gradle.api.attributes.Usage
 
 plugins {
     id("buildsrc.convention.kotlin-jvm")
@@ -8,7 +12,7 @@ plugins {
 description = "Spark JUnit 5 example for bigdata-test"
 
 sparkPlatform {
-    line.set("spark3")
+    line.set("cloudera")
     variants.set(listOf("iceberg"))
     addons.set(
         listOf(
@@ -28,7 +32,6 @@ dependencies {
     testImplementation("org.apache.spark:spark-sql-kafka-0-10_2.12")
     testImplementation("org.apache.spark:spark-avro_2.12")
     testImplementation("org.apache.hadoop:hadoop-aws")
-    testImplementation("org.apache.iceberg:iceberg-spark-runtime-3.5_2.12")
     testImplementation("org.apache.iceberg:iceberg-aws-bundle")
     testImplementation("com.google.cloud.bigdataoss:gcs-connector")
     testImplementation("com.google.cloud.bigdataoss:gcsio")
@@ -38,8 +41,50 @@ dependencies {
     testRuntimeOnly(libs.junitPlatformLauncher)
 }
 
+val apacheSparkVersion = libs.versions.spark.get()
+val apacheHadoopVersion = libs.versions.hadoop.get()
+val icebergVersion = libs.versions.iceberg.get()
+val testImplementationConfiguration = configurations.named("testImplementation")
+val testRuntimeOnlyConfiguration = configurations.named("testRuntimeOnly")
+
+fun createSparkRuntimeClasspath(name: String, dependencyLine: String) = configurations.create(name) {
+    isCanBeConsumed = false
+    isCanBeResolved = true
+    extendsFrom(testImplementationConfiguration.get(), testRuntimeOnlyConfiguration.get())
+
+    attributes {
+        attribute(Usage.USAGE_ATTRIBUTE, objects.named(Usage::class, Usage.JAVA_RUNTIME))
+        attribute(Category.CATEGORY_ATTRIBUTE, objects.named(Category::class, Category.LIBRARY))
+        attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, objects.named(LibraryElements::class, LibraryElements.JAR))
+        attribute(Bundling.BUNDLING_ATTRIBUTE, objects.named(Bundling::class, Bundling.EXTERNAL))
+    }
+
+    if (dependencyLine == "apache") {
+        resolutionStrategy.eachDependency {
+            when (requested.group) {
+                "org.apache.spark" -> useVersion(apacheSparkVersion)
+                "org.apache.hadoop" -> useVersion(apacheHadoopVersion)
+            }
+        }
+    }
+}
+
+val apacheSparkRuntimeClasspath = createSparkRuntimeClasspath("apacheSparkRuntimeClasspath", "apache")
+val clouderaSparkRuntimeClasspath = createSparkRuntimeClasspath("clouderaSparkRuntimeClasspath", "cloudera")
+
+dependencies {
+    add(apacheSparkRuntimeClasspath.name, "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:$icebergVersion")
+    add(apacheSparkRuntimeClasspath.name, "org.apache.logging.log4j:log4j-slf4j-impl:2.20.0")
+    add(clouderaSparkRuntimeClasspath.name, "org.apache.iceberg:iceberg-spark-runtime-3.3_2.12")
+}
+
+fun Test.useSparkRuntimeClasspath(runtimeClasspath: Configuration, dependencyLine: String) {
+    classpath = sourceSets.test.get().output + sourceSets.main.get().output + runtimeClasspath
+    systemProperty("bigdata.spark.dependency.line", dependencyLine)
+}
 
 tasks.withType<Test>().configureEach {
+    outputs.upToDateWhen { false }
     minHeapSize = "2048m"
     maxHeapSize = "8192m"
     jvmArgs(
@@ -49,64 +94,143 @@ tasks.withType<Test>().configureEach {
     )
 }
 
+tasks.named<Test>("test") {
+    useSparkRuntimeClasspath(clouderaSparkRuntimeClasspath, "cloudera")
+}
+
 val sparkBigDataTestClass = "org.openprojectx.bigdata.test.example.spark.SparkBigDataTestExample"
 val sparkCommonConfig = "classpath:spark-bigdata-test-common.toml"
 
 fun registerSparkMatrixTest(
     name: String,
     descriptionText: String,
+    dependencyLine: String,
+    runtimeClasspath: Configuration,
     variantConfig: String,
 ) = tasks.register<Test>(name) {
     description = descriptionText
     group = "verification"
     testClassesDirs = sourceSets.test.get().output.classesDirs
-    classpath = sourceSets.test.get().runtimeClasspath
+    useSparkRuntimeClasspath(runtimeClasspath, dependencyLine)
     useJUnitPlatform()
     filter.includeTestsMatching(sparkBigDataTestClass)
     systemProperty("bigdata.test.config.replace", "true")
     systemProperty("bigdata.test.config", "$sparkCommonConfig,$variantConfig")
 }
 
-val sparkApacheHmsTest = registerSparkMatrixTest(
-    name = "sparkApacheHmsTest",
-    descriptionText = "Runs the Spark example with open-source Hive 3 HMS and plaintext Kafka.",
+val sparkApacheDepsApacheHmsTest = registerSparkMatrixTest(
+    name = "sparkApacheDepsApacheHmsTest",
+    descriptionText = "Runs the Spark example with Apache Spark/Hadoop deps, open-source Hive 3 HMS, and plaintext Kafka.",
+    dependencyLine = "apache",
+    runtimeClasspath = apacheSparkRuntimeClasspath,
     variantConfig = "classpath:spark-bigdata-test-apache-hms.toml",
 )
 
-val sparkApacheHmsKerberosTest = registerSparkMatrixTest(
-    name = "sparkApacheHmsKerberosTest",
-    descriptionText = "Runs the Spark example with open-source Hive 3 HMS and Kafka Kerberos.",
+val sparkApacheDepsApacheHmsKerberosTest = registerSparkMatrixTest(
+    name = "sparkApacheDepsApacheHmsKerberosTest",
+    descriptionText = "Runs the Spark example with Apache Spark/Hadoop deps, open-source Hive 3 HMS, and Kafka Kerberos.",
+    dependencyLine = "apache",
+    runtimeClasspath = apacheSparkRuntimeClasspath,
     variantConfig = "classpath:spark-bigdata-test-apache-hms-kerberos.toml",
 )
-sparkApacheHmsKerberosTest.configure {
-    mustRunAfter(sparkApacheHmsTest)
-}
 
-val sparkClouderaHmsTest = registerSparkMatrixTest(
-    name = "sparkClouderaHmsTest",
-    descriptionText = "Runs the Spark example with Cloudera HMS and plaintext Kafka.",
+val sparkApacheDepsClouderaHmsTest = registerSparkMatrixTest(
+    name = "sparkApacheDepsClouderaHmsTest",
+    descriptionText = "Runs the Spark example with Apache Spark/Hadoop deps, Cloudera HMS, and plaintext Kafka.",
+    dependencyLine = "apache",
+    runtimeClasspath = apacheSparkRuntimeClasspath,
     variantConfig = "classpath:spark-bigdata-test-cloudera-hms.toml",
 )
-sparkClouderaHmsTest.configure {
-    mustRunAfter(sparkApacheHmsKerberosTest)
-}
 
-val sparkClouderaHmsKerberosTest = registerSparkMatrixTest(
-    name = "sparkClouderaHmsKerberosTest",
-    descriptionText = "Runs the Spark example with Cloudera HMS and Kafka Kerberos.",
+val sparkApacheDepsClouderaHmsKerberosTest = registerSparkMatrixTest(
+    name = "sparkApacheDepsClouderaHmsKerberosTest",
+    descriptionText = "Runs the Spark example with Apache Spark/Hadoop deps, Cloudera HMS, and Kafka Kerberos.",
+    dependencyLine = "apache",
+    runtimeClasspath = apacheSparkRuntimeClasspath,
     variantConfig = "classpath:spark-bigdata-test-cloudera-hms-kerberos.toml",
 )
-sparkClouderaHmsKerberosTest.configure {
-    mustRunAfter(sparkClouderaHmsTest)
+
+val sparkClouderaDepsApacheHmsTest = registerSparkMatrixTest(
+    name = "sparkClouderaDepsApacheHmsTest",
+    descriptionText = "Runs the Spark example with Cloudera Spark/Hadoop deps, open-source Hive 3 HMS, and plaintext Kafka.",
+    dependencyLine = "cloudera",
+    runtimeClasspath = clouderaSparkRuntimeClasspath,
+    variantConfig = "classpath:spark-bigdata-test-apache-hms.toml",
+)
+
+val sparkClouderaDepsApacheHmsKerberosTest = registerSparkMatrixTest(
+    name = "sparkClouderaDepsApacheHmsKerberosTest",
+    descriptionText = "Runs the Spark example with Cloudera Spark/Hadoop deps, open-source Hive 3 HMS, and Kafka Kerberos.",
+    dependencyLine = "cloudera",
+    runtimeClasspath = clouderaSparkRuntimeClasspath,
+    variantConfig = "classpath:spark-bigdata-test-apache-hms-kerberos.toml",
+)
+
+val sparkClouderaDepsClouderaHmsTest = registerSparkMatrixTest(
+    name = "sparkClouderaDepsClouderaHmsTest",
+    descriptionText = "Runs the Spark example with Cloudera Spark/Hadoop deps, Cloudera HMS, and plaintext Kafka.",
+    dependencyLine = "cloudera",
+    runtimeClasspath = clouderaSparkRuntimeClasspath,
+    variantConfig = "classpath:spark-bigdata-test-cloudera-hms.toml",
+)
+
+val sparkClouderaDepsClouderaHmsKerberosTest = registerSparkMatrixTest(
+    name = "sparkClouderaDepsClouderaHmsKerberosTest",
+    descriptionText = "Runs the Spark example with Cloudera Spark/Hadoop deps, Cloudera HMS, and Kafka Kerberos.",
+    dependencyLine = "cloudera",
+    runtimeClasspath = clouderaSparkRuntimeClasspath,
+    variantConfig = "classpath:spark-bigdata-test-cloudera-hms-kerberos.toml",
+)
+
+listOf(
+    sparkApacheDepsApacheHmsKerberosTest to sparkApacheDepsApacheHmsTest,
+    sparkApacheDepsClouderaHmsTest to sparkApacheDepsApacheHmsKerberosTest,
+    sparkApacheDepsClouderaHmsKerberosTest to sparkApacheDepsClouderaHmsTest,
+    sparkClouderaDepsApacheHmsTest to sparkApacheDepsClouderaHmsKerberosTest,
+    sparkClouderaDepsApacheHmsKerberosTest to sparkClouderaDepsApacheHmsTest,
+    sparkClouderaDepsClouderaHmsTest to sparkClouderaDepsApacheHmsKerberosTest,
+    sparkClouderaDepsClouderaHmsKerberosTest to sparkClouderaDepsClouderaHmsTest,
+).forEach { (task, predecessor) ->
+    task.configure {
+        mustRunAfter(predecessor)
+    }
 }
 
 tasks.register("sparkBigDataMatrixTest") {
-    description = "Runs all Spark HMS/Kerberos matrix combinations."
+    description = "Runs all Spark dependency/HMS/Kerberos matrix combinations."
     group = "verification"
     dependsOn(
-        sparkApacheHmsTest,
-        sparkApacheHmsKerberosTest,
-        sparkClouderaHmsTest,
-        sparkClouderaHmsKerberosTest,
+        sparkApacheDepsApacheHmsTest,
+        sparkApacheDepsApacheHmsKerberosTest,
+        sparkApacheDepsClouderaHmsTest,
+        sparkApacheDepsClouderaHmsKerberosTest,
+        sparkClouderaDepsApacheHmsTest,
+        sparkClouderaDepsApacheHmsKerberosTest,
+        sparkClouderaDepsClouderaHmsTest,
+        sparkClouderaDepsClouderaHmsKerberosTest,
     )
 }
+
+tasks.register("sparkApacheHmsTest") {
+    description = "Compatibility alias for sparkApacheDepsApacheHmsTest."
+    group = "verification"
+    dependsOn(sparkApacheDepsApacheHmsTest)
+}
+
+tasks.register("sparkApacheHmsKerberosTest") {
+    description = "Compatibility alias for sparkApacheDepsApacheHmsKerberosTest."
+    group = "verification"
+    dependsOn(sparkApacheDepsApacheHmsKerberosTest)
+}
+
+tasks.register("sparkClouderaHmsTest") {
+    description = "Compatibility alias for sparkClouderaDepsClouderaHmsTest."
+    group = "verification"
+    dependsOn(sparkClouderaDepsClouderaHmsTest)
+}
+
+tasks.register("sparkClouderaHmsKerberosTest") {
+    description = "Compatibility alias for sparkClouderaDepsClouderaHmsKerberosTest."
+    group = "verification"
+    dependsOn(sparkClouderaDepsClouderaHmsKerberosTest)
+}