feat(gradle-plugin): enhance Spark SQL prep and extension runtime management

- Updated Gradle plugin to support Spark SQL preparation inside Gradle JVM with enhanced logging and module access configuration.
- Added support for Iceberg Spark extensions and shaded runtime artifacts for improved dependency isolation.
- Introduced extension resource directories for better resource management.
- Enhanced extension runtime classpath configuration with added dependencies for AWS SDK, Iceberg AWS bundle, and servlet API.
- Updated documentation to reflect new plugin configurations, logging options, and resource handling.

Author: CoderYellow

doc/user-guide.adoc

@@ -1622,7 +1622,7 @@ If `hiveMetastore = true` fails for HMS-backed object-store tables, see link:hiv
 
 == Gradle Plugin
 
-Use the Gradle plugin when the test kit should be managed by Gradle instead of the application or test framework. This keeps container startup out of the Spring application context and out of JUnit annotations, while still starting the services before `bootRun`, `run`, `JavaExec`, or `Test` tasks launch their JVMs.
+Use the Gradle plugin when the test kit should be managed by Gradle instead of the application or test framework. This keeps container startup out of the Spring application context and out of JUnit annotations, while still starting the services before `bootRun`, `run`, or other `JavaExec` tasks launch their JVMs. `Test` task wiring is opt-in.
 
 [source,kotlin]
 ----
@@ -1689,7 +1689,11 @@ bigDataTest {
 
 Extension provisioning dependencies are resolved into the plugin-owned `bigDataTestExtensionRuntime` configuration. They are not added to the application `implementation`, `runtimeClasspath`, `testImplementation`, or `testRuntimeClasspath` configurations.
 
-By default the plugin adds the lightweight `extensions` artifact and then adds runtime libraries based on the configured extension TOML:
+By default the plugin uses the shaded `extensions:<version>:runtime` artifact. This keeps Spark, Hadoop, Kafka, Avro, thin Iceberg Spark modules, and their transitive runtime dependencies managed by bigdata-test instead of letting the application dependency graph choose those versions. The extension runtime classloader prefers its own jars and delegates shared bigdata-test core types to the Gradle plugin parent classloader.
+
+Application dependencies such as `implementation`, `runtimeOnly`, `testImplementation`, and `testRuntimeOnly` are not added to the bigdata-test extension runtime. Dependencies explicitly added to `bigDataTestExtensionRuntime` are different: they are intentionally part of the extension runtime and can affect extension execution. Use that configuration only for optional extension-only libraries that bigdata-test does not already provide.
+
+If you need to debug or override individual runtime dependencies, disable the shaded artifact. In that mode, the plugin adds the lightweight `extensions` artifact and then adds runtime libraries based on the configured extension TOML:
 
 * `[s3Jceks]` enables Hadoop client/runtime/AWS dependencies.
 * `[kafkaAvro]` enables Confluent Schema Registry, Kafka Avro, and Avro dependencies.
@@ -1701,6 +1705,7 @@ Override dependency versions with the DSL:
 ----
 bigDataTest {
     extensionRuntime {
+        useShadedArtifact.set(false)
         hadoopVersion.set("3.4.2")
         sparkVersion.set("3.5.7")
         confluentVersion.set("8.2.1")
@@ -1713,7 +1718,16 @@ bigDataTest {
 
 When Spark and Confluent Kafka dependencies are both present, the plugin selects `at.yawk.lz4:lz4-java` for Gradle's shared `org.lz4:lz4-java` capability. Override `lz4Version` if the selected Confluent line uses a different compatible lz4 artifact.
 
-For Hadoop S3A, the plugin excludes `software.amazon.awssdk:bundle` from `hadoop-aws` and adds the modular `software.amazon.awssdk:s3` dependency instead. This keeps the extension runtime focused on S3 support instead of shading every AWS service client.
+For Hadoop S3A, the plugin excludes `software.amazon.awssdk:bundle` from `hadoop-aws` and adds the modular `software.amazon.awssdk:s3` and `software.amazon.awssdk:s3-transfer-manager` dependencies instead. S3A loads transfer-manager classes at runtime, so both modules are required while still avoiding the full AWS SDK bundle.
+
+When `spark-sql-prep` is used by `bigDataTestStart` or `bigDataTestRun`, Spark starts inside the Gradle JVM. On Java 17, add Spark's required module access to the Gradle daemon JVM, for example in `gradle.properties`:
+
+[source,properties]
+----
+org.gradle.jvmargs=-Xmx4096M -Dfile.encoding=UTF-8 --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED
+----
+
+These flags cannot be applied after the Gradle daemon has already started. If a daemon was already running without them, stop it with `./gradlew --stop` or run the next command with `--no-daemon`.
 
 When auto-detection is not enough, force dependency groups explicitly:
 
@@ -1729,7 +1743,7 @@ bigDataTest {
 }
 ----
 
-The `extensions` module also publishes a shaded runtime artifact with classifier `runtime`. It bundles the extension runtime dependencies for users who prefer one plugin runtime artifact instead of dynamic dependency sets:
+The `extensions` module publishes the shaded runtime artifact with classifier `runtime`. This is the default plugin mode:
 
 [source,kotlin]
 ----
@@ -1741,7 +1755,75 @@ bigDataTest {
 }
 ----
 
-With `useShadedArtifact = true`, the plugin resolves `org.openprojectx.bigdata.test.core:extensions:<version>:runtime@jar` into `bigDataTestExtensionRuntime` and skips adding separate Hadoop, Spark, Kafka, and Avro extension runtime dependencies.
+With `useShadedArtifact = true`, the plugin resolves `org.openprojectx.bigdata.test.core:extensions:<version>:runtime@jar` into `bigDataTestExtensionRuntime` and skips adding separate Hadoop, Spark, Iceberg, Kafka, and Avro extension runtime dependencies.
+
+The normal `extensions` artifact does not package Spark, Hadoop, Kafka, Log4j, or SLF4J implementations; those dependencies are supplied by the plugin-owned runtime configuration when needed. The shaded `extensions:<version>:runtime` artifact does package extension runtime dependencies, including logging libraries pulled in transitively by Spark such as `slf4j-api`, `log4j-api`, `log4j-core`, and `log4j-slf4j2-impl`. Jackson is relocated inside this shaded artifact to avoid collisions with Jackson versions used by the application or Gradle build logic.
+
+=== Gradle Plugin Customization Surface
+
+The Gradle plugin has three separate classpath/resource concepts:
+
+[cols="1,2,2",options="header"]
+|===
+|Area |How to customize |Notes
+
+|Application classpath
+|Use normal Gradle configurations such as `implementation`, `runtimeOnly`, `testImplementation`, and `testRuntimeOnly`.
+|These dependencies are used by the app and tests. They are not added to `bigDataTestRun` or extension provisioning.
+
+|Extension runtime classpath
+|Use the `bigDataTestExtensionRuntime` configuration, or set `extensionRuntime.useShadedArtifact.set(false)` and configure `hadoopVersion`, `sparkVersion`, `icebergVersion`, `confluentVersion`, `avroVersion`, `awsSdkVersion`, and `lz4Version`.
+|This classpath is used only by bigdata-test extension execution. Dependencies added here can affect `spark-sql-prep`, JCEKS generation, Kafka Avro seeding, and other extensions.
+
+|Extension config and SQL/resource lookup
+|Use `extensionConfig.add(...)`. `classpath:` resources are searched from the extension runtime first and then from the project's main resource directories, including `src/main/resources` and compiled main resources.
+|There is no dedicated DSL property for extra arbitrary resource directories today. Put files under main resources, reference an absolute or relative `file:` path, or add a small resource jar to `bigDataTestExtensionRuntime`.
+
+|Gradle task JVM
+|Configure Gradle itself with `org.gradle.jvmargs`, `GRADLE_OPTS`, or `JAVA_TOOL_OPTIONS`.
+|`bigDataTestStart` and `bigDataTestRun` execute inside the Gradle JVM. Spark module-open flags and logging configuration for Spark SQL preparation must be visible when the Gradle daemon starts.
+
+|Gradle task environment
+|Set environment variables on the Gradle command, for example `TESTCONTAINERS_RYUK_DISABLED=true ./gradlew bigDataTestRun`.
+|`bigDataTestRun` is not a `JavaExec` task, so it does not have a task-local `environment {}` API. It sees the Gradle process environment.
+
+|Application task injected properties
+|Set `injectRawEndpointProperties`, `injectNamespacedEndpointProperties`, and `injectEnvironmentVariables`.
+|For auto-wired `JavaExec` and opt-in `Test` tasks, endpoint and extension output values are injected as JVM system properties and, when enabled, environment variables.
+
+|Container environment, files, mounts, ports, network mode
+|Use startup TOML `[containers.<service>.*]` tables or programmatic `withContainerCustomization(...)`.
+|This customizes Docker containers, not the Gradle task process. Prefer copied files over host mounts in CI.
+|===
+
+Example extension-only dependency override:
+
+[source,kotlin]
+----
+dependencies {
+    add("bigDataTestExtensionRuntime", "com.example:test-only-extension-helper:1.0.0")
+}
+----
+
+Example task process environment:
+
+[source,bash]
+----
+TESTCONTAINERS_RYUK_DISABLED=true GRADLE_USER_HOME=/data/.gradle ./gradlew bigDataTestRun
+----
+
+Example resource-backed Spark SQL preparation:
+
+[source,toml]
+----
+[[extensions]]
+type = "spark-sql-prep"
+scripts = ["classpath:sql/create_tables.sql"]
+----
+
+Place the script at `src/main/resources/sql/create_tables.sql`, or use a `file:` URL when the SQL lives outside the project resources.
+
+The shaded runtime jar intentionally favors isolation over small size. It uses the thin Iceberg Spark modules instead of the fat `iceberg-spark-runtime` jar, but still includes enough Hadoop, Spark, Iceberg, Kafka Avro, and AWS runtime code for config-driven provisioning without borrowing the user application's dependency graph. The largest remaining optional payload is `iceberg-aws-bundle`, which is kept for Iceberg S3 FileIO and AWS catalog support. If size or dependency inspection matters more than isolation, set `useShadedArtifact = false` and let the plugin resolve only the runtime families detected from `extensionConfig`, or add the exact artifacts you want to `bigDataTestExtensionRuntime`.
 
 Gradle configuration is Gradle-native; the plugin does not read a Maven POM. When values should come from `gradle.properties` or `-P`, wire them through Gradle providers:
 
@@ -1763,11 +1845,11 @@ bigDataTest {
 
 The plugin registers:
 
-* `bigDataTestStart` - starts the configured kit for the current Gradle build and then returns. This task is intended as a dependency of app/test tasks.
+* `bigDataTestStart` - starts the configured kit for the current Gradle build and then returns. This task is intended as a dependency of app tasks, or test tasks when explicitly opted in.
 * `bigDataTestRun` - starts the configured kit and keeps the Gradle process running until interrupted. Use this for manual troubleshooting.
 * `bigDataTestStop` - explicitly stops the kit in the current Gradle build.
 
-By default, the plugin wires `bigDataTestStart` before `JavaExec` and `Test` tasks and injects endpoint and extension output properties as JVM system properties immediately before the task launches. It also injects environment variables derived from the same property map. The kit is closed automatically when the Gradle build service closes at the end of the build.
+By default, the plugin wires `bigDataTestStart` before `JavaExec` tasks and injects endpoint and extension output properties as JVM system properties immediately before the task launches. It also injects environment variables derived from the same property map. `Test` tasks are not auto-wired by default; set `autoConfigureTestTasks.set(true)` when tests should start and receive the kit automatically. The kit is closed automatically when the Gradle build service closes at the end of the build.
 
 `bigDataTestStart` is not an interactive keep-alive command. If you run it directly, it starts the kit, prints the injected property count, and exits when the build finishes. For a long-running local environment, run:
 
@@ -1783,6 +1865,51 @@ Testcontainers runtime environment variables, such as `TESTCONTAINERS_RYUK_DISAB
 TESTCONTAINERS_RYUK_DISABLED=true GRADLE_USER_HOME=/data/.gradle ./gradlew --no-daemon bigDataTestRun
 ----
 
+=== Gradle Plugin Logging
+
+`bigDataTestRun` starts the kit and extension provisioning inside the Gradle JVM. Configure JVM logging through Gradle daemon JVM properties, not through the application `bootRun` or `test` task.
+
+For the default plugin logging backend, use SLF4J Simple properties in `gradle.properties`:
+
+[source,properties]
+----
+org.gradle.jvmargs=-Xmx4096M -Dfile.encoding=UTF-8 \
+  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED \
+  --add-opens=java.base/java.nio=ALL-UNNAMED \
+  --add-opens=java.base/java.net=ALL-UNNAMED \
+  -Dorg.slf4j.simpleLogger.defaultLogLevel=info \
+  -Dorg.slf4j.simpleLogger.log.org.openprojectx.bigdata.test=debug \
+  -Dorg.slf4j.simpleLogger.log.org.testcontainers=info
+----
+
+Spark SQL preparation is different when it runs through the Gradle plugin: Spark starts in-process inside the Gradle daemon, and Spark logs are routed through Gradle's logging bridge. Gradle controls the console rendering, so Log4j2 console pattern and ANSI color settings are not honored for those Spark extension logs. Use Gradle flags such as `--info` and extension/Spark log-level settings for verbosity. Full Log4j2 pattern/color control would require a separate Spark preparation JVM instead of the current in-process plugin execution model.
+
+If a Gradle daemon is already running, restart it before expecting new `org.gradle.jvmargs` values to apply:
+
+[source,bash]
+----
+./gradlew --stop
+GRADLE_USER_HOME=/data/.gradle ./gradlew bigDataTestRun --info
+----
+
+Container logs are configured separately from Gradle JVM logs. Use startup TOML when you want service stdout/stderr in files:
+
+[source,toml]
+----
+[containerLogs]
+mode = "FILE"
+directory = "build/container-logs"
+append = false
+----
+
+or send container logs to the Gradle console:
+
+[source,toml]
+----
+[containerLogs]
+mode = "STDOUT"
+----
+
 Disable automatic task wiring when you want explicit task dependencies:
 
 [source,kotlin]

example/spark/build.gradle.kts

@@ -97,9 +97,11 @@ val apacheSparkRuntimeClasspath = createSparkRuntimeClasspath("apacheSparkRuntim
 val clouderaSparkRuntimeClasspath = createSparkRuntimeClasspath("clouderaSparkRuntimeClasspath", "cloudera")
 
 dependencies {
-    add(apacheSparkRuntimeClasspath.name, "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:$icebergVersion")
+    add(apacheSparkRuntimeClasspath.name, "org.apache.iceberg:iceberg-spark-3.5_2.12:$icebergVersion")
+    add(apacheSparkRuntimeClasspath.name, "org.apache.iceberg:iceberg-spark-extensions-3.5_2.12:$icebergVersion")
     add(apacheSparkRuntimeClasspath.name, "org.apache.logging.log4j:log4j-slf4j-impl:2.20.0")
-    add(clouderaSparkRuntimeClasspath.name, "org.apache.iceberg:iceberg-spark-runtime-3.3_2.12")
+    add(clouderaSparkRuntimeClasspath.name, "org.apache.iceberg:iceberg-spark-3.3_2.12")
+    add(clouderaSparkRuntimeClasspath.name, "org.apache.iceberg:iceberg-spark-extensions-3.3_2.12")
 }
 
 fun Test.useSparkRuntimeClasspath(runtimeClasspath: Configuration, dependencyLine: String) {

example/spring-gradle-plugin/README.md

@@ -33,3 +33,9 @@ Spring JVM, runs the TOML extensions, and injects endpoint/extension output as J
 properties and environment variables. The application reads the JVM properties from
 `application-local.yaml`; non-Spring code can read equivalent env vars such as
 `BIGDATA_TEST_ENDPOINT_HDFS_HOST`.
+
+Spark SQL preparation runs inside the Gradle JVM. In this in-process mode, Spark logs are routed
+through Gradle's logging bridge, so Log4j2 console pattern and color settings are not honored for
+Spark extension logs. Use Gradle logging flags such as `--info` and the extension/Spark log-level
+settings for verbosity. Full pattern/color control would require running Spark preparation in an
+isolated JVM instead of inside the Gradle daemon.

example/spring-gradle-plugin/gradle.properties

@@ -1 +1 @@
-org.gradle.jvmargs=-Xmx4096M -Dfile.encoding=UTF-8
+org.gradle.jvmargs=-Xmx4096M -Dfile.encoding=UTF-8 --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED

example/spring-gradle-plugin/src/main/resources/spring-bigdata-extensions.toml

@@ -8,3 +8,21 @@ fileName = "s3.jceks"
 
 [[s3Buckets]]
 bucket = "spring-example"
+
+[[extensions]]
+type = "spark-sql-prep"
+id = "prepare-iceberg-s3"
+appName = "spring-gradle-plugin-iceberg-prep"
+enableHiveSupport = false
+statements = [
+    "CREATE NAMESPACE IF NOT EXISTS spring_s3.demo",
+    "CREATE TABLE IF NOT EXISTS spring_s3.demo.events (id BIGINT, name STRING) USING iceberg",
+    "INSERT INTO spring_s3.demo.events VALUES (1, 'spring-gradle-plugin')",
+]
+
+[extensions.configs]
+"spark.sql.extensions" = "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"
+"spark.sql.catalog.spring_s3" = "org.apache.iceberg.spark.SparkCatalog"
+"spark.sql.catalog.spring_s3.type" = "hadoop"
+"spark.sql.catalog.spring_s3.warehouse" = "s3a://spring-example/iceberg"
+"spark.sql.shuffle.partitions" = "1"

extensions/build.gradle.kts

@@ -10,6 +10,12 @@ description = "Config-driven bigdata-test extensions for Hadoop credential provi
 val shadedRuntime by configurations.creating {
     isCanBeConsumed = false
     isCanBeResolved = true
+    resolutionStrategy.eachDependency {
+        if (requested.group.startsWith("com.fasterxml.jackson")) {
+            useVersion("2.15.2")
+            because("Spark 3.5.x expects Jackson 2.15.x at runtime.")
+        }
+    }
     resolutionStrategy.capabilitiesResolution.withCapability("org.lz4:lz4-java") {
         select("at.yawk.lz4:lz4-java:${libs.versions.lz4.get()}")
         because("Kafka/Confluent and Spark publish different providers for the same lz4 capability.")
@@ -36,6 +42,7 @@ dependencies {
     compileOnly(libs.avro)
     compileOnly(libs.sparkSql)
     compileOnly(libs.sparkHive)
+    compileOnly(libs.servletApi)
     implementation(libs.kotlinxSerialization)
 
     shadedRuntime(libs.hadoopClientApi)
@@ -44,12 +51,21 @@ dependencies {
         exclude(group = "software.amazon.awssdk", module = "bundle")
     }
     shadedRuntime(libs.awsSdkS3)
+    shadedRuntime(libs.awsSdkS3TransferManager)
     shadedRuntime(libs.kafkaAvroSerializer)
     shadedRuntime(libs.kafkaSchemaRegistryClient)
     shadedRuntime(libs.lz4Java)
     shadedRuntime(libs.avro)
-    shadedRuntime(libs.sparkSql)
-    shadedRuntime(libs.sparkHive)
+    shadedRuntime(libs.sparkSql) {
+        exclude(group = "org.rocksdb", module = "rocksdbjni")
+    }
+    shadedRuntime(libs.sparkHive) {
+        exclude(group = "org.rocksdb", module = "rocksdbjni")
+    }
+    shadedRuntime(libs.icebergSpark)
+    shadedRuntime(libs.icebergSparkExtensions)
+    shadedRuntime(libs.icebergAwsBundle)
+    shadedRuntime(libs.servletApi)
     shadedRuntime(libs.kotlinxSerialization)
 
     testImplementation(libs.junitJupiterApi)
@@ -70,6 +86,7 @@ tasks.named<com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar>("shadowJ
     configurations = listOf(shadedRuntime)
     isZip64 = true
     mergeServiceFiles()
+    relocate("com.fasterxml.jackson", "org.openprojectx.bigdata.test.shaded.fasterxml.jackson")
 }
 
 configurations.matching { it.name.startsWith("test") }.configureEach {

extensions/src/main/kotlin/org/openprojectx/bigdata/test/extensions/core/BigDataExtensionResourceLoader.kt

@@ -5,12 +5,21 @@ import java.nio.file.Path
 
 class BigDataExtensionResourceLoader(
     private val classLoader: ClassLoader = Thread.currentThread().contextClassLoader,
+    private val resourceDirectories: Iterable<Path> = emptyList(),
 ) {
     fun readText(location: String): String = when {
         location.startsWith("classpath:") -> {
             val resource = location.removePrefix("classpath:").trimStart('/')
             classLoader.getResource(resource)?.readText()
-                ?: error("Classpath resource '$resource' was not found")
+                ?: resourceDirectories
+                    .asSequence()
+                    .map { it.resolve(resource) }
+                    .firstOrNull(Files::isRegularFile)
+                    ?.let(Files::readString)
+                ?: error(
+                    "Classpath resource '$resource' was not found in the extension runtime classpath " +
+                        "or configured resource directories: ${resourceDirectories.joinToString()}",
+                )
         }
         location.startsWith("file:") -> Files.readString(Path.of(location.removePrefix("file:")))
         else -> location