user-guide.adoc

bigdata-test User Guide

Table of Contents

• 1. Overview
  • 1.1. S3 naming migration
• 2. Module Layout
• 3. Dependency Management
• 4. Publishing Local Snapshots
• 5. Programmatic Usage
  • 5.1. Service Endpoints
• 6. JUnit 5 Usage
  • 6.1. Service Flags
  • 6.2. Service Image Configuration
  • 6.3. Container Customization Escape Hatch
    • 6.3.1. HDFS Environment Variable Names
  • 6.4. Startup Health Checks
  • 6.5. HTTP TLS Gateway
    • 6.5.1. Stable CA Files
    • 6.5.2. JVM Truststore Handling
    • 6.5.3. Importing the CA into a JDK
  • 6.6. Dynamic and Fixed Ports
  • 6.7. Service Endpoint Reference
• 7. Container Logs
  • 7.1. Programmatic Log Routing
  • 7.2. JUnit Log Routing
• 8. Kerberos
• 9. Extensions Module
  • 9.1. JUnit Extension Declaration
  • 9.2. Programmatic Extension Declaration
  • 9.3. Object Store Bucket Extensions
  • 9.4. Object Store Upload Extensions
  • 9.5. S3 JCEKS Extension
  • 9.6. Kafka Avro Seed Extension
  • 9.7. Spark SQL Preparation Extension
  • 9.8. Extension Lifecycle
• 10. Hive Metastore and Object Stores
  • 10.1. Spark Datasource Tables
  • 10.2. Hive External Tables
  • 10.3. HMS Server-Side S3A
  • 10.4. HMS Server-Side HDFS
  • 10.5. HMS Server-Side GCS
• 11. Gradle Plugin
  • 11.1. Gradle Plugin Customization Surface
  • 11.2. Gradle Plugin Logging
• 12. Spring Boot
• 13. Example Projects
  • 13.1. JUnit Example
  • 13.2. Spring Example
  • 13.3. Spark Example
• 14. Troubleshooting
  • 14.1. HADOOP_HOME and hadoop.home.dir are unset
  • 14.2. S3 Hive External Table Fails with Credential Errors
  • 14.3. GCS Hive External Table Fails with ClassNotFoundException
  • 14.4. Container Startup Failures

1. Overview

bigdata-test provides composable Testcontainers fixtures for local big-data integration tests. It is intended for tests that need real service protocols without maintaining a long-lived development cluster.

The project currently provides containers and test integration for:

• HDFS

• Open-source Hive Metastore backed by PostgreSQL or MySQL

• Cloudera HMS with embedded PostgreSQL

• Kafka

• Schema Registry

• Kafka UI

• Floci S3-compatible AWS emulator

• Floci GCP-compatible GCS emulator

• Kerberos KDC and service principals

The project is split so lightweight users can depend on core or junit5 without pulling Spark, Hadoop client utilities, or storage SDKs. Heavier setup such as JCEKS creation, Kafka Avro seeding, and Spark SQL preparation lives in the extensions module.

1.1. S3 naming migration

LocalStack support and its legacy naming have been removed. The S3 emulator is Floci, and all configuration and API names now describe the S3 service directly:

Previous name	Current name
localStackS3	s3
localStackS3Tls	s3Tls
BigDataService.LOCALSTACK_S3	BigDataService.S3
withLocalStackS3()	withS3()
Endpoint port edge	Endpoint port http

The old names are not retained as aliases. Update annotations, TOML, Gradle DSL, Spring properties, container customization keys, and endpoint property references together.

2. Module Layout

Module	Purpose
core	Container builder, endpoint model, service options, log routing, and service startup.
junit5	JUnit 5 annotation and parameter injection for BigDataTestKit.
extensions	Config-driven lifecycle extensions. Current built-ins create S3 JCEKS files, create object-store buckets, seed Kafka Avro records, and run Spark SQL preparation.
gradle-plugin	Gradle plugin that starts BigDataTestKit in the Gradle daemon and injects endpoint properties into app and test JVMs.
bigdata-test-spring-boot-autoconfigure	Spring Boot auto-configuration that starts a BigDataTestKit from application properties.
bigdata-test-spring-boot-starter	Starter artifact that brings in the Spring Boot auto-configuration.
example:junit	Plain JUnit 5 examples.
example:spring	Spring Boot example.
example:spark	Spark integration example using HDFS, HMS, Kafka, Schema Registry, S3, GCS, Iceberg, and Hive Parquet tables.

3. Dependency Management

The repository imports the Testcontainers BOM once in the root subprojects configuration:

org.testcontainers:testcontainers-bom:2.0.4

Individual modules should depend on Testcontainers modules without repeating the BOM.

External consumers should import the same BOM in their own dependency-management setup before adding bigdata-test and any direct Testcontainers dependencies. With Testcontainers 2.x, module coordinates use the testcontainers- prefix, for example:

• org.testcontainers:testcontainers-junit-jupiter

• org.testcontainers:testcontainers-postgresql

The extensions module compiles against Hadoop, Kafka, Avro, Schema Registry, and Spark as provided dependencies. They are needed only when the corresponding extension runs, and they are not part of the extension module’s compile or runtime API. This reduces dependency conflicts for user applications that already use their own Hadoop, Kafka, or Spark client versions.

Kafka-only or JCEKS-only users do not need Spark on their runtime classpath. Tests that enable Spark SQL preparation must provide either Apache Spark or the matching vendor Spark distribution themselves.

If an application needs to force its own runtime versions, use normal Gradle dependency constraints, platforms, or exclusions in the consuming build. Future extension modules can split these integrations further by feature so users can depend only on the extension families they need.

4. Publishing Local Snapshots

Publish all snapshot artifacts to Maven local when another local project needs to consume the current checkout:

GRADLE_USER_HOME=/data/.gradle ./gradlew publishToMavenLocal

Publish only selected modules when you do not need the full repository:

GRADLE_USER_HOME=/data/.gradle ./gradlew :core:publishToMavenLocal
GRADLE_USER_HOME=/data/.gradle ./gradlew :junit5:publishToMavenLocal
GRADLE_USER_HOME=/data/.gradle ./gradlew :extensions:publishToMavenLocal

The consuming build must include mavenLocal() before remote repositories if it should prefer the local snapshot:

repositories {
    mavenLocal()
    mavenCentral()
}

Use the current project version from gradle.properties or the root build. For example, if this checkout is 0.1.1-SNAPSHOT, consume org.openprojectx.bigdata.test:core:0.1.1-SNAPSHOT, org.openprojectx.bigdata.test:junit5:0.1.1-SNAPSHOT, or org.openprojectx.bigdata.test:extensions:0.1.1-SNAPSHOT.

5. Programmatic Usage

Use BigDataTestKit.builder() when you want direct lifecycle control.

val kit = BigDataTestKit.builder()
    .withHiveMetastore()
    .withKafka(KafkaOptions(enabled = true, schemaRegistryEnabled = true))
    .withS3()
    .build()

kit.use {
    it.start()

    val metastoreUri = it.endpoint(BigDataService.HIVE_METASTORE)
        .property("hive.metastore.uris")

    val bootstrapServers = it.endpoint(BigDataService.KAFKA)
        .property("bootstrap.servers")
}

The kit starts only the services requested through builder methods. It stops containers in reverse order when close() is called.

5.1. Service Endpoints

After startup, each service exposes a BigDataEndpoint.

val endpoint = kit.endpoint(BigDataService.KAFKA)
val bootstrapServers = endpoint.property("bootstrap.servers")
val host = endpoint.host
val port = endpoint.ports["bootstrap"]

kit.springProperties() flattens all endpoint properties into one map, which is useful for application context initializers and Spring Boot tests.

6. JUnit 5 Usage

Annotate a test class with @BigDataTest and request BigDataTestKit as a test parameter.

@BigDataTest(
    hiveMetastore = true,
    kafka = true,
    schemaRegistry = true,
    s3 = true,
)
class MyIntegrationTest {
    @Test
    fun test(kit: BigDataTestKit) {
        val hms = kit.endpoint(BigDataService.HIVE_METASTORE)
        val kafka = kit.endpoint(BigDataService.KAFKA)
    }
}

The JUnit extension starts the kit before all tests in the class and closes it after all tests complete.

6.1. Service Flags

Annotation field	Effect
config	Loads TOML config before containers start. Use this for startup-time settings such as service image overrides.
kerberos	Starts the shared Kerberos KDC.
hdfs	Starts HDFS.
hdfsKerberos	Starts HDFS with Kerberos service configuration.
hiveMetastore	Starts open-source Hive Metastore and an external PostgreSQL support container.
clouderaHms	Starts the Cloudera HMS image, which includes embedded PostgreSQL. Do not combine this with hiveMetastore.
hiveMetastoreKerberos	Starts Hive Metastore with Kerberos service configuration.
kafka	Starts Kafka.
kafkaKerberos	Starts Kafka with Kerberos/SASL configuration.
schemaRegistry	Starts Schema Registry.
kafkaUi	Starts Kafka UI.
kafkaUiKerberos	Configures Kafka UI for Kerberos/SASL Kafka access.
s3	Starts Floci S3-compatible AWS emulation.
fakeGcs	Starts Floci GCP-compatible GCS emulation.

6.2. Service Image Configuration

Service images can be overridden from TOML with an [images] table. The JUnit extension reads this table before creating containers, so these values affect startup.

[images]
kerberos = "ghcr.io/openprojectx/directory-kerby/kerby-kdc:latest"
hdfs = "apache/hadoop:3.5.0"
hiveMetastore = "ghcr.io/openprojectx/hive:3.1.3-hadoop-3.4.2-gcs-4.0.4-jdk17-0.1.5"
hiveMetastorePostgres = "postgres:16-alpine"
hiveMetastoreMysql = "mysql:8.0.44-bookworm"
clouderaHms = "ghcr.io/openprojectx/cloudera-hms:0.1.74"
clouderaHmsMariadb = "ghcr.io/openprojectx/cloudera-hms:0.1.74-mariadb"
kafka = "apache/kafka:4.1.2"
schemaRegistry = "confluentinc/cp-schema-registry:7.8.0"
kafkaUi = "ghcr.io/kafbat/kafka-ui:latest"
s3 = "ghcr.io/openprojectx/dockerhub/floci/floci:1.5.32"
fakeGcs = "ghcr.io/openprojectx/dockerhub/floci/floci-gcp:0.5.0"

Kafka and the open-source HMS database sidecar use typed Testcontainers classes instead of plain generic containers. When you override those images, bigdata-test marks the configured Kafka image as a compatible substitute for apache/kafka, PostgreSQL as a compatible substitute for postgres, and MySQL as a compatible substitute for mysql, so private mirrors or rebuilt images can still pass Testcontainers image checks.

You can place [images] in the same TOML files used by @BigDataExtensions, because @BigDataTest reads those locations before startup when the extensions annotation is present:

@BigDataExtensions("classpath:bigdata-extensions.toml")
@BigDataTest(hdfs = true, kafka = true, schemaRegistry = true)
class MyIntegrationTest

You can also point @BigDataTest directly at startup config:

@BigDataTest(
    config = ["classpath:bigdata-test.toml"],
    hdfs = true,
    kafka = true,
)
class MyIntegrationTest

When both are used, files from @BigDataExtensions are loaded first and files from @BigDataTest(config = …​) are loaded after them, so direct @BigDataTest config wins for duplicate image keys.

Gradle or any other test launcher can add startup config at runtime with the bigdata.test.config system property. Use a comma-separated list when a task should compose common config with one matrix-specific override:

tasks.register<Test>("sparkApacheHmsKerberosTest") {
    useJUnitPlatform()
    filter.includeTestsMatching("org.openprojectx.bigdata.test.example.spark.SparkBigDataTestExample")
    systemProperty("bigdata.test.config.replace", "true")
    systemProperty(
        "bigdata.test.config",
        "classpath:spark-bigdata-test-common.toml,classpath:spark-bigdata-test-apache-hms-kerberos.toml",
    )
}

When bigdata.test.config.replace=true, task config replaces startup config from @BigDataExtensions and @BigDataTest(config = …​). Without it, task config is appended last and wins for scalar values. Service booleans are additive, so replacement mode is the right choice when a matrix needs to turn Kerberos or HMS implementations on and off.

Extension config has the same launcher hook: bigdata.extensions.config and bigdata.extensions.config.replace.

Container data-driven tests can be TOML-driven because service images, ports, Kerberos flags, and extension inputs are runtime configuration. Dependency data-driven tests must be Gradle-task/classpath-driven because Spark, Hadoop, Iceberg, and vendor distributions are resolved before the test JVM starts. The Spark example uses separate resolvable runtime classpaths for Apache and Cloudera dependency lines, then pairs those classpaths with TOML service variants.

HDFS-specific startup settings use the [hdfs] table. Keep the default dataNodeHostname = "hdfs" for Docker-network clients. Use localhost, or a stable hostname that resolves to 127.0.0.1, only for host-side clients that connect through mapped host ports. Host-side clients must also use dfs.client.use.datanode.hostname=true; the framework exposes this endpoint property by default.

[hdfs]
dataNodeHostname = "localhost"
localHdfsSitePath = "build/bigdata-test/hadoop/hdfs-site.xml"

[ports]
hdfsNameNode = 8020
hdfsDataNode = 9866
hdfsWeb = 9870

localHdfsSitePath is optional. When set, the framework writes a host-side Hadoop XML file with the resolved HDFS client properties, including the mapped NameNode URI, DataNode hostname mode, and Kerberos client settings when HDFS Kerberos is enabled. When omitted, the file is generated in framework-managed temporary material.

HMS client XML files can be provisioned the same way. These files are host-side client resources and intentionally contain only client-safe settings such as hive.metastore.uris, Kerberos client flags, TLS truststore settings, and object-store/Hadoop properties. They do not include HMS server database credentials.

[hiveMetastore]
localHiveSitePath = "build/bigdata-test/hive/hive-site.xml"
localMetastoreSitePath = "build/bigdata-test/hive/metastore-site.xml"

When omitted, both files are generated in framework-managed temporary material. The same options apply to Apache HMS and Cloudera HMS.

6.3. Container Customization Escape Hatch

Typed service options should cover the normal path, but integration-test containers sometimes need image-specific workarounds before the framework has first-class support. Use container customizations as an escape hatch for troubleshooting or temporary overrides.

Container customizations run before the service container starts. Use them for Docker/Testcontainers-level concerns such as network mode, extra ports, environment variables, copied files, host mounts, image-specific startup flags, and temporary workarounds. Use extensions for post-start provisioning such as creating buckets, generating JCEKS files, creating Kafka topics, or producing seed records.

TOML supports environment variables, copied files, host mounts, and extra exposed ports:

[containers.hdfs]
networkMode = "host"

[containers.hdfs.env]
HADOOP_OPTS = "-Dsun.security.krb5.debug=true"

[containers.hdfs.files]
"/opt/hadoop/etc/hadoop/extra-site.xml" = "classpath:hadoop/extra-site.xml"
"/tmp/message.txt" = "text:created by bigdata-test"

[containers.hdfs.mounts]
"/tmp/local-hadoop-conf" = "/opt/hadoop-extra:ro"

[containers.hdfs.ports]
"9866" = 9866

File sources must use one of these prefixes:

• classpath: reads a test classpath resource and copies its bytes into the container.

• file: copies a host file path into the container.

• text: writes inline UTF-8 text into the container.

networkMode = "host" is passed through to Testcontainers as Docker host networking. It is useful only when the local Docker runtime supports it for the target platform. The default bridge network remains the most portable option, especially when multiple test containers need to communicate through Testcontainers network aliases.

Mounts are useful for local debugging, but copied files are more reliable in CI because they do not depend on Docker host path sharing. Prefer copied files for certificates, keytabs, Hadoop XML, JAAS, and other test fixtures.

Available service names in container TOML are kerberos, hdfs, hiveMetastore, hms, kafka, schemaRegistry, kafkaUi, s3, and fakeGcs.

6.3.1. HDFS Environment Variable Names

The HDFS service currently defaults to the apache/hadoop:3.5.0 image. Its entrypoint runs the image’s own envtoconf.py bootstrap before the bigdata-test HDFS startup command. This is an Apache Hadoop Docker image convention, not a general Hadoop runtime specification.

That image parser treats environment variable names whose second token is a known config format as instructions to generate files under HADOOP_CONF_DIR. Tokens are split on _ and .. For example:

CORE_XML_fs_defaultFS=hdfs://hdfs:8020
HDFS_XML_dfs_replication=1

Those variables ask the image to generate core.xml and hdfs.xml content. This can coexist with bigdata-test when you intentionally use the image-native convention.

Do not use arbitrary HDFS custom environment variable names whose second token is one of the image’s reserved config format names:

xml, properties, yaml, yml, env, sh, cfg, conf

For example, this looks harmless but conflicts with the image parser:

[containers.hdfs.env]
TEST_ENV = "TEST"

TEST_ENV is interpreted as "create test.env`" with no config key. In `apache/hadoop:3.5.0, the env, sh, cfg, and conf transforms are also buggy because they iterate parsed keys incorrectly. Names such as TEST_ENV, TEST_ENV_FOO, APP_CONF, and FOO_SH_BAR can fail before HDFS starts.

Use a non-reserved name instead:

[containers.hdfs.env]
TEST_VALUE = "TEST"
MYAPP_ENV_VALUE = "TEST"

For Hadoop XML settings, prefer the typed bigdata-test options or copied XML files. If you intentionally use the image convention, use one of the working formats with a property suffix, such as CORE_XML_fs_defaultFS. Avoid the image’s env, sh, cfg, and conf formats with apache/hadoop:3.5.0.

Programmatic tests can use the same portable operations:

val kit = BigDataTestKit.builder()
    .withHdfs()
    .withContainerNetworkMode(BigDataService.HDFS, "host")
    .withContainerEnv(BigDataService.HDFS, "HADOOP_OPTS", "-Dsun.security.krb5.debug=true")
    .withContainerFile(
        BigDataService.HDFS,
        ContainerFileTransferOptions.content(
            "<configuration/>",
            "/opt/hadoop/etc/hadoop/extra-site.xml",
        ),
    )
    .withContainerPort(BigDataService.HDFS, ContainerPortOptions(containerPort = 9866, hostPort = 9866))
    .build()

The lower-level withContainerCustomization method is useful when a test wants to build the customization object dynamically:

val customization = ContainerCustomizationOptions(
    networkMode = if (useHostNetwork) "host" else null,
    environment = mapOf("HADOOP_OPTS" to "-Dsun.security.krb5.debug=true"),
    files = listOf(
        ContainerFileTransferOptions.content(
            "<configuration/>",
            "/opt/hadoop/etc/hadoop/extra-site.xml",
        ),
    ),
    ports = listOf(ContainerPortOptions(containerPort = 9866, hostPort = 9866)),
)

val kit = BigDataTestKit.builder()
    .withHdfs()
    .withContainerCustomization(BigDataService.HDFS, customization)
    .build()

For cases that cannot be represented declaratively, use the Testcontainers callback. The callback runs after bigdata-test applies its built-in container settings and before the container starts:

val kit = BigDataTestKit.builder()
    .withHdfs()
    .customizeContainer(BigDataService.HDFS) { container ->
        container.withCommand("sh", "-lc", "env && exec /entrypoint.sh")
    }
    .build()

The callback receives the actual GenericContainer<*>, so users can call Testcontainers APIs directly. This is the most flexible workaround path when bigdata-test does not expose a typed option yet. For example, before networkMode was first-class, the same host-network workaround could be written as:

val kit = BigDataTestKit.builder()
    .withHdfs()
    .customizeContainer(BigDataService.HDFS) { container ->
        container.withNetworkMode("host")
    }
    .build()

Do not use the callback as the normal configuration API for reusable tests. If a setting is stable and generally useful, prefer a typed option or TOML field so it can be documented, data-driven from Gradle tasks, and used from Java.

6.4. Startup Health Checks

Every service still uses a Testcontainers wait strategy to detect process startup. Optional health checks run after a service starts and after its endpoint is registered. Use these when a test should fail early if a service is not actually usable.

Health check modes:

• basic keeps the default behavior: rely on the service’s Testcontainers wait strategy.

• cli runs a service-specific command inside the service container.

• none disables the post-start health check.

[healthChecks]
hdfs = "cli"
kafka = "cli"
s3 = "cli"

[healthCheckTimeouts]
hdfs = 90
kafka = 60

Programmatic setup uses the same options:

val kit = BigDataTestKit.builder()
    .withHdfs()
    .withHealthCheck(
        BigDataService.HDFS,
        BigDataHealthCheckOptions(mode = BigDataHealthCheckMode.CLI, timeoutSeconds = 90),
    )
    .build()

Current CLI probes:

Service	CLI probe
Kerberos	Verifies generated client krb5.conf and keytab files exist in the Kerby container.
HDFS	Uses hdfs dfs to create a directory, write a small file, read it back, and remove it. This validates NameNode plus DataNode block transfer from the container network.
Hive Metastore	Checks the Thrift port from inside the HMS container. HMS does not provide a good standalone metadata CLI unless a client stack is also present.
Kafka	Runs kafka-topics.sh --list against the in-container broker endpoint. This is best for plaintext-compatible listeners; TLS or Kerberos Kafka may need service-specific client properties before this probe can be enabled.
Schema Registry	Uses wget or curl against /subjects.
Kafka UI	Uses wget or curl against /.
Floci S3-compatible AWS emulator	Checks the Floci /_floci/health endpoint.
Floci GCP-compatible GCS emulator	Uses wget or curl against the configured HTTP health endpoint.

CLI probes validate the container’s own environment. For host-side behavior, such as Windows HDFS clients connecting through mapped ports, use a host/JVM smoke test as well because an in-container hdfs dfs command cannot prove host network reachability.

hiveMetastore is the open-source HMS image and defaults to ghcr.io/openprojectx/hive:3.1.3-hadoop-3.4.2-gcs-4.0.4-jdk17-0.1.5. Hive 4 images can be tested by overriding this image key, but Spark 3.x brings a Hive 2.3 metastore client and should use the Hive 3 image unless that client stack is changed. The open-source HMS path starts an external database support container. By default it uses PostgreSQL from hiveMetastorePostgres. Set [hiveMetastore] databaseType = "mysql" to use MySQL from hiveMetastoreMysql. The support database uses a random host port by default; set [hiveMetastore] databaseHostPort when you need a stable local database port for troubleshooting.

clouderaHms is the embedded-database Cloudera HMS image. Use the clouderaHms = true annotation flag when you want this implementation. It defaults to embedded PostgreSQL with ghcr.io/openprojectx/cloudera-hms:0.1.74; set [clouderaHms] databaseType = "mariadb" to use ghcr.io/openprojectx/cloudera-hms:0.1.74-mariadb. Its default warehouse path is /tmp/cloudera-hms/warehouse, which is writable for the non-root database user in the Cloudera HMS images. If you override [clouderaHms] warehouseDir, use a path that the image user can create and write. A test class must not enable both hiveMetastore and clouderaHms.

6.5. HTTP TLS Gateway

HTTP services can be exposed through an HAProxy HTTPS gateway. The service container still speaks its normal HTTP protocol internally; HAProxy listens on container port 443, terminates TLS, and forwards to the service over the shared Docker network. This matches local environments where users normally expose one HTTPS endpoint and configure local DNS or /etc/hosts for the chosen domain.

TLS certificates are generated in the JVM. No openssl binary is required. By default, bigdata-test generates a temporary root CA, service certificates, and a PKCS12 truststore. The root CA signs the per-service certificates. The truststore contains the root CA, so clients trust any service certificate generated from that CA.

6.5.1. Stable CA Files

For a stable project CA, generate the CA once and keep both files. The test framework will load that CA and generate per-service leaf certificates from it:

BigDataTlsCertificates.ensureCertificateAuthority(
    Path.of("build/tls/root-ca.crt"),
    Path.of("build/tls/root-ca.key"),
)

BigDataTlsCertificates.ensureCertificateAuthority(
    Path.of("build/tls/root-ca.crt"),
    Path.of("build/tls/root-ca.key")
);

ensureCertificateAuthority creates both files if neither exists and reuses them if both exist. If only one file exists, it fails so the CA is not accidentally rotated. Use generateCertificateAuthority(…​, overwrite = true) only when you intentionally want to replace the CA.

The CA certificate is PEM. JVM clients normally trust certificates through a keystore/truststore, not directly from PEM. Generate a JVM-ready PKCS12 truststore from the CA when you want to use javax.net.ssl.trustStore yourself:

BigDataTlsCertificates.generateTrustStore(
    certificatePath = Path.of("build/tls/root-ca.crt"),
    trustStorePath = Path.of("build/tls/truststore.p12"),
    password = "changeit",
    overwrite = true,
)

BigDataTlsCertificates.generateTrustStore(
    Path.of("build/tls/root-ca.crt"),
    Path.of("build/tls/truststore.p12"),
    "changeit",
    "bigdata-test-root-ca",
    "PKCS12",
    true
);

Reference the generated CA from TOML. trustStorePath is the PKCS12 truststore that bigdata-test writes for the current run. It can be stable as a path, but the CA is only stable when caCertPath and caKeyPath are stable:

[tls]
caCertPath = "build/tls/root-ca.crt"
caKeyPath = "build/tls/root-ca.key"
trustStorePath = "build/bigdata-test-truststore.p12"
trustStorePassword = "changeit"
haproxyImage = "haproxy:3.0-alpine"

[s3Tls]
enabled = true
domain = "localhost"

[fakeGcsTls]
enabled = true
domain = "storage.googleapis.com"

[schemaRegistryTls]
enabled = true
domain = "schema-registry.local"

[kafkaUiTls]
enabled = true
domain = "kafka-ui.local"

[kafkaTls]
enabled = true
domain = "localhost"

[hiveMetastoreTls]
enabled = true
domain = "localhost"

[hdfsWebTls]
enabled = true
domain = "hdfs.local"

Do not commit the CA private key for a real shared environment. For local-only test certificates, keep it under a developer-controlled path such as .bigdata-test/tls/root-ca.key and add it to .gitignore. In CI, store the CA files as secrets or generate them before the test task and point TOML at the generated files.

The HTTPS endpoint replaces the normal endpoint property for that service. For example, aws.endpoint-url.s3 becomes https://<domain>:<mapped-port>; when s3Tls.enabled=true.

6.5.2. JVM Truststore Handling

Every TLS-enabled endpoint includes JVM truststore settings:

• javax.net.ssl.trustStore

• javax.net.ssl.trustStorePassword

• javax.net.ssl.trustStoreType

• bigdata.test.tls.ca-cert

With @BigDataTest, the JUnit extension applies those settings automatically before tests run:

• saves the current javax.net.ssl.trustStore, javax.net.ssl.trustStorePassword, and javax.net.ssl.trustStoreType values;

• sets them to the test truststore generated from [tls];

• installs a matching default SSLContext;

• restores the previous JVM properties and default SSLContext after the test class.

This means HTTP clients and libraries that use the default JSSE trust configuration can connect during the test without extra setup.

With direct BigDataTestKit.builder() usage, the core module does not mutate global JVM properties. Read the endpoint properties and apply them yourself when the client needs JVM-default trust:

val endpoint = kit.endpoint(BigDataService.S3)
System.setProperty("javax.net.ssl.trustStore", endpoint.property("javax.net.ssl.trustStore"))
System.setProperty("javax.net.ssl.trustStorePassword", endpoint.property("javax.net.ssl.trustStorePassword"))
System.setProperty("javax.net.ssl.trustStoreType", endpoint.property("javax.net.ssl.trustStoreType"))

For a permanent test JVM setup, use a stable CA and stable truststore path in TOML. The JUnit extension will still set and restore the JVM properties around the test, but the truststore file and CA identity remain predictable:

[tls]
caCertPath = ".bigdata-test/tls/root-ca.crt"
caKeyPath = ".bigdata-test/tls/root-ca.key"
trustStorePath = ".bigdata-test/tls/truststore.p12"
trustStorePassword = "changeit"

If code runs outside the JUnit extension, pass the same truststore to the JVM:

./gradlew test \
  -Djavax.net.ssl.trustStore=.bigdata-test/tls/truststore.p12 \
  -Djavax.net.ssl.trustStorePassword=changeit \
  -Djavax.net.ssl.trustStoreType=PKCS12

The truststore must exist before those JVM properties are used. bigdata-test writes it when the kit starts, so external clients usually need either the JUnit extension, a small setup step that calls BigDataTlsCertificates.ensureCertificateAuthority(…​), or a checked-in/generated truststore prepared before the external client starts.

6.5.3. Importing the CA into a JDK

Prefer a project-local PKCS12 truststore plus javax.net.ssl.trustStore for tests. Importing into a JDK modifies that JDK installation and usually requires write permission. Use it only for developer machines or controlled CI images.

Manual keytool import:

keytool -importcert \
  -alias bigdata-test-root-ca \
  -file .bigdata-test/tls/root-ca.crt \
  -keystore "$JAVA_HOME/lib/security/cacerts" \
  -storepass changeit \
  -noprompt

The API can do the same operation and creates a backup by default:

BigDataTlsCertificates.installCertificateAuthorityToJavaTrustStore(
    javaHome = Path.of(System.getProperty("java.home")),
    certificatePath = Path.of(".bigdata-test/tls/root-ca.crt"),
)

BigDataTlsCertificates.installCertificateAuthorityToJavaTrustStore(
    Path.of(System.getProperty("java.home")),
    Path.of(".bigdata-test/tls/root-ca.crt")
);

By default the API writes to <javaHome>/lib/security/cacerts, uses store password changeit, alias bigdata-test-root-ca, and writes a backup file next to cacerts. If the alias already exists, it fails unless overwriteAlias=true.

TLS host ports are random by default even when sameHostPorts=true, because multiple enabled HTTP services cannot all bind localhost port 443. Use explicit TOML port fields when a stable HTTPS port is required:

[ports]
s3Tls = 443
schemaRegistryTls = 8443

Available TLS port fields are hdfsWebTls, schemaRegistryTls, kafkaUiTls, s3Tls, and fakeGcsTls.

Non-HTTP services use their native TLS implementation rather than HAProxy to avoid protocol compatibility issues. Kafka TLS is native broker TLS because Kafka clients depend on broker metadata and advertised listeners. Hive Metastore TLS is native Thrift TLS on the normal metastore port.

Kafka TLS can be enabled from the annotation or TOML:

@BigDataTest(kafkaTls = true)

[kafkaTls]
enabled = true
domain = "localhost"

Hive Metastore TLS can be enabled for either hiveMetastore or clouderaHms:

@BigDataTest(hiveMetastore = true, hiveMetastoreTls = true)

[services]
hiveMetastore = true

[hiveMetastoreTls]
enabled = true
domain = "localhost"

The HMS endpoint keeps the normal thrift://host:port URI. Hive clients switch to TLS through endpoint properties:

• hive.metastore.use.SSL=true

• hive.metastore.truststore.path

• hive.metastore.truststore.password

The JUnit extension also installs the generated truststore as JVM TLS properties for clients that use JSSE defaults.

The Kafka endpoint then exposes security.protocol=SSL for plaintext Kafka or security.protocol=SASL_SSL when Kafka Kerberos is also enabled. It also exposes ssl.truststore.location, ssl.truststore.password, and ssl.truststore.type for Kafka clients.

6.6. Dynamic and Fixed Ports

Host ports are dynamic by default. Keep port fields set to 0 unless an external local tool needs a stable port.

@BigDataTest(
    hdfs = true,
    kafka = true,
    hdfsDataNodeHostname = "localhost",
    hdfsNameNodePort = 18020,
    hdfsDataNodePort = 9866,
    hdfsWebPort = 19870,
    kafkaPort = 19092,
)
class MyFixedPortTest

For host-side HDFS clients, including Windows clients using Docker Desktop, keep the HDFS DataNode transfer port stable and set the advertised DataNode hostname to a host-reachable name. The client first connects to the NameNode, then the NameNode returns DataNode block locations. The client only uses the reported hostname when dfs.client.use.datanode.hostname=true; otherwise it may use the DataNode IP from the report and ignore the advertised hostname.

By default the framework advertises hdfs, which is correct for other containers on the Testcontainers network. Host-side clients usually need hdfsDataNodeHostname = "localhost" plus sameHostPorts=true or hdfsDataNodePort = 9866.

If both host processes and other Docker containers need to access the same HDFS instance, use an advertised hostname that is reachable from both sides. In practice that means either keeping the default hdfs for container-only clients, using localhost for host-only clients, or providing a user-managed DNS/hosts name that resolves appropriately in both environments. For JVM tests, the org.openprojectx.java.dns Gradle plugin can map a stable name such as hdfs.test.local to 127.0.0.1 without editing the OS hosts file:

plugins {
    id("org.openprojectx.java.dns") version "0.1.1"
}

javadns {
    hosts.put("hdfs.test.local", "127.0.0.1")
}

Then configure HDFS to advertise hdfs.test.local and keep the DataNode host port fixed. The Java DNS plugin affects only the client JVM’s hostname resolution; the DataNode still has to register the stable hostname with the NameNode.

Docker host networking can also be requested for a service:

[containers.hdfs]
networkMode = "host"

Host networking support depends on the Docker runtime. On Linux it maps directly to the host network namespace; on Docker Desktop it depends on the Desktop version and settings. Prefer the fixed-port plus stable-hostname approach for portable tests.

To bind each enabled service to the same localhost port it uses inside the container, enable sameHostPorts.

@BigDataTest(
    hdfs = true,
    hiveMetastore = true,
    kafka = true,
    schemaRegistry = true,
    s3 = true,
    fakeGcs = true,
    sameHostPorts = true,
)
class MySameHostPortTest

This maps common service ports directly, for example HDFS 8020, 9866, and 9870, Hive Metastore 9083, Kafka 9092, Schema Registry 8085, Floci S3 4566, and Floci GCS 4588. Explicit per-service port fields still take priority over sameHostPorts.

TLS gateway ports are not affected by sameHostPorts, because every gateway listens on container port 443. Configure the per-service TLS port fields explicitly if a stable HTTPS host port is needed.

The core builder exposes the same behavior:

val kit = BigDataTestKit.builder()
    .withHdfs()
    .withKafka()
    .withSameHostPorts()
    .build()

Available fixed host-port config:

Service endpoint	TOML key under [ports]	Annotation/Gradle property	Container port
Kerberos KDC	kerberosKdc	kerberosKdcPort	88
HDFS NameNode RPC	hdfsNameNode	hdfsNameNodePort	8020
HDFS DataNode data transfer	hdfsDataNode	hdfsDataNodePort	9866
HDFS Web UI	hdfsWeb	hdfsWebPort	9870
HDFS Web UI TLS gateway	hdfsWebTls	hdfsWebTls	443 on the HAProxy gateway
Hive Metastore Thrift, including Cloudera HMS	hiveMetastore	hiveMetastorePort	9083
Kafka broker	kafka	kafkaPort	9092
Schema Registry HTTP	schemaRegistry	schemaRegistryPort	8085
Schema Registry TLS gateway	schemaRegistryTls	schemaRegistryTls	443 on the HAProxy gateway
Kafka UI HTTP	kafkaUi	kafkaUiPort	8080
Kafka UI TLS gateway	kafkaUiTls	kafkaUiTls	443 on the HAProxy gateway
Floci S3 HTTP	s3	s3Port	4566
Floci S3 TLS gateway	s3Tls	s3Tls	443 on the HAProxy gateway
Floci GCS HTTP	fakeGcs	fakeGcsPort	4588 by default, or 4443 when overriding the image to fsouza/fake-gcs-server
Floci GCS TLS gateway	fakeGcsTls	fakeGcsTls	443 on the HAProxy gateway

There is no [ports] clouderaHms key. Cloudera HMS shares the Hive Metastore service endpoint, so its Thrift port is configured with [ports] hiveMetastore = 9083.

The embedded Cloudera HMS database port is not a service endpoint port. Configure it under [clouderaHms]:

[services]
clouderaHms = true

[ports]
hiveMetastore = 9083

[clouderaHms]
databaseType = "mariadb" # or "postgresql"
databaseHostPort = 3306  # 0 means random

For Cloudera HMS, databaseType = "postgresql" exposes embedded database container port 5432; databaseType = "mariadb" exposes embedded database container port 3306.

Endpoint properties always report the actual mapped host ports.

6.7. Service Endpoint Reference

Each started service exposes a BigDataEndpoint with named ports and connection properties. With dynamic ports, the host ports are assigned by Testcontainers. With sameHostPorts = true, the host ports below are used directly unless an explicit per-service port field overrides them.

Service	Port names	Default ports	Endpoint property keys
BigDataService.KERBEROS	kdc	88	bigdata.test.kerberos.realm, bigdata.test.kerberos.kdc, bigdata.test.kerberos.krb5-conf, bigdata.test.kerberos.client-principal, bigdata.test.kerberos.client-password, bigdata.test.kerberos.client-keytab
BigDataService.HDFS	namenode, datanode, web, web-tls	8020, 9866, 9870, 443	fs.defaultFS, spring.hadoop.fs-uri, dfs.client.use.datanode.hostname, dfs.datanode.hostname, dfs.namenode.https-address
BigDataService.HIVE_METASTORE	thrift	9083	hive.metastore.uris, spring.bigdata.test.hive-metastore.thrift-uri, hive.metastore.use.SSL, hive.metastore.truststore.path, hive.metastore.truststore.password
BigDataService.KAFKA	bootstrap	9092	bootstrap.servers, spring.kafka.bootstrap-servers, bootstrap.servers.internal, security.protocol, kafka.kerberos.service-name, sasl.jaas.config, ssl.truststore.location, ssl.truststore.password, ssl.truststore.type
BigDataService.SCHEMA_REGISTRY	http, https	8085, 443	schema.registry.url
BigDataService.KAFKA_UI	http, https	8080, 443	bigdata.test.kafka-ui.url
BigDataService.S3	edge, https	4566, 443	spring.cloud.aws.s3.endpoint, aws.endpoint-url.s3, aws.accessKeyId, aws.secretAccessKey, aws.region
BigDataService.FAKE_GCS	http, https	4588, 443	bigdata.test.gcs.endpoint, google.cloud.storage.host

The same reference data is exposed in code through BigDataService.defaultPorts and BigDataService.endpointProperties.

7. Container Logs

Container logs are disabled by default. Enable them when debugging service startup or service-side behavior.

7.1. Programmatic Log Routing

Send logs to the main JVM console:

val kit = BigDataTestKit.builder()
    .withKafka()
    .withContainerLogsToStdout()
    .build()

Write one file per service:

val kit = BigDataTestKit.builder()
    .withHiveMetastore()
    .withContainerLogs(
        ContainerLogOptions(
            mode = ContainerLogMode.FILE,
            directory = "build/container-logs",
            append = false,
        ),
    )
    .build()

Set service container log levels programmatically:

val kit = BigDataTestKit.builder()
    .withKafka()
    .withContainerLogLevel(BigDataService.KAFKA, "DEBUG")
    .withContainerLogsToStdout()
    .build()

7.2. JUnit Log Routing

@BigDataTest(
    kafka = true,
    schemaRegistry = true,
    containerLogMode = ContainerLogMode.FILE,
    containerLogDirectory = "build/container-logs",
    containerLogAppend = false,
)
class MyTroubleshootingTest

Set service container log levels from TOML:

[containerLogLevels]
kafka = "DEBUG"
schemaRegistry = "DEBUG"
hdfs = "INFO"
s3 = "TRACE"

ContainerLogMode.STDOUT prefixes each line with the service name. ContainerLogMode.FILE writes files such as:

• kafka.log

• schema-registry.log

• hive-metastore.log

• hive-metastore-postgres.log

File logs append by default for backward compatibility. Set append = false in TOML, containerLogAppend = false on @BigDataTest, or ContainerLogOptions(append = false) programmatically to overwrite the files at the start of each kit run.

TOML example:

[containerLogs]
mode = "FILE"
directory = "build/container-logs"
append = false

For HDFS troubleshooting, use DEBUG or TRACE. The framework maps that to Hadoop’s container-side log settings and tails the NameNode/DataNode log files so startup failures show the Hadoop process output, not only the Testcontainers wait-strategy error.

val kit = BigDataTestKit.builder()
    .withHdfs()
    .withContainerLogLevel(BigDataService.HDFS, "DEBUG")
    .withContainerLogsToStdout()
    .build()

Use file mode when the run happens in CI or when the container may be removed before you can run docker logs:

val kit = BigDataTestKit.builder()
    .withHdfs()
    .withContainerLogLevel(BigDataService.HDFS, "TRACE")
    .withContainerLogsToDirectory("build/bigdata-test-container-logs")
    .build()

hive-metastore-postgres.log is only written for the open-source HMS distribution. The Cloudera HMS image keeps its PostgreSQL process inside hive-metastore.log.

Log level and log routing are separate. [containerLogLevels] changes service verbosity by setting the common environment variables for each image family. containerLogMode or withContainerLogs…​ controls where container stdout/stderr goes. If an image uses a different log-level variable, override it directly:

[containers.kafka.env]
KAFKA_LOG4J_ROOT_LOGLEVEL = "TRACE"

Supported typed levels are TRACE, DEBUG, INFO, WARN, ERROR, and OFF.

8. Kerberos

Kerberos can be enabled globally and per service. Set kerberos = true to start the KDC, then enable Kerberos on the services that should use it.

@BigDataTest(
    kerberos = true,
    hdfs = true,
    hdfsKerberos = true,
    hiveMetastore = true,
    hiveMetastoreKerberos = true,
    kafka = true,
    kafkaKerberos = true,
)
class MyKerberosIntegrationTest

For host-side Kafka Kerberos clients, prefer an explicit fixed kafkaPort. Kafka must advertise the same host and port that the client uses during the SASL/GSSAPI exchange; with a random host port the framework cannot know that advertised port before the broker starts. When Schema Registry is enabled with Kafka Kerberos, it uses Kafka’s internal plaintext listener while host clients keep using the Kerberos listener.

Kerberos bootstrap timing is configurable from TOML. The defaults are intentionally modest for local runs; CI can use a longer window when KDC admin startup is slower:

[kerberos]
clientPrincipal = "app_user@EXAMPLE.COM"
clientPassword = "app-user-secret"
materialDirectory = "build/bigdata-test/kerberos"
localKrb5ConfPath = "build/bigdata-test/kerberos/krb5.conf"
localClientKeytabPath = "build/bigdata-test/kerberos/app_user.keytab"
startupTimeoutSeconds = 180
materialTimeoutSeconds = 120
adminAttempts = 120
adminRetryDelaySeconds = 1
debug = true

clientPrincipal and clientPassword configure the generated app/client user and keytab. materialDirectory makes generated Kerberos material land in a stable local directory instead of a temporary directory. localKrb5ConfPath and localClientKeytabPath optionally pin the exact local files exposed through bigdata.test.kerberos.krb5-conf and bigdata.test.kerberos.client-keytab. startupTimeoutSeconds controls how long Testcontainers waits for the KDC image to print its ready log. materialTimeoutSeconds controls the post-start wait for generated keytabs and client config to be readable. adminAttempts and adminRetryDelaySeconds are passed to the Kerby image as KERBY_KADMIN_ATTEMPTS and KERBY_KADMIN_RETRY_DELAY_SECONDS.

Set debug = true when troubleshooting Kerberos startup in CI. It passes KERBY_DEBUG=true to the Kerby image, which enables JDK Kerberos debug output, Kerby DEBUG logs on container stdout, and dumps the generated kdc.conf, server krb5.conf, client krb5.conf, backend config, and admin server config into the container log.

Programmatic setup exposes the same options:

BigDataTestKit.builder()
    .withKerberos(
        KerberosOptions(
            enabled = true,
            clientPrincipal = "app_user@EXAMPLE.COM",
            clientPassword = "app-user-secret",
            materialDirectory = "build/bigdata-test/kerberos",
            localKrb5ConfPath = "build/bigdata-test/kerberos/krb5.conf",
            localClientKeytabPath = "build/bigdata-test/kerberos/app_user.keytab",
            startupTimeoutSeconds = 180,
            materialTimeoutSeconds = 120,
            adminAttempts = 120,
            debug = true,
        ),
    )

Default principals are defined in BigDataTestKitOptions:

Service	Default principal
Kerberos client	app_user@EXAMPLE.COM
HDFS NameNode	nn/hdfs.example.com@EXAMPLE.COM
Hive Metastore	hive/hive-metastore.example.com@EXAMPLE.COM
Kafka broker	kafka/localhost@EXAMPLE.COM
Kafka UI	kafbat-ui/kafbat-ui.example.com@EXAMPLE.COM

The KerberosMaterialExtension publishes reusable client and service material to extension output:

[kerberosMaterial]
enabled = true
localClientKeytabCopyPaths = [
  "build/app/client.keytab",
  "build/worker/client.keytab"
]
localKrb5ConfCopyPaths = [
  "build/app/krb5.conf",
  "build/worker/krb5.conf"
]

localClientKeytabCopyPaths copies the same generated client keytab to multiple stable host-local paths. This is useful when several app frameworks, tools, or IDE run configurations expect their own file location. localKrb5ConfCopyPaths does the same for the generated client krb5.conf. Existing files at those destinations are overwritten.

Programmatic configuration is also supported:

bigDataExtensions {
    kerberosMaterial {
        localClientKeytabCopyPath("build/app/client.keytab")
        localClientKeytabCopyPath("build/worker/client.keytab")
        localKrb5ConfCopyPath("build/app/krb5.conf")
    }
}

Common output keys are:

• kerberos-material.realm

• kerberos-material.kdc

• kerberos-material.krb5-conf

• kerberos-material.client.principal

• kerberos-material.client.password

• kerberos-material.client.keytab

• kerberos-material.client.keytab.copies

• kerberos-material.client.keytab.copy.0

• kerberos-material.krb5-conf.copies

• kerberos-material.krb5-conf.copy.0

• kerberos-material.kafka.service-name

• kerberos-material.kafka.principal

• kerberos-material.kafka.keytab

9. Extensions Module

The extensions module is for config-driven setup that should not live in core or junit5. It depends on heavier client libraries where needed and runs lifecycle hooks against a started BigDataTestKit.

9.1. JUnit Extension Declaration

Use @BigDataExtensions with one or more config resources.

@BigDataExtensions("classpath:bigdata-extensions.toml")
@BigDataTest(
    hdfs = true,
    kafka = true,
    schemaRegistry = true,
    s3 = true,
)
class MyIntegrationTest {
    @Test
    fun test(kit: BigDataTestKit, extensions: BigDataExtensionResult) {
        val providerPath = extensions.required("s3-jceks.credential-provider.path")
    }
}

Extension output is exposed through BigDataExtensionResult.

9.2. Programmatic Extension Declaration

Use programmatic declaration when test data, topic names, paths, or custom extensions need to be generated at runtime. The JUnit extension combines TOML resources first, then programmatic declarations.

@BigDataExtensions
@BigDataTest(
    hdfs = true,
    kafka = true,
    schemaRegistry = true,
    s3 = true,
)
class MyIntegrationTest {
    companion object : BigDataExtensionsConfigurer {
        override fun configure(extensions: BigDataExtensionsBuilder) {
            val suffix = System.nanoTime()

            extensions.s3Jceks {
                hdfsDir = "/bigdata-test/$suffix"
                fileName = "s3.jceks"
            }

            extensions.kafkaAvro {
                topic("events-$suffix", "classpath:schemas/event.avsc") {
                    record("alpha", mapOf("id" to 1, "name" to "alpha"))
                    record("beta", mapOf("id" to 2, "name" to "beta"))
                }
            }

            extensions.s3Bucket("spark-iceberg-s3-$suffix", id = "spark-s3-bucket")
            extensions.gcsBucket("spark-iceberg-gcs-$suffix", id = "spark-gcs-bucket")
        }
    }
}

Programmatic declarations can be supplied in three ways:

• A Kotlin companion object implementing BigDataExtensionsConfigurer.

• The test instance implementing BigDataExtensionsConfigurer.

• An explicit no-arg configurer class with @BigDataExtensions(configurer = MyConfigurer::class).

Java tests can use the same explicit configurer form. Kotlin’s KClass annotation parameter is exposed to Java as a normal .class value.

@BigDataExtensions(configurer = MyExtensionsConfigurer.class)
@BigDataTest(kafka = true, schemaRegistry = true)
class MyJavaIntegrationTest {
}

public final class MyExtensionsConfigurer implements BigDataExtensionsConfigurer {
    @Override
    public void configure(BigDataExtensionsBuilder extensions) {
        extensions.kafkaAvro(kafka -> kafka.topic(
            "events",
            "classpath:schemas/event.avsc",
            topic -> topic.record("alpha", Map.of("id", 1, "name", "alpha"))
        ));
    }
}

Use extensions.extension(…​) to add a custom BigDataExtension directly when the built-in builder methods are not enough.

When TOML and programmatic declarations are used together, extension.id is the identity. If the same id appears more than once, the later declaration replaces the earlier one before any lifecycle hook runs. Programmatic declarations therefore override TOML declarations with the same id. Use distinct ids when multiple extensions of the same type should run.

9.3. Object Store Bucket Extensions

The S3 and GCS bucket extensions create buckets after the object-store services start. They are useful when bucket names are generated per test class.

extensions.s3Bucket("spark-iceberg-s3-$suffix", id = "spark-s3-bucket")
extensions.gcsBucket("spark-iceberg-gcs-$suffix", id = "spark-gcs-bucket")

Outputs:

• spark-s3-bucket.bucket

• spark-s3-bucket.s3a.uri

• spark-gcs-bucket.bucket

• spark-gcs-bucket.gs.uri

9.4. Object Store Upload Extensions

The S3 and GCS upload extensions create the target bucket by default and upload local files or folders after the object-store services start. They use the AWS S3 SDK and Google Cloud Storage SDK against the endpoints exposed by BigDataTestKit.

[[s3Uploads]]
id = "seed-s3"
bucket = "demo-s3"
prefix = "input"
sources = [
  { source = "file:src/test/resources/data/users.json", key = "input/users.json", contentType = "application/json" },
  { source = "src/test/resources/data/events", prefix = "events" },
]

[[gcsUploads]]
id = "seed-gcs"
bucket = "demo-gcs"
project = "bigdata-test"
prefix = "input"
createBucket = true
sources = [
  { source = "classpath:data/products.json", key = "input/products.json", contentType = "application/json" },
]

The equivalent table-array form is also supported and is easier to edit when each source has several fields:

[[s3Uploads]]
id = "seed-s3"
bucket = "demo-s3"
prefix = "input"

[[s3Uploads.sources]]
source = "file:src/test/resources/data/users.json"
key = "input/users.json"
contentType = "application/json"

[[s3Uploads.sources]]
source = "src/test/resources/data/events"
prefix = "events"

[[gcsUploads]]
id = "seed-gcs"
bucket = "demo-gcs"
project = "bigdata-test"
prefix = "input"
createBucket = true

[[gcsUploads.sources]]
source = "classpath:data/products.json"
key = "input/products.json"
contentType = "application/json"

Both sources = [ { …​ } ] and / are valid TOML and supported by the extension config loader. Use inline arrays for short declarations, and table arrays when the source list gets longer.

Programmatic config:

extensions.s3Upload("demo-s3") {
    id = "seed-s3"
    prefix = "input"
    file("src/test/resources/data/users.json", key = "input/users.json", contentType = "application/json")
    directory("src/test/resources/data/events", prefix = "events")
}

extensions.gcsUpload("demo-gcs") {
    id = "seed-gcs"
    project = "bigdata-test"
    file("classpath:data/products.json", key = "input/products.json", contentType = "application/json")
}

source accepts file:<path> or a plain filesystem path for files and directories. Single files can also use classpath:<resource>. Directory uploads should use filesystem paths because classpath directories are not reliably listable after resources are packaged in a jar. When key is omitted for a file, the object key is built from the extension prefix, source prefix, and file name. For directories, object keys are built from the extension prefix, source prefix, and the relative file path under the directory.

Path mapping:

Config	Source type	Final object key	Readable URI
bucket = "demo-s3", extension prefix = "input", source key = "input/users.json"	Single file with explicit key	input/users.json	s3a://demo-s3/input/users.json
bucket = "demo-s3", extension prefix = "input", source source = "file:…​/users.json", no source key or source prefix	Single file with implicit key	input/users.json	s3a://demo-s3/input/users.json
bucket = "demo-s3", extension prefix = "input", source prefix = "events", directory file 2026/01.json	Directory	input/events/2026/01.json	s3a://demo-s3/input/events/2026/01.json
bucket = "demo-gcs", extension prefix = "input", source key = "input/products.json"	Single file with explicit key	input/products.json	gs://demo-gcs/input/products.json

key is already the full object key inside the bucket. It is not prefixed again by the extension prefix. Use either key for an exact object path, or omit key and let the extension combine prefix values with the file name or directory-relative path.

Outputs:

• <id>.bucket

• <id>.s3a.uri or <id>.gs.uri

• <id>.uploaded-count

• <id>.uploaded.<index>.key

9.5. S3 JCEKS Extension

The s3Jceks extension creates a Hadoop credential provider file in HDFS and stores the S3 emulator credentials in it. It builds its Hadoop client configuration from the active HDFS endpoint properties, including fs.defaultFS, dfs.client.use.datanode.hostname, dfs.datanode.hostname, and any HDFS TLS properties exposed by the test kit.

[s3Jceks]
enabled = true
hdfsDir = "/bigdata-test/spark"
fileName = "s3.jceks"
accessKeyAlias = "fs.s3a.access.key"
secretKeyAlias = "fs.s3a.secret.key"
additionalHdfsPaths = [
  "/bigdata-test/app/s3.jceks",
  "/bigdata-test/worker/s3.jceks"
]

[s3Jceks.aliases]
"fs.s3a.encryption.algorithm" = ""
"fs.s3a.server-side-encryption-algorithm" = ""
"fs.s3a.encryption.key" = ""
"fs.s3a.server-side-encryption.key" = ""

aliases writes additional entries into the same JCEKS file. The extension writes empty defaults for Hadoop 3.4 S3A encryption aliases because S3AUtils.lookupPassword() reads those options through the credential provider even when encryption is disabled. The JCEKS provider cannot store a zero-length secret, so empty alias values are stored as a single space; S3A trims the value back to blank while reading. Override them only when your test needs S3A encryption options.

additionalHdfsPaths writes the same generated credentials to more HDFS JCEKS files. The primary file remains hdfsDir + fileName for backward compatibility. Use the plural output when the client should search all generated providers.

Outputs:

• s3-jceks.hdfs.path

• s3-jceks.hdfs.paths

• s3-jceks.credential-provider.path

• s3-jceks.credential-provider.paths

• s3-jceks.credential-provider.path.0

• s3-jceks.credential-provider.path.1

Use s3-jceks.credential-provider.path as hadoop.security.credential.provider.path or spark.hadoop.hadoop.security.credential.provider.path for the primary generated JCEKS. Use s3-jceks.credential-provider.paths when additionalHdfsPaths is configured; it is a comma-separated Hadoop credential provider path.

9.6. Kafka Avro Seed Extension

The kafkaAvro extension creates topics and produces Avro records using Schema Registry.

[kafkaAvro]
enabled = true

[[kafkaAvro.topics]]
name = "events"
schema = "classpath:schemas/event.avsc"
records = [
  { key = "alpha", value = { id = 1, name = "alpha" } },
  { key = "beta", value = { id = 2, name = "beta" } },
]

Records can be supplied inline as compact TOML arrays or from a resource. Use this extension when test data should be declarative and reproducible instead of manually wired in every test.

9.7. Spark SQL Preparation Extension

The spark-sql-prep extension starts an in-process SparkSession after the requested containers are running, executes configured SQL statements or SQL script resources, then stops Spark and clears Spark session state by default. Use it to prepare Hive Metastore databases, Iceberg tables, Hive Parquet tables, and seed data against the test HMS, S3, and GCS services before the test method runs.

The extension uses Spark from the consuming test runtime classpath. This is important for matrix tests: Apache Spark tasks should put Apache Spark on the test runtime classpath, and Cloudera Spark tasks should put the Cloudera Spark dependency line on the test runtime classpath. The extension does not choose or publish a Spark version.

[[extensions]]
type = "spark-sql-prep"
id = "prepare-tables"
master = "local[2]"
enableHiveSupport = true
stopAfterRun = true
statements = [
  "CREATE DATABASE IF NOT EXISTS demo",
]
scripts = [
  "classpath:sql/create_tables.sql",
]

[extensions.configs]
"spark.sql.extensions" = "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"
"spark.sql.catalog.s3" = "org.apache.iceberg.spark.SparkCatalog"
"spark.sql.catalog.s3.type" = "hadoop"
"spark.sql.catalog.gcs_local" = "org.apache.iceberg.spark.SparkCatalog"
"spark.sql.catalog.gcs_local.type" = "hadoop"

When useKitEndpoints = true, the default, the extension wires known endpoint properties from the active kit:

• Hive Metastore URI and HMS TLS properties

• HDFS default filesystem and DataNode hostname setting

• S3 emulator endpoint and test credentials

• GCS emulator endpoint using unauthenticated GCS connector settings

• Kerberos client krb5.conf, principal, and keytab properties when Kerberos is enabled

Explicit configs are applied after endpoint defaults and therefore override generated Spark config.

Programmatic declaration is useful when table names or warehouse locations include a runtime suffix:

extensions.sparkSqlPreparation {
    id = "prepare-tables"
    statement("CREATE DATABASE IF NOT EXISTS demo_$suffix")
    script("classpath:sql/create_tables.sql")
    config("spark.sql.shuffle.partitions", "1")
}

Java configurers can use the same builder through the Consumer overload:

extensions.sparkSqlPreparation(spark -> {
    spark.setId("prepare-tables");
    spark.statement("CREATE DATABASE IF NOT EXISTS demo");
    spark.config("spark.sql.shuffle.partitions", "1");
});

Outputs:

• prepare-tables.engine

• prepare-tables.executed-statements

The cleanup defaults are intentionally conservative for user application isolation: stopAfterRun = true, clearSparkSessions = true, and closeHadoopFileSystems = true. They reduce pollution of the test JVM, but in-process Spark is still less isolated than a separate Spark container because Spark, Hadoop, and connector libraries share the same JVM classpath as the test.

9.8. Extension Lifecycle

Extensions implement BigDataExtension and receive lifecycle events:

• AFTER_KIT_START

• BEFORE_TEST_EXECUTION

• AFTER_ALL

Built-in extensions are loaded by BigDataExtensionsConfigLoader. Future extension modules can expose a BigDataExtensionProvider through ServiceLoader; providers are selected from config entries under extensions by type.

10. Hive Metastore and Object Stores

Hive Metastore can be used in two different ways, and the distinction matters for object-store tables.

bigdata-test also supports two HMS implementations:

• hiveMetastore = true starts open-source HMS from ghcr.io/openprojectx/hive:3.1.3-hadoop-3.4.2-gcs-4.0.4-jdk17-0.1.5 by default and starts a separate PostgreSQL support container. Set [hiveMetastore] databaseType = "mysql" to use a MySQL support container instead. Set [hiveMetastore] databaseHostPort to bind the support database to a stable local port.

• clouderaHms = true starts ghcr.io/openprojectx/cloudera-hms:0.1.74, which already contains PostgreSQL in the same container. Set [clouderaHms] databaseType = "mariadb" to use the embedded MariaDB image ghcr.io/openprojectx/cloudera-hms:0.1.74-mariadb.

The endpoint contract is the same for both implementations: BigDataService.HIVE_METASTORE exposes the Thrift URI through hive.metastore.uris. Choose one implementation per test class.

To run open-source HMS with MySQL instead of PostgreSQL:

[services]
hiveMetastore = true

[hiveMetastore]
databaseType = "mysql"
databaseName = "metastore"
databaseUser = "hive"
databasePassword = "hive"
# Optional. 0 means Testcontainers picks a random host port.
databaseHostPort = 0

[images]
hiveMetastore = "ghcr.io/openprojectx/hive:3.1.3-hadoop-3.4.2-gcs-4.0.4-jdk17-0.1.5"
hiveMetastoreMysql = "mysql:8.0.44-bookworm"

databaseType is explicit: setting hiveMetastoreMysql alone only changes the MySQL image that will be used when databaseType = "mysql" is selected. databaseHostPort controls only the host-side mapped port of the support database; the HMS container still connects to the database through the Docker network alias and the database container port.

To run Cloudera HMS with the embedded MariaDB image:

[services]
clouderaHms = true

[clouderaHms]
databaseType = "mariadb"
databaseName = "metastore"
databaseUser = "hive"
databasePassword = "hive"
# Optional. 0 means Testcontainers picks a random host port for 3306.
databaseHostPort = 0

[images]
clouderaHms = "ghcr.io/openprojectx/cloudera-hms:0.1.74"
clouderaHmsMariadb = "ghcr.io/openprojectx/cloudera-hms:0.1.74-mariadb"

For databaseType = "postgresql", the Cloudera image exposes embedded PostgreSQL on container port 5432. For databaseType = "mariadb", it exposes embedded MariaDB on container port 3306. databaseHostPort controls only the host-side mapped port for that embedded database; HMS itself still runs in the same container and listens on Thrift port 9083.

The open-source image can be replaced with Hive 4 or standalone-metastore images for clients that use a compatible Hive metastore API:

[images]
hiveMetastore = "ghcr.io/openprojectx/hive:4.2.0-hadoop-3.4.2-gcs-4.0.4-jdk21-0.1.4"

or:

[images]
hiveMetastore = "ghcr.io/openprojectx/hive-standalone-metastore:4.2.0-hadoop-3.4.2-gcs-4.0.4-jdk21-0.1.4"

10.1. Spark Datasource Tables

Spark datasource DDL looks like this:

CREATE TABLE db.table (
  id INT,
  name STRING
)
USING PARQUET
LOCATION 's3a://bucket/path'

Spark writes and reads the Parquet files using the Spark driver’s Hadoop configuration. HMS stores Spark datasource metadata, such as spark.sql.sources.provider, but does not need to fully resolve the path in the same way a Hive external table does.

10.2. Hive External Tables

Hive-style DDL looks like this:

CREATE EXTERNAL TABLE db.table (
  id INT,
  name STRING
)
STORED AS PARQUET
LOCATION 's3a://bucket/path'

This asks HMS to validate table metadata and location using HMS server-side Hadoop configuration. The HMS container itself must have the right filesystem implementation, endpoint, and credentials.

10.3. HMS Server-Side S3A

When Hive Metastore starts with the S3 emulator enabled, bigdata-test injects HMS-side S3A configuration:

• fs.s3a.endpoint=http://s3:4566

• fs.s3a.endpoint.region=us-east-1

• fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider

• fs.s3a.access.key=test

• fs.s3a.secret.key=test

• fs.s3a.path.style.access=true

• fs.s3a.connection.ssl.enabled=false

• fs.s3a.change.detection.mode=none

This allows Hive external table DDL with s3a:// locations to validate against the internal S3 emulator endpoint. The Spark example includes this path and verifies the HMS table location through HiveMetaStoreClient.

10.4. HMS Server-Side HDFS

When Hive Metastore starts with HDFS enabled, bigdata-test injects HMS-side HDFS client configuration:

• fs.defaultFS=hdfs://hdfs:<namenode-port>

• dfs.client.use.datanode.hostname=true

• dfs.datanode.hostname=<configured-data-node-hostname>

This lets HMS resolve hdfs:// warehouse and table locations from inside the Docker network. The default namenode host is the container network alias hdfs; if you override HDFS ports or datanode advertisement, the HMS config follows the same test-kit options.

10.5. HMS Server-Side GCS

When Hive Metastore starts with the GCS emulator enabled, bigdata-test injects HMS-side GCS configuration based on the Google Cloud Storage connector configuration model:

• fs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem

• fs.AbstractFileSystem.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS

• fs.gs.project.id=bigdata-test

• fs.gs.storage.root.url=http://fake-gcs:4588/

• fs.gs.storage.service.path=storage/v1/

• fs.gs.client.type=HTTP_API_CLIENT

• fs.gs.auth.type=UNAUTHENTICATED

• fs.gs.status.parallel.enable=false

• fs.gs.create.items.conflict.check.enable=false

• fs.gs.implicit.dir.repair.enable=false

• fs.gs.hierarchical.namespace.folders.enable=false

• fs.gs.max.requests.per.batch=1

• fs.gs.operation.move.enable=false

• fs.gs.copy.with.rewrite.enable=false

• fs.gs.client.upload.type=WRITE_TO_DISK_THEN_UPLOAD

• fs.gs.outputstream.direct.upload.enable=false

This configuration does not fail HMS startup by itself. HMS can start with these keys even if the GCS connector jar is not present. Creating or validating a gs:// Hive external table requires the HMS image to include the GCS connector and its runtime dependencies. The default open-source HMS image and the ghcr.io/openprojectx/cloudera-hms image include those dependencies.

If hiveMetastore = true fails for HMS-backed object-store tables, see Open-Source HMS Image Issues. That note captures the expected /data/Git/hive-docker image contract and the remaining Hive 4 compatibility note for Spark 3.x clients.

11. Gradle Plugin

Use the Gradle plugin when the test kit should be managed by Gradle instead of the application or test framework. This keeps container startup out of the Spring application context and out of JUnit annotations, while still starting the services before bootRun, run, or other JavaExec tasks launch their JVMs. Test task wiring is opt-in.

plugins {
    id("org.openprojectx.bigdata-test") version "0.1.12-SNAPSHOT"
}

bigDataTest {
    config.add("classpath:bigdata-test.toml")
    extensionConfig.add("file:${projectDir}/src/main/resources/bigdata-extensions.toml")
    extensionRuntime {
        hadoopVersion.set("3.4.2")
        sparkVersion.set("3.5.7")
    }
}

config loads startup TOML using the same table names as JUnit startup config where the Gradle plugin has matching fields. classpath: locations are resolved from Gradle source-set resources, including src/main/resources and src/test/resources. It is applied as Gradle property conventions, so explicit DSL values win over TOML values:

[services]
kerberos = true
hdfsKerberos = true
s3 = true

[kerberos]
clientPrincipal = "app_user@EXAMPLE.COM"
clientPassword = "app-user-secret"
debug = true

[ports]
hdfsNameNode = 8020
s3 = 4566

Use extensionConfig for post-start provisioning TOML such as bucket creation, JCEKS generation, Kafka messages, and Spark SQL preparation. The same startup values can also be configured directly with the DSL:

bigDataTest {
    services {
        kerberos.set(true)
        hdfs.set(true)
        s3.set(true)
    }
    hdfs {
        kerberosEnabled.set(true)
        dataNodeHostname.set("localhost")
        localHdfsSitePath.set("build/bigdata-test/hadoop/hdfs-site.xml")
    }
    ports {
        hdfsNameNode.set(8020)
        hdfsDataNode.set(9866)
        hdfsWeb.set(9870)
        s3.set(4566)
    }
    containerLogs {
        mode.set(ContainerLogMode.STDOUT)
    }
    containerLogLevels.put("hdfs", "DEBUG")
}

Extension provisioning dependencies are resolved into the plugin-owned bigDataTestExtensionRuntime configuration. They are not added to the application implementation, runtimeClasspath, testImplementation, or testRuntimeClasspath configurations.

By default the plugin uses the shaded extensions:<version>:runtime artifact. This keeps Spark, Hadoop, Kafka, Avro, thin Iceberg Spark/Hive catalog modules, and their transitive runtime dependencies managed by bigdata-test instead of letting the application dependency graph choose those versions. The extension runtime classloader prefers its own jars and delegates shared bigdata-test core types to the Gradle plugin parent classloader.

Application dependencies such as implementation, runtimeOnly, testImplementation, and testRuntimeOnly are not added to the bigdata-test extension runtime. Dependencies explicitly added to bigDataTestExtensionRuntime are different: they are intentionally part of the extension runtime and can affect extension execution. Use that configuration only for optional extension-only libraries that bigdata-test does not already provide.

If you need to debug or override individual runtime dependencies, disable the shaded artifact. In that mode, the plugin adds the lightweight extensions artifact and then adds runtime libraries based on the configured extension TOML:

• [s3Jceks] enables Hadoop client/runtime/AWS dependencies.

• [kafkaAvro] enables Confluent Schema Registry, Kafka Avro, and Avro dependencies.

• sparkSqlPreparation enables Spark SQL, Spark Hive, and Hadoop dependencies.

Override dependency versions with the DSL:

bigDataTest {
    extensionRuntime {
        useShadedArtifact.set(false)
        hadoopVersion.set("3.4.2")
        sparkVersion.set("3.5.7")
        confluentVersion.set("8.2.1")
        avroVersion.set("1.12.1")
        lz4Version.set("1.10.1")
        awsSdkVersion.set("2.41.5")
    }
}

When Spark and Confluent Kafka dependencies are both present, the plugin selects at.yawk.lz4:lz4-java for Gradle’s shared org.lz4:lz4-java capability. Override lz4Version if the selected Confluent line uses a different compatible lz4 artifact.

For Hadoop S3A, the plugin excludes software.amazon.awssdk:bundle from hadoop-aws and adds the modular software.amazon.awssdk:s3 and software.amazon.awssdk:s3-transfer-manager dependencies instead. S3A loads transfer-manager classes at runtime, so both modules are required while still avoiding the full AWS SDK bundle.

When spark-sql-prep is used by bigDataTestStart or bigDataTestRun, Spark starts inside the Gradle JVM. On Java 17, add Spark’s required module access to the Gradle daemon JVM, for example in gradle.properties:

org.gradle.jvmargs=-Xmx4096M -Dfile.encoding=UTF-8 --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED

These flags cannot be applied after the Gradle daemon has already started. If a daemon was already running without them, stop it with ./gradlew --stop or run the next command with --no-daemon.

When auto-detection is not enough, force dependency groups explicitly:

bigDataTest {
    extensionRuntime {
        autoDetect.set(false)
        includeHadoop.set(true)
        includeKafkaAvro.set(true)
        includeSpark.set(true)
    }
}

The extensions module publishes the shaded runtime artifact with classifier runtime. This is the default plugin mode:

bigDataTest {
    extensionRuntime {
        useShadedArtifact.set(true)
        extensionsVersion.set("0.1.12-SNAPSHOT")
    }
}

With useShadedArtifact = true, the plugin resolves org.openprojectx.bigdata.test.core:extensions:<version>:runtime@jar into bigDataTestExtensionRuntime and skips adding separate Hadoop, Spark, Iceberg, Kafka, and Avro extension runtime dependencies.

The normal extensions artifact does not package Spark, Hadoop, Kafka, Log4j, or SLF4J implementations; those dependencies are supplied by the plugin-owned runtime configuration when needed. The shaded extensions:<version>:runtime artifact does package extension runtime dependencies, including logging libraries pulled in transitively by Spark such as slf4j-api, log4j-api, log4j-core, and log4j-slf4j2-impl. Jackson is relocated inside this shaded artifact to avoid collisions with Jackson versions used by the application or Gradle build logic.

11.1. Gradle Plugin Customization Surface

The Gradle plugin has three separate classpath/resource concepts:

Area	How to customize	Notes
Application classpath	Use normal Gradle configurations such as implementation, runtimeOnly, testImplementation, and testRuntimeOnly.	These dependencies are used by the app and tests. They are not added to bigDataTestRun or extension provisioning.
Extension runtime classpath	Use the bigDataTestExtensionRuntime configuration, or set extensionRuntime.useShadedArtifact.set(false) and configure hadoopVersion, sparkVersion, icebergVersion, confluentVersion, avroVersion, awsSdkVersion, and lz4Version.	This classpath is used only by bigdata-test extension execution. Dependencies added here can affect spark-sql-prep, JCEKS generation, Kafka Avro seeding, and other extensions.
Extension config and SQL/resource lookup	Use extensionConfig.add(…​). classpath: resources are searched from the extension runtime first and then from the project’s Gradle source-set resource directories, including src/main/resources, src/test/resources, and compiled resource outputs.	There is no dedicated DSL property for extra arbitrary resource directories today. Put files under a Gradle source-set resource directory, reference an absolute or relative file: path, or add a small resource jar to bigDataTestExtensionRuntime.
Gradle task JVM	Configure Gradle itself with org.gradle.jvmargs, GRADLE_OPTS, or JAVA_TOOL_OPTIONS.	bigDataTestStart and bigDataTestRun execute inside the Gradle JVM. Spark module-open flags and logging configuration for Spark SQL preparation must be visible when the Gradle daemon starts.
Gradle task environment	Set environment variables on the Gradle command, for example TESTCONTAINERS_RYUK_DISABLED=true ./gradlew bigDataTestRun.	bigDataTestRun is not a JavaExec task, so it does not have a task-local environment {} API. It sees the Gradle process environment.
Application task injected properties	Set injectRawEndpointProperties, injectNamespacedEndpointProperties, and injectEnvironmentVariables.	For auto-wired JavaExec and opt-in Test tasks, endpoint and extension output values are injected as JVM system properties and, when enabled, environment variables.
Container environment, files, mounts, ports, network mode	Use startup TOML [containers.<service>.*] tables or programmatic withContainerCustomization(…​).	This customizes Docker containers, not the Gradle task process. Prefer copied files over host mounts in CI.

Example extension-only dependency override:

dependencies {
    add("bigDataTestExtensionRuntime", "com.example:test-only-extension-helper:1.0.0")
}

Example task process environment:

TESTCONTAINERS_RYUK_DISABLED=true GRADLE_USER_HOME=/data/.gradle ./gradlew bigDataTestRun

Example resource-backed Spark SQL preparation:

[[extensions]]
type = "spark-sql-prep"
scripts = ["classpath:sql/create_tables.sql"]

Place the script at src/main/resources/sql/create_tables.sql, or use a file: URL when the SQL lives outside the project resources.

The shaded runtime jar intentionally favors isolation over small size. It uses the thin Iceberg Spark modules plus iceberg-hive-metastore instead of the fat iceberg-spark-runtime jar, but still includes enough Hadoop, Spark, Iceberg, Kafka Avro, and AWS runtime code for config-driven provisioning without borrowing the user application’s dependency graph. iceberg-hive-metastore is required when Spark SQL preparation uses an Iceberg HiveCatalog. The largest remaining optional payload is iceberg-aws-bundle, which is kept for Iceberg S3 FileIO and AWS catalog support. If size or dependency inspection matters more than isolation, set useShadedArtifact = false and let the plugin resolve only the runtime families detected from extensionConfig, or add the exact artifacts you want to bigDataTestExtensionRuntime.

Gradle configuration is Gradle-native; the plugin does not read a Maven POM. When values should come from gradle.properties or -P, wire them through Gradle providers:

bigDataTest {
    services {
        kerberos.set(true)
    }
    kerberos {
        clientPrincipal.set(providers.gradleProperty("bigdata.test.kerberos.clientPrincipal"))
        clientPassword.set(providers.gradleProperty("bigdata.test.kerberos.clientPassword"))
        materialDirectory.set(layout.buildDirectory.dir("bigdata-test/kerberos").map { it.asFile.absolutePath })
        localKrb5ConfPath.set(layout.buildDirectory.file("bigdata-test/kerberos/krb5.conf").map { it.asFile.absolutePath })
        localClientKeytabPath.set(layout.buildDirectory.file("bigdata-test/kerberos/app_user.keytab").map { it.asFile.absolutePath })
    }
}

The plugin registers:

• bigDataTestStart - starts the configured kit for the current Gradle build and then returns. This task is intended as a dependency of app tasks, or test tasks when explicitly opted in.

• bigDataTestRun - starts the configured kit and keeps the Gradle process running until interrupted. Use this for manual troubleshooting.

• bigDataTestStop - explicitly stops the kit in the current Gradle build.

By default, the plugin wires bigDataTestStart before JavaExec tasks and injects endpoint and extension output properties as JVM system properties immediately before the task launches. It also injects environment variables derived from the same property map. Test tasks are not auto-wired by default; set autoConfigureTestTasks.set(true) when tests should start and receive the kit automatically. The kit is closed automatically when the Gradle build service closes at the end of the build.

bigDataTestStart is not an interactive keep-alive command. If you run it directly, it starts the kit, prints the injected property count, and exits when the build finishes. For a long-running local environment, run:

GRADLE_USER_HOME=/data/.gradle ./gradlew bigDataTestRun

Testcontainers runtime environment variables, such as TESTCONTAINERS_RYUK_DISABLED, must be visible to the Gradle JVM that runs the kit. If a Gradle daemon is already running, a one-off shell prefix might not be visible to that daemon. Use --no-daemon, or stop existing daemons before running the task:

TESTCONTAINERS_RYUK_DISABLED=true GRADLE_USER_HOME=/data/.gradle ./gradlew --no-daemon bigDataTestRun

11.2. Gradle Plugin Logging

bigDataTestRun starts the kit and extension provisioning inside the Gradle JVM. Configure JVM logging through Gradle daemon JVM properties, not through the application bootRun or test task.

For the default plugin logging backend, use SLF4J Simple properties in gradle.properties:

org.gradle.jvmargs=-Xmx4096M -Dfile.encoding=UTF-8 \
  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED \
  --add-opens=java.base/java.nio=ALL-UNNAMED \
  --add-opens=java.base/java.net=ALL-UNNAMED \
  -Dorg.slf4j.simpleLogger.defaultLogLevel=info \
  -Dorg.slf4j.simpleLogger.log.org.openprojectx.bigdata.test=debug \
  -Dorg.slf4j.simpleLogger.log.org.testcontainers=info

Spark SQL preparation is different when it runs through the Gradle plugin: Spark starts in-process inside the Gradle daemon, and Spark logs are routed through Gradle’s logging bridge. Gradle controls the console rendering, so Log4j2 console pattern and ANSI color settings are not honored for those Spark extension logs. Use Gradle flags such as --info and extension/Spark log-level settings for verbosity. Full Log4j2 pattern/color control would require a separate Spark preparation JVM instead of the current in-process plugin execution model.

If a Gradle daemon is already running, restart it before expecting new org.gradle.jvmargs values to apply:

./gradlew --stop
GRADLE_USER_HOME=/data/.gradle ./gradlew bigDataTestRun --info

Container logs are configured separately from Gradle JVM logs. Use startup TOML when you want service stdout/stderr in files:

[containerLogs]
mode = "FILE"
directory = "build/container-logs"
append = false

or send container logs to the Gradle console:

[containerLogs]
mode = "STDOUT"

Disable automatic task wiring when you want explicit task dependencies:

bigDataTest {
    autoConfigureJavaExecTasks.set(false)
    autoConfigureTestTasks.set(false)
}

tasks.named("bootRun") {
    dependsOn(tasks.named("bigDataTestStart"))
    finalizedBy(tasks.named("bigDataTestStop"))
}

The plugin injects raw endpoint properties, such as fs.defaultFS, hive.metastore.uris, bootstrap.servers, and object-store endpoint keys, and namespaced copies:

bigdata.test.endpoint.hdfs.properties.fs.defaultFS
bigdata.test.endpoint.hdfs.properties.bigdata.test.hdfs.hdfs-site
bigdata.test.endpoint.hdfs.ports.namenode
bigdata.test.endpoint.hive-metastore.properties.bigdata.test.hive-metastore.hive-site
bigdata.test.endpoint.hive-metastore.properties.bigdata.test.hive-metastore.metastore-site
bigdata.test.endpoint.hive-metastore.properties.hive.metastore.uris
bigdata.test.endpoint.s3.properties.aws.endpoint-url.s3
bigdata.test.extensions.s3-jceks.credential-provider.path

For HDFS clients that prefer Hadoop XML resources, read bigdata.test.endpoint.hdfs.properties.bigdata.test.hdfs.hdfs-site and add that file to the Hadoop Configuration. The Gradle plugin also injects the same value as an environment variable for launched tasks.

For HMS clients that prefer XML resources, read bigdata.test.endpoint.hive-metastore.properties.bigdata.test.hive-metastore.hive-site or bigdata.test.endpoint.hive-metastore.properties.bigdata.test.hive-metastore.metastore-site and add the file to the Hive/Hadoop client configuration. The Gradle plugin injects both values as system properties and environment variables for launched tasks.

Environment variable names are generated by uppercasing property keys and replacing non-alphanumeric characters with _:

bigdata.test.endpoint.hdfs.properties.fs.defaultFS
BIGDATA_TEST_ENDPOINT_HDFS_PROPERTIES_FS_DEFAULTFS

bigdata.test.endpoint.s3.properties.aws.endpoint-url.s3
BIGDATA_TEST_ENDPOINT_S3_PROPERTIES_AWS_ENDPOINT_URL_S3

Disable environment-variable injection when an application should only receive JVM properties:

bigDataTest {
    injectEnvironmentVariables.set(false)
}

This lets application configuration reference Gradle-managed services without including bigdata-test Spring auto-configuration:

example:
  s3:
    hadoop:
      fs.defaultFS: ${bigdata.test.endpoint.hdfs.properties.fs.defaultFS}
      fs.s3a.endpoint: ${bigdata.test.endpoint.s3.properties.aws.endpoint-url.s3}
      hadoop.security.authentication: kerberos
      dfs.data.transfer.protection: authentication
      hadoop.security.credential.provider.path: ${bigdata.test.extensions.s3-jceks.credential-provider.path}

The plugin classpath is isolated from the application runtime classpath. Hadoop, Kafka, Avro, and Spark dependencies needed by provisioning extensions run in Gradle and are not added to the user application unless the user declares them separately. This is launched-JVM isolation, not per-extension isolation inside the Gradle daemon.

12. Spring Boot

The Spring Boot starter creates and starts a BigDataTestKit bean when enabled.

bigdata:
  test:
    enabled: true
    hive-metastore:
      enabled: true
      image: ghcr.io/openprojectx/hive:3.1.3-hadoop-3.4.2-gcs-4.0.4-jdk17-0.1.5
      database-type: postgresql
      database-image: postgres:16-alpine
    kafka:
      enabled: true
      kerberos-enabled: true
      schema-registry-enabled: true
      schema-registry-kerberos-enabled: true
    s3:
      enabled: true

The bean is closed with the application context.

Use bigdata.test.cloudera-hms.enabled=true instead of bigdata.test.hive-metastore.enabled=true for the embedded-Postgres Cloudera HMS image. The two HMS implementations are mutually exclusive.

The extensions module can also run from Spring Boot. This is opt-in and uses the same TOML extension files as JUnit tests. Add the extensions artifact to the application runtime classpath, then enable extension startup:

bigdata:
  extensions:
    enabled: true
    config:
      - classpath:bigdata-extensions.toml

Spring starts the BigDataTestKit first, then runs configured extensions as an early ApplicationRunner. This allows post-start provisioning such as creating buckets and writing an S3A JCEKS file to HDFS before application runners use the services.

The Spring extension bridge also accepts task-time overrides:

./gradlew bootRun \
  -Dbigdata.extensions.config=file:/path/to/extensions.toml \
  -Dbigdata.extensions.config.replace=true

Without bigdata.extensions.config.replace=true, system-property locations are appended after the YAML locations.

Spring Boot configuration also supports the same fixed-port model used by JUnit and programmatic tests:

bigdata:
  test:
    ports:
      same-host-ports: false
      s3: 4566
      hdfs-name-node: 8020
      hdfs-data-node: 9866
      hdfs-web: 9870
      kerberos-kdc: 88

Leave a port as 0 or omit it to use a random host port.

Spring Boot configuration also supports container log routing and per-service log levels:

bigdata:
  test:
    container-logs:
      mode: stdout
      directory: build/bigdata-test-container-logs
    container-log-levels:
      hdfs: DEBUG
      kerberos: DEBUG

mode: stdout streams logs to the main application process. Use mode: file to keep one file per service under directory; this is the better default for workflow artifacts and post-failure diagnostics. For HDFS, DEBUG or TRACE enables verbose Hadoop NameNode/DataNode logs.

13. Example Projects

13.1. JUnit Example

The JUnit examples live under example/junit. They are annotated with @Disabled by default so ordinary builds do not start containers unexpectedly. Remove @Disabled from an example class to run it.

13.2. Spring Example

The Spring example is a normal Spring Boot web application. Its default configuration keeps the S3A REST API disabled, so CI builds do not start containers. The bigdata-test Spring starter is added to the application runtime classpath only outside CI, which lets local IDE runs use the local profile without making CI resolve or package the bigdata-test runtime.

GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spring:check

The local developer profile is fully configuration driven. The app source does not call BigDataTestKit; application-local.yaml starts Kerberos, HDFS, and the S3 emulator and enables the REST API.

The local profile also enables HDFS and Kerberos container logs on stdout:

bigdata:
  test:
    container-logs:
      mode: stdout
    container-log-levels:
      hdfs: DEBUG
      kerberos: DEBUG

For Gradle, use bootRunLocal. It runs the app with spring.profiles.active=local; the Spring extension bridge reads spring-bigdata-extensions.toml, creates the S3 bucket, and writes the S3A JCEKS file to HDFS:

GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spring:bootRunLocal

For IntelliJ IDEA, run org.openprojectx.bigdata.test.example.spring.BigDataTestExampleApplicationKt with active profile local. Because the starter and extensions dependencies are runtimeOnly for non-CI builds, the same config-driven container startup and provisioning works from the IDE.

The generated JCEKS file is written to HDFS:

/bigdata-test/spring/s3.jceks

The local profile configures S3A with:

• fs.s3a.endpoint=http://localhost:4566

• fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider

• fs.defaultFS=hdfs://localhost:8020

• hadoop.security.credential.provider.path=jceks://hdfs/bigdata-test/spring/s3.jceks

• hadoop.tmp.dir=${user.dir}/build/spring-local/hadoop-tmp

The Spring extension bootstrap runs before normal application beans are instantiated, so eager Hadoop clients can rely on extension-provisioned resources such as HDFS JCEKS files.

Use the API after bootRunLocal starts:

curl -sS -X PUT \
  -H 'Content-Type: text/plain' \
  --data 'hello-s3a' \
  http://localhost:8080/api/s3/objects/demo.txt

curl -sS http://localhost:8080/api/s3/objects/demo.txt

curl -sS http://localhost:8080/api/s3/objects

curl -sS -X DELETE http://localhost:8080/api/s3/objects/demo.txt

13.3. Spark Example

The Spark example lives under example/spark. It starts HDFS, Hive Metastore, Kafka, Schema Registry, Floci S3, and Floci GCS.

The example uses hiveMetastore = true with the Hive 3 open-source HMS image. This keeps Spark 3.x on a compatible HMS protocol while still exercising server-side S3A and GCS configuration. Use clouderaHms = true when the embedded-Postgres Cloudera HMS image is the target under test.

The test demonstrates:

• Creating an S3 JCEKS file on HDFS through the extensions module.

• Producing Avro records to Kafka through Schema Registry.

• Reading Kafka data from Spark.

• Writing Iceberg tables to S3 and GCS through Hadoop catalogs.

• Creating an HMS-backed Iceberg table and asserting HMS metadata with HiveMetaStoreClient.

• Creating Hive external Parquet tables stored on S3 and GCS and asserting HMS location and Parquet metadata.

The GCS Hive external table stores a gs:// location in HMS and Spark reads the Parquet data through the GCS connector.

Run it with:

GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:test

The Spark example also has Gradle tasks for a dependency/HMS/Kerberos matrix:

GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkBigDataMatrixTest

The aggregate task runs these cells:

Task	Dependency line	HMS	Kerberos
sparkApacheDepsApacheHmsTest	Apache Spark/Hadoop	Open-source Hive 3 HMS	No
sparkApacheDepsApacheHmsKerberosTest	Apache Spark/Hadoop	Open-source Hive 3 HMS	Yes
sparkApacheDepsClouderaHmsTest	Apache Spark/Hadoop	Cloudera HMS	No
sparkApacheDepsClouderaHmsKerberosTest	Apache Spark/Hadoop	Cloudera HMS	Opt-in
sparkClouderaDepsApacheHmsTest	Cloudera Spark/Hadoop	Cloudera HMS	No
sparkClouderaDepsApacheHmsKerberosTest	Cloudera Spark/Hadoop	Cloudera HMS	Opt-in
sparkClouderaDepsClouderaHmsTest	Cloudera Spark/Hadoop	Cloudera HMS	No
sparkClouderaDepsClouderaHmsKerberosTest	Cloudera Spark/Hadoop	Cloudera HMS	Opt-in

The Kerberos axis currently enables the shared KDC and Kafka Kerberos path used by the Spark Kafka source. The HMS axis switches between the open-source Hive 3 HMS image and the Cloudera HMS image for Apache Spark/Hadoop dependencies. Cloudera Spark/Hadoop dependencies always run with Cloudera HMS because that Hive client calls HMS Thrift APIs, such as get_database_req, that the open-source Hive 3.1.3 HMS server does not implement. The dependency axis switches the test JVM runtime classpath; it is intentionally implemented in Gradle rather than TOML.

sparkClouderaDepsApacheHmsTest and sparkClouderaDepsApacheHmsKerberosTest are compatibility task names only; both use Cloudera HMS.

Cloudera HMS Kerberos cells are skipped by default while the Cloudera HMS image-side auth configuration is being stabilized. Enable them explicitly when troubleshooting that path:

GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkBigDataMatrixTest \
  -Pbigdata.spark.enableClouderaHmsKerberos=true

Current constraints:

• The default matrix still exercises Kerberos with the open-source Hive HMS path. That includes KDC startup, client keytab login, HDFS Kerberos, HMS access to secured HDFS, Kafka Kerberos, and the Spark S3/GCS smoke checks.

• Cloudera HMS plaintext cells run by default.

• Cloudera HMS Kerberos cells are disabled by default because the current Cloudera HMS image path can fail before the thrift endpoint is usable during HMS server-side Kerberos transport setup. In the failing path, HMS accepts the server principal but the client-facing Kerberos principal is not resolved correctly by the server image, so the process exits before Spark can connect.

• Enabling HDFS Kerberos can require HMS Kerberos when HMS needs to access HDFS-backed configuration or warehouse paths. The test framework wires HMS HDFS client security and HDFS proxy-user properties for that path.

• HMS TLS is independent from HMS Kerberos. The Kerberos matrix files do not enable HMS TLS unless the TOML explicitly sets hiveMetastoreTls.

To inspect the resolved dependency tree for each Spark dependency line, use Gradle’s dependencies task with the resolvable classpath configurations from example/spark/build.gradle.kts:

GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:dependencies \
  --configuration apacheSparkRuntimeClasspath

GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:dependencies \
  --configuration clouderaSparkRuntimeClasspath

Use dependencyInsight when checking why a specific module or version is present:

GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:dependencyInsight \
  --configuration apacheSparkRuntimeClasspath \
  --dependency org.apache.spark:spark-sql_2.12

GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:dependencyInsight \
  --configuration clouderaSparkRuntimeClasspath \
  --dependency org.apache.hadoop:hadoop-common

GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:dependencyInsight \
  --configuration clouderaSparkRuntimeClasspath \
  --dependency org.apache.iceberg

testRuntimeClasspath is not the right configuration for comparing the matrix lines because the Spark matrix overrides each test task’s runtime classpath. Use apacheSparkRuntimeClasspath for the Apache Spark/Hadoop line and clouderaSparkRuntimeClasspath for the Cloudera line.

Legacy HMS-only task aliases are still present:

GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkApacheHmsTest
GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkApacheHmsKerberosTest
GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkClouderaHmsTest
GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:sparkClouderaHmsKerberosTest

The sparkClouderaHmsTest and sparkClouderaHmsKerberosTest aliases select the Cloudera dependency line and Cloudera HMS. Use sparkApacheDepsClouderaHmsTest when you specifically want Apache Spark/Hadoop dependencies against Cloudera HMS.

Spark example test tasks always execute instead of reusing previous test outputs, because these are container smoke tests. The shared Spark TOML writes service container logs to example/spark/build/bigdata-test-container-logs; Gradle’s test output contains the test JVM logs and points to the HTML test report on failure.

14. Troubleshooting

14.1. HADOOP_HOME and hadoop.home.dir are unset

This Hadoop message can appear in test output when Hadoop checks for native utilities. It is usually not fatal. Hadoop falls back to built-in Java classes.

14.2. S3 Hive External Table Fails with Credential Errors

If a Hive external table on s3a:// fails with an AWS credential error, check whether HMS was started with the S3 emulator enabled and whether custom HiveMetastoreOptions.extraConfiguration overrides the S3A settings.

The important distinction is where the path is resolved:

• Spark datasource table: Spark client-side Hadoop config is usually enough.

• Hive external table: HMS server-side Hadoop config must be correct.

14.3. GCS Hive External Table Fails with ClassNotFoundException

If HMS returns ClassNotFoundException: com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem, the HMS image does not include the GCS connector jar. The config can be present and HMS can still start, but the jar is required when HMS validates a gs:// path. Use the default open-source HMS image, the Cloudera HMS image, or another HMS image that includes the GCS connector and runtime dependencies.

14.4. Container Startup Failures

Enable container logs for the affected test:

@BigDataTest(
    hiveMetastore = true,
    containerLogMode = ContainerLogMode.FILE,
    containerLogDirectory = "build/container-logs",
)
class HmsDebugTest

Then inspect the service-specific log files under the configured directory.