You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feat(core): replace hardcoded S3 and GCS emulator images with constants
- Introduced `DEFAULT_LOCALSTACK_S3_IMAGE` and `DEFAULT_FAKE_GCS_IMAGE` constants for centralizing default image configuration.
- Updated `BigDataTestKit`, `BigDataContainerFactory`, Gradle plugin, and related test configurations to reference constants instead of hardcoded image strings.
- Enhanced support for Floci S3 and GCS emulators with dynamic health endpoint selection and conditional configuration.
- Updated documentation and examples to reflect emulator-specific changes and default configurations.
Copy file name to clipboardExpand all lines: doc/user-guide.adoc
+26-26Lines changed: 26 additions & 26 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -16,8 +16,8 @@ The project currently provides containers and test integration for:
16
16
* Kafka
17
17
* Schema Registry
18
18
* Kafka UI
19
-
* LocalStack S3
20
-
* fake-gcs-server
19
+
* Floci S3-compatible AWS emulator
20
+
* Floci GCP-compatible GCS emulator
21
21
* Kerberos KDC and service principals
22
22
23
23
The project is split so lightweight users can depend on `core` or `junit5` without pulling Spark, Hadoop client utilities, or storage SDKs. Heavier setup such as JCEKS creation, Kafka Avro seeding, and Spark SQL preparation lives in the `extensions` module.
@@ -213,10 +213,10 @@ The JUnit extension starts the kit before all tests in the class and closes it a
213
213
|Configures Kafka UI for Kerberos/SASL Kafka access.
Kafka and the open-source HMS database sidecar use typed Testcontainers classes instead of plain generic containers. When you override those images, `bigdata-test` marks the configured Kafka image as a compatible substitute for `apache/kafka`, PostgreSQL as a compatible substitute for `postgres`, and MySQL as a compatible substitute for `mysql`, so private mirrors or rebuilt images can still pass Testcontainers image checks.
@@ -521,11 +521,11 @@ Current CLI probes:
521
521
|Kafka UI
522
522
|Uses `wget` or `curl` against `/`.
523
523
524
-
|LocalStack S3
524
+
|Floci S3-compatible AWS emulator
525
525
|Uses `awslocal s3 ls` when available, then falls back to `aws --endpoint-url`, then HTTP health.
526
526
527
-
|fake-gcs-server
528
-
|Uses `wget` or `curl` against `/storage/v1/b`.
527
+
|Floci GCP-compatible GCS emulator
528
+
|Uses `wget` or `curl` against the configured HTTP health endpoint.
529
529
|===
530
530
531
531
CLI probes validate the container's own environment. For host-side behavior, such as Windows HDFS clients connecting through mapped ports, use a host/JVM smoke test as well because an in-container `hdfs dfs` command cannot prove host network reachability.
@@ -834,7 +834,7 @@ To bind each enabled service to the same localhost port it uses inside the conta
834
834
class MySameHostPortTest
835
835
----
836
836
837
-
This maps common service ports directly, for example HDFS `8020`, `9866`, and `9870`, Hive Metastore `9083`, Kafka `9092`, Schema Registry `8085`, LocalStack `4566`, and fake GCS `4443`. Explicit per-service port fields still take priority over `sameHostPorts`.
837
+
This maps common service ports directly, for example HDFS `8020`, `9866`, and `9870`, Hive Metastore `9083`, Kafka `9092`, Schema Registry `8085`, Floci S3 `4566`, and Floci GCS `4588`. Explicit per-service port fields still take priority over `sameHostPorts`.
838
838
839
839
TLS gateway ports are not affected by `sameHostPorts`, because every gateway listens on container port `443`. Configure the per-service TLS port fields explicitly if a stable HTTPS host port is needed.
840
840
@@ -913,22 +913,22 @@ Available fixed host-port config:
913
913
|`kafkaUiTls`
914
914
|`443` on the HAProxy gateway
915
915
916
-
|LocalStack S3 HTTP
916
+
|Floci S3 HTTP
917
917
|`localStackS3`
918
918
|`localStackS3Port`
919
919
|`4566`
920
920
921
-
|LocalStack S3 TLS gateway
921
+
|Floci S3 TLS gateway
922
922
|`localStackS3Tls`
923
923
|`localStackS3Tls`
924
924
|`443` on the HAProxy gateway
925
925
926
-
|fake-gcs-server HTTP
926
+
|Floci GCS HTTP
927
927
|`fakeGcs`
928
928
|`fakeGcsPort`
929
-
|`4443`
929
+
|`4588` by default, or `4443` when overriding the image to `fsouza/fake-gcs-server`
930
930
931
-
|fake-gcs-server TLS gateway
931
+
|Floci GCS TLS gateway
932
932
|`fakeGcsTls`
933
933
|`fakeGcsTls`
934
934
|`443` on the HAProxy gateway
@@ -1003,7 +1003,7 @@ Each started service exposes a `BigDataEndpoint` with named ports and connection
The `s3Jceks` extension creates a Hadoop credential provider file in HDFS and stores LocalStack S3 credentials in it.
1351
+
The `s3Jceks` extension creates a Hadoop credential provider file in HDFS and stores the S3 emulator credentials in it.
1352
1352
It builds its Hadoop client configuration from the active HDFS endpoint properties, including `fs.defaultFS`, `dfs.client.use.datanode.hostname`, `dfs.datanode.hostname`, and any HDFS TLS properties exposed by the test kit.
1353
1353
1354
1354
[source,toml]
@@ -1421,8 +1421,8 @@ When `useKitEndpoints = true`, the default, the extension wires known endpoint p
1421
1421
1422
1422
* Hive Metastore URI and HMS TLS properties
1423
1423
* HDFS default filesystem and DataNode hostname setting
1424
-
* LocalStack S3 endpoint and test credentials
1425
-
* fake-gcs-server endpoint using unauthenticated GCS connector settings
1424
+
* S3 emulator endpoint and test credentials
1425
+
* GCS emulator endpoint using unauthenticated GCS connector settings
1426
1426
* Kerberos client `krb5.conf`, principal, and keytab properties when Kerberos is enabled
1427
1427
1428
1428
Explicit `configs` are applied after endpoint defaults and therefore override generated Spark config.
@@ -1572,7 +1572,7 @@ This asks HMS to validate table metadata and location using HMS server-side Hado
1572
1572
1573
1573
=== HMS Server-Side S3A
1574
1574
1575
-
When Hive Metastore starts with LocalStack S3 enabled, `bigdata-test` injects HMS-side S3A configuration:
1575
+
When Hive Metastore starts with the S3 emulator enabled, `bigdata-test` injects HMS-side S3A configuration:
1576
1576
1577
1577
* `fs.s3a.endpoint=http://localstack:4566`
1578
1578
* `fs.s3a.endpoint.region=us-east-1`
@@ -1583,7 +1583,7 @@ When Hive Metastore starts with LocalStack S3 enabled, `bigdata-test` injects HM
1583
1583
* `fs.s3a.connection.ssl.enabled=false`
1584
1584
* `fs.s3a.change.detection.mode=none`
1585
1585
1586
-
This allows Hive external table DDL with `s3a://` locations to validate against the internal LocalStack endpoint. The Spark example includes this path and verifies the HMS table location through `HiveMetaStoreClient`.
1586
+
This allows Hive external table DDL with `s3a://` locations to validate against the internal S3 emulator endpoint. The Spark example includes this path and verifies the HMS table location through `HiveMetaStoreClient`.
1587
1587
1588
1588
=== HMS Server-Side HDFS
1589
1589
@@ -1597,12 +1597,12 @@ This lets HMS resolve `hdfs://` warehouse and table locations from inside the Do
1597
1597
1598
1598
=== HMS Server-Side GCS
1599
1599
1600
-
When Hive Metastore starts with fake GCS enabled, `bigdata-test` injects HMS-side GCS configuration based on the Google Cloud Storage connector configuration model:
1600
+
When Hive Metastore starts with the GCS emulator enabled, `bigdata-test` injects HMS-side GCS configuration based on the Google Cloud Storage connector configuration model:
The local developer profile is fully configuration driven. The app source does not call `BigDataTestKit`; `application-local.yaml` starts Kerberos, HDFS, and LocalStack S3 and enables the REST API.
2081
+
The local developer profile is fully configuration driven. The app source does not call `BigDataTestKit`; `application-local.yaml` starts Kerberos, HDFS, and the S3 emulator and enables the REST API.
2082
2082
2083
2083
The local profile also enables HDFS and Kerberos container logs on stdout:
The Spark example lives under `example/spark`. It starts HDFS, Hive Metastore, Kafka, Schema Registry, LocalStack S3, and fake GCS.
2140
+
The Spark example lives under `example/spark`. It starts HDFS, Hive Metastore, Kafka, Schema Registry, Floci S3, and Floci GCS.
2141
2141
2142
2142
The example uses `hiveMetastore = true` with the Hive 3 open-source HMS image. This keeps Spark 3.x on a compatible HMS protocol while still exercising server-side S3A and GCS configuration. Use `clouderaHms = true` when the embedded-Postgres Cloudera HMS image is the target under test.
2143
2143
@@ -2284,7 +2284,7 @@ This Hadoop message can appear in test output when Hadoop checks for native util
2284
2284
2285
2285
=== S3 Hive External Table Fails with Credential Errors
2286
2286
2287
-
If a Hive external table on `s3a://` fails with an AWS credential error, check whether HMS was started with LocalStack S3 enabled and whether custom `HiveMetastoreOptions.extraConfiguration` overrides the S3A settings.
2287
+
If a Hive external table on `s3a://` fails with an AWS credential error, check whether HMS was started with the S3 emulator enabled and whether custom `HiveMetastoreOptions.extraConfiguration` overrides the S3A settings.
2288
2288
2289
2289
The important distinction is where the path is resolved:
0 commit comments