feat: enhance GCS integration and update HMS image version

- Updated HMS container image to `ghcr.io/openprojectx/cloudera-hms:0.1.16`.
- Enhanced GCS integration for Hive external tables in both Spark example and HMS configuration.
- Added support for creating and verifying GCS-backed Hive external Parquet tables.
- Improved documentation to clarify HMS requirements for GCS connector dependencies.

Author: CoderYellow

README.md

@@ -159,7 +159,7 @@ class MyFixedPortTest
 
 Available JUnit port fields are `kerberosKdcPort`, `hdfsNameNodePort`, `hdfsWebPort`, `hiveMetastorePort`, `kafkaPort`, `schemaRegistryPort`, `kafkaUiPort`, `localStackS3Port`, and `fakeGcsPort`. Endpoint properties still use the actual mapped host ports returned by Testcontainers.
 
-When Hive Metastore is started with LocalStack S3 or fake GCS, the kit also injects server-side Hadoop filesystem configuration into HMS. S3 external table DDL can validate `s3a://` locations against the internal LocalStack endpoint. GCS config is also injected, but HMS still needs a GCS connector jar in the HMS image before it can validate `gs://` locations.
+When Hive Metastore is started with LocalStack S3 or fake GCS, the kit also injects server-side Hadoop filesystem configuration into HMS. External table DDL can validate `s3a://` locations against the internal LocalStack endpoint and `gs://` locations against the internal fake GCS endpoint when the HMS image includes the matching filesystem connector.
 
 ## Spring Boot
 
@@ -204,7 +204,7 @@ GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spring:bootRun --args='--sprin
 
 The JUnit examples in `example/junit` are annotated with `@Disabled`; remove that annotation from an example class to start the configured stack.
 
-The Spark example in `example/spark` creates a `SparkSession` from `BigDataTestKit` endpoints, starts HDFS, Hive Metastore, Kafka, Schema Registry, LocalStack S3, and fake GCS, then uses `extensions` config to create the S3 JCEKS file on HDFS and produce Avro records to Kafka. HDFS is used as the config store for the JCEKS file; S3 and GCS are object-store Iceberg smoke checks through Hadoop catalogs. The example also creates an HMS-backed Iceberg table and a Hive external Parquet table stored on S3, then verifies the HMS database/table location and format metadata through `HiveMetaStoreClient`.
+The Spark example in `example/spark` creates a `SparkSession` from `BigDataTestKit` endpoints, starts HDFS, Hive Metastore, Kafka, Schema Registry, LocalStack S3, and fake GCS, then uses `extensions` config to create the S3 JCEKS file on HDFS and produce Avro records to Kafka. HDFS is used as the config store for the JCEKS file; S3 and GCS are object-store Iceberg smoke checks through Hadoop catalogs. The example also creates an HMS-backed Iceberg table and Hive external Parquet tables on S3 and GCS, then verifies the HMS database/table location and format metadata through `HiveMetaStoreClient`.
 
 ```bash
 GRADLE_USER_HOME=/data/.gradle ./gradlew :example:spark:test

autoconfigure/src/main/kotlin/org/openprojectx/bigdata/test/autoconfigure/BigDataTestProperties.kt

@@ -37,7 +37,7 @@ data class BigDataTestProperties(
 
     data class HiveMetastore(
         var enabled: Boolean = false,
-        var image: String = "ghcr.io/openprojectx/cloudera-hms:0.1.15",
+        var image: String = "ghcr.io/openprojectx/cloudera-hms:0.1.16",
         var databaseName: String = "metastore_db",
         var databaseUser: String = "hive",
         var databasePassword: String = "hive-password",

core/src/main/kotlin/org/openprojectx/bigdata/test/core/BigDataTestKitOptions.kt

@@ -47,7 +47,7 @@ data class HdfsOptions(
 
 data class HiveMetastoreOptions(
     val enabled: Boolean = false,
-    val image: String = "ghcr.io/openprojectx/cloudera-hms:0.1.15",
+    val image: String = "ghcr.io/openprojectx/cloudera-hms:0.1.16",
     val apacheHiveImage: String = "apache/hive:3.1.3",
     val databaseName: String = "metastore_db",
     val databaseUser: String = "hive",

doc/user-guide.adoc

@@ -449,7 +449,7 @@ When Hive Metastore starts with fake GCS enabled, `bigdata-test` injects HMS-sid
 * `fs.gs.implicit.dir.repair.enable=false`
 * `fs.gs.hierarchical.namespace.folders.enable=false`
 
-This configuration does not fail HMS startup by itself. HMS can start with these keys even if the GCS connector jar is not present. Creating or validating a `gs://` Hive external table still requires the HMS image to include the GCS connector and its runtime dependencies.
+This configuration does not fail HMS startup by itself. HMS can start with these keys even if the GCS connector jar is not present. Creating or validating a `gs://` Hive external table requires the HMS image to include the GCS connector and its runtime dependencies. The default `ghcr.io/openprojectx/cloudera-hms` image includes those dependencies.
 
 == Spring Boot
 
@@ -506,7 +506,7 @@ The test demonstrates:
 * Reading Kafka data from Spark.
 * Writing Iceberg tables to S3 and GCS through Hadoop catalogs.
 * Creating an HMS-backed Iceberg table and asserting HMS metadata with `HiveMetaStoreClient`.
-* Creating a Hive external Parquet table stored on S3 and asserting HMS location and Parquet metadata.
+* Creating Hive external Parquet tables stored on S3 and GCS and asserting HMS location and Parquet metadata.
 
 Run it with:
 
@@ -532,7 +532,7 @@ The important distinction is where the path is resolved:
 
 === GCS Hive External Table Fails with `ClassNotFoundException`
 
-If HMS returns `ClassNotFoundException: com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem`, the HMS image does not include the GCS connector jar. The config can be present and HMS can still start, but the jar is required when HMS validates a `gs://` path.
+If HMS returns `ClassNotFoundException: com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem`, the HMS image does not include the GCS connector jar. The config can be present and HMS can still start, but the jar is required when HMS validates a `gs://` path. Use the default `ghcr.io/openprojectx/cloudera-hms` image or another HMS image that includes the GCS connector and runtime dependencies.
 
 === Container Startup Failures
 

example/spark/src/test/kotlin/org/openprojectx/bigdata/test/example/spark/SparkBigDataTestExample.kt

@@ -42,6 +42,7 @@ class SparkBigDataTestExample {
         val topic = "spark-avro-events"
         val s3Bucket = "spark-iceberg-s3-$runId"
         val gcsBucket = "spark-iceberg-gcs-$runId"
+        val gcsIcebergDataPath = "gs://$gcsBucket/data/demo_$runId/events_gcs"
         val s3CredentialProviderPath = extensions.required("s3-jceks.credential-provider.path")
         val s3CredentialProviderHdfsPath = extensions.required("s3-jceks.hdfs.path")
         createS3Bucket(s3.property("aws.endpoint-url.s3"), s3Bucket)
@@ -65,7 +66,7 @@ class SparkBigDataTestExample {
                 namespace = "demo_$runId",
                 table = "events_gcs",
                 storageName = "gcs",
-                dataPath = "gs://$gcsBucket/data/demo_$runId/events_gcs",
+                dataPath = gcsIcebergDataPath,
             )
             assertIcebergTable(spark, catalog = "hms", namespace = "hms_demo_$runId", table = "events_hms", storageName = "hms")
             assertHiveMetastoreTable(
@@ -81,6 +82,15 @@ class SparkBigDataTestExample {
                 location = "s3a://$s3Bucket/hive-parquet/events_parquet_s3",
                 storageName = "hive-s3-parquet",
             )
+            assertHiveExternalParquetTable(
+                spark = spark,
+                hiveMetastoreUri = hiveMetastore.property("hive.metastore.uris"),
+                database = "hive_gcs_demo_$runId",
+                table = "events_parquet_gcs",
+                location = gcsIcebergDataPath,
+                storageName = "gcs",
+                seedData = false,
+            )
         }
     }
 
@@ -203,9 +213,19 @@ class SparkBigDataTestExample {
         table: String,
         location: String,
         storageName: String,
+        seedData: Boolean = true,
     ) {
         val identifier = "$database.$table"
         spark.sql("CREATE DATABASE IF NOT EXISTS $database")
+        if (seedData) {
+            spark.sql(
+                """
+                SELECT 1 AS id, 'alpha' AS name, '$storageName' AS storage
+                UNION ALL
+                SELECT 2 AS id, 'beta' AS name, '$storageName' AS storage
+                """.trimIndent(),
+            ).write().mode("overwrite").parquet(location)
+        }
         spark.sql(
             """
             CREATE EXTERNAL TABLE $identifier (
@@ -216,7 +236,6 @@ class SparkBigDataTestExample {
             LOCATION '$location'
             """.trimIndent(),
         )
-        spark.sql("INSERT INTO $identifier VALUES (1, 'alpha', '$storageName'), (2, 'beta', '$storageName')")
         val count = spark.table(identifier).where("storage = '$storageName'").count()
         check(count == 2L) { "Expected two Hive external Parquet rows in $identifier" }
         assertParquetFiles(spark, location)