Skip to content

Commit 80a41a4

Browse files
CoderYellowCoderYellow
committed
feat: add dual Hive Metastore distribution support with updated configurations
- Introduced support for both open-source and Cloudera Hive Metastore distributions. - Enhanced `BigDataTest` and `BigDataTestKit` to enable selective HMS distribution activation. - Added new `clouderaHms` property in annotations, configuration, and extension logic. - Updated Spark and HMS examples to demonstrate configuration of open-source and Cloudera HMS images. - Improved filesystem configuration and object store integration for Hive Metastore. - Implemented validation to prevent simultaneous activation of multiple HMS distributions. - Enhanced documentation with usage details for dual HMS configurations and scenarios.
1 parent d0b9fa5 commit 80a41a4

16 files changed

Lines changed: 559 additions & 105 deletions

File tree

‎README.md‎

Lines changed: 20 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,14 @@ records = [
101101
]
102102
```
103103

104-
Image overrides are read before containers start. You can put `[images]` in the same TOML file referenced by `@BigDataExtensions`, or in files listed directly on `@BigDataTest(config = ["classpath:bigdata-test.toml"])`; direct `@BigDataTest` config files take priority when the same image key appears in both places.
104+
Image overrides are read before containers start. `hiveMetastore` is the open-source HMS image and `hiveMetastorePostgres` is its external PostgreSQL image. `clouderaHms` is the embedded-Postgres Cloudera HMS image. You can put `[images]` in the same TOML file referenced by `@BigDataExtensions`, or in files listed directly on `@BigDataTest(config = ["classpath:bigdata-test.toml"])`; direct `@BigDataTest` config files take priority when the same image key appears in both places.
105+
106+
```toml
107+
[images]
108+
hiveMetastore = "ghcr.io/openprojectx/hive:3.1.3-hadoop-3.4.2-gcs-4.0.4-jdk17-0.1.4"
109+
hiveMetastorePostgres = "postgres:16-alpine"
110+
clouderaHms = "ghcr.io/openprojectx/cloudera-hms:0.1.16"
111+
```
105112

106113
The same setup can be declared programmatically when names, records, or options need to be generated dynamically:
107114

@@ -163,6 +170,10 @@ class MyIntegrationTest {
163170
}
164171
```
165172

173+
`hiveMetastore = true` starts the open-source HMS distribution and an external Postgres support container. Use `clouderaHms = true` when you want the Cloudera HMS image with embedded Postgres. Enable only one HMS implementation for a test class.
174+
175+
The Spark example uses `hiveMetastore = true` with the Hive 3 open-source HMS image so Spark 3.x uses a compatible metastore server while still exercising server-side S3A and GCS filesystem configuration. Use `clouderaHms = true` when you want the embedded-Postgres Cloudera HMS image instead.
176+
166177
Kerberos can be enabled per service. Hadoop/HDFS, Hive Metastore, Kafka, and Kafka UI expose dedicated Kerberos switches. Set `kerberos = true` to start the shared KDC and then enable auth for the services that should use it. Schema Registry can still be enabled with Kafka Kerberos; it uses Kafka's internal plaintext listener while host clients keep using Kerberos.
167178

168179
```kotlin
@@ -193,7 +204,7 @@ JUnit tests can also route container logs to the main test process console or to
193204
class MyTroubleshootingTest
194205
```
195206

196-
`ContainerLogMode.STDOUT` prefixes each line with the service name. `ContainerLogMode.FILE` writes files such as `kafka.log`, `schema-registry.log`, `hive-metastore.log`, and `hive-metastore-postgres.log`.
207+
`ContainerLogMode.STDOUT` prefixes each line with the service name. `ContainerLogMode.FILE` writes files such as `kafka.log`, `schema-registry.log`, `hive-metastore.log`, and `hive-metastore-postgres.log`. The Postgres log is only produced for the open-source HMS distribution.
197208

198209
JUnit host ports are random by default. Leave port fields at `0` for Testcontainers dynamic port mapping, or set a positive value when a local tool needs a stable host port:
199210

@@ -242,7 +253,9 @@ Default service ports and endpoint property keys:
242253

243254
The same metadata is available programmatically from `BigDataService.defaultPorts` and `BigDataService.endpointProperties`.
244255

245-
When Hive Metastore is started with LocalStack S3 or fake GCS, the kit also injects server-side Hadoop filesystem configuration into HMS. External table DDL can validate `s3a://` locations against the internal LocalStack endpoint and `gs://` locations against the internal fake GCS endpoint when the HMS image includes the matching filesystem connector.
256+
When Hive Metastore is started with LocalStack S3 or fake GCS, the kit injects server-side Hadoop filesystem configuration into HMS. External table DDL can validate `s3a://` locations against the internal LocalStack endpoint and `gs://` locations against the internal fake GCS endpoint when the HMS image includes the matching filesystem connector.
257+
258+
`hiveMetastore` defaults to `ghcr.io/openprojectx/hive:3.1.3-hadoop-3.4.2-gcs-4.0.4-jdk17-0.1.4`. Hive 4 images can be tested by overriding `[images].hiveMetastore`, but Spark 3.x brings a Hive 2.3 metastore client and should use the Hive 3 image unless that client stack is changed. The Cloudera image remains available through `clouderaHms`.
246259

247260
## Spring Boot
248261

@@ -254,6 +267,8 @@ bigdata:
254267
enabled: true
255268
hive-metastore:
256269
enabled: true
270+
image: ghcr.io/openprojectx/hive:3.1.3-hadoop-3.4.2-gcs-4.0.4-jdk17-0.1.4
271+
database-image: postgres:16-alpine
257272
kafka:
258273
enabled: true
259274
kerberos-enabled: true
@@ -263,6 +278,8 @@ bigdata:
263278
enabled: true
264279
```
265280
281+
For the embedded-Postgres Cloudera HMS image, use `bigdata.test.cloudera-hms.enabled=true` instead of `bigdata.test.hive-metastore.enabled=true`.
282+
266283
The auto-configuration exposes a started `BigDataTestKit` bean and closes it with the application context.
267284

268285
## Dependency Management

‎autoconfigure/src/main/kotlin/org/openprojectx/bigdata/test/autoconfigure/BigDataTestAutoConfiguration.kt‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ package org.openprojectx.bigdata.test.autoconfigure
22

33
import org.openprojectx.bigdata.test.core.BigDataTestKit
44
import org.openprojectx.bigdata.test.core.HdfsOptions
5+
import org.openprojectx.bigdata.test.core.HiveMetastoreDistribution
56
import org.openprojectx.bigdata.test.core.HiveMetastoreOptions
67
import org.openprojectx.bigdata.test.core.KafkaOptions
78
import org.openprojectx.bigdata.test.core.KerberosAuthOptions
@@ -61,11 +62,16 @@ class BigDataTestAutoConfiguration {
6162
)
6263
}
6364

65+
require(!(properties.hiveMetastore.enabled && properties.clouderaHms.enabled)) {
66+
"Use only one HMS implementation: bigdata.test.hive-metastore or bigdata.test.cloudera-hms"
67+
}
6468
if (properties.hiveMetastore.enabled) {
6569
builder.withHiveMetastore(
6670
HiveMetastoreOptions(
6771
enabled = true,
72+
distribution = HiveMetastoreDistribution.OPEN_SOURCE,
6873
image = properties.hiveMetastore.image,
74+
databaseImage = properties.hiveMetastore.databaseImage,
6975
databaseName = properties.hiveMetastore.databaseName,
7076
databaseUser = properties.hiveMetastore.databaseUser,
7177
databasePassword = properties.hiveMetastore.databasePassword,
@@ -80,6 +86,23 @@ class BigDataTestAutoConfiguration {
8086
)
8187
}
8288

89+
if (properties.clouderaHms.enabled) {
90+
builder.withClouderaHms(
91+
HiveMetastoreOptions(
92+
enabled = true,
93+
distribution = HiveMetastoreDistribution.CLOUDERA,
94+
image = properties.clouderaHms.image,
95+
warehouseDir = properties.clouderaHms.warehouseDir,
96+
extraConfiguration = properties.clouderaHms.extraConfiguration,
97+
kerberos = KerberosAuthOptions(
98+
enabled = properties.clouderaHms.kerberosEnabled,
99+
servicePrincipal = "hive/hive-metastore.example.com@${properties.kerberos.realm}",
100+
keytabPath = "/kerby/keytabs/hive-metastore.keytab",
101+
),
102+
),
103+
)
104+
}
105+
83106
if (properties.kafka.enabled) {
84107
builder.withKafka(
85108
KafkaOptions(

‎autoconfigure/src/main/kotlin/org/openprojectx/bigdata/test/autoconfigure/BigDataTestProperties.kt‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ data class BigDataTestProperties(
99
var tls: Tls = Tls(),
1010
var hdfs: Hdfs = Hdfs(),
1111
var hiveMetastore: HiveMetastore = HiveMetastore(),
12+
var clouderaHms: ClouderaHms = ClouderaHms(),
1213
var kafka: Kafka = Kafka(),
1314
var localstackS3: ObjectStore = ObjectStore(image = "localstack/localstack:4.14.0"),
1415
var fakeGcs: ObjectStore = ObjectStore(image = "fsouza/fake-gcs-server:1.54"),
@@ -37,10 +38,19 @@ data class BigDataTestProperties(
3738

3839
data class HiveMetastore(
3940
var enabled: Boolean = false,
40-
var image: String = "ghcr.io/openprojectx/cloudera-hms:0.1.16",
41-
var databaseName: String = "metastore_db",
41+
var image: String = "ghcr.io/openprojectx/hive:3.1.3-hadoop-3.4.2-gcs-4.0.4-jdk17-0.1.4",
42+
var databaseImage: String = "postgres:16-alpine",
43+
var databaseName: String = "metastore",
4244
var databaseUser: String = "hive",
43-
var databasePassword: String = "hive-password",
45+
var databasePassword: String = "hive",
46+
var warehouseDir: String = "/user/hive/warehouse",
47+
var extraConfiguration: Map<String, String> = emptyMap(),
48+
var kerberosEnabled: Boolean = false,
49+
)
50+
51+
data class ClouderaHms(
52+
var enabled: Boolean = false,
53+
var image: String = "ghcr.io/openprojectx/cloudera-hms:0.1.16",
4454
var warehouseDir: String = "/user/hive/warehouse",
4555
var extraConfiguration: Map<String, String> = emptyMap(),
4656
var kerberosEnabled: Boolean = false,

‎core/build.gradle.kts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,4 +7,5 @@ dependencies {
77
api(libs.testcontainers)
88
api(libs.testcontainersPostgresql)
99
api(libs.testcontainersKafka)
10+
api(libs.hiveDockerTestcontainers)
1011
}

‎core/src/main/kotlin/org/openprojectx/bigdata/test/core/BigDataTestKit.kt‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,15 @@ class BigDataTestKit private constructor(
6767
fun withHiveMetastore(options: HiveMetastoreOptions = HiveMetastoreOptions(enabled = true)): Builder =
6868
apply { hiveMetastore = options.copy(enabled = true) }
6969

70+
fun withClouderaHms(
71+
options: HiveMetastoreOptions = HiveMetastoreOptions(
72+
enabled = true,
73+
distribution = HiveMetastoreDistribution.CLOUDERA,
74+
image = "ghcr.io/openprojectx/cloudera-hms:0.1.16",
75+
),
76+
): Builder =
77+
apply { hiveMetastore = options.copy(enabled = true, distribution = HiveMetastoreDistribution.CLOUDERA) }
78+
7079
fun withKafka(options: KafkaOptions = KafkaOptions(enabled = true)): Builder =
7180
apply { kafka = options.copy(enabled = true) }
7281

‎core/src/main/kotlin/org/openprojectx/bigdata/test/core/BigDataTestKitOptions.kt‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -54,11 +54,12 @@ data class HdfsOptions(
5454

5555
data class HiveMetastoreOptions(
5656
val enabled: Boolean = false,
57-
val image: String = "ghcr.io/openprojectx/cloudera-hms:0.1.16",
58-
val apacheHiveImage: String = "apache/hive:3.1.3",
59-
val databaseName: String = "metastore_db",
57+
val distribution: HiveMetastoreDistribution = HiveMetastoreDistribution.OPEN_SOURCE,
58+
val image: String = "ghcr.io/openprojectx/hive:3.1.3-hadoop-3.4.2-gcs-4.0.4-jdk17-0.1.4",
59+
val databaseImage: String = "postgres:16-alpine",
60+
val databaseName: String = "metastore",
6061
val databaseUser: String = "hive",
61-
val databasePassword: String = "hive-password",
62+
val databasePassword: String = "hive",
6263
val warehouseDir: String = "/user/hive/warehouse",
6364
val extraConfiguration: Map<String, String> = emptyMap(),
6465
val kerberos: KerberosAuthOptions = KerberosAuthOptions(
@@ -67,6 +68,11 @@ data class HiveMetastoreOptions(
6768
),
6869
)
6970

71+
enum class HiveMetastoreDistribution {
72+
OPEN_SOURCE,
73+
CLOUDERA,
74+
}
75+
7076
data class KafkaOptions(
7177
val enabled: Boolean = false,
7278
val image: String = "apache/kafka:4.1.2",

‎core/src/main/kotlin/org/openprojectx/bigdata/test/core/container/BigDataContainerFactory.kt‎

Lines changed: 127 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,11 @@ import org.openprojectx.bigdata.test.core.BigDataEndpoint
44
import org.openprojectx.bigdata.test.core.BigDataService
55
import org.openprojectx.bigdata.test.core.BigDataTestKitOptions
66
import org.openprojectx.bigdata.test.core.ContainerLogMode
7+
import org.openprojectx.bigdata.test.core.HiveMetastoreDistribution
78
import org.openprojectx.bigdata.test.core.KerberosAuthOptions
89
import org.openprojectx.bigdata.test.core.KafkaOptions
910
import org.openprojectx.bigdata.test.core.KerberosOptions
11+
import org.openprojectx.hive.docker.testcontainers.HiveMetastoreContainer
1012
import org.testcontainers.containers.BindMode
1113
import org.testcontainers.containers.GenericContainer
1214
import org.testcontainers.containers.Network
@@ -158,28 +160,60 @@ internal class BigDataContainerFactory(
158160

159161
private fun hiveMetastore(): BigDataServiceContainer {
160162
val hive = options.hiveMetastore
161-
val image = DockerImageName.parse("postgres:14")
162-
val postgres = PostgreSQLContainer(image)
163+
if (hive.distribution == HiveMetastoreDistribution.CLOUDERA) {
164+
return clouderaHms()
165+
}
166+
val postgres = PostgreSQLContainer(DockerImageName.parse(hive.databaseImage))
167+
.withNetwork(network)
168+
.withNetworkAliases("hive-metastore-postgres")
163169
.withDatabaseName(hive.databaseName)
164170
.withUsername(hive.databaseUser)
165171
.withPassword(hive.databasePassword)
172+
supportContainers += attachLogs("hive-metastore-postgres", postgres)
173+
postgres.start()
174+
175+
val container = FixedPortHiveMetastoreContainer(hive.image)
176+
container
166177
.withNetwork(network)
167-
.withNetworkAliases("hms-postgres")
168-
supportContainers += postgres
169-
attachLogs("hive-metastore-postgres", postgres)
178+
.withNetworkAliases("hive-metastore", "hive-metastore.example.com")
179+
.withEnv("SERVICE_NAME", "metastore")
180+
.withPostgres("hive-metastore-postgres", 5432, hive.databaseName, hive.databaseUser, hive.databasePassword)
181+
.withWarehousePath(hive.warehouseDir)
182+
container.withServicePort(9083, options.portBindings.hostPort(9083, options.portBindings.hiveMetastore))
183+
if (hive.kerberos.enabled) {
184+
mountKerberos(container)
185+
container
186+
.withEnv("KRB5_CONFIG", "/kerby/client/krb5.conf")
187+
.withEnv("SERVICE_OPTS", "-Djava.security.krb5.conf=/kerby/client/krb5.conf")
188+
}
189+
configureHiveDockerObjectStores(container)
190+
if (hive.extraConfiguration.isNotEmpty() || hive.kerberos.enabled) {
191+
container
192+
.withEnv("HIVE_CUSTOM_CONF_DIR", "/bigdata-test/hive-conf")
193+
.withFileSystemBind(openSourceHiveConfigurationDirectory().toString(), "/bigdata-test/hive-conf", BindMode.READ_ONLY)
194+
}
170195

196+
return BigDataServiceContainer(BigDataService.HIVE_METASTORE, attachLogs("hive-metastore", container)) {
197+
val thriftUri = container.thriftUri
198+
BigDataEndpoint(
199+
service = BigDataService.HIVE_METASTORE,
200+
host = container.host,
201+
ports = mapOf("thrift" to container.getMappedPort(9083)),
202+
properties = mapOf(
203+
"hive.metastore.uris" to thriftUri,
204+
"spring.bigdata.test.hive-metastore.thrift-uri" to thriftUri,
205+
) + kerberosProperties("hive.metastore", hive.kerberos),
206+
)
207+
}
208+
}
209+
210+
private fun clouderaHms(): BigDataServiceContainer {
211+
val hive = options.hiveMetastore
171212
val container = GenericBigDataContainer(hive.image)
172213
.withNetwork(network)
173214
.withNetworkAliases("hive-metastore", "hive-metastore.example.com")
174215
.withServicePort(9083, options.portBindings.hostPort(9083, options.portBindings.hiveMetastore))
175-
.withEnv("POSTGRES_DB", hive.databaseName)
176-
.withEnv("POSTGRES_USER", hive.databaseUser)
177-
.withEnv("POSTGRES_PASSWORD", hive.databasePassword)
178-
.withEnv("HMS_JDBC_URL", "jdbc:postgresql://hms-postgres:5432/${hive.databaseName}")
179-
.withEnv("HMS_JDBC_USER", hive.databaseUser)
180-
.withEnv("HMS_JDBC_PASSWORD", hive.databasePassword)
181216
.withEnv("HMS_WAREHOUSE_DIR", hive.warehouseDir)
182-
.dependsOn(postgres)
183217
.waitingFor(Wait.forListeningPort().withStartupTimeout(Duration.ofMinutes(3)))
184218
if (hive.kerberos.enabled) {
185219
mountKerberos(container)
@@ -421,12 +455,92 @@ internal class BigDataContainerFactory(
421455
put("fs.gs.storage.service.path", "storage/v1/")
422456
put("fs.gs.client.type", "HTTP_API_CLIENT")
423457
put("fs.gs.auth.type", "UNAUTHENTICATED")
458+
put("fs.gs.status.parallel.enable", "false")
424459
put("fs.gs.create.items.conflict.check.enable", "false")
425460
put("fs.gs.implicit.dir.repair.enable", "false")
426461
put("fs.gs.hierarchical.namespace.folders.enable", "false")
427462
}
428463
}
429464

465+
private fun configureHiveDockerObjectStores(container: HiveMetastoreContainer) {
466+
if (options.localStackS3.enabled) {
467+
container
468+
.withEnv("S3A_FILE_SYSTEM_IMPL", "org.apache.hadoop.fs.s3a.S3AFileSystem")
469+
.withEnv("S3_ENDPOINT_URL", "http://localstack:4566")
470+
.withEnv("AWS_ACCESS_KEY_ID", "test")
471+
.withEnv("AWS_SECRET_ACCESS_KEY", "test")
472+
.withEnv("S3_PATH_STYLE_ACCESS", "true")
473+
.withEnv("S3_SSL_ENABLED", "false")
474+
}
475+
if (options.fakeGcs.enabled) {
476+
container
477+
.withEnv("GCS_FILE_SYSTEM_IMPL", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")
478+
.withEnv("GCS_ABSTRACT_FILE_SYSTEM_IMPL", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS")
479+
.withEnv("GCS_AUTH_TYPE", "UNAUTHENTICATED")
480+
.withEnv("GCS_PROJECT_ID", "bigdata-test")
481+
.withEnv("GCS_STORAGE_ROOT_URL", "http://fake-gcs:4443")
482+
.withEnv("GCS_STORAGE_SERVICE_PATH", "/storage/v1/")
483+
.withEnv("GCS_CREATE_ITEMS_CONFLICT_CHECK_ENABLED", "false")
484+
.withEnv("GCS_CLIENT_UPLOAD_TYPE", "WRITE_TO_DISK_THEN_UPLOAD")
485+
.withEnv("GCS_DIRECT_UPLOAD_ENABLED", "false")
486+
.withEnv("GCS_PERFORMANCE_CACHE_ENABLED", "false")
487+
.withEnv("GCS_STORAGE_CLIENT_CACHE_ENABLED", "false")
488+
.withEnv("GCS_GRPC_ENABLED", "false")
489+
}
490+
}
491+
492+
private fun openSourceHiveConfigurationDirectory(): Path {
493+
val hive = options.hiveMetastore
494+
val dir = Files.createTempDirectory("bigdata-test-hive-conf-")
495+
val metastoreProperties = linkedMapOf(
496+
"hive.metastore.warehouse.dir" to hive.warehouseDir,
497+
"javax.jdo.option.ConnectionURL" to "jdbc:postgresql://hive-metastore-postgres:5432/${hive.databaseName}",
498+
"javax.jdo.option.ConnectionDriverName" to "org.postgresql.Driver",
499+
"javax.jdo.option.ConnectionUserName" to hive.databaseUser,
500+
"javax.jdo.option.ConnectionPassword" to hive.databasePassword,
501+
)
502+
if (hive.kerberos.enabled) {
503+
metastoreProperties += mapOf(
504+
"hive.metastore.sasl.enabled" to "true",
505+
"hive.metastore.kerberos.principal" to hive.kerberos.servicePrincipal,
506+
"hive.metastore.kerberos.keytab.file" to hive.kerberos.keytabPath,
507+
"hadoop.security.authentication" to "kerberos",
508+
)
509+
}
510+
metastoreProperties += hive.extraConfiguration
511+
val hadoopProperties = hiveMetastoreObjectStoreConfiguration()
512+
513+
writeConfigurationXml(dir.resolve("hive-site.xml"), metastoreProperties + hadoopProperties)
514+
writeConfigurationXml(dir.resolve("metastore-site.xml"), metastoreProperties + hadoopProperties)
515+
writeConfigurationXml(dir.resolve("core-site.xml"), hadoopProperties)
516+
return dir
517+
}
518+
519+
private fun writeConfigurationXml(path: Path, properties: Map<String, String>) {
520+
Files.writeString(
521+
path,
522+
buildString {
523+
appendLine("<configuration>")
524+
properties.forEach { (key, value) ->
525+
appendLine(" <property>")
526+
appendLine(" <name>${xmlEscape(key)}</name>")
527+
appendLine(" <value>${xmlEscape(value)}</value>")
528+
appendLine(" </property>")
529+
}
530+
appendLine("</configuration>")
531+
},
532+
StandardCharsets.UTF_8,
533+
)
534+
}
535+
536+
private fun xmlEscape(value: String): String =
537+
value
538+
.replace("&", "&amp;")
539+
.replace("<", "&lt;")
540+
.replace(">", "&gt;")
541+
.replace("\"", "&quot;")
542+
.replace("'", "&apos;")
543+
430544

431545
private fun <T : GenericContainer<*>> attachLogs(name: String, container: T): T {
432546
when (options.containerLogs.mode) {
@@ -511,7 +625,7 @@ internal class BigDataContainerFactory(
511625
if (!contains(entry)) add(entry)
512626
}
513627

514-
private fun mountKerberos(container: GenericBigDataContainer) {
628+
private fun mountKerberos(container: GenericContainer<*>) {
515629
container.withFileSystemBind(kerberosDirectory(), "/kerby", BindMode.READ_ONLY)
516630
}
517631

0 commit comments

Comments
 (0)