Skip to content

Commit d0b9fa5

Browse files
CoderYellowCoderYellow
committed
feat: support service image overrides via TOML configuration
- Added `[images]` table in TOML to override service container images at startup. - Extended `@BigDataTest` annotation with a `config` property to load TOML resources. - Updated JUnit extension to prioritize direct `@BigDataTest(config)` files over `@BigDataExtensions`. - Added support for image configuration in Kerberos, HDFS, Kafka, Schema Registry, LocalStack S3, and fake GCS. - Enhanced documentation with examples for using `[images]` in tests and extensions. - Integrated new image loading logic, enabling flexible and centralized service configuration.
1 parent 89a5650 commit d0b9fa5

7 files changed

Lines changed: 273 additions & 4 deletions

File tree

‎README.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,12 @@ class MyIntegrationTest {
7878
Example config:
7979

8080
```toml
81+
[images]
82+
hdfs = "apache/hadoop:3.5.0"
83+
kafka = "apache/kafka:4.1.2"
84+
schemaRegistry = "confluentinc/cp-schema-registry:7.8.0"
85+
localStackS3 = "localstack/localstack:4.14.0"
86+
8187
[s3Jceks]
8288
enabled = true
8389
hdfsDir = "/bigdata-test/spark"
@@ -95,6 +101,8 @@ records = [
95101
]
96102
```
97103

104+
Image overrides are read before containers start. You can put `[images]` in the same TOML file referenced by `@BigDataExtensions`, or in files listed directly on `@BigDataTest(config = ["classpath:bigdata-test.toml"])`; direct `@BigDataTest` config files take priority when the same image key appears in both places.
105+
98106
The same setup can be declared programmatically when names, records, or options need to be generated dynamically:
99107

100108
```kotlin

‎doc/user-guide.adoc‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -136,6 +136,9 @@ The JUnit extension starts the kit before all tests in the class and closes it a
136136
|===
137137
|Annotation field |Effect
138138

139+
|`config`
140+
|Loads TOML config before containers start. Use this for startup-time settings such as service image overrides.
141+
139142
|`kerberos`
140143
|Starts the shared Kerberos KDC.
141144

@@ -173,6 +176,47 @@ The JUnit extension starts the kit before all tests in the class and closes it a
173176
|Starts fake-gcs-server.
174177
|===
175178

179+
=== Service Image Configuration
180+
181+
Service images can be overridden from TOML with an `[images]` table. The JUnit extension reads this table before creating containers, so these values affect startup.
182+
183+
[source,toml]
184+
----
185+
[images]
186+
kerberos = "openprojectx/kerby-kdc:latest"
187+
hdfs = "apache/hadoop:3.5.0"
188+
hiveMetastore = "ghcr.io/openprojectx/cloudera-hms:0.1.16"
189+
hiveMetastoreApache = "apache/hive:3.1.3"
190+
kafka = "apache/kafka:4.1.2"
191+
schemaRegistry = "confluentinc/cp-schema-registry:7.8.0"
192+
kafkaUi = "ghcr.io/kafbat/kafka-ui:latest"
193+
localStackS3 = "localstack/localstack:4.14.0"
194+
fakeGcs = "fsouza/fake-gcs-server:1.54"
195+
----
196+
197+
You can place `[images]` in the same TOML files used by `@BigDataExtensions`, because `@BigDataTest` reads those locations before startup when the extensions annotation is present:
198+
199+
[source,kotlin]
200+
----
201+
@BigDataExtensions("classpath:bigdata-extensions.toml")
202+
@BigDataTest(hdfs = true, kafka = true, schemaRegistry = true)
203+
class MyIntegrationTest
204+
----
205+
206+
You can also point `@BigDataTest` directly at startup config:
207+
208+
[source,kotlin]
209+
----
210+
@BigDataTest(
211+
config = ["classpath:bigdata-test.toml"],
212+
hdfs = true,
213+
kafka = true,
214+
)
215+
class MyIntegrationTest
216+
----
217+
218+
When both are used, files from `@BigDataExtensions` are loaded first and files from `@BigDataTest(config = ...)` are loaded after them, so direct `@BigDataTest` config wins for duplicate image keys.
219+
176220
=== Dynamic and Fixed Ports
177221

178222
Host ports are dynamic by default. Keep port fields set to `0` unless an external local tool needs a stable port.

‎example/spark/src/test/resources/spark-bigdata-extensions.toml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,14 @@
1+
[images]
2+
kerberos = "openprojectx/kerby-kdc:latest"
3+
hdfs = "apache/hadoop:3.5.0"
4+
hiveMetastore = "ghcr.io/openprojectx/cloudera-hms:0.1.16"
5+
hiveMetastoreApache = "apache/hive:3.1.3"
6+
kafka = "apache/kafka:4.1.2"
7+
schemaRegistry = "confluentinc/cp-schema-registry:7.8.0"
8+
kafkaUi = "ghcr.io/kafbat/kafka-ui:latest"
9+
localStackS3 = "localstack/localstack:4.14.0"
10+
fakeGcs = "fsouza/fake-gcs-server:1.54"
11+
112
[s3Jceks]
213
enabled = true
314
hdfsDir = "/bigdata-test/spark"

‎extensions/src/main/kotlin/org/openprojectx/bigdata/test/extensions/kerberos/KerberosMaterialExtension.kt‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
package org.openprojectx.bigdata.test.extensions.kerberos
2+
3+
import org.openprojectx.bigdata.test.core.BigDataService
4+
import org.openprojectx.bigdata.test.extensions.core.BigDataExtension
5+
import org.openprojectx.bigdata.test.extensions.core.BigDataExtensionContext
6+
import org.openprojectx.bigdata.test.extensions.core.BigDataExtensionEvent
7+
8+
data class KerberosMaterialExtension(
9+
override val id: String = "kerberos-material",
10+
) : BigDataExtension {
11+
override val requiredServices: Set<BigDataService> = setOf(BigDataService.KERBEROS)
12+
override val events: Set<BigDataExtensionEvent> = setOf(BigDataExtensionEvent.AFTER_KIT_START)
13+
14+
override fun onEvent(event: BigDataExtensionEvent, context: BigDataExtensionContext) {
15+
val kerberos = context.endpoint(BigDataService.KERBEROS)
16+
put(context, "realm", kerberos.property("bigdata.test.kerberos.realm"))
17+
put(context, "kdc", kerberos.property("bigdata.test.kerberos.kdc"))
18+
put(context, "krb5-conf", kerberos.property("bigdata.test.kerberos.krb5-conf"))
19+
put(context, "client.principal", kerberos.property("bigdata.test.kerberos.client-principal"))
20+
put(context, "client.password", kerberos.property("bigdata.test.kerberos.client-password"))
21+
put(context, "client.keytab", kerberos.property("bigdata.test.kerberos.client-keytab"))
22+
put(context, "client.keytab.container", kerberos.property("bigdata.test.kerberos.client-keytab.container"))
23+
24+
context.kit.endpoints().forEach { (service, endpoint) ->
25+
endpoint.properties.forEach { (key, value) ->
26+
if (key.endsWith(".kerberos.principal")) put(context, "${service.key}.principal", value)
27+
if (key.endsWith(".kerberos.service-name")) put(context, "${service.key}.service-name", value)
28+
if (key.endsWith(".kerberos.keytab")) put(context, "${service.key}.keytab.container", value)
29+
if (key.endsWith(".kerberos.keytab.local")) put(context, "${service.key}.keytab", value)
30+
}
31+
}
32+
}
33+
34+
private fun put(context: BigDataExtensionContext, key: String, value: String) {
35+
context.putOutput("$id.$key", value)
36+
}
37+
38+
private val BigDataService.key: String
39+
get() = name.lowercase().replace('_', '-')
40+
}

‎junit5/src/main/kotlin/org/openprojectx/bigdata/test/junit5/BigDataTest.kt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ import org.openprojectx.bigdata.test.core.ContainerLogMode
77
@Retention(AnnotationRetention.RUNTIME)
88
@ExtendWith(BigDataTestExtension::class)
99
annotation class BigDataTest(
10+
val config: Array<String> = [],
1011
val kerberos: Boolean = false,
1112
val kerberosClientPrincipal: String = "app_user@EXAMPLE.COM",
1213
val kerberosClientPassword: String = "app-user-secret",

‎junit5/src/main/kotlin/org/openprojectx/bigdata/test/junit5/BigDataTestConfig.kt‎

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
package org.openprojectx.bigdata.test.junit5
2+
3+
import java.nio.file.Files
4+
import java.nio.file.Path
5+
6+
internal data class BigDataTestImageConfig(
7+
val kerberos: String? = null,
8+
val hdfs: String? = null,
9+
val hiveMetastore: String? = null,
10+
val hiveMetastoreApache: String? = null,
11+
val kafka: String? = null,
12+
val schemaRegistry: String? = null,
13+
val kafkaUi: String? = null,
14+
val localStackS3: String? = null,
15+
val fakeGcs: String? = null,
16+
) {
17+
fun merge(override: BigDataTestImageConfig): BigDataTestImageConfig =
18+
BigDataTestImageConfig(
19+
kerberos = override.kerberos ?: kerberos,
20+
hdfs = override.hdfs ?: hdfs,
21+
hiveMetastore = override.hiveMetastore ?: hiveMetastore,
22+
hiveMetastoreApache = override.hiveMetastoreApache ?: hiveMetastoreApache,
23+
kafka = override.kafka ?: kafka,
24+
schemaRegistry = override.schemaRegistry ?: schemaRegistry,
25+
kafkaUi = override.kafkaUi ?: kafkaUi,
26+
localStackS3 = override.localStackS3 ?: localStackS3,
27+
fakeGcs = override.fakeGcs ?: fakeGcs,
28+
)
29+
}
30+
31+
internal class BigDataTestConfigLoader(
32+
private val classLoader: ClassLoader,
33+
) {
34+
fun load(locations: Iterable<String>): BigDataTestImageConfig =
35+
locations.fold(BigDataTestImageConfig()) { current, location ->
36+
current.merge(load(location))
37+
}
38+
39+
private fun load(location: String): BigDataTestImageConfig {
40+
val text = readText(location)
41+
val images = parseImagesTable(text)
42+
return BigDataTestImageConfig(
43+
kerberos = images["kerberos"],
44+
hdfs = images["hdfs"],
45+
hiveMetastore = images["hiveMetastore"],
46+
hiveMetastoreApache = images["hiveMetastoreApache"],
47+
kafka = images["kafka"],
48+
schemaRegistry = images["schemaRegistry"],
49+
kafkaUi = images["kafkaUi"],
50+
localStackS3 = images["localStackS3"],
51+
fakeGcs = images["fakeGcs"],
52+
)
53+
}
54+
55+
private fun readText(location: String): String =
56+
when {
57+
location.startsWith("classpath:") -> {
58+
val path = location.removePrefix("classpath:").removePrefix("/")
59+
classLoader.getResource(path)?.readText()
60+
?: error("BigDataTest config resource '$location' was not found")
61+
}
62+
else -> Files.readString(Path.of(location))
63+
}
64+
65+
private fun parseImagesTable(text: String): Map<String, String> {
66+
val values = linkedMapOf<String, String>()
67+
var inImages = false
68+
text.lineSequence().forEachIndexed { index, raw ->
69+
val line = stripComment(raw).trim()
70+
if (line.isEmpty()) return@forEachIndexed
71+
if (line.startsWith("[") && line.endsWith("]")) {
72+
inImages = line == "[images]"
73+
return@forEachIndexed
74+
}
75+
if (!inImages) return@forEachIndexed
76+
val separator = line.indexOf('=')
77+
require(separator > 0) { "TOML line ${index + 1}: expected key = value in [images]" }
78+
val key = line.substring(0, separator).trim()
79+
val value = line.substring(separator + 1).trim()
80+
values[key] = parseString(value, index)
81+
}
82+
return values
83+
}
84+
85+
private fun stripComment(line: String): String {
86+
var quoted = false
87+
var escaped = false
88+
line.forEachIndexed { index, char ->
89+
when {
90+
escaped -> escaped = false
91+
char == '\\' && quoted -> escaped = true
92+
char == '"' -> quoted = !quoted
93+
char == '#' && !quoted -> return line.substring(0, index)
94+
}
95+
}
96+
return line
97+
}
98+
99+
private fun parseString(value: String, line: Int): String {
100+
require(value.startsWith('"') && value.endsWith('"')) {
101+
"TOML line ${line + 1}: [images] values must be quoted strings"
102+
}
103+
return buildString {
104+
var escaped = false
105+
value.substring(1, value.length - 1).forEach { char ->
106+
when {
107+
escaped -> {
108+
append(
109+
when (char) {
110+
'"' -> '"'
111+
'\\' -> '\\'
112+
'n' -> '\n'
113+
'r' -> '\r'
114+
't' -> '\t'
115+
else -> char
116+
},
117+
)
118+
escaped = false
119+
}
120+
char == '\\' -> escaped = true
121+
else -> append(char)
122+
}
123+
}
124+
require(!escaped) { "TOML line ${line + 1}: unterminated escape in string" }
125+
}
126+
}
127+
}

‎junit5/src/main/kotlin/org/openprojectx/bigdata/test/junit5/BigDataTestExtension.kt‎

Lines changed: 42 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -13,13 +13,14 @@ import org.openprojectx.bigdata.test.core.ContainerLogOptions
1313
import org.openprojectx.bigdata.test.core.KafkaOptions
1414
import org.openprojectx.bigdata.test.core.KerberosAuthOptions
1515
import org.openprojectx.bigdata.test.core.KerberosOptions
16+
import org.openprojectx.bigdata.test.core.ObjectStoreOptions
1617
import org.openprojectx.bigdata.test.core.PortBindingOptions
1718

1819
class BigDataTestExtension : BeforeAllCallback, AfterAllCallback, ParameterResolver {
1920
override fun beforeAll(context: ExtensionContext) {
2021
val annotation = context.requiredTestClass.getAnnotation(BigDataTest::class.java)
2122
?: return
22-
val kit = kitFrom(annotation)
23+
val kit = kitFrom(annotation, context)
2324
kit.start()
2425
BigDataTestKitStore.put(context, kit)
2526
}
@@ -34,7 +35,9 @@ class BigDataTestExtension : BeforeAllCallback, AfterAllCallback, ParameterResol
3435
override fun resolveParameter(parameterContext: ParameterContext, extensionContext: ExtensionContext): Any =
3536
BigDataTestKitStore.get(extensionContext)
3637

37-
private fun kitFrom(annotation: BigDataTest): BigDataTestKit {
38+
private fun kitFrom(annotation: BigDataTest, context: ExtensionContext): BigDataTestKit {
39+
val configLocations = bigDataExtensionsLocations(context) + annotation.config.asIterable()
40+
val images = BigDataTestConfigLoader(context.requiredTestClass.classLoader).load(configLocations)
3841
val builder = BigDataTestKit.builder()
3942
.withPortBindings(
4043
PortBindingOptions(
@@ -62,6 +65,7 @@ class BigDataTestExtension : BeforeAllCallback, AfterAllCallback, ParameterResol
6265
builder.withKerberos(
6366
KerberosOptions(
6467
enabled = true,
68+
image = images.kerberos ?: "openprojectx/kerby-kdc:latest",
6569
clientPrincipal = annotation.kerberosClientPrincipal,
6670
clientPassword = annotation.kerberosClientPassword,
6771
),
@@ -71,6 +75,7 @@ class BigDataTestExtension : BeforeAllCallback, AfterAllCallback, ParameterResol
7175
builder.withHdfs(
7276
HdfsOptions(
7377
enabled = true,
78+
image = images.hdfs ?: "apache/hadoop:3.5.0",
7479
kerberos = KerberosAuthOptions(
7580
enabled = annotation.hdfsKerberos,
7681
servicePrincipal = "nn/hdfs.example.com@EXAMPLE.COM",
@@ -83,6 +88,8 @@ class BigDataTestExtension : BeforeAllCallback, AfterAllCallback, ParameterResol
8388
builder.withHiveMetastore(
8489
HiveMetastoreOptions(
8590
enabled = true,
91+
image = images.hiveMetastore ?: "ghcr.io/openprojectx/cloudera-hms:0.1.16",
92+
apacheHiveImage = images.hiveMetastoreApache ?: "apache/hive:3.1.3",
8693
kerberos = KerberosAuthOptions(
8794
enabled = annotation.hiveMetastoreKerberos,
8895
servicePrincipal = "hive/hive-metastore.example.com@EXAMPLE.COM",
@@ -95,8 +102,11 @@ class BigDataTestExtension : BeforeAllCallback, AfterAllCallback, ParameterResol
95102
builder.withKafka(
96103
KafkaOptions(
97104
enabled = true,
105+
image = images.kafka ?: "apache/kafka:4.1.2",
98106
schemaRegistryEnabled = annotation.schemaRegistry,
107+
schemaRegistryImage = images.schemaRegistry ?: "confluentinc/cp-schema-registry:7.8.0",
99108
kafkaUiEnabled = annotation.kafkaUi,
109+
kafkaUiImage = images.kafkaUi ?: "ghcr.io/kafbat/kafka-ui:latest",
100110
kerberos = KerberosAuthOptions(
101111
enabled = annotation.kafkaKerberos,
102112
servicePrincipal = "kafka/localhost@EXAMPLE.COM",
@@ -110,8 +120,22 @@ class BigDataTestExtension : BeforeAllCallback, AfterAllCallback, ParameterResol
110120
),
111121
)
112122
}
113-
if (annotation.localStackS3) builder.withLocalStackS3()
114-
if (annotation.fakeGcs) builder.withFakeGcs()
123+
if (annotation.localStackS3) {
124+
builder.withLocalStackS3(
125+
ObjectStoreOptions(
126+
enabled = true,
127+
image = images.localStackS3 ?: "localstack/localstack:4.14.0",
128+
),
129+
)
130+
}
131+
if (annotation.fakeGcs) {
132+
builder.withFakeGcs(
133+
ObjectStoreOptions(
134+
enabled = true,
135+
image = images.fakeGcs ?: "fsouza/fake-gcs-server:1.54",
136+
),
137+
)
138+
}
115139
return builder.build()
116140
}
117141

@@ -120,4 +144,18 @@ class BigDataTestExtension : BeforeAllCallback, AfterAllCallback, ParameterResol
120144
hiveMetastoreKerberos ||
121145
kafkaKerberos ||
122146
kafkaUiKerberos
147+
148+
private fun bigDataExtensionsLocations(context: ExtensionContext): List<String> =
149+
context.requiredTestClass.annotations
150+
.firstOrNull { it.annotationClass.qualifiedName == BIG_DATA_EXTENSIONS_ANNOTATION }
151+
?.let { annotation ->
152+
val method = annotation.javaClass.getMethod("value")
153+
@Suppress("UNCHECKED_CAST")
154+
(method.invoke(annotation) as Array<String>).toList()
155+
}
156+
.orEmpty()
157+
158+
private companion object {
159+
const val BIG_DATA_EXTENSIONS_ANNOTATION = "org.openprojectx.bigdata.test.extensions.junit5.BigDataExtensions"
160+
}
123161
}

0 commit comments

Comments
 (0)