Skip to content

Commit ce4feb6

Browse files
CoderYellowCoderYellow
committed
feat: add sameHostPorts support for consistent host-container port mappings
- Introduced `sameHostPorts` flag in `BigDataTest` annotation and `BigDataTestKit` builder to enable port alignment. - Updated containers to support binding same host ports as service container ports. - Enhanced metadata with default ports and endpoint property mappings for all services. - Extended documentation and examples to demonstrate `sameHostPorts` usage. - Added `FixedPortKafkaContainer` for handling fixed Kafka host port bindings.
1 parent dc2116b commit ce4feb6

10 files changed

Lines changed: 211 additions & 20 deletions

File tree

‎README.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -199,8 +199,40 @@ JUnit host ports are random by default. Leave port fields at `0` for Testcontain
199199
class MyFixedPortTest
200200
```
201201

202+
If you want the host to use each service's normal container port, enable `sameHostPorts`:
203+
204+
```kotlin
205+
@BigDataTest(
206+
hdfs = true,
207+
hiveMetastore = true,
208+
kafka = true,
209+
schemaRegistry = true,
210+
localStackS3 = true,
211+
fakeGcs = true,
212+
sameHostPorts = true,
213+
)
214+
class MySameHostPortTest
215+
```
216+
217+
This binds ports such as HDFS `8020`/`9870`, HMS `9083`, Kafka `9092`, Schema Registry `8085`, LocalStack `4566`, and fake GCS `4443` on localhost. Explicit per-service port fields still take priority over `sameHostPorts`.
218+
202219
Available JUnit port fields are `kerberosKdcPort`, `hdfsNameNodePort`, `hdfsWebPort`, `hiveMetastorePort`, `kafkaPort`, `schemaRegistryPort`, `kafkaUiPort`, `localStackS3Port`, and `fakeGcsPort`. Endpoint properties still use the actual mapped host ports returned by Testcontainers.
203220

221+
Default service ports and endpoint property keys:
222+
223+
| Service | Port names | Default ports | Main endpoint properties |
224+
| --- | --- | --- | --- |
225+
| `KERBEROS` | `kdc` | `88` | `bigdata.test.kerberos.kdc` |
226+
| `HDFS` | `namenode`, `web` | `8020`, `9870` | `fs.defaultFS`, `spring.hadoop.fs-uri` |
227+
| `HIVE_METASTORE` | `thrift` | `9083` | `hive.metastore.uris`, `spring.bigdata.test.hive-metastore.thrift-uri` |
228+
| `KAFKA` | `bootstrap` | `9092` | `bootstrap.servers`, `spring.kafka.bootstrap-servers`, `bootstrap.servers.internal` |
229+
| `SCHEMA_REGISTRY` | `http` | `8085` | `schema.registry.url` |
230+
| `KAFKA_UI` | `http` | `8080` | `bigdata.test.kafka-ui.url` |
231+
| `LOCALSTACK_S3` | `edge` | `4566` | `aws.endpoint-url.s3`, `spring.cloud.aws.s3.endpoint` |
232+
| `FAKE_GCS` | `http` | `4443` | `google.cloud.storage.host`, `bigdata.test.gcs.endpoint` |
233+
234+
The same metadata is available programmatically from `BigDataService.defaultPorts` and `BigDataService.endpointProperties`.
235+
204236
When Hive Metastore is started with LocalStack S3 or fake GCS, the kit also injects server-side Hadoop filesystem configuration into HMS. External table DDL can validate `s3a://` locations against the internal LocalStack endpoint and `gs://` locations against the internal fake GCS endpoint when the HMS image includes the matching filesystem connector.
205237

206238
## Spring Boot

‎core/src/main/kotlin/org/openprojectx/bigdata/test/core/BigDataService.kt‎

Lines changed: 47 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,50 @@
11
package org.openprojectx.bigdata.test.core
22

3-
enum class BigDataService {
4-
KERBEROS,
5-
HDFS,
6-
HIVE_METASTORE,
7-
KAFKA,
8-
SCHEMA_REGISTRY,
9-
KAFKA_UI,
10-
LOCALSTACK_S3,
11-
FAKE_GCS,
3+
enum class BigDataService(
4+
val defaultPorts: Map<String, Int>,
5+
val endpointProperties: Set<String>,
6+
) {
7+
KERBEROS(
8+
defaultPorts = mapOf("kdc" to 88),
9+
endpointProperties = setOf(
10+
"bigdata.test.kerberos.realm",
11+
"bigdata.test.kerberos.kdc",
12+
"bigdata.test.kerberos.client-principal",
13+
"bigdata.test.kerberos.client-password",
14+
),
15+
),
16+
HDFS(
17+
defaultPorts = mapOf("namenode" to 8020, "web" to 9870),
18+
endpointProperties = setOf("fs.defaultFS", "spring.hadoop.fs-uri"),
19+
),
20+
HIVE_METASTORE(
21+
defaultPorts = mapOf("thrift" to 9083),
22+
endpointProperties = setOf("hive.metastore.uris", "spring.bigdata.test.hive-metastore.thrift-uri"),
23+
),
24+
KAFKA(
25+
defaultPorts = mapOf("bootstrap" to 9092),
26+
endpointProperties = setOf("bootstrap.servers", "spring.kafka.bootstrap-servers", "bootstrap.servers.internal"),
27+
),
28+
SCHEMA_REGISTRY(
29+
defaultPorts = mapOf("http" to 8085),
30+
endpointProperties = setOf("schema.registry.url"),
31+
),
32+
KAFKA_UI(
33+
defaultPorts = mapOf("http" to 8080),
34+
endpointProperties = setOf("bigdata.test.kafka-ui.url"),
35+
),
36+
LOCALSTACK_S3(
37+
defaultPorts = mapOf("edge" to 4566),
38+
endpointProperties = setOf(
39+
"spring.cloud.aws.s3.endpoint",
40+
"aws.endpoint-url.s3",
41+
"aws.accessKeyId",
42+
"aws.secretAccessKey",
43+
"aws.region",
44+
),
45+
),
46+
FAKE_GCS(
47+
defaultPorts = mapOf("http" to 4443),
48+
endpointProperties = setOf("bigdata.test.gcs.endpoint", "google.cloud.storage.host"),
49+
),
1250
}

‎core/src/main/kotlin/org/openprojectx/bigdata/test/core/BigDataTestKit.kt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,9 @@ class BigDataTestKit private constructor(
7979
fun withPortBindings(options: PortBindingOptions): Builder =
8080
apply { portBindings = options }
8181

82+
fun withSameHostPorts(): Builder =
83+
apply { portBindings = portBindings.copy(sameHostPorts = true) }
84+
8285
fun withContainerLogs(options: ContainerLogOptions): Builder =
8386
apply { containerLogs = options }
8487

‎core/src/main/kotlin/org/openprojectx/bigdata/test/core/BigDataTestKitOptions.kt‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,7 @@ data class ObjectStoreOptions(
9090

9191

9292
data class PortBindingOptions(
93+
val sameHostPorts: Boolean = false,
9394
val kerberosKdc: Int = 0,
9495
val hdfsNameNode: Int = 0,
9596
val hdfsWeb: Int = 0,
@@ -99,7 +100,16 @@ data class PortBindingOptions(
99100
val kafkaUi: Int = 0,
100101
val localStackS3: Int = 0,
101102
val fakeGcs: Int = 0,
102-
)
103+
) {
104+
fun hostPort(containerPort: Int, configuredHostPort: Int): Int {
105+
require(configuredHostPort >= 0) { "Host port must be 0 for random binding or a positive fixed port" }
106+
return when {
107+
configuredHostPort > 0 -> configuredHostPort
108+
sameHostPorts -> containerPort
109+
else -> 0
110+
}
111+
}
112+
}
103113

104114
enum class ContainerLogMode {
105115
NONE,

‎core/src/main/kotlin/org/openprojectx/bigdata/test/core/container/BigDataContainerFactory.kt‎

Lines changed: 19 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ internal class BigDataContainerFactory(
7777
val container = GenericBigDataContainer(kerberos.image)
7878
.withNetwork(network)
7979
.withNetworkAliases("kerby-kdc")
80-
.withServicePort(88, options.portBindings.kerberosKdc)
80+
.withServicePort(88, options.portBindings.hostPort(88, options.portBindings.kerberosKdc))
8181
.withFileSystemBind(kerberosDirectory(), "/var/lib/kerby")
8282
.withEnv("KERBY_REALM", kerberos.realm)
8383
.withEnv("KERBY_KDC_HOST", "kerby-kdc")
@@ -113,8 +113,11 @@ internal class BigDataContainerFactory(
113113
val container = GenericBigDataContainer(hdfs.image)
114114
.withNetwork(network)
115115
.withNetworkAliases("hdfs", "hdfs.example.com")
116-
.withServicePort(hdfs.nameNodePort, options.portBindings.hdfsNameNode)
117-
.withServicePort(hdfs.webPort, options.portBindings.hdfsWeb)
116+
.withServicePort(
117+
hdfs.nameNodePort,
118+
options.portBindings.hostPort(hdfs.nameNodePort, options.portBindings.hdfsNameNode),
119+
)
120+
.withServicePort(hdfs.webPort, options.portBindings.hostPort(hdfs.webPort, options.portBindings.hdfsWeb))
118121
.withCommand("sh", "-lc", hdfsStartupCommand(hdfs.nameNodePort, hdfs.webPort))
119122
.waitingFor(Wait.forHttp("/").forPort(hdfs.webPort).withStartupTimeout(Duration.ofMinutes(3)))
120123
if (hdfs.kerberos.enabled) {
@@ -159,7 +162,7 @@ internal class BigDataContainerFactory(
159162
val container = GenericBigDataContainer(hive.image)
160163
.withNetwork(network)
161164
.withNetworkAliases("hive-metastore", "hive-metastore.example.com")
162-
.withServicePort(9083, options.portBindings.hiveMetastore)
165+
.withServicePort(9083, options.portBindings.hostPort(9083, options.portBindings.hiveMetastore))
163166
.withEnv("POSTGRES_DB", hive.databaseName)
164167
.withEnv("POSTGRES_USER", hive.databaseUser)
165168
.withEnv("POSTGRES_PASSWORD", hive.databasePassword)
@@ -207,7 +210,7 @@ internal class BigDataContainerFactory(
207210
val container = GenericBigDataContainer(kafka.image)
208211
.withNetwork(network)
209212
.withNetworkAliases("kafka", "broker1.example.com")
210-
.withServicePort(9092, options.portBindings.kafka)
213+
.withServicePort(9092, options.portBindings.hostPort(9092, options.portBindings.kafka))
211214
.withEnv("KAFKA_NODE_ID", "1")
212215
.withEnv("KAFKA_PROCESS_ROLES", "broker,controller")
213216
.withEnv("KAFKA_CONTROLLER_QUORUM_VOTERS", "1@kafka:29093")
@@ -253,7 +256,13 @@ internal class BigDataContainerFactory(
253256
}
254257

255258
private fun plaintextKafka(kafka: KafkaOptions): BigDataServiceContainer {
256-
val container = KafkaContainer(DockerImageName.parse(kafka.image))
259+
val kafkaHostPort = options.portBindings.hostPort(9092, options.portBindings.kafka)
260+
val container = if (kafkaHostPort == 0) {
261+
KafkaContainer(DockerImageName.parse(kafka.image))
262+
} else {
263+
FixedPortKafkaContainer(DockerImageName.parse(kafka.image)).withServicePort(9092, kafkaHostPort)
264+
}
265+
container
257266
.withNetwork(network)
258267
.withNetworkAliases("kafka")
259268
.withListener("kafka:19092")
@@ -279,7 +288,7 @@ internal class BigDataContainerFactory(
279288
val container = GenericBigDataContainer(kafka.schemaRegistryImage)
280289
.withNetwork(network)
281290
.withNetworkAliases("schema-registry")
282-
.withServicePort(8085, options.portBindings.schemaRegistry)
291+
.withServicePort(8085, options.portBindings.hostPort(8085, options.portBindings.schemaRegistry))
283292
.withEnv("SCHEMA_REGISTRY_HOST_NAME", "schema-registry")
284293
.withEnv("SCHEMA_REGISTRY_LISTENERS", "http://0.0.0.0:8085")
285294
.withEnv("SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS", "PLAINTEXT://kafka:19092")
@@ -314,7 +323,7 @@ internal class BigDataContainerFactory(
314323
val container = GenericBigDataContainer(kafka.kafkaUiImage)
315324
.withNetwork(network)
316325
.withNetworkAliases("kafka-ui")
317-
.withServicePort(8080, options.portBindings.kafkaUi)
326+
.withServicePort(8080, options.portBindings.hostPort(8080, options.portBindings.kafkaUi))
318327
.withEnv("KAFKA_CLUSTERS_0_NAME", "local")
319328
.withEnv("KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS", "kafka:9092")
320329
.withEnv("DYNAMIC_CONFIG_ENABLED", "false")
@@ -346,7 +355,7 @@ internal class BigDataContainerFactory(
346355
val container = GenericBigDataContainer(objectStore.image)
347356
.withNetwork(network)
348357
.withNetworkAliases("localstack")
349-
.withServicePort(4566, options.portBindings.localStackS3)
358+
.withServicePort(4566, options.portBindings.hostPort(4566, options.portBindings.localStackS3))
350359
.withEnv("SERVICES", "s3")
351360
.waitingFor(Wait.forHttp("/_localstack/health").forStatusCode(200).withStartupTimeout(Duration.ofMinutes(3)))
352361

@@ -372,7 +381,7 @@ internal class BigDataContainerFactory(
372381
val container = GenericBigDataContainer(objectStore.image)
373382
.withNetwork(network)
374383
.withNetworkAliases("fake-gcs")
375-
.withServicePort(4443, options.portBindings.fakeGcs)
384+
.withServicePort(4443, options.portBindings.hostPort(4443, options.portBindings.fakeGcs))
376385
.withCommand("-scheme", "http", "-port", "4443")
377386
.waitingFor(Wait.forHttp("/storage/v1/b").forStatusCode(200).withStartupTimeout(Duration.ofMinutes(2)))
378387

‎core/src/main/kotlin/org/openprojectx/bigdata/test/core/container/FixedPortKafkaContainer.kt‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
package org.openprojectx.bigdata.test.core.container
2+
3+
import org.testcontainers.containers.InternetProtocol
4+
import org.testcontainers.kafka.KafkaContainer
5+
import org.testcontainers.utility.DockerImageName
6+
7+
internal class FixedPortKafkaContainer(image: DockerImageName) : KafkaContainer(image) {
8+
fun withServicePort(containerPort: Int, hostPort: Int): FixedPortKafkaContainer =
9+
apply {
10+
require(hostPort > 0) { "Fixed Kafka host port must be a positive port" }
11+
addFixedExposedPort(hostPort, containerPort, InternetProtocol.TCP)
12+
}
13+
}

‎doc/user-guide.adoc‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -192,6 +192,35 @@ Host ports are dynamic by default. Keep port fields set to `0` unless an externa
192192
class MyFixedPortTest
193193
----
194194

195+
To bind each enabled service to the same localhost port it uses inside the container, enable `sameHostPorts`.
196+
197+
[source,kotlin]
198+
----
199+
@BigDataTest(
200+
hdfs = true,
201+
hiveMetastore = true,
202+
kafka = true,
203+
schemaRegistry = true,
204+
localStackS3 = true,
205+
fakeGcs = true,
206+
sameHostPorts = true,
207+
)
208+
class MySameHostPortTest
209+
----
210+
211+
This maps common service ports directly, for example HDFS `8020` and `9870`, Hive Metastore `9083`, Kafka `9092`, Schema Registry `8085`, LocalStack `4566`, and fake GCS `4443`. Explicit per-service port fields still take priority over `sameHostPorts`.
212+
213+
The core builder exposes the same behavior:
214+
215+
[source,kotlin]
216+
----
217+
val kit = BigDataTestKit.builder()
218+
.withHdfs()
219+
.withKafka()
220+
.withSameHostPorts()
221+
.build()
222+
----
223+
195224
Available port fields:
196225

197226
* `kerberosKdcPort`
@@ -206,6 +235,60 @@ Available port fields:
206235

207236
Endpoint properties always report the actual mapped host ports.
208237

238+
=== Service Endpoint Reference
239+
240+
Each started service exposes a `BigDataEndpoint` with named ports and connection properties. With dynamic ports, the host ports are assigned by Testcontainers. With `sameHostPorts = true`, the host ports below are used directly unless an explicit per-service port field overrides them.
241+
242+
[cols="1,2,2,4", options="header"]
243+
|===
244+
|Service
245+
|Port names
246+
|Default ports
247+
|Endpoint property keys
248+
249+
|`BigDataService.KERBEROS`
250+
|`kdc`
251+
|`88`
252+
|`bigdata.test.kerberos.realm`, `bigdata.test.kerberos.kdc`, `bigdata.test.kerberos.client-principal`, `bigdata.test.kerberos.client-password`
253+
254+
|`BigDataService.HDFS`
255+
|`namenode`, `web`
256+
|`8020`, `9870`
257+
|`fs.defaultFS`, `spring.hadoop.fs-uri`
258+
259+
|`BigDataService.HIVE_METASTORE`
260+
|`thrift`
261+
|`9083`
262+
|`hive.metastore.uris`, `spring.bigdata.test.hive-metastore.thrift-uri`
263+
264+
|`BigDataService.KAFKA`
265+
|`bootstrap`
266+
|`9092`
267+
|`bootstrap.servers`, `spring.kafka.bootstrap-servers`, `bootstrap.servers.internal`
268+
269+
|`BigDataService.SCHEMA_REGISTRY`
270+
|`http`
271+
|`8085`
272+
|`schema.registry.url`
273+
274+
|`BigDataService.KAFKA_UI`
275+
|`http`
276+
|`8080`
277+
|`bigdata.test.kafka-ui.url`
278+
279+
|`BigDataService.LOCALSTACK_S3`
280+
|`edge`
281+
|`4566`
282+
|`spring.cloud.aws.s3.endpoint`, `aws.endpoint-url.s3`, `aws.accessKeyId`, `aws.secretAccessKey`, `aws.region`
283+
284+
|`BigDataService.FAKE_GCS`
285+
|`http`
286+
|`4443`
287+
|`bigdata.test.gcs.endpoint`, `google.cloud.storage.host`
288+
|===
289+
290+
The same reference data is exposed in code through `BigDataService.defaultPorts` and `BigDataService.endpointProperties`.
291+
209292
== Container Logs
210293

211294
Container logs are disabled by default. Enable them when debugging service startup or service-side behavior.

‎example/spark/src/test/kotlin/org/openprojectx/bigdata/test/example/spark/SparkBigDataTestExample.kt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ import org.openprojectx.bigdata.test.junit5.BigDataTest
1515
localStackS3 = true,
1616
fakeGcs = true,
1717
containerLogMode = ContainerLogMode.FILE,
18+
sameHostPorts = true,
1819
)
1920
class SparkBigDataTestExample : SparkBigDataScenario() {
2021
override val runId: String get() = scenarioRunId

‎junit5/src/main/kotlin/org/openprojectx/bigdata/test/junit5/BigDataTest.kt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ annotation class BigDataTest(
2020
val kafkaUiKerberos: Boolean = false,
2121
val localStackS3: Boolean = false,
2222
val fakeGcs: Boolean = false,
23+
val sameHostPorts: Boolean = false,
2324
val kerberosKdcPort: Int = 0,
2425
val hdfsNameNodePort: Int = 0,
2526
val hdfsWebPort: Int = 0,

‎junit5/src/main/kotlin/org/openprojectx/bigdata/test/junit5/BigDataTestExtension.kt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ class BigDataTestExtension : BeforeAllCallback, AfterAllCallback, ParameterResol
3737
val builder = BigDataTestKit.builder()
3838
.withPortBindings(
3939
PortBindingOptions(
40+
sameHostPorts = annotation.sameHostPorts,
4041
kerberosKdc = annotation.kerberosKdcPort,
4142
hdfsNameNode = annotation.hdfsNameNodePort,
4243
hdfsWeb = annotation.hdfsWebPort,

0 commit comments

Comments
 (0)