Skip to content

Commit dad923f

Browse files
committed
feat: migrate configuration format from JSON to TOML
- Replaced JSON configuration with TOML for `BigDataExtensions` across examples, tests, and documentation. - Updated `BigDataExtensionsConfigLoader` to support parsing TOML files via a custom parser. - Converted all JSON examples in documentation and tests to TOML for improved clarity and compactness. - Added new TOML resource file `spark-bigdata-extensions.toml` in Spark example. - Improved inline array and table handling for declarative test data configurations with TOML.
1 parent 43aea5e commit dad923f

6 files changed

Lines changed: 299 additions & 74 deletions

File tree

README.md

Lines changed: 18 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -56,10 +56,10 @@ The built-in extensions currently support:
5656
- `s3Jceks`: creates an HDFS-backed JCEKS file from the LocalStack S3 endpoint credentials and exposes `s3-jceks.credential-provider.path`.
5757
- `kafkaAvro`: creates Kafka topics and produces Avro records through Schema Registry from inline JSON records or a records resource.
5858

59-
JUnit usage is declaration-driven. The test declares the services with `@BigDataTest`, then points `@BigDataExtensions` at one or more JSON resources:
59+
JUnit usage is declaration-driven. The test declares the services with `@BigDataTest`, then points `@BigDataExtensions` at one or more TOML resources:
6060

6161
```kotlin
62-
@BigDataExtensions("classpath:bigdata-extensions.json")
62+
@BigDataExtensions("classpath:bigdata-extensions.toml")
6363
@BigDataTest(
6464
hdfs = true,
6565
kafka = true,
@@ -76,27 +76,22 @@ class MyIntegrationTest {
7676

7777
Example config:
7878

79-
```json
80-
{
81-
"s3Jceks": {
82-
"enabled": true,
83-
"hdfsDir": "/bigdata-test/spark",
84-
"fileName": "s3.jceks"
85-
},
86-
"kafkaAvro": {
87-
"enabled": true,
88-
"topics": [
89-
{
90-
"name": "events",
91-
"schema": "classpath:schemas/event.avsc",
92-
"records": [
93-
{"key": "alpha", "value": {"id": 1, "name": "alpha"}},
94-
{"key": "beta", "value": {"id": 2, "name": "beta"}}
95-
]
96-
}
97-
]
98-
}
99-
}
79+
```toml
80+
[s3Jceks]
81+
enabled = true
82+
hdfsDir = "/bigdata-test/spark"
83+
fileName = "s3.jceks"
84+
85+
[kafkaAvro]
86+
enabled = true
87+
88+
[[kafkaAvro.topics]]
89+
name = "events"
90+
schema = "classpath:schemas/event.avsc"
91+
records = [
92+
{ key = "alpha", value = { id = 1, name = "alpha" } },
93+
{ key = "beta", value = { id = 2, name = "beta" } },
94+
]
10095
```
10196

10297
For future extension modules, implement `BigDataExtension` directly for programmatic use or publish a `BigDataExtensionProvider` via `ServiceLoader`. Providers are selected from config entries under `extensions` by `type`, and extensions can hook lifecycle events such as `AFTER_KIT_START`, `BEFORE_TEST_EXECUTION`, and `AFTER_ALL`.

doc/user-guide.adoc

Lines changed: 21 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -307,7 +307,7 @@ Use `@BigDataExtensions` with one or more config resources.
307307

308308
[source,kotlin]
309309
----
310-
@BigDataExtensions("classpath:bigdata-extensions.json")
310+
@BigDataExtensions("classpath:bigdata-extensions.toml")
311311
@BigDataTest(
312312
hdfs = true,
313313
kafka = true,
@@ -328,17 +328,14 @@ Extension output is exposed through `BigDataExtensionResult`.
328328

329329
The `s3Jceks` extension creates a Hadoop credential provider file in HDFS and stores LocalStack S3 credentials in it.
330330

331-
[source,json]
331+
[source,toml]
332332
----
333-
{
334-
"s3Jceks": {
335-
"enabled": true,
336-
"hdfsDir": "/bigdata-test/spark",
337-
"fileName": "s3.jceks",
338-
"accessKeyAlias": "fs.s3a.access.key",
339-
"secretKeyAlias": "fs.s3a.secret.key"
340-
}
341-
}
333+
[s3Jceks]
334+
enabled = true
335+
hdfsDir = "/bigdata-test/spark"
336+
fileName = "s3.jceks"
337+
accessKeyAlias = "fs.s3a.access.key"
338+
secretKeyAlias = "fs.s3a.secret.key"
342339
----
343340

344341
Outputs:
@@ -352,26 +349,21 @@ Use `s3-jceks.credential-provider.path` as `hadoop.security.credential.provider.
352349

353350
The `kafkaAvro` extension creates topics and produces Avro records using Schema Registry.
354351

355-
[source,json]
356-
----
357-
{
358-
"kafkaAvro": {
359-
"enabled": true,
360-
"topics": [
361-
{
362-
"name": "events",
363-
"schema": "classpath:schemas/event.avsc",
364-
"records": [
365-
{"key": "alpha", "value": {"id": 1, "name": "alpha"}},
366-
{"key": "beta", "value": {"id": 2, "name": "beta"}}
367-
]
368-
}
369-
]
370-
}
371-
}
352+
[source,toml]
353+
----
354+
[kafkaAvro]
355+
enabled = true
356+
357+
[[kafkaAvro.topics]]
358+
name = "events"
359+
schema = "classpath:schemas/event.avsc"
360+
records = [
361+
{ key = "alpha", value = { id = 1, name = "alpha" } },
362+
{ key = "beta", value = { id = 2, name = "beta" } },
363+
]
372364
----
373365

374-
Records can be supplied inline or from a resource. Use this extension when test data should be declarative and reproducible instead of manually wired in every test.
366+
Records can be supplied inline as compact TOML arrays or from a resource. Use this extension when test data should be declarative and reproducible instead of manually wired in every test.
375367

376368
=== Extension Lifecycle
377369

example/spark/src/test/kotlin/org/openprojectx/bigdata/test/example/spark/SparkBigDataTestExample.kt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ import java.nio.charset.StandardCharsets
1818
import java.nio.file.Files
1919
import java.time.Duration
2020

21-
@BigDataExtensions("classpath:spark-bigdata-extensions.json")
21+
@BigDataExtensions("classpath:spark-bigdata-extensions.toml")
2222
@BigDataTest(
2323
hdfs = true,
2424
hiveMetastore = true,

example/spark/src/test/resources/spark-bigdata-extensions.json

Lines changed: 0 additions & 20 deletions
This file was deleted.
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
[s3Jceks]
2+
enabled = true
3+
hdfsDir = "/bigdata-test/spark"
4+
fileName = "s3.jceks"
5+
6+
[kafkaAvro]
7+
enabled = true
8+
9+
[[kafkaAvro.topics]]
10+
name = "spark-avro-events"
11+
schema = "classpath:schemas/spark-event.avsc"
12+
records = [
13+
{ key = "alpha", value = { id = 1, name = "alpha" } },
14+
{ key = "beta", value = { id = 2, name = "beta" } },
15+
]

0 commit comments

Comments
 (0)