kotlin

Stream processing with Kafka, ksqlDB and Kotlin

Experiment with stream processing using Kotlin, Kafka, and ksqlDB, providing a playground for seamless data integration and analysis

Getting Started

1. Install JDK 21 or 17 (earlier versions are deprecated) with SDKMan:

GraalVM (Native)

sdk i java 23.1.9.r21-nik   # JDK 21
sdk i java 23.0.11.r17-nik  # JDK 17

Standard JDK

sdk i java 21.0.10-librca
sdk i java 17.0.18-librca

2. (Optional) Install pre-commit:

brew install pre-commit

# From root folder where `.pre-commit-config.yaml` is located, run:
pre-commit install

3. Build and generate the application artifact:

# Native - requires GraalVM
./gradlew nativeCompile

# fatJar - Standard JDK
./gradlew clean shadowJar

4. Start Kafka, Schema Registry and others from the parent directory:

docker compose up -d

5. Run the application with and check the subcommands:

# Native - requires GraalVM
./build/native/nativeCompile/kotlin-sp

# fatJar - Standard JDK
java -jar build/libs/kotlin-sp-2.0.jar

Commands:
  help      Display help information about the specified command.
  producer  Parse data from source dataset and publish as JSON to Kafka
  consumer  Subscribe and consume records from Kafka topic

5.1. CLI for ProducerApp

# Native - requires GraalVM
./build/native/nativeCompile/kotlin-sp producer

# fatJar - Standard JDK
java -jar build/libs/kotlin-sp-2.0.jar producer

Parse data from source dataset and publish as JSON to Kafka
Commands:
  green   Process GreenTaxiDTO data from CSV file and publish to Kafka topic
  yellow  Process YellowTaxiDTO data from CSV file and publish to Kafka topic
  fhv     Process FhvTaxiDTO data from CSV file and publish to Kafka topic

# Native - requires GraalVM
./build/native/nativeCompile/kotlin-sp producer [green|yellow|fhv]

# fatJar - Standard JDK
java -jar build/libs/kotlin-sp-2.0.jar producer [green|yellow|fhv]

Process [GreenTaxiDTO|YellowTaxiDTO|FhvTaxiDTO] data from CSV file and publish to Kafka topic
  -i, --csv-file=<csvFilePath>
                        CSV file path
  -t, --topic=<topic>   Target Kafka topic for records

5.2. CLI for ConsumerApp

# Native - requires GraalVM
./build/native/nativeCompile/kotlin-sp consumer

# fatJar - Standard JDK
java -jar build/libs/kotlin-sp-2.0.jar consumer

Subscribe and consume records from Kafka topic
Commands:
  green   Deserialize ConsumerRecords from source Kafka topic to GreenTaxiDTO
  yellow  Deserialize ConsumerRecords from source Kafka topic to YellowTaxiDTO
  fhv     Deserialize ConsumerRecords from source Kafka topic to FhvTaxiDTO

# Native - requires GraalVM
./build/native/nativeCompile/kotlin-sp consumer [green|yellow|fhv]

# fatJar - Standard JDK
java -jar build/libs/kotlin-sp-2.0.jar consumer [green|yellow|fhv]

Deserialize ConsumerRecords from source Kafka topic to [GreenTaxiDTO|YellowTaxiDTO|FhvTaxiDTO]
  -g, --consumer-group=<consumerGroup>
                        Consumer group to subscribe to the Source kafka topic
  -t, --topic=<topic>   Source Kafka topic for records

TODO's:

• Set up an environment for Kotlin, Kafka Client and KafkaStreams
• Set up a Schema Registry
• Build a Cli Application for Producer and Consumer
• Explore serialization with JSON
• Explore serialization with Protobuf
• Adds support to Native Image builds (GraalVM)
• Static code analysis with detekt