module6-stream-processing

Stream processing with Kafka & ksqlDB

This submodule focuses on various approaches for stream processing, such as:

• Kotlin: Producing and Consuming messages from/to Kafka with Kotlin/Java
• ksqlDB: Stream Processing with ksqlDB (Kafka Streams)

Getting Started

1. Spin up Kafka Broker with one of the following options:

1.1. Kafka Cluster (Single-broker) with KRaft:

docker compose up -d

1.2. Kafka Cluster (Multi-broker) with KRaft:

docker compose -f compose.kraft-multi-broker.yaml up -d

1.3. Kafka Cluster (Multi-broker) with Zookeeper:

docker compose -f compose.zookeeper-multi-broker.yaml up -d

1.4. Redpanda:

docker compose -f compose.redpanda.yaml up -d

2. Access Conduktor Web UI for Kafka

open http://localhost:8080

TODO's:

• Single-broker Kafka Cluster (with KRaft)
• Multi-broker Kafka Cluster (with KRaft)
• Multi-broker Kafka Cluster (with Zookeeper)
• Confluent Schema Registry
• Confluent Rest Proxy
• ksqlDB Server and CLI
• Kafka Admin UI: Conduktor Console
• Deploy Kafka Cluster on Kubernetes using Helm Charts