Skip to content

Commit cb25b9f

Browse files
kibertoadkibertoad
authored
Implement load tests for Kafka (#411)
1 parent 87b1614 commit cb25b9f

26 files changed

Lines changed: 2607 additions & 1 deletion

‎biome.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
"@lokalise/biome-config/configs/biome-package.jsonc"
88
],
99
"files": {
10-
"includes": ["**", "!.turbo", "!**/dist", "!**/coverage"]
10+
"includes": ["**", "!.turbo", "!**/dist", "!**/coverage", "!**/load-tests"]
1111
},
1212
"linter": {
1313
"rules": {

‎packages/kafka/load-tests/.gitattributes‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
scripts/*.sh text eol=lf

‎packages/kafka/load-tests/README.md‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
# Kafka Load Tests
2+
3+
Load tests for `@message-queue-toolkit/kafka` measuring throughput, latency, and backlog under configurable load.
4+
5+
Two producer modes and two consumer modes can be combined:
6+
7+
- **CDC**: CockroachDB inserts → CDC changefeed → Kafka (realistic end-to-end)
8+
- **Direct**: `AbstractKafkaPublisher` → Kafka (isolates Kafka consumer performance)
9+
- **Single**: One message at a time (`batchProcessingEnabled: false`)
10+
- **Batch**: Batched consumption via `KafkaMessageBatchStream` (`batchProcessingEnabled: true`)
11+
12+
## Prerequisites
13+
14+
- Docker & Docker Compose
15+
- Node.js >= 22.14.0
16+
17+
## Quick Start
18+
19+
```bash
20+
# Install dependencies
21+
npm install
22+
23+
# Start infrastructure (Kafka + CockroachDB + CDC changefeed)
24+
npm run docker:start
25+
26+
# Stop and clean up
27+
npm run docker:stop
28+
```
29+
30+
## Test Scripts
31+
32+
### CDC (CockroachDB → CDC → Kafka)
33+
34+
```bash
35+
# Single-message consumer
36+
npm run load:cdc:light # 100 rows/sec, 30s
37+
npm run load:cdc:medium # 1000 rows/sec, 60s
38+
npm run load:cdc:heavy # 5000 rows/sec, 120s
39+
npm run load:cdc -- --rate 500 --duration 45 --batch 50
40+
41+
# Batch consumer
42+
npm run load:cdc:batch:light # 100 rows/sec, 30s
43+
npm run load:cdc:batch:medium # 1000 rows/sec, 60s
44+
npm run load:cdc:batch:heavy # 5000 rows/sec, 120s
45+
npm run load:cdc:batch -- --rate 500 --consumer-batch 100 --consumer-timeout 500
46+
```
47+
48+
### Direct (Kafka publisher → Kafka)
49+
50+
```bash
51+
# Single-message consumer
52+
npm run load:direct:light # 100 msgs/sec, 30s
53+
npm run load:direct:medium # 1000 msgs/sec, 60s
54+
npm run load:direct:heavy # 5000 msgs/sec, 120s
55+
npm run load:direct -- --rate 500 --duration 45 --batch 50
56+
57+
# Batch consumer
58+
npm run load:direct:batch:light # 100 msgs/sec, 30s
59+
npm run load:direct:batch:medium # 1000 msgs/sec, 60s
60+
npm run load:direct:batch:heavy # 5000 msgs/sec, 120s
61+
npm run load:direct:batch -- --rate 500 --consumer-batch 100 --consumer-timeout 500
62+
```
63+
64+
## CLI Options
65+
66+
### Common
67+
68+
| Flag | Short | Default | Description |
69+
|------|-------|---------|-------------|
70+
| `--rate` | `-r` | 1000 | Target produce rate (rows or msgs per sec) |
71+
| `--duration` | `-d` | 60 | Test duration (seconds) |
72+
| `--batch` | `-b` | 100 | Producer batch size |
73+
74+
### Batch consumer only
75+
76+
| Flag | Default | Description |
77+
|------|---------|-------------|
78+
| `--consumer-batch` | 50 | Messages per consumer batch |
79+
| `--consumer-timeout` | 200 | Batch flush timeout (ms) |
80+
81+
## Architecture
82+
83+
### CDC mode
84+
85+
```
86+
Load Generator → CockroachDB (inserts) → CDC Changefeed → Kafka → Consumer → Metrics
87+
```
88+
89+
1. **CockroachDB** tables (`events`, `orders`) with CDC changefeed targeting Kafka
90+
2. **Load generator** inserts rows into CRDB at configurable rate with fire-and-forget concurrency
91+
3. **CDC changefeed** publishes row changes to Kafka topics
92+
4. **Consumer** (single or batch) processes messages and records metrics
93+
94+
### Direct mode
95+
96+
```
97+
Load Generator → AbstractKafkaPublisher → Kafka → Consumer → Metrics
98+
```
99+
100+
1. **Publisher** sends messages directly to Kafka topics (`direct-events`, `direct-orders`)
101+
2. **Consumer** (single or batch) processes messages and records metrics
102+
103+
## Services
104+
105+
| Service | Port | Description |
106+
|---------|------|-------------|
107+
| Kafka | 9092 | Message broker (KRaft, 6 partitions) |
108+
| CockroachDB | 26257 | SQL database |
109+
| CockroachDB UI | 8181 | DB admin console |
110+
| Kafka UI | 8080 | Topic browser |
111+
112+
## Latency Measurement
113+
114+
Each `events` row embeds `{"loadtest_ts": <epoch_ms>}` in its payload. The consumer extracts this timestamp and computes end-to-end latency (insert/publish → consume). Reported as avg, p50, p95, p99.

‎packages/kafka/load-tests/docker-compose.yml‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
services:
2+
3+
kafka:
4+
image: apache/kafka:3.7.1
5+
ports:
6+
- 9092:9092
7+
environment:
8+
KAFKA_NODE_ID: 1
9+
KAFKA_PROCESS_ROLES: broker,controller
10+
KAFKA_LISTENERS: LOCAL://0.0.0.0:9092,DOCKER://kafka:9093,CONTROLLER://localhost:9094
11+
KAFKA_ADVERTISED_LISTENERS: LOCAL://localhost:9092,DOCKER://kafka:9093
12+
KAFKA_INTER_BROKER_LISTENER_NAME: LOCAL
13+
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
14+
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,LOCAL:PLAINTEXT,DOCKER:PLAINTEXT
15+
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9094
16+
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
17+
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
18+
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
19+
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
20+
KAFKA_NUM_PARTITIONS: 6
21+
healthcheck:
22+
test: /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list
23+
interval: 5s
24+
timeout: 10s
25+
retries: 10
26+
start_period: 15s
27+
restart: on-failure
28+
29+
cockroachdb:
30+
image: cockroachdb/cockroach:latest-v25.1
31+
command: start-single-node --insecure
32+
ports:
33+
- 26257:26257
34+
- 8181:8080
35+
depends_on:
36+
kafka:
37+
condition: service_healthy
38+
healthcheck:
39+
test: curl -f http://localhost:8080/health?ready=1 || exit 1
40+
interval: 5s
41+
timeout: 5s
42+
retries: 10
43+
start_period: 10s
44+
restart: on-failure
45+
46+
crdb-init:
47+
image: cockroachdb/cockroach:latest-v25.1
48+
volumes:
49+
- ./scripts/init-crdb.sh:/init-crdb.sh
50+
entrypoint: ["/bin/bash", "/init-crdb.sh"]
51+
depends_on:
52+
cockroachdb:
53+
condition: service_healthy
54+
restart: "no"
55+
56+
kafka-ui:
57+
image: provectuslabs/kafka-ui:latest
58+
ports:
59+
- 8080:8080
60+
depends_on:
61+
kafka:
62+
condition: service_healthy
63+
environment:
64+
DYNAMIC_CONFIG_ENABLED: 'true'
65+
KAFKA_CLUSTERS_0_NAME: LoadTest
66+
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9093
67+
restart: on-failure

0 commit comments

Comments
 (0)