Skip to content

Commit 273b69c

Browse files
graytaylor0graytaylor0
committed
Add kafka buffer backward compatibility test
Signed-off-by: Taylor Gray <tylgry@amazon.com>
1 parent 638b049 commit 273b69c

10 files changed

Lines changed: 845 additions & 0 deletions

File tree

‎.github/workflows/data-prepper-kafka-backward-compatibility-e2e-tests.yml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# This workflow will build a Java project with Gradle
2+
# For more information see: https://help.github.com/actions/language-and-framework-guides/building-and-testing-java-with-gradle
3+
4+
name: Data Prepper Kafka Backward Compatibility End-to-end test with Gradle
5+
6+
on:
7+
push:
8+
branches: [ main ]
9+
pull_request:
10+
workflow_dispatch:
11+
12+
jobs:
13+
build:
14+
strategy:
15+
matrix:
16+
java: [11, 17, 21, docker]
17+
fail-fast: false
18+
19+
runs-on: ubuntu-latest
20+
21+
steps:
22+
- name: Set up JDK 11
23+
uses: actions/setup-java@v1
24+
with:
25+
java-version: 11
26+
- name: Checkout Data Prepper
27+
uses: actions/checkout@v2
28+
- name: Run Kafka backward compatibility end-to-end tests with Gradle
29+
run: ./gradlew -PendToEndJavaVersion=${{ matrix.java }} :e2e-test:kafka-backward-compatibility:kafkaBackwardCompatibilityTest

‎e2e-test/kafka-backward-compatibility/QUICKSTART.md‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# ✅ Kafka Backward Compatibility Test - Ready to Run!
2+
3+
## 🚀 Quick Start
4+
5+
### **Single Command:**
6+
7+
```bash
8+
cd /Users/tylgry/data-prepper
9+
./gradlew :e2e-test:kafka-backward-compatibility:kafkaBackwardCompatibilityTest
10+
```
11+
12+
---
13+
14+
## 📋 **What It Does:**
15+
16+
1. ✅ Starts **Kafka** using Testcontainers (no manual setup needed!)
17+
2. ✅ Starts **OpenSearch** container
18+
3. ✅ Pulls **Data Prepper 2.10.0** from Docker Hub
19+
4. ✅ Starts released Data Prepper → sends 2 test records → writes to Kafka
20+
5. ✅ Stops released Data Prepper
21+
6. ✅ Starts **current build** Data Prepper → reads from Kafka → writes to OpenSearch
22+
7. ✅ Verifies both records are in OpenSearch
23+
8. ✅ Cleanup all containers
24+
25+
**Expected time: ~5 minutes**
26+
27+
---
28+
29+
## ✅ **What Changed from Original Build.gradle Error:**
30+
31+
### **Before (Failed):**
32+
- ❌ Tried to use `confluentinc/cp-kafka:3.6.0` (doesn't exist)
33+
- ❌ Manual Docker Kafka container management
34+
- ❌ Complex networking setup
35+
36+
### **After (Fixed):**
37+
- ✅ Uses **Testcontainers** `KafkaContainer` (automatic Kafka management)
38+
- ✅ Valid image: `confluentinc/cp-kafka:7.5.0`
39+
- ✅ Kafka dependencies from Maven: `org.apache.kafka:kafka-clients:3.9.1`
40+
- ✅ Simpler networking with `host.docker.internal`
41+
42+
---
43+
44+
## 🔧 **Prerequisites:**
45+
46+
```bash
47+
# 1. Docker running
48+
docker ps
49+
50+
# 2. Build Data Prepper Docker image first
51+
cd /Users/tylgry/data-prepper
52+
./gradlew :release:docker:docker
53+
```
54+
55+
---
56+
57+
## 🎯 **Test Different Versions:**
58+
59+
```bash
60+
# Test with Data Prepper 2.9.0
61+
./gradlew :e2e-test:kafka-backward-compatibility:kafkaBackwardCompatibilityTest \
62+
-PbackwardCompatVersion=2.9.0
63+
64+
# Test with Data Prepper 2.8.0
65+
./gradlew :e2e-test:kafka-backward-compatibility:kafkaBackwardCompatibilityTest \
66+
-PbackwardCompatVersion=2.8.0
67+
```
68+
69+
---
70+
71+
## 🐛 **Troubleshooting:**
72+
73+
### **If test fails, check:**
74+
75+
```bash
76+
# 1. Docker logs for Kafka (managed by Testcontainers)
77+
docker logs $(docker ps -q --filter "ancestor=confluentinc/cp-kafka:7.5.0")
78+
79+
# 2. Released Data Prepper logs
80+
docker logs data-prepper-writer
81+
82+
# 3. Current Data Prepper logs
83+
docker logs data-prepper-reader
84+
85+
# 4. OpenSearch logs
86+
docker logs node-0.example.com
87+
```
88+
89+
### **Clean up stuck containers:**
90+
91+
```bash
92+
docker stop data-prepper-writer data-prepper-reader
93+
docker rm data-prepper-writer data-prepper-reader
94+
```
95+
96+
---
97+
98+
## 📊 **Key Changes Made:**
99+
100+
### **Files Created:**
101+
```
102+
e2e-test/kafka-backward-compatibility/
103+
├── build.gradle (Testcontainers + Docker orchestration)
104+
├── README.md (Full documentation)
105+
└── src/integrationTest/
106+
├── java/.../KafkaBackwardCompatibilityTest.java (Test logic with Testcontainers)
107+
└── resources/
108+
├── writer-pipeline.yaml (HTTP → Kafka)
109+
├── reader-pipeline.yaml (Kafka → OpenSearch)
110+
└── data-prepper-config.yaml
111+
```
112+
113+
### **Key Dependencies Added:**
114+
```gradle
115+
'org.apache.kafka:kafka-clients:3.9.1'
116+
'org.apache.kafka:connect-json:3.9.1'
117+
'org.testcontainers:testcontainers:1.19.0'
118+
'org.testcontainers:kafka:1.19.0'
119+
```
120+
121+
---
122+
123+
## ✅ **Ready to Run!**
124+
125+
```bash
126+
cd /Users/tylgry/data-prepper && \
127+
./gradlew :e2e-test:kafka-backward-compatibility:kafkaBackwardCompatibilityTest
128+
```
129+
130+
🎉 **That's it!** The test will automatically manage Kafka, verify backward compatibility, and clean up!

‎e2e-test/kafka-backward-compatibility/README.md‎

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
# Kafka Backward Compatibility End-to-End Test
2+
3+
## Overview
4+
5+
This test verifies that the current build of Data Prepper can successfully read and process messages written to Kafka by previous released versions.
6+
7+
## Test Scenario
8+
9+
### Phase 1: Write with Released Version
10+
1. Start Kafka container
11+
2. Start **released** Data Prepper from Docker Hub (e.g., `opensearchproject/data-prepper:2.10.0`)
12+
3. Send 2 test records via HTTP endpoint
13+
4. Records are written to Kafka buffer using the released version's format
14+
5. Stop the released Data Prepper container
15+
16+
### Phase 2: Read with Current Build
17+
1. Start **current** Data Prepper (built from source)
18+
2. Data Prepper reads the 2 messages from Kafka
19+
3. Messages are processed and written to OpenSearch
20+
21+
### Phase 3: Verification
22+
1. Query OpenSearch to verify both records were correctly processed
23+
2. Validate message content matches expected values
24+
25+
## Running the Test
26+
27+
### Prerequisites
28+
- Docker running locally
29+
- Java 11 or later
30+
- Gradle
31+
32+
### Run the Test
33+
34+
```bash
35+
# From the repository root
36+
./gradlew :e2e-test:kafka-backward-compatibility:kafkaBackwardCompatibilityTest
37+
```
38+
39+
### Run with Specific Version
40+
41+
You can test backward compatibility with a specific Data Prepper version:
42+
43+
```bash
44+
./gradlew :e2e-test:kafka-backward-compatibility:kafkaBackwardCompatibilityTest \
45+
-PbackwardCompatVersion=2.9.0
46+
```
47+
48+
## Test Components
49+
50+
### Docker Containers
51+
52+
1. **Kafka** (`kafka-backward-compat-test`)
53+
- Confluent Kafka 3.6.0
54+
- KRaft mode (no ZooKeeper)
55+
- Port: 9092
56+
57+
2. **OpenSearch** (`node-0.example.com`)
58+
- OpenSearch 1.3.14
59+
- Port: 9200
60+
61+
3. **Released Data Prepper** (`data-prepper-writer`)
62+
- Pulled from Docker Hub
63+
- Writes data to Kafka
64+
65+
4. **Current Data Prepper** (`data-prepper-reader`)
66+
- Built from current source
67+
- Reads data from Kafka
68+
69+
### Configuration Files
70+
71+
- **`writer-pipeline.yaml`**: HTTP source → Kafka buffer
72+
- **`reader-pipeline.yaml`**: Kafka source → OpenSearch sink
73+
- **`data-prepper-config.yaml`**: Basic Data Prepper configuration
74+
75+
## What This Tests
76+
77+
✅ **Kafka message format compatibility**
78+
- Protobuf serialization format
79+
- Message envelope structure
80+
- Field naming and types
81+
82+
✅ **Consumer offset management**
83+
- Offset commits and reads
84+
- Consumer group handling
85+
86+
✅ **Data integrity**
87+
- Messages written by old version can be read by new version
88+
- No data loss or corruption
89+
- Field values preserved correctly
90+
91+
## Troubleshooting
92+
93+
### View Docker Logs
94+
95+
```bash
96+
# Kafka logs
97+
docker logs kafka-backward-compat-test
98+
99+
# Released Data Prepper logs
100+
docker logs data-prepper-writer
101+
102+
# Current Data Prepper logs
103+
docker logs data-prepper-reader
104+
105+
# OpenSearch logs
106+
docker logs node-0.example.com
107+
```
108+
109+
### Manual Container Control
110+
111+
```bash
112+
# Start Kafka manually
113+
./gradlew :e2e-test:kafka-backward-compatibility:startKafkaDockerContainer
114+
115+
# Start released Data Prepper manually
116+
./gradlew :e2e-test:kafka-backward-compatibility:startReleasedDataPrepperContainer
117+
118+
# Start current Data Prepper manually
119+
./gradlew :e2e-test:kafka-backward-compatibility:startCurrentDataPrepperContainer
120+
121+
# Stop all
122+
./gradlew :e2e-test:kafka-backward-compatibility:stopKafkaDockerContainer
123+
./gradlew :e2e-test:kafka-backward-compatibility:stopReleasedDataPrepperContainer
124+
./gradlew :e2e-test:kafka-backward-compatibility:stopCurrentDataPrepperContainer
125+
```
126+
127+
### Check Kafka Topics
128+
129+
```bash
130+
docker exec kafka-backward-compat-test kafka-topics --list --bootstrap-server localhost:9092
131+
```
132+
133+
### Query OpenSearch Directly
134+
135+
```bash
136+
curl -k -u admin:admin "https://localhost:9200/backward-compat-test-index/_search?pretty"
137+
```
138+
139+
## Expected Behavior
140+
141+
When the test passes:
142+
1. Released Data Prepper successfully writes 2 messages to Kafka
143+
2. Current Data Prepper successfully reads both messages from Kafka
144+
3. Both messages appear in OpenSearch with correct content
145+
4. Test completes with ✅ SUCCESS message
146+
147+
## CI/CD Integration
148+
149+
This test should be run:
150+
- ✅ Before releasing new versions
151+
- ✅ When making changes to Kafka buffer serialization
152+
- ✅ When upgrading Kafka client libraries
153+
- ✅ As part of the release candidate verification process
154+
155+
## Notes
156+
157+
- The test uses **unencrypted** Kafka messages for simplicity
158+
- If you need to test encrypted backward compatibility, modify the pipeline configs to enable encryption
159+
- The released version can be configured via `-PbackwardCompatVersion` property
160+
- Default released version is `2.10.0` (configured in `build.gradle`)

0 commit comments

Comments
 (0)