Skip to content

Commit eed7755

Browse files
graytaylor0graytaylor0
authored
Add kafka buffer backward compatibility test (#6406)
Signed-off-by: Taylor Gray <tylgry@amazon.com>
1 parent c25f82f commit eed7755

9 files changed

Lines changed: 714 additions & 0 deletions

File tree

‎.github/workflows/data-prepper-kafka-backward-compatibility-e2e-tests.yml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# This workflow will build a Java project with Gradle
2+
# For more information see: https://help.github.com/actions/language-and-framework-guides/building-and-testing-java-with-gradle
3+
4+
name: Data Prepper Kafka Backward Compatibility End-to-end test with Gradle
5+
6+
on:
7+
push:
8+
branches: [ main ]
9+
pull_request:
10+
workflow_dispatch:
11+
12+
jobs:
13+
build:
14+
strategy:
15+
matrix:
16+
java: [11, 17, 21, docker]
17+
fail-fast: false
18+
19+
runs-on: ubuntu-latest
20+
21+
steps:
22+
- name: Set up JDK 11
23+
uses: actions/setup-java@v1
24+
with:
25+
java-version: 11
26+
- name: Checkout Data Prepper
27+
uses: actions/checkout@v2
28+
- name: Run Kafka backward compatibility end-to-end tests with Gradle
29+
run: ./gradlew -PendToEndJavaVersion=${{ matrix.java }} :e2e-test:kafka-buffer-backward-compatibility:kafkaBufferBackwardCompatibilityTest

‎e2e-test/kafka-buffer-backward-compatibility/README.md‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
# Kafka Backward Compatibility End-to-End Test
2+
3+
## Overview
4+
5+
This test verifies that the current build of Data Prepper can successfully read and process messages written to Kafka by previous released versions.
6+
7+
## Test Scenario
8+
9+
### Phase 1: Write with Released Version
10+
1. Start Kafka container
11+
2. Start **released** Data Prepper from Docker Hub (e.g., `opensearchproject/data-prepper:2`)
12+
3. Send 2 test records via HTTP endpoint
13+
4. Records are written to Kafka buffer using the released version's format
14+
5. Stop the released Data Prepper container
15+
16+
### Phase 2: Read with Current Build
17+
1. Start **current** Data Prepper (built from source)
18+
2. Data Prepper reads the 2 messages from Kafka
19+
3. Messages are processed and written to OpenSearch
20+
21+
### Phase 3: Verification
22+
1. Query OpenSearch to verify both records were correctly processed
23+
2. Validate message content matches expected values
24+
25+
## Running the Test
26+
27+
### Prerequisites
28+
- Docker running locally
29+
- Java 11 or later
30+
- Gradle
31+
32+
### Run the Test
33+
34+
```bash
35+
# From the repository root
36+
./gradlew :e2e-test:kafka-buffer-backward-compatibility:kafkaBufferBackwardCompatibilityTest
37+
```
38+
39+
### Run with Specific Version
40+
41+
You can test backward compatibility with a specific Data Prepper version:
42+
43+
```bash
44+
./gradlew :e2e-test:kafka-buffer-backward-compatibility:kafkaBufferBackwardCompatibilityTest \
45+
-PreleasedVersion=2.9.0
46+
```
47+
48+
## Test Components
49+
50+
### Docker Containers
51+
52+
1. **Kafka** (`kafka-buffer-backward-compatibility-test`)
53+
- Confluent Kafka 3.6.0
54+
- KRaft mode (no ZooKeeper)
55+
- Port: 9092
56+
57+
2. **OpenSearch** (`node-0.example.com`)
58+
- OpenSearch 1.3.14
59+
- Port: 9200
60+
61+
3. **Released Data Prepper** (`data-prepper-writer`)
62+
- Pulled from Docker Hub
63+
- Writes data to Kafka
64+
65+
4. **Current Data Prepper** (`data-prepper-reader`)
66+
- Built from current source
67+
- Reads data from Kafka
68+
69+
### Configuration Files
70+
71+
- **`writer-pipeline.yaml`**: HTTP source → Kafka buffer
72+
- **`reader-pipeline.yaml`**: Kafka source → OpenSearch sink
73+
- **`data-prepper-config.yaml`**: Basic Data Prepper configuration
74+
75+
## What This Tests
76+
77+
**Kafka message format compatibility**
78+
- Protobuf serialization format
79+
- Message envelope structure
80+
- Field naming and types
81+
82+
**Consumer offset management**
83+
- Offset commits and reads
84+
- Consumer group handling
85+
86+
**Data integrity**
87+
- Messages written by old version can be read by new version
88+
- No data loss or corruption
89+
- Field values preserved correctly
90+
91+
## Troubleshooting
92+
93+
### View Docker Logs
94+
95+
```bash
96+
# Kafka logs
97+
docker logs kafka-buffer-backward-compatibility-test
98+
99+
# Released Data Prepper logs
100+
docker logs data-prepper-writer
101+
102+
# Current Data Prepper logs
103+
docker logs data-prepper-reader
104+
105+
# OpenSearch logs
106+
docker logs node-0.example.com
107+
```
108+
109+
### Manual Container Control
110+
111+
```bash
112+
# Start Kafka manually
113+
./gradlew :e2e-test:kafka-buffer-backward-compatibility:startKafkaDockerContainer
114+
115+
# Start released Data Prepper manually
116+
./gradlew :e2e-test:kafka-buffer-backward-compatibility:startReleasedDataPrepperContainer
117+
118+
# Start current Data Prepper manually
119+
./gradlew :e2e-test:kafka-buffer-backward-compatibility:startCurrentDataPrepperContainer
120+
121+
# Stop all
122+
./gradlew :e2e-test:kafka-buffer-backward-compatibility:stopKafkaDockerContainer
123+
./gradlew :e2e-test:kafka-buffer-backward-compatibility:stopReleasedDataPrepperContainer
124+
./gradlew :e2e-test:kafka-buffer-backward-compatibility:stopCurrentDataPrepperContainer
125+
```
126+
127+
### Check Kafka Topics
128+
129+
```bash
130+
docker exec kafka-buffer-backward-compatibility-test kafka-topics --list --bootstrap-server localhost:9092
131+
```
132+
133+
### Query OpenSearch Directly
134+
135+
```bash
136+
curl -k -u admin:admin "https://localhost:9200/kafka-backward-compatibility-test-index/_search?pretty"
137+
```
138+
139+
## Expected Behavior
140+
141+
When the test passes:
142+
1. Released Data Prepper successfully writes 2 messages to Kafka
143+
2. Current Data Prepper successfully reads both messages from Kafka
144+
3. Both messages appear in OpenSearch with correct content
145+
4. Test completes with SUCCESS message
146+
147+
## Notes
148+
149+
- The test uses **unencrypted** Kafka messages for simplicity
150+
- If you need to test encrypted backward compatibility, modify the pipeline configs to enable encryption
151+
- The released version can be configured via `-PreleasedVersion` property
152+
- Default released version is `2` (configured in `build.gradle`)

0 commit comments

Comments
 (0)