Skip to content

Commit 66ae4fa

Browse files
sarkarshuvojitsarkarshuvojit
committed
added draft whitepaper
1 parent 8d29acd commit 66ae4fa

1 file changed

Lines changed: 230 additions & 0 deletions

File tree

‎whitepaper.md‎

Lines changed: 230 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,230 @@
1+
# Kafka Synchronous Proxy: Technical Whitepaper
2+
3+
## Abstract
4+
5+
The Kafka Synchronous Proxy is a web service that bridges the gap between traditional synchronous REST APIs and asynchronous Kafka message streaming. It enables request-response patterns over Kafka by implementing a correlation-based message routing system with timeout handling. This proxy allows systems to interact with Kafka-based microservices using familiar HTTP semantics while maintaining the benefits of message-driven architecture.
6+
7+
## 1. Introduction
8+
9+
Modern distributed systems often employ event-driven architectures using Apache Kafka for scalability and decoupling. However, many client applications and legacy systems still require synchronous request-response patterns. The Kafka Synchronous Proxy addresses this architectural mismatch by providing a REST API that translates synchronous HTTP requests into asynchronous Kafka message exchanges.
10+
11+
## 2. System Architecture
12+
13+
### 2.1 Core Components
14+
15+
The system consists of four main architectural layers:
16+
17+
1. **HTTP Server Layer** (`internal/server/server.go`): Built on Fiber v2.43.0, handles HTTP requests and provides graceful shutdown capabilities
18+
2. **Controller Layer** (`internal/controller/controller.go`): Manages request parsing, service orchestration, and response formatting
19+
3. **Service Layer** (`pkg/service/blocking_service.go`): Implements the core business logic for synchronous message exchange
20+
4. **Messaging Layer** (`pkg/messaging/kafka/kafka.go`): Provides Kafka-specific implementation of the messaging interface
21+
22+
### 2.2 Request Flow Architecture
23+
24+
```
25+
HTTP Client → REST Endpoint → Controller → Blocking Service → Kafka Producer
26+
↓
27+
HTTP Response ← JSON Formatter ← Service Layer ← Kafka Consumer ← Response Topic
28+
```
29+
30+
## 3. Technical Implementation
31+
32+
### 3.1 Correlation-Based Message Routing
33+
34+
The system implements a correlation ID mechanism to match requests with responses:
35+
36+
- **Correlation ID Generation**: Uses UUID v4 for unique message identification
37+
- **Message Key Mapping**: Kafka message keys serve as correlation identifiers
38+
- **Response Matching**: Consumer filters messages by correlation ID to find corresponding responses
39+
40+
### 3.2 Request Processing Flow
41+
42+
1. **Request Ingestion**: HTTP POST requests are received at `/v1/` endpoint
43+
2. **Payload Parsing**: Request body is parsed into `BlockingRequestDto` structure containing:
44+
- `requestTopic`: Target Kafka topic for the request
45+
- `responseTopic`: Topic to monitor for responses
46+
- `payload`: Message payload (JSON)
47+
- `headers`: Custom headers to forward
48+
- `brokers`: Kafka broker addresses
49+
50+
3. **Message Production**:
51+
- Generates unique correlation ID
52+
- Serializes payload and headers to JSON
53+
- Produces message to request topic with correlation ID as key
54+
55+
4. **Response Consumption**:
56+
- Starts consumer on response topic
57+
- Filters messages by correlation ID
58+
- Implements configurable timeout (default: 5 seconds)
59+
60+
### 3.3 Timeout Management
61+
62+
The system implements a robust timeout mechanism:
63+
64+
- **Configuration**: Timeout is configurable via `KSP_TIMEOUT` environment variable
65+
- **Default Value**: 5 seconds if not specified
66+
- **Context Cancellation**: Uses Go context for graceful timeout handling
67+
- **HTTP Status Mapping**: Returns HTTP 408 (Request Timeout) on timeout
68+
69+
### 3.4 Error Handling
70+
71+
Comprehensive error handling covers:
72+
73+
- **Kafka Connection Errors**: Producer/consumer initialization failures
74+
- **Message Parsing Errors**: Invalid JSON in request/response
75+
- **Timeout Errors**: No response received within timeout period
76+
- **Context Cancellation**: Graceful shutdown scenarios
77+
78+
## 4. Message Format Specification
79+
80+
### 4.1 Request Message Format
81+
82+
**HTTP Request:**
83+
```json
84+
{
85+
"requestTopic": "com.example.service.request",
86+
"responseTopic": "com.example.service.response",
87+
"payload": {
88+
"data": "example payload"
89+
},
90+
"headers": {
91+
"authorization": "Bearer token"
92+
},
93+
"brokers": ["localhost:9092"]
94+
}
95+
```
96+
97+
**Kafka Message:**
98+
- **Key**: UUID v4 correlation ID
99+
- **Value**: JSON serialized payload
100+
- **Headers**: Key-value pairs from request headers
101+
102+
### 4.2 Response Message Format
103+
104+
**Kafka Response:**
105+
- **Key**: Same correlation ID from request
106+
- **Value**: Service response payload
107+
- **Headers**: Response headers from processing service
108+
109+
**HTTP Response:**
110+
```json
111+
{
112+
"response": {
113+
"result": "processed data"
114+
},
115+
"headers": {
116+
"authorization": "Bearer token",
117+
"custom-header": "value"
118+
}
119+
}
120+
```
121+
122+
## 5. Performance Characteristics
123+
124+
### 5.1 Scalability Features
125+
126+
- **Stateless Design**: Each request is independent, enabling horizontal scaling
127+
- **Connection Pooling**: Kafka producers and consumers are created per request
128+
- **Graceful Shutdown**: 30-second graceful shutdown timeout for request completion
129+
- **Memory Efficiency**: Minimal memory footprint with automatic resource cleanup
130+
131+
### 5.2 Throughput Considerations
132+
133+
- **Synchronous Blocking**: Each request blocks until response or timeout
134+
- **Concurrency**: Multiple requests handled concurrently by Fiber framework
135+
- **Kafka Partition Strategy**: Currently uses partition 0 for consumption (single partition)
136+
137+
### 5.3 Latency Factors
138+
139+
- **Network Latency**: Kafka broker communication overhead
140+
- **Processing Time**: Downstream service processing duration
141+
- **Timeout Boundary**: Maximum 5-second response time (configurable)
142+
143+
## 6. Deployment and Configuration
144+
145+
### 6.1 Deployment Options
146+
147+
The service supports multiple deployment modes:
148+
149+
1. **Standalone Binary**: Direct execution with Go runtime
150+
2. **Docker Container**: Containerized deployment with port 8420 exposure
151+
3. **Docker Compose**: Multi-service orchestration support
152+
153+
### 6.2 Configuration Parameters
154+
155+
- **Port**: Fixed at 8420 (HTTP server)
156+
- **Timeout**: Configurable via `KSP_TIMEOUT` environment variable
157+
- **Kafka Brokers**: Specified per request (dynamic configuration)
158+
- **Log Level**: Standard Go log output
159+
160+
## 7. Security Considerations
161+
162+
### 7.1 Authentication and Authorization
163+
164+
- **Header Passthrough**: Custom authentication headers forwarded to Kafka
165+
- **Broker Security**: Supports standard Kafka security protocols
166+
- **No Built-in Authentication**: Relies on external authentication mechanisms
167+
168+
### 7.2 Data Protection
169+
170+
- **Payload Encryption**: Supports encrypted payloads (transparent handling)
171+
- **Header Sanitization**: Headers processed without modification
172+
- **Memory Safety**: Automatic cleanup of sensitive data
173+
174+
## 8. Use Cases and Applications
175+
176+
### 8.1 Legacy System Integration
177+
178+
- **API Gateway Pattern**: Expose Kafka-based services as REST APIs
179+
- **Microservice Migration**: Gradual migration from synchronous to asynchronous patterns
180+
- **Protocol Translation**: Bridge HTTP and Kafka ecosystems
181+
182+
### 8.2 Event-Driven Architecture
183+
184+
- **Command-Query Separation**: Separate command and query operations
185+
- **Service Orchestration**: Coordinate multiple service interactions
186+
- **Request-Response Patterns**: Implement synchronous behavior in event-driven systems
187+
188+
## 9. Limitations and Considerations
189+
190+
### 9.1 Current Limitations
191+
192+
- **Single Partition Consumption**: Only consumes from partition 0
193+
- **No Message Persistence**: No retry mechanism for failed messages
194+
- **Fixed Timeout**: Global timeout applies to all requests
195+
- **No Load Balancing**: No built-in consumer group management
196+
197+
### 9.2 Operational Considerations
198+
199+
- **Resource Usage**: Each request creates new Kafka connections
200+
- **Error Recovery**: Manual intervention required for broker failures
201+
- **Monitoring**: Limited built-in metrics and health checks
202+
203+
## 10. Future Enhancements
204+
205+
### 10.1 Proposed Improvements
206+
207+
- **Multi-Partition Support**: Distribute load across multiple partitions
208+
- **Connection Pooling**: Reuse Kafka connections for better performance
209+
- **Health Checks**: Implement health and readiness endpoints
210+
- **Metrics Integration**: Add Prometheus/OpenTelemetry support
211+
212+
### 10.2 Architectural Evolution
213+
214+
- **Message Persistence**: Add message replay capabilities
215+
- **Circuit Breaker**: Implement fault tolerance patterns
216+
- **Dynamic Configuration**: Runtime configuration updates
217+
- **Consumer Groups**: Support for scalable consumer patterns
218+
219+
## 11. Conclusion
220+
221+
The Kafka Synchronous Proxy provides a practical solution for integrating synchronous client applications with asynchronous Kafka-based services. Its correlation-based message routing, timeout handling, and stateless design make it suitable for production environments where request-response patterns are required over Kafka infrastructure.
222+
223+
The system's modular architecture and clear separation of concerns enable easy extension and customization for specific use cases. While current limitations exist, the foundation provides a solid base for future enhancements and enterprise-grade features.
224+
225+
## References
226+
227+
- **Apache Kafka**: https://kafka.apache.org/
228+
- **Fiber Web Framework**: https://gofiber.io/
229+
- **Shopify Sarama**: https://github.com/Shopify/sarama
230+
- **Go Context Package**: https://pkg.go.dev/context

0 commit comments

Comments
 (0)