You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: blog/batch-vs-stream-processing/index.md
+17-11Lines changed: 17 additions & 11 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -123,24 +123,30 @@ In batch, this doesn't exist as a problem. In streaming, it's a constant enginee
123
123
124
124
125
125
## Hidden Cost #3 - Exactly-Once Is Harder Than It Sounds
126
+
Handling failures in batch pipelines is usually predictable.
127
+
If a batch job fails, you typically resolve the issue and rerun the pipeline from the beginning. Since the processing happens on bounded data, recovery is relatively straightforward.
126
128
127
-
In a batch pipeline, if a job fails, you rerun it. Simple.
129
+
Streaming systems work very differently.
128
130
129
-
In a streaming pipeline, failure recovery is a distributed systems problem.
131
+
In platforms like Kafka and Flink, data is continuously flowing through the system. If a streaming job crashes midway through processing, recovery becomes much more complex than simply restarting the job.
130
132
131
-
Say your Flink job crashes halfway through processing a Kafka topic partition. When it restarts, does it reprocess events it already processed? Does it skip events it hadn't gotten to yet? Does it process some events twice?
133
+
For example, after recovery:
134
+
- Should previously processed events be replayed?
135
+
- Could some records get skipped unintentionally?
136
+
- Is there a possibility that certain events are processed more than once?
132
137
133
-
This is the **exactly-once semantics** problem ensuring each event is processed exactly once, not zero times, not twice.
138
+
This challenge is commonly addressed through **exactly-once processing guarantees**, where the goal is to ensure that every event affects the system exactly one time even during failures and restarts.
134
139
135
-
Getting this right requires:
136
-
- Kafka consumer offset management
137
-
- Flink checkpoint configuration
138
-
- Idempotent writes to your output store
139
-
- Careful handling of state during recovery
140
+
Achieving reliable exactly-once behavior usually depends on several components working together correctly:
140
141
141
-
We had a production incident in our first month where a Flink job restart caused approximately 2,000 order events to be processed twice. Customers received duplicate status update notifications. Support tickets spiked. It took us two days to identify, patch, and backfill the correct state.
142
+
- Proper Kafka offset management
143
+
- Reliable Flink checkpointing and state recovery
144
+
- Idempotent writes to downstream systems
145
+
- Consistent state synchronization during failover scenarios
142
146
143
-
A batch job that fails leaves your data unchanged. You fix it, rerun, done. A streaming job that fails mid-stream can leave your data in a partially-updated state that's genuinely difficult to reason about.
147
+
In practice, recovery bugs in streaming systems can have real operational impact. A single restart issue can lead to duplicate event processing, inconsistent downstream data, repeated customer notifications, or inaccurate analytics until the state is corrected.
148
+
149
+
Unlike batch systems, where failures often leave datasets untouched until rerun, streaming failures can leave systems in partially updated states that are significantly harder to debug and recover from.
144
150
145
151
146
152
## Hidden Cost #4 - Testing Is a Different Discipline
0 commit comments