Skip to content

Fix batch listener backoff state reset in multi-partition containers - #4518

Open
29shivam wants to merge 1 commit into
spring-projects:mainfrom
29shivam:fix/batch-listener-backoff-partition-scoped-clear
Open

Fix batch listener backoff state reset in multi-partition containers#4518
29shivam wants to merge 1 commit into
spring-projects:mainfrom
29shivam:fix/batch-listener-backoff-partition-scoped-clear

Conversation

@29shivam

Copy link
Copy Markdown

Problem

In a non-concurrent KafkaMessageListenerContainer with multiple partitions, when a successful batch follows a prior failure, doInvokeBatchListener calls commonErrorHandler.clearThreadState(). This wipes the entire FailedRecordTracker map for the current thread, resetting the failure count for all partitions — including partitions that are still inside their backoff/retry cycle.

The result: a broken record that should be retried N times (or sent to the DLQ after N attempts) restarts its counter every time any other partition successfully processes a batch, so it never exhausts its retries and never reaches the DLQ.

Reproduction scenario (from #4371):

  1. Two-partition topic, non-concurrent container, backoff+DLQ configured
  2. Partition A has a broken record, partition B has healthy records
  3. Partition A fails → batchFailed = true, FailedRecordTracker records attempt 1 for A
  4. Error handler seeks A back, commits B
  5. Next poll only returns B's records → success → clearThreadState() → A's failure count wiped
  6. A is un-paused; failure count is 1 again → cycle repeats, DLQ never reached

Fix

Introduce clearThreadStateFor(Collection<TopicPartition> partitions) throughout the error-handler stack:

  • FailedRecordTracker — removes only the specified partitions from the per-thread failure map
  • FailedRecordProcessor — public delegate to tracker
  • CommonErrorHandler — new default (no-op) so existing custom handlers need no changes
  • CommonMixedErrorHandler / CommonDelegatingErrorHandler — delegate to sub-handlers
  • KafkaMessageListenerContainer.doInvokeBatchListener — calls clearThreadStateFor(records.partitions()) instead of clearThreadState()

This scopes the reset to only the partitions that were actually part of the successful batch, leaving backoff state intact for partitions still retrying.

The existing clearThreadState() call (on thread termination) is unchanged.

Checklist

  • Tests added/updated
  • @since tag added to new API (CommonErrorHandler.clearThreadStateFor)

Closes #4371

In a non-concurrent container with multiple partitions, a successful
batch invocation after a prior failure called clearThreadState() on the
CommonErrorHandler, wiping FailedRecordTracker state for ALL partitions.
This caused partitions that were still inside their backoff/retry cycle
to lose their failure count and restart from attempt 1, preventing broken
records from ever reaching the DLQ.

Root cause: clearThreadState() removes the entire Thread->Map entry from
FailedRecordTracker, regardless of which partitions were actually part of
the successful batch.

Fix: introduce clearThreadStateFor(Collection<TopicPartition>) so the
container can scope the reset to only the partitions present in the
current successful batch. doInvokeBatchListener() now calls this narrower
method, leaving backoff state intact for partitions that are still retrying.

The full clearThreadState() path (called on thread termination) is
unchanged. Composite handlers (CommonMixedErrorHandler,
CommonDelegatingErrorHandler) delegate the new method to their children.

Closes spring-projectsgh-4371

Signed-off-by: 29shivam <shivamsingh6126@gmail.com>
@29shivam
29shivam force-pushed the fix/batch-listener-backoff-partition-scoped-clear branch from 174b0df to 1470bba Compare June 26, 2026 04:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Backoff breaks recovery on batching listener container in multi partition environment

1 participant