Skip to content

[WIP] Fix race condition in PayloadAssembler OnTimedEvent method#543

Open
WillButAgain wants to merge 1 commit into
Project-MONAI:mainfrom
WillButAgain:feature/fix-payload-assembler-race-condition
Open

[WIP] Fix race condition in PayloadAssembler OnTimedEvent method#543
WillButAgain wants to merge 1 commit into
Project-MONAI:mainfrom
WillButAgain:feature/fix-payload-assembler-race-condition

Conversation

@WillButAgain

Copy link
Copy Markdown

Description

Fixes race condition in PayloadAssembler causing message processing failures and dead-lettered RabbitMQ messages.

This PR addresses a critical race condition in the PayloadAssembler.OnTimedEvent method that was causing InvalidOperationException when concurrent threads modified the _payloads dictionary during enumeration. This resulted in:

  • RabbitMQ messages failing to be acknowledged
  • Messages accumulating in dead-letter queues
  • Intermittent export workflow failures

Root Cause:
The OnTimedEvent method iterates over _payloads.Keys while other threads (processing incoming DICOM files via Queue()) can simultaneously add/remove entries. Even though ConcurrentDictionary is thread-safe for individual operations, iterating over its .Keys property while the collection is being modified throws InvalidOperationException.

Solution:
Implemented a snapshot pattern that captures the dictionary keys before iteration, preventing the collection-modified-during-enumeration exception:

  • Changed foreach (var key in _payloads.Keys) to use _payloads.Keys.ToList() snapshot
  • Added TryGetValue() guard to safely handle keys that may have been removed between snapshot and access
  • No locks required—leverages existing ConcurrentDictionary thread safety

Testing:
Added two comprehensive concurrency tests to verify the fix:

  • GivenConcurrentAccess_WhenProcessingPayloads_ExpectNoRaceCondition - Tests 100 concurrent queue operations
  • GivenHighConcurrentLoad_WhenProcessingPayloads_ExpectNoExceptions - Stress test with 500 concurrent operations

Status

Ready

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • All tests passed locally.
  • Documentation comments included/updated.
  • User guide updated.
  • I have updated the changelog

Impact

  • Severity: High
  • Affected Component: PayloadAssembler.OnTimedEvent
  • Performance Impact: Minimal (small memory allocation for key snapshot)
  • Risk: Low - Simple, well-established pattern with no architectural changes

Related Issues

Related to production incidents with:

  • 7,928+ messages in dead-letter queues
  • Unacknowledged RabbitMQ messages
  • InvalidOperationException: Collection was modified; enumeration operation may not execute

See PAYLOAD_ASSEMBLER_RACE_CONDITION_REPORT.md for detailed analysis.

Signed-off-by: will tepe <will.tepe@cchmc.org>
@bluna301 bluna301 changed the title Fix race condition in PayloadAssembler OnTimedEvent method [WIP] Fix race condition in PayloadAssembler OnTimedEvent method Mar 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant