[WIP] Fix race condition in PayloadAssembler OnTimedEvent method#543
Open
WillButAgain wants to merge 1 commit into
Open
[WIP] Fix race condition in PayloadAssembler OnTimedEvent method#543WillButAgain wants to merge 1 commit into
WillButAgain wants to merge 1 commit into
Conversation
Signed-off-by: will tepe <will.tepe@cchmc.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes race condition in PayloadAssembler causing message processing failures and dead-lettered RabbitMQ messages.
This PR addresses a critical race condition in the
PayloadAssembler.OnTimedEventmethod that was causingInvalidOperationExceptionwhen concurrent threads modified the_payloadsdictionary during enumeration. This resulted in:Root Cause:
The
OnTimedEventmethod iterates over_payloads.Keyswhile other threads (processing incoming DICOM files viaQueue()) can simultaneously add/remove entries. Even thoughConcurrentDictionaryis thread-safe for individual operations, iterating over its.Keysproperty while the collection is being modified throwsInvalidOperationException.Solution:
Implemented a snapshot pattern that captures the dictionary keys before iteration, preventing the collection-modified-during-enumeration exception:
foreach (var key in _payloads.Keys)to use_payloads.Keys.ToList()snapshotTryGetValue()guard to safely handle keys that may have been removed between snapshot and accessConcurrentDictionarythread safetyTesting:
Added two comprehensive concurrency tests to verify the fix:
GivenConcurrentAccess_WhenProcessingPayloads_ExpectNoRaceCondition- Tests 100 concurrent queue operationsGivenHighConcurrentLoad_WhenProcessingPayloads_ExpectNoExceptions- Stress test with 500 concurrent operationsStatus
Ready
Types of changes
Impact
PayloadAssembler.OnTimedEventRelated Issues
Related to production incidents with:
InvalidOperationException: Collection was modified; enumeration operation may not executeSee
PAYLOAD_ASSEMBLER_RACE_CONDITION_REPORT.mdfor detailed analysis.