DATAGO-137655: time-bound producer close to fix unbind hang on reconnect#496
Open
sunil-solace wants to merge 1 commit into
Open
DATAGO-137655: time-bound producer close to fix unbind hang on reconnect#496sunil-solace wants to merge 1 commit into
sunil-solace wants to merge 1 commit into
Conversation
When the JCSMP session is reconnecting (e.g. a broker / Message VPN outage with unlimited reconnect retries), XMLMessageProducer.close() and TransactedSession.close() block indefinitely in JCSMPBasicSession.waitUntilSessionReconnectDone. These closes run on the binding-lifecycle thread during Binding.unbind() -> JCSMPOutboundMessageHandler.stop(), so the unbind never returns and the producer binding lifecycle stays wedged until the process is restarted. Run the producer-side closes (the per-handler producer/transacted session and the shared producer) on a daemon-thread executor with an upper time bound. If a close does not complete within the timeout, the worker is interrupted and the stop proceeds, so a producer unbind can no longer hang while the session is reconnecting. The bound is configurable via spring.cloud.stream.solace.binder.producer-close-timeout-in-millis (default 10000 ms). Adds JCSMPCloseFlowReconnectHangIT, which reproduces the hang against a broker by disabling the Message VPN to force a perpetual reconnect, then asserts the unbind completes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR prevents producer-side unbind/stop from hanging indefinitely while a JCSMP session is reconnecting by running producer/transacted-session close logic on a daemon-thread executor with a configurable timeout, and adds an integration test that reproduces the reconnect-hang scenario.
Changes:
- Add time-bounded, interruptible close logic for producer-side resources via
JCSMPSessionProducerManager.closeSafely(...). - Plumb a new binder configuration property
spring.cloud.stream.solace.binder.producer-close-timeout-in-millis(default 10000ms) into binder construction. - Add integration test
JCSMPCloseFlowReconnectHangITthat forces perpetual reconnect (by disabling the Message VPN) and asserts unbind completes.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| solace-spring-cloud-stream-binder/solace-spring-cloud-stream-binder/src/test/java/com/solace/spring/cloud/stream/binder/JCSMPCloseFlowReconnectHangIT.java | Adds broker-backed IT reproducing unbind hang during reconnect and asserting bounded completion. |
| solace-spring-cloud-stream-binder/solace-spring-cloud-stream-binder/src/main/java/com/solace/spring/cloud/stream/binder/SolaceMessageChannelBinder.java | Constructs JCSMPSessionProducerManager with a close timeout and ensures executor shutdown on binder destroy. |
| solace-spring-cloud-stream-binder/solace-spring-cloud-stream-binder/src/main/java/com/solace/spring/cloud/stream/binder/config/SolaceMessageChannelBinderConfiguration.java | Wires the new producer-close timeout property into binder instantiation. |
| solace-spring-cloud-stream-binder/solace-spring-cloud-stream-binder-core/src/main/java/com/solace/spring/cloud/stream/binder/util/SharedResourceManager.java | Refactors shared resource close to occur outside the shared lock to avoid blocking other callers. |
| solace-spring-cloud-stream-binder/solace-spring-cloud-stream-binder-core/src/main/java/com/solace/spring/cloud/stream/binder/util/JCSMPSessionProducerManager.java | Introduces bounded close execution with timeout + logging, plus executor lifecycle management. |
| solace-spring-cloud-stream-binder/solace-spring-cloud-stream-binder-core/src/main/java/com/solace/spring/cloud/stream/binder/properties/SolaceBinderConfigurationProperties.java | Adds producerCloseTimeoutInMillis configuration property and default. |
| solace-spring-cloud-stream-binder/solace-spring-cloud-stream-binder-core/src/main/java/com/solace/spring/cloud/stream/binder/outbound/JCSMPOutboundMessageHandler.java | Uses bounded close for per-handler producer + transacted session and hardens producer recreation checks. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+358
to
+360
| // Drop references even if the close timed out; the handler no longer owns them. | ||
| producer = null; | ||
| transactedSession = null; |
Comment on lines
+32
to
34
| private final long closeTimeoutInMillis; | ||
| private final ExecutorService closeExecutor = Executors.newCachedThreadPool(newCloseThreadFactory()); | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When the JCSMP session is reconnecting (e.g. a broker / Message VPN outage with unlimited reconnect retries), XMLMessageProducer.close() and TransactedSession.close() block indefinitely in
JCSMPBasicSession.waitUntilSessionReconnectDone. These closes run on the binding-lifecycle thread during Binding.unbind() -> JCSMPOutboundMessageHandler.stop(), so the unbind never returns and the producer binding lifecycle stays wedged until the process is restarted.
Run the producer-side closes (the per-handler producer/transacted session and the shared producer) on a daemon-thread executor with an upper time bound. If a close does not complete within the timeout, the worker is interrupted and the stop proceeds, so a producer unbind can no longer hang while the session is reconnecting.
The bound is configurable via
spring.cloud.stream.solace.binder.producer-close-timeout-in-millis (default 10000 ms).
Adds JCSMPCloseFlowReconnectHangIT, which reproduces the hang against a broker by disabling the Message VPN to force a perpetual reconnect, then asserts the unbind completes.