We are currently running a Bone Age MAP on MONAI Deploy Express v0.6.0 in shadow mode. We are receiving clinical studies via Compass (COMPASS-SCP) and launching the MAP. We are exporting the produced DICOM SR to the MDE ORTHANC instance and back to Compass, and we are also exporting the source DICOM study to MDE ORTHANC. Workflow definition is attached.
The problem we are seeing is that a slew of RabbitMQ errors are being produced during workflow execution. They seem to come in two main types.
Timeout Waiting for Delivery - looks like RabbitMQ is expecting the export service to acknowledge a message, but it never does, causing timeout after 30 minutes:
Consumer 1 on channel 1 has timed out waiting for delivery acknowledgement. Timeout used: 1800000 ms.
basic.ack Precondition Failed with Unknown Delivery Tag - I believe this occurs when a consumer acknowledges a message that RabbitMQ no longer recognizes:
operation basic.ack caused a channel exception precondition_failed: unknown delivery tag X
Possible error causes.
These timeouts are causing exports to either be delayed (for around 30 minutes until the timeout occurs) or cancelled entirely (export tasks timeout, as can be seen in the Workflow Manager logs).
Any insights into a possible cause or solution for this problem would be greatly appreciated. As some additional context, in all cases (irrespective of export failure or not), MAPs are launched successfully, and the input DICOM study and output DICOM SR are populated in the minio and mdtm folders. The RabbitMQ issues seem to only block the export of DICOM data.
We are currently running a Bone Age MAP on MONAI Deploy Express v0.6.0 in shadow mode. We are receiving clinical studies via Compass (
COMPASS-SCP) and launching the MAP. We are exporting the produced DICOM SR to the MDE ORTHANC instance and back to Compass, and we are also exporting the source DICOM study to MDE ORTHANC. Workflow definition is attached.The problem we are seeing is that a slew of RabbitMQ errors are being produced during workflow execution. They seem to come in two main types.
Timeout Waiting for Delivery - looks like RabbitMQ is expecting the export service to acknowledge a message, but it never does, causing timeout after 30 minutes:
Consumer 1 on channel 1 has timed out waiting for delivery acknowledgement. Timeout used: 1800000 ms.basic.ackPrecondition Failed with Unknown Delivery Tag - I believe this occurs when a consumer acknowledges a message that RabbitMQ no longer recognizes:Possible error causes.
These timeouts are causing exports to either be delayed (for around 30 minutes until the timeout occurs) or cancelled entirely (export tasks timeout, as can be seen in the Workflow Manager logs).
Any insights into a possible cause or solution for this problem would be greatly appreciated. As some additional context, in all cases (irrespective of export failure or not), MAPs are launched successfully, and the input DICOM study and output DICOM SR are populated in the
minioandmdtmfolders. The RabbitMQ issues seem to only block the export of DICOM data.