The sample CRT configs in example-configs.data.xml fail to successfully complete the stop-trigger-sources transition

[bieryAtFnal]

These tests were run with fddaq-v5.3.2-rc4.

When I ran each of the following commands:

drunc-unified-shell ssh-standalone config/daqsystemtest/example-configs.data.xml local-crt-bern-1x1-config ${USER}-local-test boot wait 5 conf wait 3 start --run-number 111 enable-triggers wait 10 disable-triggers drain-dataflow stop-trigger-sources stop scrap terminate

or

drunc-unified-shell ssh-standalone config/daqsystemtest/example-configs.data.xml local-crt-grenoble-1x1-config ${USER}-local-test boot wait 5 conf wait 3 start --run-number 121 enable-triggers wait 10 disable-triggers drain-dataflow stop-trigger-sources stop scrap terminate

the system seemed to time out while trying to complete the stop-trigger-sources transition.

I noticed the following error messages in the logs (although these may have been symptoms of the problem rather than causes):

$ egrep 'ERROR|WARN' log_*.txt
log_biery_biery-local-test_crt-grenoble-root-controller.txt:[2025/06/30 12:12:44] ERROR      controller.py:510              controller:                                   Something wrong happened while sending the command to crt-grenoble-controller: Error raised: Socket closed
log_biery_biery-local-test_crt-grenoble-root-controller.txt:[2025/06/30 12:12:44] ERROR      controller.py:513              controller:                                   Socket closed
log_biery_biery-local-test_crt-grenoble-root-controller.txt:[2025/06/30 12:12:44] ERROR      controller.py:532              controller:                                   Failed to propagate execute_fsm_command to crt-grenoble-controller (crt-grenoble-controller) EXCEPTION THROWN: Socket closed

And, I noticed that the HDF5 raw data file did not successfully get closed in these tests (it still had the ".writing" suffix).

[wesketchum]

I can't manage to reproduce this ...

I'm running on np04-srv-015. I've tried both with a fddaq-v5.3.2-rc4 dev area, and with the release, and in both cases with multiple runs I don't see this happen: no errors or warnings in the logs, hdf5 files closed properly, CRT logs indicating the run_produce() function stopped well ...

@denizergonul , perhaps you're able to try and take a look?

Regardless though, I do think something fishy may be happening here: https://github.com/DUNE-DAQ/crtmodules/blob/4db05aa5a3d98c7571d91ccd8d72100f8a935065/plugins/CRTGrenobleReaderModule.cpp#L243. It's not clear to me how the worker thread exits cleanly, as the m_run_marker is only set to false at do_scrap. We could change that to be done at do_stop instead, and see if that helps? But I'd like to be able to reproduce it first ...

[denizergonul]

I also couldn't reproduce the issue with fddaq-v5.3.2-rc4. Do we still have the log files of the unsuccessful run? I will try to understand why the socket was closed.

[wesketchum]

As a best guess, here may be a setup file for a fix to try @bieryAtFnal ?

dbtx-create-workarea-from-recipe.py --use-ref https://gist.githubusercontent.com/wesketchum/c64eb0dedccdf43219b86c41d68f7fcc/raw

[bieryAtFnal]

I realized that I should have sent the set of commands that I'm using. Those are the following:

drunc-unified-shell ssh-standalone config/daqsystemtest/example-configs.data.xml local-crt-grenoble-1x1-config ${USER}-local-test boot wait 5 conf wait 3 start --run-number 112 enable-triggers wait 10 disable-triggers drain-dataflow stop-trigger-sources stop scrap terminate

Maybe that would help with reproducing the problem.

In any case, the changes that Wes sent did remove the timeout at the stop-trigger-sources transition for me.

There were still some complaints in the logs about action plans, and I was able to get rid of some of them by removing the following lines:

(dbt) [biery@np04-srv-003 daqsystemtest]$ git diff moduleconfs.data.xml
diff --git a/config/daqsystemtest/moduleconfs.data.xml b/config/daqsystemtest/moduleconfs.data.xml
index 8a29f47..f83f72c 100644
--- a/config/daqsystemtest/moduleconfs.data.xml
+++ b/config/daqsystemtest/moduleconfs.data.xml
@@ -98,9 +98,6 @@
  <attr name="execution_policy" type="enum" val="modules-in-parallel"/>
  <rel name="command" class="FSMCommand" id="start"/>
  <rel name="steps">
-  <ref class="DaqModulesGroupByType" id="aggregator-step"/>
-  <ref class="DaqModulesGroupByType" id="tp-handler-step"/>
-  <ref class="DaqModulesGroupByType" id="dlh-step"/>
   <ref class="DaqModulesGroupByType" id="socket-writer-data-source-step"/>
   <ref class="DaqModulesGroupByType" id="crt-bern-reader-data-source-step"/>
  </rel>
@@ -112,9 +109,6 @@
  <rel name="steps">
   <ref class="DaqModulesGroupByType" id="crt-bern-reader-data-source-step"/>
   <ref class="DaqModulesGroupByType" id="socket-writer-data-source-step"/>
-  <ref class="DaqModulesGroupByType" id="dlh-step"/>
-  <ref class="DaqModulesGroupByType" id="tp-handler-step"/>
-  <ref class="DaqModulesGroupByType" id="aggregator-step"/>
  </rel>
 </obj>
 
@@ -122,9 +116,6 @@
  <attr name="execution_policy" type="enum" val="modules-in-parallel"/>
  <rel name="command" class="FSMCommand" id="start"/>
  <rel name="steps">
-  <ref class="DaqModulesGroupByType" id="aggregator-step"/>
-  <ref class="DaqModulesGroupByType" id="tp-handler-step"/>
-  <ref class="DaqModulesGroupByType" id="dlh-step"/>
   <ref class="DaqModulesGroupByType" id="socket-writer-data-source-step"/>
   <ref class="DaqModulesGroupByType" id="crt-grenoble-reader-data-source-step"/>
  </rel>
@@ -136,9 +127,6 @@
  <rel name="steps">
   <ref class="DaqModulesGroupByType" id="crt-grenoble-reader-data-source-step"/>
   <ref class="DaqModulesGroupByType" id="socket-writer-data-source-step"/>
-  <ref class="DaqModulesGroupByType" id="dlh-step"/>
-  <ref class="DaqModulesGroupByType" id="tp-handler-step"/>
-  <ref class="DaqModulesGroupByType" id="aggregator-step"/>
  </rel>
 </obj>

[bieryAtFnal]

This was determined to be related to an uninitialized variable in the asiolibs code, which caused a do_stop() method to take a very long time to run occasionally. That was fixed in this PR and ones that followed it. Closing this Issue.