enable and fix msqt in CI

[Aurashk]

Description

Leaving in draft as we require a runner with 6 cores to get this working.
https://github.com/DUNE-DAQ/drunc/actions/runs/26099828067/job/76747435708
I think this PR can supersede #610 as subprocess process manager is not needed to run MSQT in CI

• Note also that the MSQT workflow linked above passes even though the tests were not actually run. Need to address that.

Type of change

• New feature / enhancement
• Optimization
• Bug fix
• Breaking change
• Documentation

List of required branches from other repositories

WHAT PRs NEED TO BE INCLUDED TO MAKE THE CHANGE.

Change log

WHAT HAS CHANGED.

Suggested manual testing checklist

LIST COMMANDS TO DEMONSTRATE CHANGE

Developer checklist

Prior to marking this as "Ready for Review"

Tests ran on: WHAT HOSTNAME from release RELEASE_NAME

Unit tests - some tests can't be ran on the CI. This is documented. If this PR checks a feature that can't be tested with CI, this has been marked appropriately.

Integration tests - the daqsystemtest_integtest_bundle requires a lot of resources, and connections to the EHN1 infrastructure. Check the cross referenced list if you can't run these. The developer needs to run at least the .

• Unit tests (pytest --marker) passed
  • With relevant marker
  • Without marker
• Integration tests passed
  • Only daqsystemtest_integtest_bundle.sh -k minimal_system_quick_test.py
  • Full daqsystemtest_integtest_bundle.sh
• Testing skipped as there are no core code changes in this PR, this only relates to documentation/CI workflows
• Drunc integration tests pass (./scripts/drunc_integtest_bundle.sh)

Final checklist prior to marking this as "Ready for Review"

• Code is clearly commented.
• New unit tests have been added, or is documented in # ISSUE NUMBER
• A suitable reviewer has been chosen from this list.

Reviewer checklist

• This branch has been rebased with develop prior to testing.
• Suggested manual tests show changes.
• CI workflows fails documented (if present)
• Integration tests passed (on either np0x or IC HEP clusters)
  • Use the following guidelines to determine which of the integration tests you need to run
    • You do not need to run any integration tests if
      • Code changes are not associated with src/
      • PR changes only affect docstrings
      • In this case, be sure to validate any suggested manual testing.
    • Run only the minimum integration test as daqsystemtest_integtest_bundle.sh -k minimal_system_quick_test.py if
      • PR changes only affect a few log entries
      • PR changes are small, and do not have a large impact on the workflow (use carefully)
    • Otherwise run the full integration test bundle as daqsystemtest_integtest_bundle.sh
  • What to do if the integration tests fail?
    • Only concern yourself if failures related to drunc are in the log files
    • If non-drunc failure appears:
      • Validate failure in fresh working area
      • Contact Pawel if unsure
• If you have ran the full integration test bundle, leave a comment on the PR stating
  • Which host the integration tests have ran on
  • [Optional] A copy of the test summary
• Drunc integration tests pass (scripts/drunc_integtest_bundle.sh)

Once the above boxes are checked, the PR(s) can be merged following the steps below.

Prior to merging

Choose one of the following an complete all substeps

• Changes only affect the Run Control, are in a single repository, and do not affect the end user.
  • Changes are documented in docstrings and code comments
  • Wiki has been updated if architectural or endpoint changes
• Otherwise
  • Workflow changes demonstrated in the Change Log (if necessary)
  • Wiki has been updated (if necessary)
  • #dunedaq-integration Slack channel notified (see below)

Once completed, the reviewer can merge the PR.

Notification message for a Slack channel

Note - this should be to #dunedaq-integration for general workflow that isn't during a release candidate period, and to #daq-release-prep otherwise.

For an single merge that changes the user workflow

The CCM WG has an isolated PR ready to merge that affects user workflows. The PR is:

_URL_

I will leave time for any comments, otherwise will merge these at the end of the work day _Insert your time zone_.

For co-ordinated merge

The CCM WG has a set of co-ordinated merges ready to merge. The PRs are:

_URL_

_URL_


I will leave time for any comments, otherwise will merge these at the end of the day.

[PawelPlesniak]

Hi @Aurashk, I spoke with Kurt who suggested that we can do a few things. The first is to keep attempting to run the MSQT, and we can skip the system requirement checks using the --skip-resource-checks option. I think this is likely not the best option, and I look for how to increase the limits. If this is not possible, we can also use a test that was specifically designed to run on GitHub workflows, this is small_footprint_quick_test. If you can use this test instead of the MSQT, that would be helpful. Thank you!

[Aurashk]

current status - there are no further warnings about resources etc but the test is failing with cryptic error. Attempts to print logs haven't been successful so far

[PawelPlesniak]

The jq exception is typically associated with not being found. The logs of the failures are usually at the top of the workflow. For the latest run, this was

WARNING: Unable to copy the pytest directory for this failed test ().

Usually the test logs are either placed in an /nfs or /tmp path. The choice of which I am unclear of, but @bieryAtFnal may be able to clarify. We may need to manually override the place these are written to

[bieryAtFnal]

Hi, yes, when run from a user account, the DAQ application and controller logs files appear in /tmp/pytest-of-${USER}/pytest-current/runcurrent. I'm not sure where they are stored when running from a GitHub workflow, but I see that the ones run on daq.fnal.gov can be viewed at locations like the following: /home/nfs/dunedaq/github-actions/pytest-logs-2026-06-21-0634UTC.

If you are using the global bundle script (daqsystemtest/scripts/dunedaq_integtest_bundle.sh), there is a --tmpdir command-line argument that you can use to direct the logs to a different location. And, if you are using that script, it should tell you where its log is located when it finishes running.

If you are calling pytest directly in a Linux shell, you can set the PYTEST_DEBUG_TEMPROOT env var to specify the location of the pytest output.