Separate Nexus coverage from QMCPACK coverage with Nexus-only workflow

[brockdyer03]

Current

A challenge I am not sure is surmountable in #5894 is the difficulties with parallel coverage gathering through pytest-cov. The primary problem is that pytest-cov grabs any .coverage* files and aggregates the results, so if you have 4 tests generating their own coverage reports then a test might accidentally "steal" the coverage report from another test, resulting in an error.

I genuinely am not sure if it is possible to even fix that because there doesn't seem to be a way to individually label the coverage reports for each test.

Proposal

I think the best way to fix this is to just create a separate Nexus-only workflow that reports coverage completely separately from the QMCPACK coverage.

I've made a branch on my fork of QMCPACK that is what I envision for a Nexus-only workflow that tests Python 3.10 and Python 3.14, accounts for Numpy 1.22.0 and 2.4.4, and tests on both ubuntu:latest and macos:latest. It would be relatively trivial to also collect coverage there, and since the Nexus tests are quite rapid it would not add any significant runtime to our current workflows.

CodeCov already supplies tools for submitting separate coverage reports for different languages, detailed here.

[prckent]

Potential solution: don't use pytest-cov. coverage.py does everything we need already, and we have already configured it to work properly in parallel. => drop pytest-cov usage, uninstall it, use existing coverage config, and everything will work.

Should this not work, please name the unique features of pytest-cov that are needed.

[brockdyer03]

Potential solution: don't use pytest-cov. coverage.py does everything we need already, and we have already configured it to work properly in parallel. => drop pytest-cov usage, uninstall it, use existing coverage config, and everything will work.

Should this not work, please name the unique features of pytest-cov that are needed.

This is an less than desirable solution because you then have two different Python environments when you are running normal tests versus coverage.

Per the pytest-cov documentation:

Consistent pytest behavior. If you run coverage run -m pytest you will have slightly different sys.path (CWD will be in it, unlike when running pytest).

As much as we want to say the environments are the same, we would need far more rigorous checks to ensure zero contamination.

[brockdyer03]

These are some additional benefits of a Nexus-only workflow:

1. More complex matrices (multiple Python versions, any combination of package versions)
2. Removing reliance on apt or dnf for Python packages
3. Rapid testing of Nexus; no need to wait 20 minutes for QMCPACK to build
4. Allow for conditional QMCPACK workflows (e.g. don't run if we know Nexus is going to fail)

[prckent]

As we discussed: It makes sense to have a nexus dedicated workflow action that runs quickly to (in future) gate the full QMCPACK set of actions. However, there is not currently a way to handle the coverage reporting from codecov back to the PR - we have it set to delay for 2 reports (merged real, complex build) and there will be additional python coverage in the QMCPACK coverage builds that we wish to monitor. => I suggest to not run coverage in the new nexus action. Use it for early exit and to test a small matrix of supported python versions.