Nexus: Switch to Pytest

[brockdyer03]

Proposed changes

This PR does a few things to further #5881, which are quite necessary to happen in one fell swoop so that develop is not broken.

nexus/nexus/tests/CMakeLists.txt

This PR modifies nexus/nexus/tests/CMakeLists.txt to require, then use pytest for running Nexus tests. The test list is acquired in a nearly identical fashion to how it was acquired via nxs-test, just with some extra logic to take the output of pytest --collect-only -qq and transform it into a format that is compatible with CMake, using sed and tr. Then, there are similar transforms for the tests, but I've done some string replacement to retain the test names while also being able to point Pytest to the correct tests.

One thing that got removed is the cached test list. It is simply not worth the effort to try and cache the test list as it would take a significant amount of time to build in the logic that actually checks if you need to regenerate the tests.

nexus/nexus/tests/test_*.py

I had to make some changes to the tests in this that were not previously present because of the requirement that the tests be runnable 1) through ctest and 2) in parallel. The gist of the change is just that each test has to individually set generic_settings.raise_error = True, which normally would've been done by nxs-test, and for reasons I haven't pinned down yet, is not an issue when running the tests serially through Pytest, but results in errors when running tests individually/in parallel.

nexus/nexus/tests/test_optional_dependencies.py

This file will almost certainly get removed after the dust has settled with the Pytest migration, but for now it must remain. Because that test module has asserts that require optional dependencies be present, it would normally cause test fails, so to avoid that with Pytest I've used pytest.importorskip() to ensure those tests are not run unless the dependencies are there. This will eventually be unnecessary because there is not a reason to check those imports with a Pytest framework.

What type(s) of changes does this code introduce?

• New feature
• Build related changes
• Testing changes (e.g. new unit/integration/performance tests)

Does this introduce a breaking change?

• Yes

What systems has this change been tested on?

Laptop, Fedora 43 KDE Plasma
Python 3.14.2
NumPy 2.4.4
uv 0.11.3
CMake 3.31.11
gcc 15.2.1

Checklist

• • I have read the pull request guidance and develop docs
• • This PR is up to date with the current state of 'develop'

[brockdyer03]

Ok, I think I know why some of these tests are failing. Turns out the command pytest --collect-only -qq changed its behavior at some point, so if the container is using pytest==6.2.5 then the format of the output is along the lines of

nexus/nexus/tests/test_basisset.py: 5

but locally I am using pytest==9.0.2, where the output is

nexus/tests/test_basisset.py: 5

I think that is fixable, but it'll require a bit more logic than I had hoped.

Additionally, it seems that only after pytest==7.4.0 the -c command line argument was extended to include --config-file, so I'll have to change that too.

[brockdyer03]

I made a change that should fix the problems in coverage, but it comes with the potential downside that the Nexus coverage run is only 1 process, and so will take a bit longer than the parallel runs. That being said, I don't think this is actually much of an issue since the coverage workflows are already very long, and from my own testing the entire Pytest run for Nexus (through ctest, with coverage) only takes ~60 seconds.

[brockdyer03]

Ok, after checking the logs, the problem that is arising is that the containers in the coverage runs are still on Ubuntu 22.04LTS, which means they use Python 3.10, which doesn't have native TOML parsing, which is required for the way that coverage is configured here.

There are three solutions:

1. Upgrade the containers to Ubuntu 24.04LTS.
2. Implement Use pip for installing Pytest in containers #5896 and ensure we are installing coverage with the [toml] extension.
3. Implement Separate Nexus coverage from QMCPACK coverage with Nexus-only workflow #5897 and completely separate Nexus's coverage measuring from the rest of QMCPACK's coverage.

I personally am partial to #5897, since that additionally opens the door to a great deal more flexibility in terms of Nexus's testing, which I've enumerated in a comment on that issue.

[prckent]

Blocking until enough testing can be done and thought given. Given various deadlines it will likely be next week before this can be revisited.

[brockdyer03]

I am working on trying to figure out what is going on with the CI. I think it's a CMake bug that only occurs in older CMake versions, but I can't really tell what is causing it since the problem is not reproducible locally.

It looks like the issue is with the output of the pytest version check command

message("Checking pytest version >= 6.2.4")
execute_process(COMMAND ${Python3_EXECUTABLE} -m pytest --version
                RESULT_VARIABLE PYTEST_VERSION_RESULT
                OUTPUT_VARIABLE PYTEST_VERSION
                OUTPUT_STRIP_TRAILING_WHITESPACE)
message("Pytest version result is ' ${PYTEST_VERSION_RESULT} '")
message("Pytest version before replace is ' ${PYTEST_VERSION} '")
string(REPLACE "pytest " "" PYTEST_VERSION "${PYTEST_VERSION}")
message("Pytest version after replace is ' ${PYTEST_VERSION} '")

On my end, for CMake 3.31.11, the output from this is

Checking pytest version >= 6.2.4
Pytest version result is ' 0 '
Pytest version before replace is ' pytest 9.0.3 '
Pytest version after replace is ' 9.0.3 '

From a workflow that is, in theory, running CMake 3.22.1,

Checking pytest version >= 6.2.4
pytest 6.2.5
Pytest version result is ' 0 '
Pytest version before replace is '  '
Pytest version after replace is '  '

It seems to not be properly storing the output of the command in the specified OUTPUT_VARIABLE.

[brockdyer03]

Big shoutout to some obscure pytest behavior that was changed 5 years ago that made that bug very difficult to track down.

Once again, another compelling argument in favor of not using the system package manager to install Python dependencies.

[prckent]

So close!

Unfortunately the test generation is not clean with my setup. Clearly I have a "too new" version of something that is printing deprecation warnings. These are being incorrectly picked up by the test generation code. I have a very reproducible setup and will be able to investigate further if needed. Execution of a few individual ctest tests looked to work as expected.

$ ctest -N -R nexus
Test project /home/pk7/projects/qmc/git_QMCPACK_prckent/build_spack
  Test #2716: ntest_nexus_basisset
  Test #2717: ntest_nexus_bundle
  Test #2718: ntest_nexus_developer
  Test #2719: ntest_nexus_execute
  Test #2720: ntest_nexus_fileio
  Test #2721: ntest_nexus_gamess_analyzer
  Test #2722: ntest_nexus_gamess_input
  Test #2723: ntest_nexus_gamess_simulation
  Test #2724: ntest_nexus_generic
  Test #2725: ntest_nexus_grid_functions
  Test #2726: ntest_nexus_hdfreader
  Test #2727: ntest_nexus_machines
  Test #2728: ntest_nexus_memory
  Test #2729: ntest_nexus_nexus_base
  Test #2730: ntest_nexus_nexus_imports
  Test #2731: ntest_nexus_numerics
  Test #2732: ntest_nexus_nxs_redo
  Test #2733: ntest_nexus_nxs_sim
  Test #2734: ntest_nexus_observables
  Test #2735: ntest_nexus_periodic_table
  Test #2736: ntest_nexus_physical_system
  Test #2737: ntest_nexus_project_manager
  Test #2738: ntest_nexus_pseudopotential
  Test #2739: ntest_nexus_pwscf_analyzer
  Test #2740: ntest_nexus_pwscf_input
  Test #2741: ntest_nexus_pwscf_postprocessor_analyzers
  Test #2742: ntest_nexus_pwscf_postprocessor_input
  Test #2743: ntest_nexus_pwscf_postprocessor_simulations
  Test #2744: ntest_nexus_pwscf_simulation
  Test #2745: ntest_nexus_pyscf_analyzer
  Test #2746: ntest_nexus_pyscf_input
  Test #2747: ntest_nexus_pyscf_simulation
  Test #2748: ntest_nexus_qdens
  Test #2749: ntest_nexus_qmc_fit
  Test #2750: ntest_nexus_qmca
  Test #2751: ntest_nexus_qmcpack_analyzer
  Test #2752: ntest_nexus_qmcpack_converter_analyzers
  Test #2753: ntest_nexus_qmcpack_converter_input
  Test #2754: ntest_nexus_qmcpack_converter_simulations
  Test #2755: ntest_nexus_qmcpack_input
  Test #2756: ntest_nexus_qmcpack_simulation
  Test #2757: ntest_nexus_quantum_package_analyzer
  Test #2758: ntest_nexus_quantum_package_input
  Test #2759: ntest_nexus_quantum_package_simulation
  Test #2760: ntest_nexus_required_dependencies
  Test #2761: ntest_nexus_rmg_analyzer
  Test #2762: ntest_nexus_rmg_input
  Test #2763: ntest_nexus_rmg_simulation
  Test #2764: ntest_nexus_settings
  Test #2765: ntest_nexus_simulation_module
  Test #2766: ntest_nexus_structure
  Test #2767: ntest_nexus_testing
  Test #2768: ntest_nexus_unit_converter
  Test #2769: ntest_nexus_user_examples_alt
  Test #2770: ntest_nexus_vasp_analyzer
  Test #2771: ntest_nexus_vasp_input
  Test #2772: ntest_nexus_vasp_simulation
  Test #2773: ntest_nexus_versions
  Test #2774: ntest_nexus_xmlreader
  Test #2775: ntest_nexus_=============================== warnings summary ===============================
  Test #2776: ntest_nexus_../../../../apps/spack/var/spack/environments/envgccnewmpi/.spack-env/view/lib/python3.14/site-packages/dateutil/tz/tz
  Test #2777: ntest_nexus_  /home/pk7/apps/spack/var/spack/environments/envgccnewmpi/.spack-env/view/lib/python3.14/site-packages/dateutil/tz/tz: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC).
  Test #2778: ntest_nexus_    EPOCH = datetime.datetime.utcfromtimestamp(0)
  Test #2779: ntest_nexus_-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

[ye-luo]

Prefer using --disable-warnings when extracting list of tests.

[brockdyer03]

Prefer using --disable-warnings when extracting list of tests.

I would have done that except I forgot that I could do that...

I'll make that change now haha.

[prckent]

When building with python coverage enabled, only a single test is created. Why? Is there a fundamental reason for this? It is not appropriate to change the number of tests when we change a build option. i.e. We should have ~59 tests created when coverage is enabled, not one. Currently with ntest ~59 tests are created independent of coverage settings. The coalescence in this PR hides which tests are actually being run (all? some?) , disables the ability to filter tests, and may result in the tests being serialized. Please restore the original test layout or give very strong reasons why they need to be fused.

$cmake -GNinja -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpiCC -DQMC_DATA=/scratch/pk7/QMC_DATA_FULL -DENABLE_TIMERS=1 -DQMC_PERFORMANCE_NIO_MAX_ATOMS=16 -DQMC_PERFORMANCE_C_GRAPHITE_MAX_ATOMS=16  -DCMAKE_BUILD_TYPE=Debug -DENABLE_GCOV=TRUE -DENABLE_PYCOV=TRUE -DPYTHON_COVERAGE_EXECUTABLE=coverage ../qmcpack/
$ ctest -N -R nexus
Test project /home/pk7/projects/qmc/git_QMCPACK_prckent/build_spack
  Test #2716: ntest_nexus_coverage_all

Total Tests: 1

compares with the standard no ENABLE_PYCOV results:

$ cmake -GNinja -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpiCC -DQMC_DATA=/scratch/pk7/QMC_DATA_FULL -DENABLE_TIMERS=1 -DQMC_PERFORMANCE_NIO_MAX_ATOMS=16 -DQMC_PERFORMANCE_C_GRAPHITE_MAX_ATOMS=16  -DCMAKE_BUILD_TYPE=Debug -DENABLE_GCOV=TRUE ../qmcpack/
$ ctest -N -R nexus
Test project /home/pk7/projects/qmc/git_QMCPACK_prckent/build_spack
  Test #2716: ntest_nexus_basisset
  Test #2717: ntest_nexus_bundle
  Test #2718: ntest_nexus_developer
  Test #2719: ntest_nexus_execute
  Test #2720: ntest_nexus_fileio
  Test #2721: ntest_nexus_gamess_analyzer
...
  Test #2773: ntest_nexus_versions
  Test #2774: ntest_nexus_xmlreader

Total Tests: 59

[brockdyer03]

When building with python coverage enabled, only a single test is created. Why? Is there a fundamental reason for this? It is not appropriate to change the number of tests when we change a build option. i.e. We should have ~59 tests created when coverage is enabled, not one. Currently with ntest ~59 tests are created independent of coverage settings. The coalescence in this PR hides which tests are actually being run (all? some?) , disables the ability to filter tests, and may result in the tests being serialized. Please restore the original test layout or give very strong reasons why.

The reason this is necessary (I can not emphasize how much time I spent trying to find an alternative) is because pytest-cov/coverage.py automatically concatenates subprocess coverage at the end of a test module's run.

It does not discriminate the subprocess coverage files based on the test name, and thus if you are running multiple tests from the same directory (as we always are), if test 1 finishes while test 2 is halfway done, the coverage reports from the partial test 2 will be gobbled up by the concatenation step from the end of test 1.

As best I can tell, there's only two ways around this: either run the entire Nexus test suite as one test, or don't get coverage through pytest-cov and instead run the tests through coverage.py.

The latter is not a good option, as running through coverage.py changes $PYTHONPATH when you use it versus using pytest-cov. That means we would have two different testing environments, one for regular tests, and one for coverage, which is very much not acceptable in my eyes.

I chose to have the GCov runs run the entire Nexus test suite as one instance of pytest instead, since that means no chance of gobbling up reports accidentally, and a consistent testing environment. I additionally think this is not a concern since, while yes it may put all of the Nexus tests into one suite and make it harder to select a specific Nexus test (keeping in mind that this only applies to running through ctest, not pytest), should the Nexus test suite fail it will still print out the specific subtest that fails.

[prckent]

Fair enough.

Happily the combined coverage runs in ~1 minute. I can imagine big python projects needing workarounds, but Nexus is small.

This pytest limitation confirms that future workflow tests (tests that actually run calculations) will have to run through ctest, in keeping with what we were expecting and had discussed. Only cmake/ctest currently knows the MPI runner seetings for example, and the workflows have to run in parallel for sufficient throughput. Since the workflow tests are more on the QMCPACK/PySCF/QE side and codecov will merge reports for us, I don’t see this as being a problem. These workflow tests are also in the future.

[prckent]

Happy to approve this after so much effort and discussion.

As now noted in the work plan #5881 , a small mention of pytest will also be needed in the QMCPACK docs.

[prckent]

Test this please