[WIP] Update of Perlmutter build script#5926
Draft
brockdyer03 wants to merge 5 commits into
Draft
Conversation
Contributor
|
Leave out mixed precision builds by default as they are generally special interest. |
Contributor
Author
|
It'd be pretty trivial to make them only on by selection, but I believe the other build scripts all do both mixed and full precision by default, meaning this script would be the only one that doesn't. |
prckent
reviewed
Jul 6, 2026
Contributor
There was a problem hiding this comment.
Testing myself, I'll note that the python environment setup needs a bit more building out - we need numpy and h5py to have a worthwhile set of tests run. Did they work for you?
-- Found Python3: /pscratch/sd/p/pkent/test_qmc/qmcpack/.qmcvenv/bin/python (found version "3.13.11") found components: Interpreter
-...
-- Ready to parse QMCPACK source tree
...
-- Adding integration tests for QMCPACK
Use --log-level=VERBOSE CMake option for details of which tests will be enabled.
Adding estimator tests for QMCPACK
Missing python module numpy, not adding test estimator-skinetic
Missing python module h5py, not adding test estimator-skinetic
...
Adding example tests for QMCPACK
Missing python module numpy, not adding test nexus base
Skipping Nexus tests because numpy is not present in python installation
Contributor
|
Also: The module loads should not be directed to /dev/null since if anything goes wrong we want to see the error. |
…void bash being weird
Contributor
|
Fixing perlmutter recipe and improving user interface are better to be handled in two steps. It is nice to improving the user interface. However, it complicates individual scripts in the current way. To avoid code duplication, it will be better to make |
Contributor
Author
I was initially going to go this route, however I am completely unfamiliar with other supercomputing systems and can't test on anything other than Perlmutter. It would be nearly impossible for me to write something that is guaranteed to work on other systems. |
Contributor
|
I am experimenting with this now. I am leaning in favor of updating the existing script the minimal amount in a seperate PR to get it to work fully before complicating things. Of note so far:
|
Contributor
You don't need to. Probably just need to show
|
Contributor
Author
Most of the code here is just for a nice user experience. I don't think it really complicates things since the default behavior is the same as the previous script. It may look complicated, but I think the only part that actually changes anything in terms of the build environment is lines 259-end.
On Perlmutter the no-modules-loaded CMake is 3.28.3
It does indeed have conflicts, that is why I load |
Contributor
|
The tests in my build (LLVM21, OpenMPI5) fail with the pattern reported by Brock: |
Contributor
|
The CPU code has the same failure pattern. Looking quickly the stack trace (below) is similar in all the failing cases => My suspicion is that there is an ABI compatibility problem with the currently used HDF5 parallel and OpenMPI modules. The old build script presumably did not suffer from this because we used cray-mpich. |
…dy being loaded to avoid re-loading unnecessarily.
Contributor
|
Working on a solution with |
Contributor
|
Walked through landmines #6026 |
Contributor
|
Catalyzed by this PR, #6026 now solves the fundamental problem. I'll note that the fancy features in this PR currently don't tackle the fundamental problems of getting a user setup on Perlmutter unless they only need to run QMCPACK "by hand". Before investing further any effort I suggest a discussion on that to ensure that effort is spent efficiently. |
Contributor
Author
I'm not sure I understand what you mean by that. The script updates I've proposed also, in the event the user does not have their Python environment set up, creates a virtual environment and installs all of the necessary packages to use Nexus, and installs Nexus into the environment. That should be enough to get people going, right? |
Contributor
|
It is enough to get people going, but to clarify: to do QMC in general the user also needs QE or PySCF or some other source of trial wavefunctions. Locally I have also been installing these; the development dockerfile does the same. => A generally useful install involves a lot more than just having Nexus available. It would be all too easy for someone to e.g. bring in conflicting HDF5 versions or make a "slow" install of PySCF. This is obviously a big step up from the currently provided build scripts but I think this path is work discussing to make sure that we don't have to rework any intermediate steps and there is buy in across the people who update these scripts. The question of how to tackle this applies to all the build scripts except for perhaps the OLCF and ALCF ones where users will likely copy in the trial wavefunctions because small jobs are not encouraged. |
Contributor
Author
If you'd really like something for running QMCPACK on Perlmutter, then just write the documentation for how to use the container. It's not hard to use a container on Perlmutter, they've got pretty good documentation on it, and you don't really need to download anything.
I don't really understand enough about how QMCPACK is built or run to know whether or not it's likely that someone will accidentally grab conflicting HDF5 versions, but I have not personally had any such problems. I am not sure if it is really feasible to be giving users a "one-stop-shop" sort of script that does QMCPACK, QE, PySCF, and whatever other programs they may need. I personally have a customized installation of QE on Perlmutter, and I would not want to be stuck with whatever QMCPACK thinks is the correct version. Perhaps the script could provide a link to documentation for how to get a "good" build of QE or PySCF done on Perlmutter, but now we are opening the door to needing to maintain those doc pages for as long as Perlmutter exists.
The big thing, in my opinion, is a friendly user interface (e.g. the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
The previous build script for Perlmutter no longer functions, so I have decided to update it. While doing so, I decided to add a more friendly interface so that users aren't stuck just building every single build variant every time.
Here is the new usage:
I also added code in that creates a virtual environment and installs Nexus to it if the user does not specify an existing Nexus installation. I imagine this should probably be toggleable, but I haven't gotten to that yet.
Importantly, the behavior of this script is supposed to be identical if no arguments are passed, which I think is useful for not creating unexpected behavior.
GPU Test Failures
I am not entirely sure what I need to do to fix the GPU builds, this is decidedly not my area of expertise. I'm attaching several files that others may find useful in helping to diagnose the problem.
./config/build_nersc_perlmutter_Clang.sh -s . -i ./build -e ../qe_custom_2/qe-build/bin -t gpu -rf > build_script_updated.log 2>&1 &build_script_updated.log
ctest -R deterministic -j 32 --output-on-failure > perlmutter_Clang21_gpu_real_deterministic.log 2>&1 &perlmutter_Clang21_gpu_real_deterministic.log
What type(s) of changes does this code introduce?
Does this introduce a breaking change?
What systems has this change been tested on?
Perlmutter
Checklist