Establish automated testing of performance benchmarks

[paulromano]

[jtramm]

Currently, we have a CI test setup on the ANL JLSE repository. It tracks performance and correctness on a set of LWR benchmarks each night, using a nightly build of LLVM and OpenMC, and testing on both NVIDIA and AMD GPUs. Each morning, an email is delivered to me (and anyone else interested) that reports the results of the testing, and alerts us to any breakages.

The current CI allows us to track any performance degradations over time that result from changes either in OpenMC or in the LLVM compiler itself. It has been very useful to identify correctness issues that pop up due to ongoing development of the LLVM OpenMP offloading compiler.

The CI itself is very primitive, being composed of a set of shell scripts and python scripts. It is launched nightly as a cron job on one of the JLSE login nodes. The scripts will pull in the newest version of OpenMC, and then launch jobs on the various NVIDIA and AMD nodes. Those jobs compile OpenMC with the nightly LLVM build, and run a series of benchmarks. More scripts then process the std out from these benchmarks and boil them down to a "particles/sec" FOM and an eigenvalue. The scripts then store these in a log .csv file.

Early in the morning, another cron jobs runs and several python scripts are launched. One converts the .csv results (which track historical performance) into plots. Another combines the plots and the nightly data from the .csv and sends an email with these results (via email API with a gmail account).

[jtramm]

In the ideal case, we would perform our CI testing using a higher level framework. It would be nice to have a web interface that can access all historical data across many different benchmarks and architectures, allowing someone to make plots with whatever granularity they want. This would also likely make the CI much easier to maintain. It would also be nice to test across a far wide variety of benchmarks.

[paulromano]

Interesting service I ran across today: https://codspeed.io

[paulromano]

I've started setting up some automated testing in the openmc-performance-bench repo using the asv Python package. My vision for how the automated testing would work is as follows:

• Set up an AMI with all of OpenMC's dependencies
• Set up a Github Action on the openmc-performance-bench repo that 1) spins up an EC2 instance, 2) installs asv, 3) runs asv on the latest openmc commit, 4) pushes results to a separate "results" repo, and 5) stops the EC2 instance.

[paulromano]

I had started to put together a script that installs all the dependencies for OpenMC here by building them from source, and my initial thought was that we would use that to create an AMI that would be used for each of our nightly runs. However, another option may be to just install everything through conda, which will certainly be faster than building everything from source. That may even allow us to do it as part of the nightly run using a vanilla Ubuntu AMI, but of course prebuilding an AMI using the conda route is also an option.

[paulromano]

@amandalund has a proof-of-concept set up! 🎉
https://openmc-dev.github.io/openmc-performance-bench/