Skip to content

add generic hooks for unsupported modules#91

Merged
ocaisa merged 4 commits intoEESSI:mainfrom
bedroge:hook_for_unsupported_modules
Sep 26, 2025
Merged

add generic hooks for unsupported modules#91
ocaisa merged 4 commits intoEESSI:mainfrom
bedroge:hook_for_unsupported_modules

Conversation

@bedroge
Copy link
Copy Markdown
Contributor

@bedroge bedroge commented Sep 25, 2025

This makes the existing zen4 pre_fetch / pre_module / post_module hooks a bit more generic, so that they can also be used for adding dummy modulefiles for other unsupported modules. The function is_unsupported_module is used to determine if something is unsupported, and if so, which environment variable can be used to ignore the LmodError of that module.

The hooks are also applied to make a module file with an LmodError for Rust 1.65.0 on A64FX.

Tested locally by building Rust 1.65.0 for A64FX with EESSI-extend:

$ eb Rust-1.65.0-GCCcore-12.2.0.eb
== Temporary log file in case of crash /tmp/eb-c40k2fnp/easybuild-5vbxdeyd.log
...
== processing EasyBuild easyconfig /home/bob/easybuild/easybuild-easyconfigs/easybuild/easyconfigs/r/Rust/Rust-1.65.0-GCCcore-12.2.0.eb
== building and installing Rust/1.65.0-GCCcore-12.2.0...
  >> installation prefix: /home/bob/eessi/versions/2023.06/software/linux/aarch64/a64fx/software/Rust/1.65.0-GCCcore-12.2.0
== fetching files and verifying checksums...
== Running pre-fetch hook...
== Updated build option 'module-only' to 'True'
== Updated build option 'force' to 'True'
  >> sources:
  >> /data/eb/sources/r/Rust/rustc-1.65.0-src.tar.gz [SHA256: 5828bb67f677eabf8c384020582b0ce7af884e1c84389484f7f8d00dd82c0038]
  >> patches:
  >> /home/bob/easybuild/easybuild-easyconfigs/easybuild/easyconfigs/r/Rust/Rust-1.60_sysroot-fix-interpreter.patch [SHA256: b59ed4c2591fc9098277299be21dd6752654f6f193d8652b7d21cb0fa0dd8716]
== ... (took < 1 sec)
== creating build dir, resetting environment...
  >> build dir: /data/eb/build/Rust/1.65.0/GCCcore-12.2.0
== Running post-ready hook...

WARNING: Deprecated functionality, will no longer work in EasyBuild v6.0: Easyconfig parameter 'parallel' is deprecated, use 'max_parallel' or the parallel property instead.; see https://docs.easybuild.io/deprecated-functionality/ for more 
information


WARNING: Deprecated functionality, will no longer work in EasyBuild v6.0: Easyconfig parameter 'parallel' is deprecated, use 'max_parallel' or the parallel property instead.; see https://docs.easybuild.io/deprecated-functionality/ for more 
information

== limiting parallelism to 2 (was 6, derived parallelism 8) for Rust on aarch64/a64fx 
== ... (took < 1 sec)
== unpacking [skipped]
== patching [skipped]
== preparing...
== Running pre-prepare hook...
  >> loading toolchain module: GCCcore/12.2.0
  >> loading modules for build dependencies:
  >>  * CMake/3.24.3-GCCcore-12.2.0
  >>  * Python/3.10.8-GCCcore-12.2.0-bare
  >>  * pkgconf/1.9.3-GCCcore-12.2.0
  >> loading modules for (runtime) dependencies:
  >>  * OpenSSL/1.1
  >> defining build environment for GCCcore/12.2.0 toolchain
== Running post-prepare hook...
== ... (took 1 secs)
== configuring [skipped]
== building [skipped]
== testing [skipped]
== installing [skipped]
== taking care of extensions [skipped]
== restore after iterating...
== ... (took < 1 sec)
== postprocessing [skipped]
== sanity checking [skipped]
== cleaning up [skipped]
== creating module...
== Running pre-module hook...
== Setting EESSI_IGNORE_LMOD_ERROR_A64FX_RUST1650 in initial environment
  >> generating module file @ /home/bob/eessi/versions/2023.06/software/linux/aarch64/a64fx/modules/all/Rust/1.65.0-GCCcore-12.2.0.lua
== Running post-module hook...
== Restored original build option 'module_only' to False
== Restored original build option 'force' to False
== Removing EESSI_IGNORE_LMOD_ERROR_A64FX_RUST1650 in initial environment
== ... (took < 1 sec)
== permissions...
== ... (took < 1 sec)
== packaging...
== ... (took < 1 sec)
  >> running shell command:
        bzip2 /home/bob/eessi/versions/2023.06/software/linux/aarch64/a64fx/software/Rust/1.65.0-GCCcore-12.2.0/easybuild/easybuild-Rust-1.65.0-20250925.133516.log
        [started at: 2025-09-25 13:35:16]
        [working dir: /home/bob/gits/EESSI/software-layer-scripts]
        [output and state saved to /tmp/eb-c40k2fnp/run-shell-cmd-output/bzip2-7wc0hadt]
  >> command completed: exit 0, ran in < 1s
== COMPLETED: Installation ended successfully (took 3 secs)
== Results of the build can be found in the log file(s) /home/bob/eessi/versions/2023.06/software/linux/aarch64/a64fx/software/Rust/1.65.0-GCCcore-12.2.0/easybuild/easybuild-Rust-1.65.0-20250925.133516.log.bz2
== Running post-easyblock hook...

== Build succeeded for 1 out of 1
== Summary:
   * [SUCCESS] Rust/1.65.0-GCCcore-12.2.0
== Temporary log file(s) /tmp/eb-c40k2fnp/easybuild-5vbxdeyd.log* have been removed.
== Temporary directory /tmp/eb-c40k2fnp has been removed.


$ tail -n2 ~/eessi/versions/2023.06/software/linux/aarch64/a64fx/modules/all/Rust/1.65.0-GCCcore-12.2.0.lua 
-- Built with EasyBuild version 5.1.1
if (not os.getenv("EESSI_IGNORE_LMOD_ERROR_A64FX_RUST1650")) then LmodError("Rust 1.65.0 is not supported on A64FX. Please use version 1.75.0 instead.") end

And also tested by building a 2022b easyconfig for Zen4:

$ eb -f ~/easybuild/easybuild-easyconfigs/easybuild/easyconfigs/f/file/file-5.43-GCCcore-12.2.0.eb 
== Temporary log file in case of crash /tmp/eb-91skhb9h/easybuild-kbx7d5vo.log
== processing EasyBuild easyconfig /home/bob/easybuild/easybuild-easyconfigs/easybuild/easyconfigs/f/file/file-5.43-GCCcore-12.2.0.eb
== building and installing file/5.43-GCCcore-12.2.0...
  >> installation prefix: /home/bob/eessi/versions/2023.06/software/linux/x86_64/amd/zen4/software/file/5.43-GCCcore-12.2.0
== fetching files and verifying checksums...
== Running pre-fetch hook...
== Updated build option 'module-only' to 'True'
== Updated build option 'force' to 'True'
  >> sources:
  >> /data/eb/sources/f/file/file-5.43.tar.gz [SHA256: 8c8015e91ae0e8d0321d94c78239892ef9dbc70c4ade0008c0e95894abfb1991]
== ... (took < 1 sec)
== creating build dir, resetting environment...
  >> build dir: /data/eb/build/file/5.43/GCCcore-12.2.0
== Running post-ready hook...

WARNING: Deprecated functionality, will no longer work in EasyBuild v6.0: Easyconfig parameter 'parallel' is deprecated, use 'max_parallel' or the parallel property instead.; see https://docs.easybuild.io/deprecated-functionality/ for more 
information

== ... (took < 1 sec)
== unpacking [skipped]
== patching [skipped]
== preparing...
== Running pre-prepare hook...
  >> loading toolchain module: GCCcore/12.2.0
  >> defining build environment for GCCcore/12.2.0 toolchain
== Running post-prepare hook...
== ... (took < 1 sec)
== configuring [skipped]
== building [skipped]
== testing [skipped]
== installing [skipped]
== taking care of extensions [skipped]
== restore after iterating...
== ... (took < 1 sec)
== postprocessing [skipped]
== sanity checking [skipped]
== cleaning up [skipped]
== creating module...
== Running pre-module hook...
== Setting EESSI_IGNORE_LMOD_ERROR_ZEN4_GCC1220 in initial environment
  >> generating module file @ /home/bob/eessi/versions/2023.06/software/linux/x86_64/amd/zen4/modules/all/file/5.43-GCCcore-12.2.0.lua
== Running post-module hook...
== Restored original build option 'module_only' to False
== Restored original build option 'force' to True
== Removing EESSI_IGNORE_LMOD_ERROR_ZEN4_GCC1220 in initial environment
== ... (took < 1 sec)
== permissions...
== ... (took < 1 sec)
== packaging...
== ... (took < 1 sec)
  >> running shell command:
        bzip2 /home/bob/eessi/versions/2023.06/software/linux/x86_64/amd/zen4/software/file/5.43-GCCcore-12.2.0/easybuild/easybuild-file-5.43-20250925.130223.log
        [started at: 2025-09-25 13:02:23]
        [working dir: /home/bob/gits/EESSI/software-layer-scripts]
        [output and state saved to /tmp/eb-91skhb9h/run-shell-cmd-output/bzip2-s98xawv1]
  >> command completed: exit 0, ran in < 1s
== COMPLETED: Installation ended successfully (took 1 secs)
== Results of the build can be found in the log file(s) /home/bob/eessi/versions/2023.06/software/linux/x86_64/amd/zen4/software/file/5.43-GCCcore-12.2.0/easybuild/easybuild-file-5.43-20250925.130223.log.bz2
== Running post-easyblock hook...

== Build succeeded for 1 out of 1
== Summary:
   * [SUCCESS] file/5.43-GCCcore-12.2.0
== Temporary log file(s) /tmp/eb-91skhb9h/easybuild-kbx7d5vo.log* have been removed.
== Temporary directory /tmp/eb-91skhb9h has been removed.


$ tail -n 2 ~/eessi/versions/2023.06/software/linux/x86_64/amd/zen4/modules/all/file/5.43-GCCcore-12.2.0.lua 
-- Built with EasyBuild version 5.1.1
if (not os.getenv("EESSI_IGNORE_LMOD_ERROR_ZEN4_GCC1220")) then LmodError("EasyConfigs using toolchains based on GCCcore-12.2.0 are not supported for the Zen4 architecture.\nSee https://www.eessi.io/docs/known_issues/eessi-2023.06/#gcc-1220-and-foss-2022b-based-modules-cannot-be-loaded-on-zen4-architecture") end

ocaisa
ocaisa reviewed Sep 26, 2025
View reviewed changes
Comment thread eb_hooks.py Outdated
@bedroge
move check for unsupported modules to the hook function themselves, i…
7b8a935 
…nstead of having them in the generic hook
@bedroge
Merge branch 'hook_for_unsupported_modules' of github.com:bedroge/sof…
4bfd2cc 
…tware-layer-scripts into hook_for_unsupported_modules
ocaisa
ocaisa approved these changes Sep 26, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@ocaisa ocaisa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, tested for the Zen4 case

@ocaisa
Copy link
Copy Markdown
Member

ocaisa commented Sep 26, 2025

bot: build repo:eessi.io-2023.06-software instance:eessi-bot-mc-aws for:arch=x86_64/amd/zen2
bot: build repo:eessi.io-2025.06-software instance:eessi-bot-mc-aws for:arch=x86_64/amd/zen2

@eessi-bot-aws
Copy link
Copy Markdown

eessi-bot-aws Bot commented Sep 26, 2025 •
edited
Loading

New job on instance eessi-bot-mc-aws for repository eessi.io-2023.06-software
Building on: amd-zen2
Building for: x86_64/amd/zen2
Job dir: /project/def-users/SHARED/jobs/2025.09/pr_91/92593

date job status comment
Sep 26 11:20:21 UTC 2025 submitted job id 92593 awaits release by job manager
Sep 26 11:21:58 UTC 2025 released job awaits launch by Slurm scheduler
Sep 26 11:59:02 UTC 2025 running job 92593 is running
Sep 26 12:02:20 UTC 2025 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-92593.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.gz created!
Artefacts
eessi-2023.06-software-linux-x86_64-amd-zen2-17588878880.tar.gzsize: 0 MiB (21344 bytes)
entries: 1
modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all
no module files in tarball
software under 2023.06/software/linux/x86_64/amd/zen2/software
no software packages in tarball
reprod directories under 2023.06/software/linux/x86_64/amd/zen2/reprod
no reprod directories in tarball
other under 2023.06/software/linux/x86_64/amd/zen2
2023.06/init/easybuild/eb_hooks.py
Sep 26 12:02:20 UTC 2025 test result
😁 SUCCESS (click triangle for details)
ReFrame Summary
[ OK ] ( 1/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos %scale=1_node /aeb2d9df @BotBuildTests:x86_64_amd_zen2+default
P: perf: 441.741 timesteps/s (r:0, l:None, u:None)
[ OK ] ( 2/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos %scale=1_node /04ff9ece @BotBuildTests:x86_64_amd_zen2+default
P: perf: 441.638 timesteps/s (r:0, l:None, u:None)
[ OK ] ( 3/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node %device_type=cpu /775175bf @BotBuildTests:x86_64_amd_zen2+default
P: latency: 1.86 us (r:0, l:None, u:None)
[ OK ] ( 4/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node %device_type=cpu /52707c40 @BotBuildTests:x86_64_amd_zen2+default
P: latency: 1.88 us (r:0, l:None, u:None)
[ OK ] ( 5/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node %device_type=cpu /b1aacda9 @BotBuildTests:x86_64_amd_zen2+default
P: latency: 4.06 us (r:0, l:None, u:None)
[ OK ] ( 6/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node %device_type=cpu /c6bad193 @BotBuildTests:x86_64_amd_zen2+default
P: latency: 4.49 us (r:0, l:None, u:None)
[ OK ] ( 7/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /15cad6c4 @BotBuildTests:x86_64_amd_zen2+default
P: latency: 0.59 us (r:0, l:None, u:None)
[ OK ] ( 8/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /6672deda @BotBuildTests:x86_64_amd_zen2+default
P: latency: 0.55 us (r:0, l:None, u:None)
[ OK ] ( 9/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /2a9a47b1 @BotBuildTests:x86_64_amd_zen2+default
P: bandwidth: 7312.6 MB/s (r:0, l:None, u:None)
[ OK ] (10/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /1b24ab8e @BotBuildTests:x86_64_amd_zen2+default
P: bandwidth: 7194.91 MB/s (r:0, l:None, u:None)
[ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted)
Details
✅ job output file slurm-92593.out
✅ no message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case
Sep 26 12:10:17 UTC 2025 uploaded transfer of eessi-2023.06-software-linux-x86_64-amd-zen2-17588878880.tar.gz to S3 bucket succeeded

@eessi-bot-aws
Copy link
Copy Markdown

eessi-bot-aws Bot commented Sep 26, 2025 •
edited
Loading

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-software
Building on: amd-zen2
Building for: x86_64/amd/zen2
Job dir: /project/def-users/SHARED/jobs/2025.09/pr_91/92594

date job status comment
Sep 26 11:20:25 UTC 2025 submitted job id 92594 awaits release by job manager
Sep 26 11:21:56 UTC 2025 released job awaits launch by Slurm scheduler
Sep 26 11:57:30 UTC 2025 running job 92594 is running
Sep 26 11:59:34 UTC 2025 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-92594.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.gz created!
Artefacts
eessi-2025.06-software-linux-x86_64-amd-zen2-17588878410.tar.gzsize: 0 MiB (21346 bytes)
entries: 1
modules under 2025.06/software/linux/x86_64/amd/zen2/modules/all
no module files in tarball
software under 2025.06/software/linux/x86_64/amd/zen2/software
no software packages in tarball
reprod directories under 2025.06/software/linux/x86_64/amd/zen2/reprod
no reprod directories in tarball
other under 2025.06/software/linux/x86_64/amd/zen2
2025.06/init/easybuild/eb_hooks.py
Sep 26 11:59:34 UTC 2025 test result
😢 FAILURE (click triangle for details)
Reason
EESSI test suite was not run, test step itself failed to execute.
Details
✅ job output file slurm-92594.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case
Sep 26 12:10:25 UTC 2025 uploaded transfer of eessi-2025.06-software-linux-x86_64-amd-zen2-17588878410.tar.gz to S3 bucket succeeded

@bedroge bedroge mentioned this pull request Sep 26, 2025
Merged
@ocaisa ocaisa merged commit 5cccf70 into EESSI:main Sep 26, 2025
66 of 68 checks passed
@bedroge bedroge deleted the hook_for_unsupported_modules branch September 26, 2025 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ocaisa ocaisa ocaisa approved these changes

Assignees

No one assigned

Labels

bot:deploy

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@bedroge @ocaisa