Skip to content

reduce number of module load commands used to set up build environment#5211

Draft
boegel wants to merge 5 commits into
easybuilders:developfrom
boegel:perf
Draft

reduce number of module load commands used to set up build environment#5211
boegel wants to merge 5 commits into
easybuilders:developfrom
boegel:perf

Conversation

@boegel

@boegel boegel commented Jun 22, 2026

Copy link
Copy Markdown
Member
  • loading of modules when setting up toolchain build environment for toolchain itself (1) + dependencies (D) is not done in one go, so a single module load command rather than 1+D `module load commands;
  • build environment for installing extensions is determine once, and environment is restored right before installing an extension (without running module commands);
    • only implemented in EasyBlock.install_extensions_sequential currently;

TODO:

  • also change EasyBlock.install_extensions_parallel to restore build environment for each extension installation;
  • fix broken tests;
  • look for further "easy" wins, like avoiding module unload commands

Essentially fixes #5052

boegel added 5 commits June 21, 2026 16:57
…lchain_and_deps which loads modules for toolchain + dependencies in one go (except when --extended-dry-run is used)
…nvironment once before installing extensions, and use restore_env to restore it before installing each extension
@boegel boegel added this to the next release (5.3.2?) milestone Jun 22, 2026
else:
# first simulate loads for toolchain dependencies, if required information is available
for tc_dep in self.tcdeps:
mod_name = tcdep['short_mod_name']

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

undefined name 'tcdep'

@branfosj

Copy link
Copy Markdown
Member

Installing the first 49 extensions from R-bundle-CRAN/2025.11-foss-2025 sequentially:

  • EB 4.9.4: 13:20
  • Current develop: 29:20
  • This PR: 18:29

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Building easyconfigs with many extensions is much slower with EasyBuild 5.x than with EasyBuild 4.x

2 participants