Easily using alternative MNS

[ocaisa]

After creating #2999 I started to think about how we can create a file that will allow us to replay module creation with a new MNS. What I was wondering is if there is a way in python to trace what self properties are being accessed by a method?

We could then sanitize that list and that should be enough to replay the module creation. We can enforce using the same easyblock (tree) as the original installation since this is now stored in the reprod dir.

[ocaisa]

If I look at self.__dict__ in a simple case (zlib-1.2.8.eb) at the time make_module_step is called, we could construct a blacklist from that and make a dict of values from what is left that represents properties that may affect module creation.

{
 'build_in_installdir': False,
 'builddir': '/home/alanc/.local/easybuild/build/zlib/1.2.8/system-system',
 'cfg': <easybuild.framework.easyconfig.easyconfig.EasyConfig object at 0x7ff0539f7c50>,
 'checksums': [],
 'config_guess': '/home/alanc/.local/easybuild/sources/generic/eb_v4.0.0.dev0/ConfigureMake/config.guess',
 'current_step': 'module',
 'dry_run': False,
 'ext_instances': [],
 'exts': [],
 'exts_all': None,
 'group': None,
 'hooks': {},
 'ignored_errors': False,
 'initial_environ': {...},
 'install_subdir': 'Core/zlib/1.2.8',
 'installdir': '/home/alanc/.local/easybuild/software/Core/zlib/1.2.8',
 'installdir_mod': '/home/alanc/.local/easybuild/modules/all',
 'is_extension': False,
 'iter_idx': 0,
 'iter_opts': {},
 'loaded_modules': [],
 'log': <easybuild.tools.build_log.EasyBuildLog object at 0x7ff05394e710>,
 'logdebug': False,
 'logfile': '/tmp/eb-groi_C/easybuild-zlib-1.2.8-20190913.110012.zvAkh.log',
 'mod_file_backup': None,
 'mod_filepath': '/home/alanc/.local/easybuild/modules/all/Core/zlib/1.2.8.lua',
 'module_extra_extensions': '',
 'module_generator': <easybuild.tools.module_generator.ModuleGeneratorLua object at 0x7ff0539f7f50>,
 'modules_footer': None,
 'modules_header': None,
 'modules_tool': <easybuild.tools.modules.Lmod object at 0x7ff0539f7e10>,
 'orig_modulepath': '/home/alanc/.local/easybuild/modules/all:/home/alanc/.local/easybuild/modules/all/Core:/home/alanc/EasyBuild_Git/EB_Devel',
 'orig_workdir': '/home/alanc/EasyBuild_Git/easybuild-framework',
 'patches': [],
 'postmsg': '',
 'reset_environ': {...},
 'robot_path': ['/home/alanc/EasyBuild_Git/easybuild-easyconfigs/easybuild/easyconfigs'],
 'rpath_filter_dirs': ['/tmp/eb-groi_C',
                       '/home/alanc/.local/easybuild/build/zlib/1.2.8/system-system'],
 'rpath_include_dirs': ['/home/alanc/.local/easybuild/software/Core/zlib/1.2.8/lib',
                        '/home/alanc/.local/easybuild/software/Core/zlib/1.2.8/lib64',
                        '$ORIGIN',
                        '$ORIGIN/../lib',
                        '$ORIGIN/../lib64'],
 'sanity_check_fail_msgs': [],
 'silent': None,
 'skip': None,
 'src': [{'checksum': '36658cb768a54c1d4dec43c3116c27ed893e88b02ecfcb44f2166f9c0b7f2a0d',
          'cmd': None,
          'finalpath': '/home/alanc/.local/easybuild/build/zlib/1.2.8/system-system/zlib-1.2.8',
          'md5': '44d667c142d7cda120332623eab69f40',
          'name': 'zlib-1.2.8.tar.gz',
          'path': '/home/alanc/.local/easybuild/sources/z/zlib/zlib-1.2.8.tar.gz',
          'sha256': '36658cb768a54c1d4dec43c3116c27ed893e88b02ecfcb44f2166f9c0b7f2a0d'}],
 'tweaked_env_vars': {}
}

I think you could blacklist everything off this list except:

 'install_subdir': 'Core/zlib/1.2.8',
 'installdir': '/home/alanc/.local/easybuild/software/Core/zlib/1.2.8',

and even these are unnecessary with #2919.

Any other property in self should be assumed to influence the final module file and stored in a dict that can be restored when using --module-only

[ocaisa]

Looking at icc, this would look like

{
 'comp_libs_subdir': 'compilers_and_libraries_2019.3.199/linux',
 'debuggerpath': 'debugger_2019',
 'home_subdir': '/p/home/jusers/ocais1/jureca/intel',
 'home_subdir_local': '/tmp/ocais1/easybuild_intel',
 'install_components': ['intel-comp__x86_64',
                        'intel-comp-32bit__x86_64',
                        'intel-comp-doc__noarch',
                        'intel-comp-l-all-common__noarch',
                        'intel-comp-l-all-vars__noarch',
                        'intel-comp-nomcu-vars__noarch',
                        'intel-comp-ps-32bit__x86_64',
                        'intel-comp-ps__x86_64',
                        'intel-comp-ps-ss__x86_64',
                        'intel-comp-ps-ss-bec__x86_64',
                        'intel-comp-ps-ss-bec-32bit__x86_64',
                        'intel-compxe__noarch',
                        'intel-compxe-doc__noarch',
                        'intel-ccompxe__noarch',
                        'intel-icc__x86_64',
                        'intel-icc-32bit__x86_64',
                        'intel-icc-common__noarch',
                        'intel-icc-common-ps__noarch',
                        'intel-icc-common-ps-ss-bec__noarch',
                        'intel-icc-doc__noarch',
                        'intel-icc-doc-ps__noarch',
                        'intel-icc-ps__x86_64',
                        'intel-icc-ps-ss-bec__x86_64',
                        'intel-icc-ps-ss-bec-32bit__x86_64',
                        'intel-openmp__x86_64',
                        'intel-openmp-32bit__x86_64',
                        'intel-openmp-common__noarch',
                        'intel-openmp-common-icc__noarch',
                        'intel-openmp-common-ifort__noarch',
                        'intel-openmp-ifort__x86_64',
                        'intel-openmp-ifort-32bit__x86_64',
                        'intel-ips__noarch',
                        'intel-ipsc__noarch'],
 'install_subdir': 'icc/2019.3.199-GCC-8.3.0',
 'installdir': '/usr/local/software/jureca/Stages/Devel-2019a/software/icc/2019.3.199-GCC-8.3.0',
 'license_env_var': 'INTEL_LICENSE_FILE',
 'license_file': '...',
 'requires_runtime_license': True,
 'serial_number': None,
}

[boegel]

@ocaisa That's cool, but I doubt that's a good approach to capture all the info required to generate a module file. What about self.module_generator.set_environment or self.module_generator.prepend_path calls in an easyblock?

What we need to do, I think, is:

• introduce a new object ModuleInfo (not sure about the name)
• change make_module_step such that it requires an instance of ModuleInfo to generate the module file (this could be opt-in initially, so we can gradually update easyblock to use it)
• create the module file using the ModuleInfo instance
• dump the data in the ModuleInfo instance to a file in the installation directory
• make --module-only pick up on that file, and use that information

That leaves open the issue where --module-only --force should not only pick up the ModuleInfo instance, but also update it, since the reason for using --module-only --force may be because the easyblock was updated to add something to the module file...

[ocaisa]

But aren't the prepends and set environment done inside the make_module_step so would still work fine?

[ocaisa]

make_module_extra would still get called within make_module_step but with the relevant module generator and module tool

[boegel]

It's not entirely clear to me what you would do with calls to self.module_generator which are done during make_module_step...

Also, there's no guarantee they're done at that time, in theory they could be done earlier (although that should be rare in practice).