[WIP] Scripts for generating easystacks based on an existing stack

[bedroge]

See https://gitlab.com/eessi/support/-/issues/145.

The first script, stack_to_json.sh, allows you to generate a json file with all the build details of an existing stack (paths to easyconfigs, easyblocks, EB version used for the builds, etc), ordered by build time. It will output a list with items that look like:

[
  {
    "build_time": "2023-11-26T20:27:55",
    "build_duration_minutes": "1",
    "name": "EasyBuild",
    "version": "4.8.2",
    "easybuild": "4.9.4",
    "toolchain": "SYSTEM",
    "easyconfig": "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/haswell/software/EasyBuild/4.8.2/easybuild/reprod/EasyBuild-4.8.2.eb",
    "easyblocks": "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/haswell/software/EasyBuild/4.8.2/easybuild/reprod/easyblocks/*.py"
  }
]

You can adjust the json as required, e.g. in order to move rebuilt apps up in the list (to match the original build time of the application, which we cannot determine automatically at the moment).

The second script, json_to_easystacks.sh, can then be used to generate a sequence of easystack files based on the json file, named like 001-eb-X.Y.Z.yml.

[eessi-bot]

Instance eessi-bot-mc-aws is configured to build for:

• architectures: x86_64/generic, x86_64/intel/haswell, x86_64/intel/sapphirerapids, x86_64/intel/skylake_avx512, x86_64/intel/cascadelake, x86_64/intel/icelake, x86_64/amd/zen2, x86_64/amd/zen3, aarch64/generic, aarch64/neoverse_n1, aarch64/neoverse_v1
• repositories: eessi.io-2023.06-compat, eessi.io-2023.06-software

[eessi-bot]

Instance eessi-bot-mc-azure is configured to build for:

• architectures: x86_64/amd/zen4
• repositories: eessi.io-2023.06-compat, eessi.io-2023.06-software

[gpu-bot-ugent]

Instance eessi-bot-vsc-ugent is configured to build for:

• architectures: x86_64/amd/zen3
• repositories: eessi-hpc.org-2023.06-software, eessi.io-2023.06-compat, eessi-hpc.org-2023.06-compat, eessi.io-2023.06-software

[eessi-bot-toprichard]

Instance rt-Grace-jr is configured to build for:

• architectures: aarch64/nvidia/grace
• repositories: eessi.io-2023.06-software

[eessi-bot-surf]

Instance eessi-bot-surf is configured to build for:

• architectures: x86_64/amd/zen4, x86_64/amd/zen2
• repositories: eessi-hpc.org-2023.06-software, eessi.io-2023.06-software, eessi.io-2023.06-compat, eessi-hpc.org-2023.06-compat

[casparvl]

As a general comment: the script should get some proper option parsing and --help. E.g. something like https://github.com/casparvl/convenience_scripts/blob/30e9be6d89cf74c6118152b180552a551f163fbe/git_cleanup#L10

[casparvl]

From what I've seen in #1037 (comment) all the bootstrapping done by load_easybuild_module.sh and load_eessi_extend_module.sh makes it really hard to enforce that e.g. EB 4.8.2 is installed with EB 4.8.2. Or even that EB 4.9.0 is installed with EB 4.8.2.

It means that even regardless of what you'd set as EB_bootstrap it is very likely that it will just use the latest EB release anyway, as that's the one being installed in all the bootstrap procedures.

I think your EB_bootstrap approach is reasonable, and we should keep it. We should just use the latest EB version as bootstrap. That will ensure that regardless of whether the bootstrap procedure from the load_* scripts is used, or whether it is an EB version actually based on the EB version in the EasyStack name, it'll be the same (namely latest) EB version.

[casparvl]

I'm now in the stage where I have to play catch-up with the rest of the architectures after we've ingested all the initially generated EasyStacks. My main issue is that I essentially have to regenerate the easystacks, figure out from the last few what is still missing, then separate those out. It's not a big deal, but it could be nice if we can improve the script to also look at e.g. the icelake prefix and then generate EasyStacks only for those. Then, if we could even indicate what number it should use to start numbering new EasyStacks... that'd make it super smooth and easy.

[laraPPr]

@bedroge Can you retarget this pr to the https://github.com/EESSI/software-layer-scripts repo?