Issues setting up period test suite runs on BSC HCA cluster

[casparvl]

While not 100% related to this repository, there wasn't any other place I could reasonably put this.

First attempt

Install a virtualenv on the login node, and run ReFrame from there (which is what our CI/run_reframe.sh from the EESSI test suite does):

That failed with

cvanleeuwe@hca-server:~$ python3 -m venv test
The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

    apt install python3.12-venv

You may need to use sudo with that command.  After installing the python3-venv
package, recreate your virtual environment.

Failing command: /home/cvanleeuwe/test/bin/python3

cvanleeuwe@hca-server:~$ virtualenv test
-bash: /usr/local/bin/virtualenv: cannot execute: required file not found

Action: @julianmorillo has asked a sysadmin to install python3.12-venv on the login host.

Second attempt

Get an interactive session on a batch node, and use ReFrame from the EESSI software env:

$ cat load_eessi_testsuite_env.sh
source eessi_test_venv/bin/activate

SCRIPT_DIR=$(dirname $(realpath $BASH_SOURCE))

export PYTHONPATH=$PYTHONPATH:$SCRIPT_DIR/test-suite
export PYTHONPATH=$PYTHONPATH:$SCRIPT_DIR/test-suite

export RFM_CONFIG_FILES=$SCRIPT_DIR/test-suite/config/bsc_hca.py
export RFM_CHECK_SEARCH_PATH=$SCRIPT_DIR/test-suite/eessi/testsuite/tests
export RFM_CHECK_SEARCH_RECURSIVE=1
export RFM_PREFIX=$SCRIPT_DIR/reframe_runs

# Make sure we also see GPU-enabled tests
export EESSI_ACCELERATOR_TARGET_OVERRIDE=accel/nvidia/cc90

# Make sure we can load CUDA-modules on the login node, where the driver symlinks in host_injections are dangling
export EESSI_OVERRIDE_GPU_CHECK=True

export EESSI_VERSION_OVERRIDE=2025.06-001
source /cvmfs/software.eessi.io/versions/2025.06/init/lmod/bash

module load ReFrame/4.7.4

Then reframe --list will initiate CPU autodetection, which vails with:

[2026-02-12T10:27:43] debug: reframe: --- /home/cvanleeuwe/EESSI/rfm.6p3ogngw/rfm-detect-job.out ---
Requirement already satisfied: pip in ./venv.reframe/lib/python3.9/site-packages (20.3.4)
Collecting pip
  Downloading pip-26.0.1-py3-none-any.whl (1.8 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.3.4
    Uninstalling pip-20.3.4:
      Successfully uninstalled pip-20.3.4
Successfully installed pip-26.0.1
Collecting reframe-hpc==4.7.4
  Using cached ReFrame_HPC-4.7.4-py3-none-any.whl.metadata (7.2 kB)
Collecting archspec>=0.2.4 (from reframe-hpc==4.7.4)
  Using cached archspec-0.2.5-py3-none-any.whl.metadata (4.4 kB)
Collecting argcomplete (from reframe-hpc==4.7.4)
  Using cached argcomplete-3.6.3-py3-none-any.whl.metadata (16 kB)
Collecting filelock (from reframe-hpc==4.7.4)
  Using cached filelock-3.19.1-py3-none-any.whl.metadata (2.1 kB)
Collecting jsonschema (from reframe-hpc==4.7.4)
  Using cached jsonschema-4.25.1-py3-none-any.whl.metadata (7.6 kB)
Collecting PyYAML (from reframe-hpc==4.7.4)
  Using cached pyyaml-6.0.3.tar.gz (130 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting requests (from reframe-hpc==4.7.4)
  Using cached requests-2.32.5-py3-none-any.whl.metadata (4.9 kB)
Collecting semver (from reframe-hpc==4.7.4)
  Using cached semver-3.0.4-py3-none-any.whl.metadata (6.8 kB)
Collecting tabulate (from reframe-hpc==4.7.4)
  Using cached tabulate-0.9.0-py3-none-any.whl.metadata (34 kB)
Collecting lxml==5.3.0 (from reframe-hpc==4.7.4)
  Using cached lxml-5.3.0.tar.gz (3.7 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
-reframe: command `pip install reframe-hpc==4.7.4' failed (exit code: 1)

--- /home/cvanleeuwe/EESSI/rfm.6p3ogngw/rfm-detect-job.out ---
[2026-02-12T10:27:43] debug: reframe: --- /home/cvanleeuwe/EESSI/rfm.6p3ogngw/rfm-detect-job.err ---
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [3 lines of output]
      Building lxml version 5.3.0.
      Building without Cython.
      Error: Please make sure the libxml2 and libxslt development packages are installed.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed to build 'lxml' when getting requirements to build wheel

(visible in the ReFrame log file)

Action: @julianmorillo has asked a sysadmin to install libxm2 and libxslt on the batch hosts.

Third attempt

Installed libxslt with EESSI-extend. Then, load both in my .bashrc, since I have no other way of changing the pre-amble of the auto-detection job:

export EESSI_VERSION_OVERRIDE=2025.06-001
source /cvmfs/software.eessi.io/versions/2023.06/init/bash
module load EESSI-extend/2025.06

module load libxslt/1.1.43-GCCcore-14.3.0
module load libxml2/2.14.3-GCCcore-14.3.0

Then, run reframe --list again. I still get the same error as above...

Fourth attempt

I login to each of the batch nodes separately, manually load the ReFrame installation from EESSI, run:

reframe --detect-host-topology=topo.json

Then, move it to the folder where ReFrame expects it, e.g.

mv topo.json ~/.reframe/topology/HCA-banana/processor.json

After having done that at each partition, I can now list without triggering autodetection jobs, since ReFrame concludes they are all there already.

[List of matched checks]
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=2_nodes %device_type=cpu /bfbdc774
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /e4bf9965
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=8_cores %device_type=cpu /13b1cbf8
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=4_cores %device_type=cpu /9855363d
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=2_cores %device_type=cpu /0535475e
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=2_nodes %device_type=cpu /1fffd703
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /3da4890b
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=8_cores %device_type=cpu /bb77df99
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=4_cores %device_type=cpu /c6ad4395
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=2_cores %device_type=cpu /3d834161
- EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=2_nodes /ba99bdd1
- EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /3255009a
- EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=2_cores /bbd790fd
- EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=2_nodes /54558a1b
- EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /59f4b331
- EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=2_cores /50b87870
Found 16 check(s)

[julianmorillo]

Action: @julianmorillo has asked a sysadmin to install python3.12-venv on the login host. --> DONE:

jmorillo@hca-server:~$ python3 -m venv test
jmorillo@hca-server:~$ cd test/
jmorillo@hca-server:~/test$ ls
bin  include  lib  lib64  pyvenv.cfg

[julianmorillo]

Action: @julianmorillo has asked a sysadmin to install libxm2 and libxslt on the batch hosts.

I asked to install these packages in Premier nodes but it looks they are already there:

$ sudo apt install libxml2-dev libxslt1-dev
. . .
libxml2-dev is already the newest version (2.9.14+dfsg-1.3ubuntu3.7).
libxslt1-dev is already the newest version (1.1.39-0exp1ubuntu0.24.04.3).

@casparvl , Was the "Second attempt" done in a Premier node or in a different one?

[casparvl]

Sorry for going silent so long on this. I'm 98% sure I tried it on a Premier node, yes. Anyway, today, I tried again: the CPU autodetectino job on arriesgado still failed with the same error as in the 2nd attempt in #34 (comment) . However, on Premier, it now succeeded. I didn't wait for the bananaf one to get scheduled, as the bananaf3-8 node was busy (which is the only one with CVMFS I believe).

Anyway, it would be convienient (but not essential) to have those libs on the arriesgado and bananaf3-8 nodes as well, as with ReFrame updates you sometimes have to re-detect the topology. But for now, I can run it.

Another thing I realized is that there is no CVMFS on the login node. This is inconvenient, as the ReFrame runtime does module discovery to figure out which tests to generate - and it can't do that from the login node if there's no CVMFS. This means I have to schedule a job, to run the reframe runtime. That's a bit of a waste of scarce resources, as the runtime is mostly just doing job management (preparing test jobs, submitting them, and then waiting for them to finish).

Would it be possible to ask the sysadmin to also install CVMFS on the login node? I know it's X64, but that's not an issue: I can override the architecture when initializing EESSI so that ReFrame gets to see the modules available for RISC-V.

[julianmorillo]

Sorry for going silent so long on this. I'm 98% sure I tried it on a Premier node, yes. Anyway, today, I tried again: the CPU autodetectino job on arriesgado still failed with the same error as in the 2nd attempt in #34 (comment) . However, on Premier, it now succeeded. I didn't wait for the bananaf one to get scheduled, as the bananaf3-8 node was busy (which is the only one with CVMFS I believe).

Asked sysadmins to install both libxml2-dev and libxslt1-dev on arriesgado-1.
Asked sysadmins to install CVMFS on HCA (login node).

[julianmorillo]

• Installation of libxml2-dev and libxslt1-dev on arriesgado-1.

[julianmorillo]

@casparvl , they are reluctant to install CVMFS on the login node. They offered me an alternative, though: there is a hidden node (arriesgado-11) that you can directly access from the HCA (login node) through ssh (without the need of scheduling a job) and it has CVMFS already installed (it is RISC-V, BTW). Would that work?

[casparvl]

I think so, I'll give that a try :)

[casparvl]

Hm... CVMFS is installed, but the repo isn't mounted for some reason.

(eessi_test_venv) cvanleeuwe@arriesgado-11:~/EESSI$ cvmfs_config stat -v software.eessi.io
software.eessi.io not mounted
(eessi_test_venv) cvanleeuwe@arriesgado-11:~/EESSI$ ls -al /cvmfs/software.eessi.io
ls: cannot access '/cvmfs/software.eessi.io': No such file or directory

[julianmorillo]

@casparvl Looks that there is an issue with the fuse module and the linux kernel used in arriesgado-11. My proposal is to move to synth-hca: it is another hidden node that you can also directly access from the HCA login node through ssh (without scheduling). This node is x86 and this time I have already checked that the repo mounts correctly:

jmorillo@synth-hca:/$ cd /cvmfs/software.eessi.io
jmorillo@synth-hca:/cvmfs/software.eessi.io$ ls
defaults  host_injections  init  README.eessi  versions

[casparvl]

Ok, I'm one step further. I can list the tests there. The issue is that since this is an x86 node, it makes it both easier and harder. Easier, because ReFrame just installs there without any issues with pip install reframe-hpc (which is what we do in our daily runs). Harder, because ReFrame checks the list of available modules, and schedules tests based on that. Since the modules available for X86 are different from the ones for riscv, it schedules e.g. tests for GROMACS, even though there is no GROMACS module yet in the riscv stack.

I figured: I can set export EESSI_SOFTWARE_SUBDIR_OVERRIDE=riscv64/generic to fix this, but... that doesn't work. The reason is the logic in the EESSI/2025.06 module that takes care of setting the modulepath on riscv64 systems:

$ cat /cvmfs/software.eessi.io/init/modules/EESSI/2025.06.lua | grep -A 10 uname
if (subprocess("uname -m"):gsub("\n$","") == "riscv64") then
    if (eessi_version == "2023.06" or eessi_version == "20240402") then
        eessi_version_override = os.getenv("EESSI_VERSION_OVERRIDE") or ""
        index_suffix = string.find(eessi_version_override, '-')
        if index_suffix then
            eessi_software_layer_version_suffix = string.sub(eessi_version_override, index_suffix)
        end
        eessi_repo = "/cvmfs/riscv.eessi.io"
        eessi_prefix = pathJoin(eessi_repo, "versions", eessi_version .. eessi_software_layer_version_suffix)
        eessi_compat_prefix = pathJoin(eessi_prefix, "compat")
        if mode() == "load" then

I.e. it runs a uname -m and parses the output. It does not respect any EESSI_SOFTWARE_SUBDIR_OVERRIDE that might be set to point to riscv.

We can adapt it, of course... but it'll require a new deployment of the module. Simply adding an or statement that checks if riscv64 is in the EESSI_SOFTWARE_SUBDIR_OVERRIDE should be sufficient, I guess...

[julianmorillo]

In any case, all of these is very outdated. I see:

eessi_repo = "/cvmfs/riscv.eessi.io"

While the EESSI RISC-V repo now is /cvmfs/dev.eessi.io/riscv.
Also versions are wrong:

if (eessi_version == "2023.06" or eessi_version == "20240402") then

while the current version in /cvmfs/dev.eessi.io/riscv is 2025.06-001:

#:/cvmfs/dev.eessi.io/riscv/versions$ ls
2025.06-001

I think @bedroge could be of help here.

[julianmorillo]

Ups, looking into the file I saw that the elseif is dealing with the current RISC-V repository, so forget my previous message:

    elseif (eessi_version == "2025.06") then
        eessi_version_override = os.getenv("EESSI_VERSION_OVERRIDE") or ""
        index_suffix = string.find(eessi_version_override, '-')
        if index_suffix then
            eessi_software_layer_version_suffix = string.sub(eessi_version_override, index_suffix)
        end
        eessi_repo = "/cvmfs/dev.eessi.io/riscv"
        eessi_prefix = pathJoin(eessi_repo, "versions", eessi_version .. eessi_software_layer_version_suffix)

[casparvl]

Yeah, I figured there must have been - I saw on the risc-v node at least that I was getting the correct repo :D

I'll see if I can make a PR later today to add an or condition that also makes sure we enter this logic block if we have specified a riscv based EESSI_SOFTWARE_SUBDIR_OVERRIDE. Shouldn't be that hard...

[casparvl]

So this

if (
    subprocess("uname -m"):gsub("\n$","") == "riscv64"
    or (eessi_software_subdir_override and string.find(eessi_software_subdir_override, "riscv64"))
    ) then

Almost works. The issue is that I'm also getting the riscv compat layer, which of course doesn't work on an x86 machine. Simple things that come from the compat layer like

$ uname -a
-bash: /cvmfs/software.eessi.io/versions/2025.06/compat/linux/riscv64/usr/bin/uname: cannot execute binary file: Exec format error

Are then suddenly broken.

I'm wondering if we should have a mechanism to get one arch for the compat layer, and another arch for the modules. But that is very specific to our usage with the test suite, where we want to be able to explore the modules for arch 1 (e.g. riscv/generic), on a system that has arch 2 (e.g. x86_64/whatever).