Use sandbox mode which is more robust in singularity cleanup phase

[julianmorillo]

No description provided.

[boegel]

@julianmorillo Is there any documentation that would explain why this works better?

No strong objections as long as it works, I just don't see why this would work better (though it seems to based on EESSI/dev.eessi.io-riscv#35 (comment))

[trz42]

Could we use this optionally? It's been tested on RISC-V, but we haven't seen this on other systems. So, to not cause problems on the other systems, maybe the safest way forward would be to add an option and use that where needed.

[julianmorillo]

@boegel
SingularityCE User Guide (runtime behavior changes)
SingularityCE 4.1 Admin/User notes (FUSE + sandbox fallback)

The critical mechanism here (from the documentation above) is that SIF images require mounting (via FUSE). In the documentation above it can be read: "SIF/SquashFS container images will now be mounted with squashfuse… when kernel mounts are not available."
In an unprivileged HPC environment this becomes:

• FUSE mount
• Which requires cleanup on exit --> where errors in {2025.06-001}[2025b] LLVM 20.1.8 dev.eessi.io-riscv#35 (comment) happen.

The documentation above also explains that if mounting fails or is unsuitable --> Singularity uses a sandbox: “If the FUSE mount fails, Singularity will fall back to extracting the container to a temporary sandbox.” and “fallback to extraction to a temporary sandbox directory”.
The key insight is that:

• Sandbox = extracted directory
• No mount needed
• No FUSE
• No teardown/cleanup complexity

[bedroge]

Could we use this optionally? It's been tested on RISC-V, but we haven't seen this on other systems. So, to not cause problems on the other systems, maybe the safest way forward would be to add an option and use that where needed.

This makes sense to me. The sandbox mode is probably (much) slower, especially when multiple jobs are extracting that image to a shared filesystem at the same time.

[julianmorillo]

Could we use this optionally? It's been tested on RISC-V, but we haven't seen this on other systems. So, to not cause problems on the other systems, maybe the safest way forward would be to add an option and use that where needed.

This makes sense to me. The sandbox mode is probably (much) slower, especially when multiple jobs are extracting that image to a shared filesystem at the same time.

I already made it optional through an environment variable that should be defined in site_config.sh. Would that work?

[julianmorillo]

Tested successfully in EESSI/dev.eessi.io-riscv#103, and EESSI/dev.eessi.io-riscv#105

[bedroge]

Could you move the eb_hooks.py changes to a separate PR, as they are unrelated?

The sandbox stuff looks good to me, though it would be good to also add an option to the script itself, as most of these things have a corresponding option. -s is already taken for --save, so we need another one (-z? or only use a long option --sandbox?)

[julianmorillo]

Implemented all the suggested changes. I took the option of just using the long --sandbox option.

[bedroge]

Tested it, and works fine. Some final nitpicks and then we can merge this.

[bedroge]

Lgtm, thanks @julianmorillo!