Skip to content

Commit b3e8a0e

Browse files
committed
First proper build
1 parent 2050903 commit b3e8a0e

15 files changed

Lines changed: 149 additions & 127 deletions

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,2 @@
11
.venv
2+
site

docs/eessi-getting-access.md

Lines changed: 46 additions & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -6,13 +6,13 @@ of the EESSI repository.
66
## Is EESSI accessible?
77

88
EESSI can be accessed via [a native (CernVM-FS) installation](#native-installation),
9-
or via [a container that includes CernVM-FS](#eessi-container).
9+
or via [a container that includes CernVM-FS](#eessi-via-a-container).
1010

1111
Before you look into these options, check if EESSI is already accessible on your system.
1212

1313
Run the following command:
1414
``` { .bash .copy }
15-
ls /cvmfs/pilot.eessi-hpc.org
15+
ls /cvmfs/software.eessi.io
1616
```
1717

1818
!!! note
@@ -22,26 +22,26 @@ ls /cvmfs/pilot.eessi-hpc.org
2222

2323
If you see output like shown below, **you already have access to EESSI on your system**. :tada:
2424
```
25-
host_injections latest versions
25+
README.eessi defaults host_injections init versions
2626
```
2727

2828
For starting to use EESSI, continue reading about
29-
[Setting up environment](eessi-usage.md#setting-up-environment).
29+
[Setting up environment](eessi-usage.md#setting-up-your-environment).
3030

3131
If you see an error message as shown below, **EESSI is not yet accessible on your
3232
system**.
3333
```
34-
ls: /cvmfs/pilot.eessi-hpc.org: No such file or directory
34+
ls: /cvmfs/software.eessi.io: No such file or directory
3535
```
3636
No worries, you don't need to be a :mage: to get access to EESSI.
3737

3838
Continue reading about the [Native installation](#native-installation) of EESSI,
39-
or access via the [EESSI container](#eessi-container).
39+
or accessing [EESSI via a container](#eessi-via-a-container).
4040

4141
## Native installation
4242

4343
Setting up native access to EESSI, that is a system-wide deployment that does not require workarounds like
44-
[using a container](../eessi_container), requires the installation and configuration of [CernVM-FS](https://cernvm.cern.ch/fs).
44+
[using a container](#eessi-via-a-container), requires the installation and configuration of [CernVM-FS](https://cernvm.cern.ch/fs).
4545

4646
This requires **admin privileges**, since you need to install CernVM-FS as an OS package.
4747

@@ -63,15 +63,13 @@ The good news is that all of this only requires a handful commands :astonished:
6363
# Installation commands for RHEL-based distros like CentOS, Rocky Linux, Almalinux, Fedora, ...
6464

6565
# install CernVM-FS
66-
sudo yum install -y https://ecsft.cern.ch/dist/cvmfs/cvmfs-release/cvmfs-release-latest.noarch.rpm
66+
sudo yum install -y https://cvmrepo.s3.cern.ch/cvmrepo/yum/cvmfs-release-latest.noarch.rpm
6767
sudo yum install -y cvmfs
6868

69-
# install EESSI configuration for CernVM-FS
70-
sudo yum install -y https://github.com/EESSI/filesystem-layer/releases/download/latest/cvmfs-config-eessi-latest.noarch.rpm
71-
7269
# create client configuration file for CernVM-FS (no squid proxy, 10GB local CernVM-FS client cache)
7370
sudo bash -c "echo 'CVMFS_CLIENT_PROFILE="single"' > /etc/cvmfs/default.local"
7471
sudo bash -c "echo 'CVMFS_QUOTA_LIMIT=10000' >> /etc/cvmfs/default.local"
72+
sudo bash -c "echo 'CVMFS_USE_CDN=yes' >> /etc/cvmfs/default.local"
7573

7674
# make sure that EESSI CernVM-FS repository is accessible
7775
sudo cvmfs_config setup
@@ -83,20 +81,18 @@ The good news is that all of this only requires a handful commands :astonished:
8381
# Installation commands for Debian-based distros like Ubuntu, ...
8482

8583
# install CernVM-FS
86-
sudo apt-get install lsb-release
87-
wget https://ecsft.cern.ch/dist/cvmfs/cvmfs-release/cvmfs-release-latest_all.deb
84+
sudo apt-get update
85+
sudo apt-get install -y lsb-release wget
86+
wget https://cvmrepo.s3.cern.ch/cvmrepo/apt/cvmfs-release-latest_all.deb
8887
sudo dpkg -i cvmfs-release-latest_all.deb
8988
rm -f cvmfs-release-latest_all.deb
9089
sudo apt-get update
9190
sudo apt-get install -y cvmfs
9291

93-
# install EESSI configuration for CernVM-FS
94-
wget https://github.com/EESSI/filesystem-layer/releases/download/latest/cvmfs-config-eessi_latest_all.deb
95-
sudo dpkg -i cvmfs-config-eessi_latest_all.deb
96-
9792
# create client configuration file for CernVM-FS (no squid proxy, 10GB local CernVM-FS client cache)
98-
sudo bash -c "echo 'CVMFS_CLIENT_PROFILE="single"' > /etc/cvmfs/default.local"
93+
sudo bash -c "echo 'CVMFS_CLIENT_PROFILE=single' > /etc/cvmfs/default.local"
9994
sudo bash -c "echo 'CVMFS_QUOTA_LIMIT=10000' >> /etc/cvmfs/default.local"
95+
sudo bash -c "echo 'CVMFS_USE_CDN=yes' >> /etc/cvmfs/default.local"
10096

10197
# make sure that EESSI CernVM-FS repository is accessible
10298
sudo cvmfs_config setup
@@ -114,53 +110,56 @@ The good news is that all of this only requires a handful commands :astonished:
114110
For more details on this, please refer to the
115111
[*Stratum 1 and proxies section* of the CernVM-FS tutorial](https://cvmfs-contrib.github.io/cvmfs-tutorial-2021/03_stratum1_proxies/).
116112

117-
## EESSI client container
113+
## EESSI via a container
118114

119-
The `eessi_container.sh` script provides a very easy yet versatile means
120-
to access EESSI.
115+
!!! warning "Prerequisite"
121116

122-
This page guides you through several example scenarios
123-
illustrating the use of the script.
124-
125-
### Prerequisites
126-
127-
- Apptainer 1.0.0 (_or newer_), or Singularity 3.7.x
117+
Apptainer 1.0.0 (_or newer_), or Singularity 3.7.x
118+
128119
- Check with `apptainer --version` or `singularity --version`
129120
- Support for the `--fusemount` option in the ``shell`` and ``run`` subcommands is required
130-
- Git
131-
- Check with `git --version`
132-
133-
### Preparation
134121

135-
Clone the [`EESSI/software-layer`](https://github.com/EESSI/software-layer.git)
136-
repository and change into the `software-layer` directory by running these commands:
122+
A small script can provide a very easy yet versatile means
123+
to access EESSI using a container.
137124

138-
``` { .bash .copy }
139-
git clone https://github.com/EESSI/software-layer.git
140-
cd software-layer
125+
```bash title="eessi-via-container.sh"
126+
--8<-- "scripts/eessi-via-container.sh"
141127
```
142128

129+
This page guides you through an example scenario
130+
illustrating the use of the script.
131+
132+
143133
### Quickstart
144134

145-
Run the `eessi_container` script (from the ``software-layer`` directory) to start a shell session in the EESSI container:
135+
Run the `eessi_via_container.sh` script to start a shell session in the container with EESSI available:
146136

147137
``` { .bash .copy }
148-
./eessi_container.sh
138+
./eessi_via_container.sh
149139
```
150140

151141
!!! Note
152142
Startup will take a bit longer the first time you run this because the container image is downloaded and converted.
153143

154144
You should see output like
155145
```
156-
Using /tmp/eessi.abc123defg as tmp storage (add '--resume /tmp/eessi.abc123defg' to resume where this session ended).
157-
Pulling container image from docker://ghcr.io/eessi/build-node:debian11 to /tmp/eessi.abc123defg/ghcr.io_eessi_build_node_debian11.sif
158-
Launching container with command (next line):
159-
singularity -q shell --fusemount container:cvmfs2 pilot.eessi-hpc.org /cvmfs/pilot.eessi-hpc.org /tmp/eessi.abc123defg/ghcr.io_eessi_build_node_debian11.sif
146+
INFO: Environment variable SINGULARITY_BIND is set, but APPTAINER_BIND is preferred
147+
INFO: Environment variable SINGULARITY_HOME is set, but APPTAINER_HOME is preferred
148+
INFO: Converting OCI blobs to SIF format
149+
INFO: Starting build...
150+
INFO: Fetching OCI image...
151+
161.9KiB / 161.9KiB [===============================================================================] 100 % 3.1 MiB/s 0s
152+
41.2MiB / 41.2MiB [=================================================================================] 100 % 3.1 MiB/s 0s
153+
5.2MiB / 5.2MiB [===================================================================================] 100 % 3.1 MiB/s 0s
154+
68.8MiB / 68.8MiB [=================================================================================] 100 % 3.1 MiB/s 0s
155+
117.8MiB / 117.8MiB [===============================================================================] 100 % 3.1 MiB/s 0s
156+
88.4MiB / 88.4MiB [=================================================================================] 100 % 3.1 MiB/s 0s
157+
INFO: Extracting OCI image...
158+
2026/05/31 08:50:54 warn rootless{usr/libexec/openssh/ssh-keysign} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
159+
INFO: Inserting Apptainer configuration...
160+
INFO: Creating SIF file...
161+
[=============================================================================================================] 100 % 0s
160162
CernVM-FS: pre-mounted on file descriptor 3
161-
Apptainer> CernVM-FS: loading Fuse module... done
162-
fuse: failed to clone device fd: Inappropriate ioctl for device
163-
fuse: trying to continue without -o clone_fd.
164163
165164
Apptainer>
166165
```
@@ -169,19 +168,14 @@ Apptainer>
169168
beginning with `CernVM-FS: ` have been printed after the first prompt
170169
`Apptainer> ` was shown.
171170

172-
In this environment, you should be able to access the EESSI pilot repository:
171+
In this environment, you should be able to access the EESSI `software.eessi.io` repository:
173172

174173
``` { .bash .copy }
175-
ls /cvmfs/pilot.eessi-hpc.org
174+
ls /cvmfs/software.eessi.io
176175
```
177176

178-
More information on using the `eessi_container` script is available in the [EESSI documentation](https://eessi.github.io/docs/getting_access/eessi_container/).
179-
180-
181177
---
182178

183179
To start using EESSI, see [Using EESSI](eessi-usage.md).
184180

185-
186-
187181
[*next: Using EESSI*](eessi-usage.md) - [*(back to overview page)*](index.md)

docs/eessi-introduction.md

Lines changed: 12 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Introduction to EESSI
22

3-
<p align="center"><a href="https://eessi.github.io/docs"><img src="../EESSI_logo_horizontal_transparant.png" alt="EESSI logo" width="600px"/></a></p>
3+
<p align="center"><a href="https://eessi.github.io/docs"><img src="img/EESSI_logo_horizontal_transparant.png" alt="EESSI logo" width="600px"/></a></p>
44

55
## What is EESSI?
66

@@ -16,7 +16,7 @@ EESSI should work on laptops, personal workstations, HPC clusters and in the clo
1616
We hope to make this work for any Linux distribution, and maybe even macOS and Windows via [WSL](https://docs.microsoft.com/en-us/windows/wsl/),
1717
and a wide variety of CPU architectures (Intel, AMD, ARM, POWER, RISC-V).
1818

19-
We focus point on the **performance** of the provided software installations, but also on automating the workflow
19+
We focus not only the **performance** of the provided software installations, but also on automating the workflow
2020
for maintaining the software stack, thoroughly testing the installations, and collaborating efficiently.
2121

2222
## Inspiration
@@ -25,7 +25,7 @@ The EESSI concept is heavily inspired by Compute Canada (now [Digital Research A
2525
which is a shared software stack used on all 5 major national systems (and a bunch of smaller ones) in Canada.
2626

2727
The design of the Compute Canada software stack is discussed in detail
28-
in the PEARC'19 paper [*"Providing a Unified Software Environment for Canadas
28+
in the PEARC'19 paper [*"Providing a Unified Software Environment for Canada's
2929
National Advanced Computing Centers"*](https://ssl.linklings.net/conferences/pearc/pearc19_program/views/includes/files/pap139s3-file1.pdf).
3030

3131
It has also been presented at the 5th EasyBuild User Meetings ([slides](https://users.ugent.be/~kehoste/eum20/eum20_03_maxime_computecanada.pdf), [recorded talk](https://www.youtube.com/watch?v=_0j5Shuf2uE&list=PLhnGtSmEGEQidEM8MZKkOaVutgt9WmqI0)), and is [well documented](https://docs.computecanada.ca/wiki/Accessing_CVMFS).
@@ -34,7 +34,7 @@ It has also been presented at the 5th EasyBuild User Meetings ([slides](https://
3434

3535
The EESSI project consists of 3 layers.
3636

37-
![EESSI overview](EESSI-overview-layers.png)
37+
![EESSI overview](img/EESSI-overview-layers.png)
3838

3939
The bottom layer is the **filesystem layer**,
4040
which is responsible for distributing the software stack across clients.
@@ -49,18 +49,18 @@ The host OS still provides a couple of things, like drivers for network and GPU,
4949

5050
### Filesystem layer
5151

52-
<img src="../EESSI-filesystem-layer.png" alt="EESSI filesystem layer" width="400px"/>
52+
<img src="img/EESSI-filesystem-layer.png" alt="EESSI filesystem layer" width="400px"/>
5353

5454
The bottom layer of the EESSI project is the **filesystem layer**,
5555
which is responsible for distributing the software stack.
5656

5757
For this we rely on [CernVM-FS](https://cernvm.cern.ch/portal/filesystem) (or CVMFS for short), a network file system used to distribute the software to the clients in a fast, reliable and scalable way.
5858

59-
CVMFS was created by CERN over a decade ago, specifically for the purpose of globally distributing a large software stack. For the experiments at the Large Hadron Collider, it hosts several hundred million files and directories that are distributed to the order of a 100,000 client computers.
59+
CVMFS was created by CERN over a decade ago, specifically for the purpose of globally distributing a large software stack. For the experiments at the Large Hadron Collider, it hosts several hundred million files and directories that are distributed to more than a 100,000 client computers.
6060

61-
<img src="../EESSI-cvmfs-hierarchy.png" alt="CernVM-FS hierarchy" width="600px"/>
61+
<img src="img/EESSI-cvmfs-hierarchy.png" alt="CernVM-FS hierarchy" width="600px"/>
6262

63-
The hierarchical structure with multiple caching layers (Stratum-0, Stratum-1's located at partner sites, and local caching proxies) ensures good performance with limited resources. Redundancy is provided by using multiple Stratum-1's at various sites. Since CVMFS is based on the HTTP protocol, the ubiquitous [Squid caching proxy](http://www.squid-cache.org/) can be leveraged to reduce server loads and improve performance at large installations (such as HPC clusters). Clients can easily mount the file system (read-only) via a [FUSE](https://en.wikipedia.org/wiki/Filesystem_in_Userspace) (Filesystem in Userspace) module.
63+
The hierarchical structure with multiple caching layers (Stratum-0, Stratum-1's located at partner sites together with additional CDN via Cloudflare, and local caching proxies) ensures good performance with limited resources. Redundancy is provided by using multiple Stratum-1's at various sites. Since CVMFS is based on the HTTP protocol, the ubiquitous [Squid caching proxy](http://www.squid-cache.org/) can be leveraged to reduce server loads and improve performance at large installations (such as HPC clusters). Clients can easily mount the file system (read-only) via a [FUSE](https://en.wikipedia.org/wiki/Filesystem_in_Userspace) (Filesystem in Userspace) module.
6464

6565
For a (basic) introduction to CernVM-FS, see [this presentation](https://www.youtube.com/watch?v=MyYx-xaL36k).
6666

@@ -69,7 +69,7 @@ Detailed information about how we configure CVMFS is available at
6969

7070
### Compatibility layer
7171

72-
<img src="../EESSI-compat-layer.png" alt="EESSI compatibility layer" width="400px"/>
72+
<img src="img/EESSI-compat-layer.png" alt="EESSI compatibility layer" width="400px"/>
7373

7474
The middle layer of the EESSI project is the **compatibility layer**,
7575
which ensures that our scientific software stack is compatible with
@@ -84,7 +84,7 @@ The compatible layer is maintained via our [https://github.com/EESSI/compatibili
8484

8585
### Software layer
8686

87-
<img src="../EESSI-software-layer.png" alt="EESSI software layer" width="400px"/>
87+
<img src="img/EESSI-software-layer.png" alt="EESSI software layer" width="400px"/>
8888

8989
The top layer of the EESSI project is the **software layer**,
9090
which provides the actual scientific software installations.
@@ -99,13 +99,12 @@ To access these software installation we provide *environment module files*
9999
and use [**Lmod**](https://lmod.readthedocs.io), a modern environment modules tool which has been widely
100100
adopted in the HPC community in recent years.
101101

102-
We leverage the [**archspec**](https://archspec.readthedocs.io) Python library
103-
to automatically select the best suited part of the software stack for
102+
We use custom architecture detection scripts
103+
to automatically select the best suited installations from the software stack for
104104
a particular host, based on its system architecture.
105105

106106
The software layer is maintained through our [https://github.com/EESSI/software-layer](https://github.com/EESSI/software-layer) GitHub repository.
107107

108-
109108
## Current status
110109

111110
The EESSI project was started mid 2020, as a loose collaboration between various Dutch universities including

0 commit comments

Comments
 (0)