Merge remote-tracking branch 'origin/main' into feat/modular-sqfs

Author: simonpintarelli

.flake8

@@ -4,7 +4,7 @@ on:
   push:
     branches:
       - feature/docs
-      - master
+      - main
 permissions:
   contents: write
 jobs:

.github/workflows/docs.yaml

@@ -1,25 +1,16 @@
 name: lint
 
-on:
-  pull_request:
-    branches: [master, main]
+on: [push, pull_request]
 
 jobs:
   lint:
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v3
-    - name: Set up Python
-      uses: actions/setup-python@v4
-      with:
-        python-version: "3.10"
-    - name: Install Tools
+    - name: Install uv
       run: |
-        python -m pip install --upgrade pip
-        python -m pip install black flake8 mypy
-    - name: Black
+        curl -LsSf https://astral.sh/uv/install.sh | sh
+    - name: ruff
       run: |
-        black --check --verbose stackinator unittests
-    - name: flake8
-      run: |
-        flake8 --count --show-source --statistics .
+        uvx ruff format --check
+        uvx ruff check

.github/workflows/lint.yaml

@@ -3,20 +3,13 @@ name: Stackinator CI
 on: [push, pull_request]
 
 jobs:
-  unittestpy36:
-    runs-on: ubuntu-20.04
-    strategy:
-      matrix:
-        python-version: ['3.6']
+  unittestpy:
+    runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v3
-    - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v4
-      with:
-        python-version: ${{ matrix.python-version }}
-    - name: Bootstrap
+    - name: Install uv
       run: |
-        ./bootstrap.sh
+        curl -LsSf https://astral.sh/uv/install.sh | sh
     - name: Generic Unittests
       run: |
         ./test_stackinator.py

.github/workflows/main.yaml

@@ -1,12 +1,18 @@
-#!/usr/bin/env python3
-
+#!/usr/bin/env -S uv run --script
+# /// script
+# requires-python = ">=3.12"
+# dependencies = [
+#   "jinja2",
+#   "jsonschema",
+#   "pyYAML",
+# ]
+# ///
 
 import pathlib
 import sys
 
 prefix = pathlib.Path(__file__).parent.parent.resolve()
-external = prefix / 'external'
-sys.path = [prefix.as_posix(), external.as_posix()] + sys.path
+sys.path = [prefix.as_posix()] + sys.path
 
 from stackinator.main import main
 

bin/stack-config

@@ -16,6 +16,15 @@ cd stackinator
 ./bootstrap.sh
 ```
 
+!!! warning
+    The `main` branch of Stackinator includes features for Spack v1.0, and may break older recipes.
+
+    For existing recipes use Spack v0.23 and earlier, use [version 5](#versions):
+
+    ```bash
+    git clone --branch=releases/v5 https://github.com/eth-cscs/stackinator.git
+    ```
+
 The `bootstrap.sh` script will install the necessary dependencies, so that Stackinator can be run as a standalone application.
 
 Once installed, add the `bin` sub-directory to your path:
@@ -36,6 +45,18 @@ pip install stackinator
     The PyPi package is only updated for releases, so you will likely be missing the latest and greatest features.
     Let us know if you need more regular PyPi updates.
 
+### Versions
+
+Stackinator version 6 will be the first release of Stackinator to support Spack 1.0, when it is released in June 2025.
+There will be significant changes introduced in Spack 1.0, which will require making some non-trivial changes to Stackinator, and possibly adding breaking changes to the Stackinator recipe specification.
+
+The git branch `releases/v5` will be maintained to provide support for all versions 0.21, 0.22 and 0.23 of Spack and existing recipes.
+
+The `main` branch of Stackinator will contain 
+
+!!! warning
+    After the release of version 5, the main development branch was changed from `master` to `main`.
+
 ## Quick Start
 
 Stackinator generates the make files and spack configurations that build the spack environments that are packaged together in the spack stack.

bootstrap.sh

@@ -1,15 +1,11 @@
 The documentation for Stackinator is built from the markdown files in this path using MkDocs and MkDocs-material.
 You can view the latest documentation online at [github.io](https://eth-cscs.github.io/stackinator/)
 
-To build a copy locally, first install `mkdocs-material`, e.g.:
+To view work in progress docs, run the serve script and follow the link it provides to view a local copy of the docs in your browser.
 ```bash
-python3 -m venv docs-env
-source docs-env/bin/activate
-pip install mkdocs-material
+./serve
 ```
 
-Then in the root of this project, build the docs and view them with your favourite browser:
-```bash
-mkdocs build
-firefox site/index.html
-```
+> [!IMPORTANT]
+> to run the serve script, you need to first install [uv](https://docs.astral.sh/uv/getting-started/installation/).
+

docs/index.md

@@ -25,13 +25,32 @@ spack:
     commit: releases/v0.20
 modules: true
 description: "HPC development tools for building MPI applications with the GNU compiler toolchain"
+version: 2
 ```
 
 * `name`: a plain text name for the environment
 * `store`: the location where the environment will be mounted.
 * `spack`: which spack repository to use for installation.
 * `modules`: _optional_ enable/diasble module file generation (default `true`).
 * `description`: _optional_ a string that describes the environment (default empty).
+* `version`:  _default = 1_ the version of the uenv recipe (see below)
+
+!!! note "uenv recipe versions"
+    Stackinator 6 introduces breaking changes to the uenv recipe format, introduced to support Spack v1.0.
+
+    We have started versioning uenv recipes:
+
+    * **version 1**: original uenv recipes for Spack v0.23 and earlier, supported by Stackinator version 5.
+    * **version 2**: uenv recipes for Spack v1.0 and later, supported by Stackinator version 6.
+
+    The default version is 1, so that old recipes that do not set a version are supported.
+
+    !!! warning "You must set version 2 explicitly to use Spack v1.0"
+
+    !!! warning "Version 1 recipes must be configured using Stackinator v5"
+        Version 5 of Stackinator is maintained in the `releases/v5` branch of stackinator.
+
+        You must also use the `releases/v5` branch of [Alps cluster config](https://github.com/eth-cscs/alps-cluster-config).
 
 ## Compilers
 
@@ -100,19 +119,19 @@ In the following sections, we will explore each of the environment configuration
 The `compiler` field describes a list compilers to use to build the software stack.
 Each compiler toolchain is specified using toolchain and spec
 
-```yaml title="compile all packages with gcc@11.3"
+```yaml title="compile all packages with gcc@11"
   compiler:
   - toolchain: gcc
-    spec: gcc@11.3
+    spec: gcc@11
 ```
 
 Sometimes two compiler toolchains are required, for example when using the `nvhpc` compilers, there are often dependencies that can't be built using the NVIDIA, or are better being built with GCC (for example `cmake`, `perl` and `netcdf-c`).
-The example below uses the `nvhpc` compilers with gcc@11.3.
+The example below uses the `nvhpc` compilers with `gcc@11`.
 
-```yaml title="compile all packages with gcc@11.3"
+```yaml title="compile all packages with gcc@11"
   compiler:
   - toolchain: gcc
-    spec: gcc@11.3
+    spec: gcc@11
   - toolchain: llvm
     spec: nvhpc@22.7
 ```
@@ -123,11 +142,14 @@ The example below uses the `nvhpc` compilers with gcc@11.3.
 !!! warning
     Stackinator does not test or support using two versions of gcc in the same toolchain.
 
+!!! note
+    It is generally advisable not to overspecify compiler version, so whenever possible constrain at most the major version.
+
 The order of the compilers is significant. The first compiler is the default, and the other compilers will only be used to build packages when explicitly added to a spec.
 For example, in the recipe below, only `netcdf-fortran` will be built with the `nvhpc` toolchain, while the root specs `cmake` and `netcdf-c` and all dependencies will be built using the `gcc` toolchain.
 
 
-```yaml title="compile all packages with gcc@11.3"
+```yaml title="compile all packages with gcc@11"
   compiler:
   - toolchain: gcc
     spec: gcc
@@ -235,7 +257,7 @@ cuda-env:
     Use `unify:true` when possible, then `unify:when_possible`, and finally `unify:false`.
 
 !!! warning
-    Don't provide a spec for MPI or Compilers, which are configured in the [`mpi:`](recipes.md#mpi) and [`compilers`](recipes.compilers) fields respecively.
+    Don't provide a spec for MPI or Compilers, which are configured in the [`mpi:`](recipes.md#mpi) and [`compilers`](recipes.md#compilers) fields respecively.
 
 !!! warning
     Stackinator does not support "spec matrices", and likely won't, because they use multiple compiler toolchains in a manner that is contrary to the Stackinator "keep it simple" principle.
@@ -358,6 +380,22 @@ Additional custom packages can be provided as part of the cluster configuration,
 These packages are all optional, and will be installed together in a single Spack package repository that is made available to downstream users of the generated uenv stack.
 See the documentation for [cluster configuration](cluster-config.md) for more detail.
 
+!!! note
+    If you need to backport a spack package from a more recent spack version, you can do it by using an already checked out spack repository like this
+
+    (disclaimer: the package might need adjustments due to spack directives changes)
+
+    ```
+    # ensure to have the folder for custom packages in your recipe
+    mkdir -p stackinator-recipe/repo/packages
+    # switch to the already checked out spack repository
+    cd $SPACK_ROOT
+    # use git to extract package files into your "custom packages" section of the stackinator recipe
+    git archive origin/develop `spack location -p fmt` | tar -x --strip-components=5 -C stackinator-recipe/repo/packages
+    ```
+
+    In the above case, the package `fmt` is backported from `origin/develop` into the `stackinator-recipe`.
+
 !!! alps
     All packages are installed under a single spack package repository called `alps`.
     The CSCS configurations in [github.com/eth-cscs/alps-cluster-config](https://github.com/eth-cscs/alps-cluster-config) provides a site configuration that defines cray-mpich, its dependencies, and the most up to date versions of cuda, nvhpc etc to all clusters on Alps.

docs/readme.md

@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+
+uvx ruff format
+uvx ruff check --fix