Lanl/mirrors (#286)

Replace the legacy -c/--cache mechanism with a unified --mirror flag
backed by a single mirrors.yaml describing five optional entities:
buildcache, bootstrap, sourcemirror, sourcecache, and concretizer cache.

- New Mirrors class (mirror.py) validates inputs and renders all spack
    config eagerly; the builder just writes the files
- Mirror config is now a CLI argument, not part of system config
- Support read-only (keyless) build caches, armored GPG keys, source
    caching, and persistent concretization caches
- Don't re-index the build cache on every push
- Replace docs/build-caches.md with a rewritten docs/mirrors.md
- Drop cache.py; add extensive mirror/recipe unit tests

Author: Paul-Ferrell

CLAUDE.md

@@ -5,3 +5,26 @@ A tool for building a scientific software stack from a recipe for vClusters on C
 Read the [documentation](https://eth-cscs.github.io/stackinator/) to get started.
 
 Create a ticket in our [GitHub issues](https://github.com/eth-cscs/stackinator/issues) if you find a bug, have a feature request or have a question.
+
+## running tests:
+
+Use uv to run the tests, which will in turn ensure that the correct dependencies from `pyproject.toml` are used:
+
+```
+uv run pytest
+```
+
+Before pushing, apply the linting rules (this calls uv under the hood):
+
+```
+./lint
+```
+## building the docs
+
+The documentation for Stackinator is built from the markdown files in the `docs` path using MkDocs and MkDocs-material.
+You can view the latest documentation online at [github.io](https://eth-cscs.github.io/stackinator/)
+
+To view work in progress docs, run the serve script and follow the link it provides to view a local copy of the docs in your browser.
+```bash
+./serve
+```

README.md

@@ -1,7 +1,8 @@
-#!/usr/bin/env -S uv run --script
+#!/usr/bin/env -S uv run --no-refresh --script
 # /// script
 # requires-python = ">=3.12"
 # dependencies = [
+#   "python-magic",
 #   "jinja2",
 #   "jsonschema",
 #   "pyYAML",

bin/stack-config

@@ -1,6 +1,7 @@
+[](){#ref-building}
 # Building Spack Stacks
 
-Once a stack has been [configured](configuring.md) using `stack-config`, it's time to build the software stack.
+Once a stack has been [configured][ref-configuring] using `stack-config`, it's time to build the software stack.
 
 ## How to Build
 
@@ -18,7 +19,7 @@ env --ignore-environment PATH=/usr/bin:/bin:`pwd -P`/spack/bin make modules stor
 The call to `make` is wrapped with with `env --ignore-env` to unset all environment variables, to improve reproducability of builds.
 
 Build times for stacks typically vary between 30 minutes to 3 hours, depending on the specific packages that have to be built.
-Using [build caches](build-caches.md) and building in shared memory (see below) are the most effective methods to speed up builds.
+Using [build caches][ref-mirrors] and building in shared memory (see below) are the most effective methods to speed up builds.
 
 ## Where to Build
 

docs/build-caches.md

@@ -1,3 +1,4 @@
+[](){#ref-cluster-config}
 # Cluster Configuration
 
 Spack stacks are built on bare-metal clusters using a minimum of dependencies from the underlying system.
@@ -10,7 +11,7 @@ A cluster configuration is a directory with the following structure:
 └─ repos.yaml       # optional reference to additional site packages
 ```
 
-The configuration is provided during the [configuration](configuring.md) step with the `--system/-s` flag.
+The configuration is provided during the [configuration][ref-configuring] step with the `--system/-s` flag.
 The following example targets the Clariden system at CSCS:
 
 ```bash
@@ -93,6 +94,13 @@ packages:
         version: ["git.59b6de6a91d9637809677c50cc48b607a91a9acb=main"]
     ```
 
+### Configuring Spack mirrors
+
+Mirrors and caches are **not** part of the system configuration.
+They are supplied per-invocation with `stack-config --mirror <file>`, because the locations involved (build caches, source caches) are often specific to the user running the build and may not be accessible to everyone using a system.
+
+See [Mirrors and Build Caches][ref-mirrors] for the full reference and examples.
+
 ## Site and System Configurations
 
 The `repo.yaml` configuration can be used to provide a list of additional Spack package repositories to use on the target system.

docs/building.md

@@ -1,3 +1,4 @@
+[](){#ref-configuring}
 # Configuring Spack Stacks
 
 Stackinator generates the make files and spack configurations that build the spack environments that are packaged together in the spack stack.
@@ -10,13 +11,14 @@ stack-config --build $BUILD_PATH --recipe $RECIPE_PATH --system $SYSTEM_CONFIG_P
 
 The following flags are required:
 
-* `-b/--build`: the path where the [build](building.md) is to be performed.
-* `-r/--recipe`: the path with the [recipe](recipes.md) yaml files that describe the environment.
-* `-s/--system`: the path containing the [system configuration](cluster-config.md) for the target cluster.
+* `-b/--build`: the path where the [build][ref-building] is to be performed.
+* `-r/--recipe`: the path with the [recipe][ref-recipes] yaml files that describe the environment.
+* `-s/--system`: the path containing the [system configuration][ref-cluster-config] for the target cluster.
 
 The following flags are optional:
 
-* `-c/--cache`: configure the [build cache](build-caches.md).
+* `--mirror`: path to a [mirrors.yaml][ref-mirrors] file configuring build caches and mirrors.
+* `-c/--cache`: legacy build cache configuration file (deprecated; use `--mirror`).
 * `-m/--mount`: override the [mount point](installing.md) where the stack will be installed.
 * `--version`: print the stackinator version.
 * `-h/--help`: print help message.

docs/cluster-config.md

@@ -1,3 +1,4 @@
+[](){#ref-installing}
 # Installing Stacks
 
 The installation path of the software stack is set when the stack is configured.
@@ -45,4 +46,6 @@ The `store` sub-directory contains the full software stack installation tree.
 ### SquashFS installation
 
 The `store.squashfs` file is a compressed [SquashFS](https://tldp.org/HOWTO/SquashFS-HOWTO/whatis.html) image of the contents of the `store` path.
-This can be mounted at runtime using [`squashfs-mount`](https://github.com/eth-cscs/squashfs-mount) or the [Slurm plugin](https://github.com/eth-cscs/slurm-uenv-mount/), or mounted by a system-administrator using [`mount`](https://man7.org/linux/man-pages/man2/mount.2.html), in order the to take advantage of the benefits of SquashFS over shared file systems.
+This can be mounted at runtime using [`uenv`](https://github.com/eth-cscs/uenv) or the uenv Slurm plugin.
+
+Images can also be mounted by a system-administrator using [`mount`](https://man7.org/linux/man-pages/man2/mount.2.html), which is used in production at CSCS to permanantly mount software stacks used by the weather service on their operational cluster when nodes boot.