add source caching

- rename sourcecache to sourcemirror (for pre-populated source mirrors)
- add sourcecache for read through source caching
    - after downloading a source, add it to the cache so that it does
      not have to be downloaded again.

Author: bcumming

docs/build-caches.md

@@ -1,101 +1,189 @@
-# Build Caches
+# Mirrors and Build Caches
+
+Spack can use *mirrors* and *caches* to speed up image builds and to build on systems with limited or no internet access.
+They are configured in a single `mirrors.yaml` file in the [system configuration](cluster-config.md).
+
+A `mirrors.yaml` can describe four kinds of entry, each optional and each documented below:
+
+| Entry | Count | Purpose |
+|-------|-------|---------|
+| [`buildcache`](#build-cache)   | one  | binary cache of built packages (the big build-time speed up) |
+| [`bootstrap`](#bootstrap-mirror) | one  | mirror used to bootstrap Spack itself |
+| [`sourcemirror`](#source-mirrors) | many | read-only mirrors that provide package sources |
+| [`sourcecache`](#source-cache)  | one  | writable local cache that fills with sources as you build |
+
+A complete example:
+
+```yaml title="mirrors.yaml"
+buildcache:
+  url: file:///capstor/scratch/team/uenv-cache
+  private_key: $SCRATCH/.keys/spack-push-key.gpg
+  mount_specific: true
+bootstrap:
+  url: https://bootstrap.spack.io
+sourcemirror:
+  mirror1:
+    url: https://example.com/spack-sources
+sourcecache:
+  path: /capstor/scratch/$USER/spack-sources
+```
 
-Stackinator facilitates using Spack's binary build caches to speed up image builds.
-Build caches are essential if you plan to build images regularly, as they generally lead to a roughly 10x speed up.
-This is the difference between half an hour or 3 minutes to build a typical image.
+To stop using any entry, remove (or comment out) it from `mirrors.yaml`.
 
-## Using Build caches
+## Build cache
 
-To use a build cache, create a simple YAML file:
+A build cache is a binary cache of built packages.
+Reusing binaries instead of rebuilding from source is roughly a 10x speed up — the difference between a 3 minute and a 30 minute image build — so a build cache is essential if you build images regularly.
 
-```yaml title='cache-config.yaml'
-root: $SCRATCH/uenv-cache
-key:  $SCRATCH/.keys/spack-push-key.gpg
+During a build Spack fetches packages from the cache when it can, and signs and pushes any package it has to build itself, so the cache improves over time.
+
+```yaml title="mirrors.yaml"
+buildcache:
+  url: file:///capstor/scratch/team/uenv-cache
+  private_key: $SCRATCH/.keys/spack-push-key.gpg
 ```
 
-To use the cache, pass the configuration as an option to `stack-config` via the `-c/--cache` flag:
+| Field | Required | Description |
+|-------|----------|-------------|
+| `url`            | yes | location of the cache (a `file://` path, or an `http(s)://`, `s3://` or `oci://` URL) |
+| `private_key`    | yes | PGP key used to sign and push packages (see [Keys](#keys)) |
+| `public_key`     | no  | PGP key used to verify downloaded packages |
+| `name`           | no  | name Spack registers the mirror under (default `buildcache`) |
+| `mount_specific` | no  | store the cache in a per-mount-point sub-directory (default `false`) |
+
+### `mount_specific`
+
+Spack binaries embed the install prefix (the image's mount point), so binaries built for `/user-environment` cannot be reused at a different mount point.
+Set `mount_specific: true` to append the mount point to the cache URL, giving each mount point its own sub-directory and avoiding relocation issues:
+
+```yaml
+buildcache:
+  url: file:///capstor/scratch/team/uenv-cache
+  private_key: $SCRATCH/.keys/spack-push-key.gpg
+  mount_specific: true   # packages stored under .../uenv-cache/user-environment
+```
+
+### Creating a build cache
+
+A build cache needs an empty directory and a PGP signing key:
 
 ```bash
-stack-config -b $build_path -r $recipe_path -s $system_config -c cache-config.yaml
+# 1. create the cache directory
+mkdir -p $SCRATCH/uenv-cache
+
+# 2. generate and export a signing key
+spack gpg create <name> <e-mail>
+spack gpg export --secret $SCRATCH/.keys/spack-push-key.gpg
 ```
 
-??? warning "If you using an old binary build cache"
-    Since v3, Stackinator creates a sub-directory in the build cache for each mount point.
-    For example, in the above example, the build cache for the `/user-environment` mount point would be `$SCRATCH/uenv-cache/user-environment`.
-    The rationale for this is so that packages for different mount points are not mixed, to avoid having to relocate binaries.
+See [Keys](#keys) for where to store the key.
+
+### Force pushing
+
+Packages are pushed to the cache after each environment builds successfully; nothing is pushed if a build fails.
+When iterating on a recipe with failing builds, force-push everything built so far with the `cache-force` target:
 
-    To continue using a build caches from before v3, first copy the `build_cache` path to a subdirectory, e.g.:
+```bash
+env --ignore-environment PATH=/usr/bin:/bin:`pwd -P`/spack/bin make cache-force
+```
 
-    ```bash
-    mkdir $SCRATCH/uenv-cache/user-environment
-    mv $SCRATCH/uenv-cache/build_cache $SCRATCH/uenv-cache/user-environment
-    ```
+## Bootstrap mirror
 
-### Build-only caches
+Spack bootstraps some of its own dependencies (such as the `clingo` concretizer) on first use.
+A bootstrap mirror lets it do this without reaching the internet — useful on air-gapped systems.
 
-A build cache can be configured to be read-only by not providing a `key` in the cache configuration file.
+```yaml title="mirrors.yaml"
+bootstrap:
+  url: https://bootstrap.spack.io
+```
 
-## Creating a Build Cache
+| Field | Required | Description |
+|-------|----------|-------------|
+| `url`        | yes | location of the bootstrap mirror |
+| `public_key` | no  | PGP key used to verify the bootstrap binaries |
 
-To create a build cache we need two things:
+## Source mirrors
 
-1. An empty directory where the cache will be populated by Spack.
-2. A private PGP key
-    *  Only required for Stackinator to push packages to the cache when it builds a package that was not in the cache.
+Source mirrors provide package **source** archives, and are read-only: Spack fetches sources from them but never writes to them.
+Use them to build on air-gapped systems — populate a mirror on a connected system, mount it read-only, and Spack will fetch sources from it.
+Any number of source mirrors can be listed; Spack searches them in order.
 
-Creating the cache directory is easy! For example, to create a cache on your scratch storage:
-```bash
-mkdir $SCRATCH/uenv-cache
+```yaml title="mirrors.yaml"
+sourcemirror:
+  internal:
+    url: https://mirror.example.com/spack-sources
+  scratch:
+    url: file:///capstor/scratch/team/spack-sources
 ```
 
-### Generating Keys
+| Field | Required | Description |
+|-------|----------|-------------|
+| `url`        | yes | location of the source mirror |
+| `public_key` | no  | PGP key used to verify sources |
 
-An installation of Spack can be used to generate the key file:
+Populate a source mirror on an internet-connected system with Spack:
 
 ```bash
-# create a key
-spack gpg create <name> <e-mail>
+spack mirror create --directory /path/to/mirror --all
+```
+
+## Source cache
 
-# export key
-spack gpg export --secret spack-push-key.gpg
+A source cache is a single, **writable** local directory that Spack fills as it downloads sources.
+On internet-connected systems Spack checks the cache first; on a miss it downloads the source and stores it, so later builds reuse it and download times shrink over time.
+
+Unlike a source mirror it needs no key, is written to automatically, and is created on demand.
+
+```yaml title="mirrors.yaml"
+sourcecache:
+  path: /capstor/scratch/$USER/spack-sources
 ```
 
-See the [spack documentation](https://spack.readthedocs.io/en/latest/getting_started.html#gpg-signing) for more information about GPG keys.
+| Field | Required | Description |
+|-------|----------|-------------|
+| `path` | yes | absolute path to a local directory (environment variables are expanded) |
 
-### Managing Keys
+## Keys
 
-The key needs to be in a location that is accessible during the build process, and secure.
-To keep your PGP key secret, you can generate it then move it to a path with appropriate permissions.
-In the example below, we create a path `.keys` for storing the key:
-```bash
-# create  .keys path is visible only to you
-mkdir $SCRATCH/.keys
-chmod 700 $SCRATCH/.keys
+The `private_key` and `public_key` fields accept either:
+
+* a **path** — absolute, or relative to the system configuration directory; or
+* a **base64-encoded key** inlined directly in `mirrors.yaml`.
+
+```yaml
+buildcache:
+  url: file:///capstor/scratch/team/uenv-cache
+  private_key: $SCRATCH/.keys/spack-push-key.gpg     # a path
+  public_key: mQINBGm4GvsBEACTyzQF...==              # inline base64
+```
+
+Generate a key with Spack, and keep the secret key somewhere private:
 
-# generate the key
+```bash
+mkdir $SCRATCH/.keys && chmod 700 $SCRATCH/.keys
 spack gpg create <name> <e-mail>
 spack gpg export --secret $SCRATCH/.keys/spack-push-key.gpg
 chmod 600 $SCRATCH/.keys/spack-push-key.gpg
 ```
 
-The cache-configuration would look like the following, where we assume that the cache is in `$SCRATCH/uenv-cache`:
-```yaml
-root: $SCRATCH/uenv-cache
-key: $SCRATCH/.keys/spack-push-key.gpg
-```
-!!! warning
-    Don't blindly copy this documentation's advice on security settings.
+See the [Spack documentation](https://spack.readthedocs.io/en/latest/getting_started.html#gpg-signing) for more on GPG keys.
 
 !!! failure "Don't use `$HOME`"
-    Don't put the keys in `$HOME`, because the build process remounts `~` as a tmpfs, and you will get error messages that Spack can't read the key.
+    The build remounts `~` as a tmpfs, so keys under `$HOME` are not visible during the build and Spack will fail to read them. Use scratch storage instead.
 
-## Force pushing to build cache
+## Legacy `--cache` option
 
-When build caches are enabled, all packages in a each Spack environment are pushed to the build cache after the whole environment has been built successfully -- nothing will be pushed to the cache if there is an error when building one of the packages.
+Before `mirrors.yaml`, a build cache was configured with a separate `cache.yaml` file passed to `stack-config` via `-c/--cache`:
 
-When debugging a recipe, where failing builds have to be run multiple times, the overheads of rebuilding all packages from scratch can be wasteful.
-To force push all packages that have been built, use the `cache-force` makefile target:
+```yaml title="cache.yaml"
+root: $SCRATCH/uenv-cache
+key:  $SCRATCH/.keys/spack-push-key.gpg
+```
 
 ```bash
-env --ignore-environment PATH=/usr/bin:/bin:`pwd -P`/spack/bin make cache-force
+stack-config -b $build -r $recipe -s $system -c cache.yaml
 ```
+
+This is **deprecated** and equivalent to a single `buildcache` entry (with `mount_specific: true`). Prefer configuring the build cache in `mirrors.yaml`.
+
+Setting `key: null` configures a read-only cache that Spack fetches from but never pushes to.

docs/cluster-config.md

@@ -95,29 +95,10 @@ packages:
 
 ### Configuring Spack mirrors: `mirrors.yaml`
 
-On air-gapped systems, Spack is unable to reach its default mirror to fetch packages. The `mirrors.yaml` configuration can be used to connect Spack to local mirrors for fetching and building packages.
+An optional `mirrors.yaml` connects Spack to local mirrors and caches, to speed up builds and to build on air-gapped systems.
+It can configure a build cache, a bootstrap mirror, read-only source mirrors, and a writable source cache.
 
-`mirrors.yaml` treats source mirrors, buildcaches, and bootstrap mirrors the same, and they may all be included in this file. Spack will search the topmost mirror first and the bottom-most mirror last, and will append the default Spack mirror to the bottom of the list when the Spack mirror config is generated.
-
-`mirrors.yaml` may include source mirrors, bootstrap mirrors, and buildcaches. Any number of source mirrors can be added, but only one bootstrap mirror and buildcache can be specified. Spack will search the source mirrors in order from first to last, and will append the default Spack mirror to the bottom of the list when the Spack mirror config is generated.
-
-If using a buildcache, a private key must be provided for signing packages. An optional public key may be specified with any type of mirror to verify packages. The buildcache is registered with Spack under the name `buildcache`, which can be overridden with an optional `name` field.
-
-To stop using a mirror, remove (or comment out) its entry from `mirrors.yaml`.
-
-```yaml title="mirrors.yaml"
-bootstrap:
-  url: file:///home/username/spack-mirror-2014-06-24
-buildcache:
-  url: https://example.com/some/buildcache/mirror
-  private_key: /user-home/.gnupg/private-keys-v1.d/my-private-key.asc
-sourcecache:
-  mirror1:
-    url: https://example.com/some/web-hosted/directory
-    public_key: ../buildcache-key.public.gpg
-  mirror2:
-    url: https://example.com/some/other-web-hosted/directory
-```
+See [Mirrors and Build Caches](build-caches.md) for the full reference and examples.
 
 ## Site and System Configurations
 

memory.md

@@ -0,0 +1 @@
+claude --resume 71a36db9-9a83-4261-bad1-855caaca9f7e

mkdocs.yml

@@ -8,7 +8,7 @@ nav:
     - 'Recipes': recipes.md
     - 'Cluster Configuration': cluster-config.md
     - 'Interfaces': interfaces.md
-    - 'Build Caches': build-caches.md
+    - 'Mirrors & Build Caches': build-caches.md
     - 'Spack 1.0': porting.md
   - 'Development': development.md
 # - Tutorial: tutorial.md

stackinator/builder.py

@@ -246,6 +246,7 @@ def generate(self, recipe):
                     spack_meta=spack_meta,
                     gpg_keys=recipe.mirrors.gpg_key_paths(config_path),
                     cache=recipe.build_cache_mirror,
+                    buildcache_push=recipe.push_to_build_cache,
                     exclude_from_cache=["nvhpc", "cuda", "perl"],
                     verbose=False,
                 )