misc cache -> explicit concretizer cache

Author: bcumming

CLAUDE.md

@@ -29,7 +29,7 @@ stackinator/           # Python package
     config.json
     compilers.json
     environments.json
-    mirror.json          # mirrors.yaml schema (buildcache/bootstrap/sourcemirror/sourcecache/misccache)
+    mirror.json          # mirrors.yaml schema (buildcache/bootstrap/sourcemirror/sourcecache/concretizer)
     cache.json           # legacy -c/--cache cache.yaml schema
     modules.json
   templates/           # Jinja2 templates for all generated files
@@ -208,7 +208,8 @@ BUILD/
     packages.yaml
     repos.yaml
     mirrors.yaml        # if --mirror provided: buildcache/sourcemirror entries
-    config.yaml         # if --mirror provided with a sourcecache/misccache (config:source_cache/misc_cache)
+    config.yaml         # if --mirror provided with a sourcecache (config:source_cache)
+    concretizer.yaml    # if --mirror provided with a concretizer cache (concretizer:concretization_cache), spack >= 1.1
     bootstrap.yaml      # if --mirror provided with a bootstrap mirror
     key_store/          # if --mirror provided with gpg keys (decoded *.gpg)
   compilers/
@@ -262,8 +263,8 @@ Writes all files to the build path. Key responsibilities:
 
 ### `Mirrors` class (`mirror.py`)
 A clean exemplar of the "recipe validates & renders, builder just prints" pattern. Constructed by `Recipe` from the `--mirror` file path; does ALL mirror input processing eagerly in `__init__` (loads + schema-validates `mirrors.yaml`, validates mirror urls, decodes/validates gpg keys to in-memory bytes, checks cache paths are absolute and expands env vars). Then presents pure static artifacts:
-- typed members: `buildcache`, `bootstrap`, `source_mirrors`, `source_cache`, `misc_cache`
-- `config_files(config_root) -> {abs_path: bytes}` — the `mirrors.yaml`, `config.yaml`, `bootstrap.yaml`, and gpg key files the builder writes verbatim
+- typed members: `buildcache`, `bootstrap`, `source_mirrors`, `source_cache`, `concretizer_cache`
+- `config_files(config_root) -> {abs_path: bytes}` — the `mirrors.yaml`, `config.yaml`, `concretizer.yaml`, `bootstrap.yaml`, and gpg key files the builder writes verbatim
 - `gpg_key_paths(config_root)` and the `build_cache_mirror` / `push_to_build_cache` properties (the latter is `None` for a keyless, read-only build cache)
 
 Mirror/cache config is supplied ONLY via `--mirror`; a `mirrors.yaml` found in the system config dir is rejected with an error (it was never a system-config artifact). Relative gpg-key paths resolve against the `--mirror` file's own directory.
@@ -310,9 +311,9 @@ Spack mirrors and caches are configured in a single `mirrors.yaml` supplied with
 - **`bootstrap`** (one): for bootstrapping Spack itself (clingo etc.). The `url` is either a local `spack bootstrap mirror` directory (referenced via its own `metadata/sources`+`metadata/binaries`) or a remote url (source-only). Needs **no key** (bootstrap binaries are sha256-verified) → emitted as `config/bootstrap.yaml` (+ a generated `metadata.yaml` only for the remote case); it is NOT a `mirrors.yaml` entry.
 - **`sourcemirror`** (many): read-only mirrors providing package source archives.
 - **`sourcecache`** (one): a writable local dir Spack fills as it fetches sources → emitted as `config:source_cache`.
-- **`misccache`** (one): a writable local dir for Spack's misc cache (package/build-cache indices and the **concretization cache** that lives under it) → emitted as `config:misc_cache`. Useful to persist across ephemeral builds. Concretization caching is on by default in Spack ≥ 1.2, opt-in in 1.1.
+- **`concretizer`** (one): a writable local dir persisting Spack's **concretization results** → emitted as `concretizer:concretization_cache:{enable,url}` in `config/concretizer.yaml`. Useful to persist across ephemeral builds. The config key requires Spack ≥ 1.1; Stackinator infers the Spack version from `config.yaml:spack.commit` (via `Recipe.find_spack_version`, defaulting to the latest supported version when the commit can't be pinned) and **skips the cache with a warning** for Spack 1.0 (which rejects the key).
 
-`sourcecache`/`misccache` are emitted to `config/config.yaml`; `bootstrap` to `config/bootstrap.yaml`; `buildcache`/`sourcemirror` to `config/mirrors.yaml` (+ decoded keys under `config/key_store/`).
+`sourcecache` is emitted to `config/config.yaml`; `concretizer` to `config/concretizer.yaml`; `bootstrap` to `config/bootstrap.yaml`; `buildcache`/`sourcemirror` to `config/mirrors.yaml` (+ decoded keys under `config/key_store/`).
 
 **Legacy:** a binary cache can still be configured with a `cache.yaml` (`root` + optional `key`) passed to `-c/--cache`. This path is deprecated in favour of a `buildcache` entry and will be removed.
 

docs/build-caches.md

@@ -18,7 +18,7 @@ A `mirrors.yaml` can describe five kinds of entry, each optional and each docume
 | [`bootstrap`](#bootstrap-mirror) | one  | mirror used to bootstrap Spack itself |
 | [`sourcemirror`](#source-mirrors) | many | read-only mirrors that provide package sources |
 | [`sourcecache`](#source-cache)  | one  | writable local cache that fills with sources as you build |
-| [`misccache`](#misc-cache)  | one  | writable local cache for Spack's indices and concretization results |
+| [`concretizer`](#concretizer-cache)  | one  | writable local cache that persists concretization results |
 
 A complete example:
 
@@ -34,8 +34,8 @@ sourcemirror:
     url: https://example.com/spack-sources
 sourcecache:
   path: /capstor/scratch/$USER/spack-sources
-misccache:
-  path: /capstor/scratch/$USER/spack-misc
+concretizer:
+  path: /capstor/scratch/$USER/spack-concretizer
 ```
 
 To stop using any entry, remove (or comment out) it from `mirrors.yaml`.
@@ -184,21 +184,31 @@ sourcecache:
 |-------|----------|-------------|
 | `path` | yes | absolute path to a local directory (environment variables are expanded) |
 
-## Misc cache
+## Concretizer cache
 
-The misc cache is a single, **writable** local directory for Spack's "misc" cache: the package and build-cache indices, and — importantly — the **concretization cache**, which stores the result of concretizing a set of specs so it does not have to be recomputed.
+The concretizer cache is a single, **writable** local directory in which Spack persists its **concretization results** — the output of concretizing a set of specs, so it does not have to be recomputed.
 Concretization can be a large fraction of build time, so pointing this at a persistent location is worthwhile when build directories are ephemeral (e.g. created in `/dev/shm` and deleted after each build).
 
 ```yaml title="mirrors.yaml"
-misccache:
-  path: /capstor/scratch/$USER/spack-misc
+concretizer:
+  path: /capstor/scratch/$USER/spack-concretizer
 ```
 
 | Field | Required | Description |
 |-------|----------|-------------|
 | `path` | yes | absolute path to a local directory (environment variables are expanded) |
 
-The concretization cache lives under this directory. Spack populates it automatically: concretization caching is on by default in Spack 1.2 and later, and is opt-in (`concretizer:concretization_cache:enable`) in Spack 1.1. It is keyed by the hash of the solver inputs, so a persistent cache can be reused safely across builds — stale entries simply miss.
+This emits a `concretizer.yaml` that sets `concretizer:concretization_cache:{enable: true, url}`.
+The cache is keyed by the hash of the solver inputs, so it can be reused safely across builds — stale entries simply miss.
+
+!!! warning "It does not eliminate concretization time"
+    The cache stores only the *result of the solve* for a given set of inputs. Before it can be consulted, Spack still has to rebuild the full concretization problem on every run — loading the package recipes and enumerating the reusable packages — and that setup work is often the larger part of concretization. So a cache hit skips the solver but not the setup: concretization gets faster, not free.
+
+    The win is therefore largest for repeated builds of the same stack against a stable build cache (the previous solve is replayed), and smallest when the bulk of concretization time is in setup. The cache never makes concretization slower.
+
+!!! note "Requires Spack ≥ 1.1"
+    The `concretizer:concretization_cache` config key was introduced in Spack 1.1, and Spack 1.0 rejects it.
+    Stackinator infers the Spack version from the `spack.commit` in `config.yaml` (defaulting to a supported version when the commit is a branch or arbitrary SHA that cannot be pinned). When it detects Spack 1.0 it skips the concretizer cache with a warning rather than producing a config that would fail the build.
 
 ## Keys
 

stackinator/mirror.py

@@ -27,6 +27,17 @@
 )
 
 
+def _supports_concretization_cache(spack_version: str) -> bool:
+    """Whether the given "major.minor" spack version supports the concretizer cache.
+
+    The concretizer:concretization_cache config key was introduced in spack 1.1;
+    spack 1.0 rejects it (its concretizer schema forbids unknown keys).
+    """
+
+    major_minor = tuple(int(x) for x in spack_version.split("."))
+    return major_minor >= (1, 1)
+
+
 class MirrorError(RuntimeError):
     """Exception class for errors thrown by mirror configuration problems."""
 
@@ -52,11 +63,12 @@ class Mirrors:
                          it fetches sources (spack config:source_cache). None if
                          absent. This is not a mirror: it has no key and no url, and
                          is emitted to config.yaml rather than mirrors.yaml.
-      * misc_cache     - at most one, a writable local directory for spack's misc
-                         cache (package/build-cache indices and the concretization
-                         cache that lives under it) (spack config:misc_cache). None
-                         if absent. Like source_cache it is not a mirror and is
-                         emitted to config.yaml.
+      * concretizer_cache - at most one, a writable local directory persisting spack's
+                         concretization results (concretizer:concretization_cache).
+                         None if absent. Like source_cache it is not a mirror; it is
+                         emitted to concretizer.yaml. It is only emitted for spack
+                         >= 1.1 (spack 1.0 rejects the config key); when requested
+                         against spack 1.0 it is skipped with a warning.
 
     All input processing - loading and schema-validating the system mirrors.yaml,
     validating urls, and reading/decoding/validating gpg keys - happens eagerly in
@@ -70,18 +82,22 @@ class Mirrors:
     MIRRORS_YAML = "mirrors.yaml"
     CONFIG_YAML = "config.yaml"
     BOOTSTRAP_YAML = "bootstrap.yaml"
+    CONCRETIZER_YAML = "concretizer.yaml"
 
     def __init__(
         self,
         system_config_root: pathlib.Path,
         mount_path: pathlib.Path,
+        spack_version: str,
         mirror_file: Optional[pathlib.Path] = None,
         cmdline_cache: Optional[pathlib.Path] = None,
     ):
         """Load and fully resolve the mirror configuration.
 
         Mirrors are supplied with the --mirror command line option (mirror_file).
         mount_path is the recipe mount point (used to make a build cache mount-specific).
+        spack_version is the best-effort "major.minor" spack version, used to gate the
+        concretizer cache (only emitted for spack >= 1.1).
         cmdline_cache is an optional legacy cache.yaml passed on the command line (--cache).
 
         Relative paths in the mirror file (e.g. gpg keys) are resolved relative to the
@@ -96,7 +112,7 @@ def __init__(
         self.bootstrap: Optional[Dict] = None
         self.source_mirrors: Dict[str, Dict] = {}
         self.source_cache: Optional[Dict] = None
-        self.misc_cache: Optional[Dict] = None
+        self.concretizer_cache: Optional[Dict] = None
 
         # The mirror configuration is supplied with --mirror, not the system
         # configuration. Reject a mirrors.yaml in the system config so it is not
@@ -164,28 +180,42 @@ def __init__(
                 f"{yaml.dump([self.buildcache], default_flow_style=False)}"
             )
 
-        # The bootstrap mirror, the read-only source mirrors, and the writable
-        # source and misc caches, if any are defined.
+        # The bootstrap mirror, the read-only source mirrors, the writable source
+        # cache, and the writable concretizer cache, if any are defined.
         self.bootstrap = raw_mirrors.get("bootstrap")
         self.source_mirrors = dict(raw_mirrors["sourcemirror"])
         self.source_cache = raw_mirrors.get("sourcecache")
-        self.misc_cache = raw_mirrors.get("misccache")
+        self.concretizer_cache = raw_mirrors.get("concretizer")
 
         # Validate that every mirror url is well-formed (see _validate_url).
         for name, mirror in self._iter_mirrors():
             self._validate_url(mirror["url"], name)
 
-        # The source and misc caches are single writable local directories (spack
-        # config:source_cache / config:misc_cache), not mirrors: validate that each is
-        # an absolute path. Expand env vars now, because the build sandbox runs
-        # `env --ignore-environment` and so would not expand them at build time.
-        for cache_name, cache in (("source", self.source_cache), ("misc", self.misc_cache)):
+        # The source and concretizer caches are single writable local directories
+        # (spack config:source_cache / concretizer:concretization_cache), not mirrors:
+        # validate that each is an absolute path. Expand env vars now, because the
+        # build sandbox runs `env --ignore-environment` and so would not expand them
+        # at build time.
+        for cache_name, cache in (("source", self.source_cache), ("concretizer", self.concretizer_cache)):
             if cache is not None:
                 path = os.path.expandvars(cache["path"])
                 if not pathlib.Path(path).is_absolute():
                     raise MirrorError(f"The {cache_name} cache path '{path}' is not absolute")
                 cache["path"] = path
 
+        # The concretizer cache (concretizer:concretization_cache) was introduced in
+        # spack 1.1; spack 1.0 rejects the config key. Determine whether to emit it
+        # based on the best-effort spack version, and warn (not error) if it was
+        # requested but cannot be configured.
+        self._emit_concretizer_cache = self.concretizer_cache is not None and _supports_concretization_cache(
+            spack_version
+        )
+        if self.concretizer_cache is not None and not self._emit_concretizer_cache:
+            self._logger.warning(
+                f"The concretizer cache is not supported by spack {spack_version} (requires >= 1.1) "
+                "and will not be configured."
+            )
+
         # Resolve the bootstrap mirror. It is either a remote url, or a local spack
         # bootstrap mirror directory (a `spack bootstrap mirror` output) whose own
         # metadata/{sources,binaries} directories we reference directly.
@@ -391,17 +421,26 @@ def config_files(self, config_root: pathlib.Path) -> Dict[pathlib.Path, bytes]:
             spack_mirrors, default_flow_style=False, sort_keys=False
         ).encode()
 
-        # the spack config.yaml setting the writable local caches: the populate-as-you-go
-        # source cache and the misc cache (which holds the concretization cache)
-        config_section = {}
+        # the spack config.yaml setting the populate-as-you-go source cache (spack
+        # config:source_cache).
         if self.source_cache is not None:
-            config_section["source_cache"] = self.source_cache["path"]
-        if self.misc_cache is not None:
-            config_section["misc_cache"] = self.misc_cache["path"]
-        if config_section:
-            config_yaml = {"config": config_section}
+            config_yaml = {"config": {"source_cache": self.source_cache["path"]}}
             files[config_root / self.CONFIG_YAML] = yaml.dump(config_yaml, default_flow_style=False).encode()
 
+        # the spack concretizer.yaml persisting concretization results. enable is set
+        # explicitly because it is opt-in in spack 1.1 (on by default only in >= 1.2).
+        # _emit_concretizer_cache gates this on the spack version (>= 1.1).
+        if self._emit_concretizer_cache:
+            concretizer_yaml = {
+                "concretizer": {
+                    "concretization_cache": {
+                        "enable": True,
+                        "url": self.concretizer_cache["path"],
+                    }
+                }
+            }
+            files[config_root / self.CONCRETIZER_YAML] = yaml.dump(concretizer_yaml, default_flow_style=False).encode()
+
         # the spack bootstrap.yaml, if a bootstrap mirror is set. Bootstrapping reads
         # bootstrap:sources (not the mirrors list), and each source's `metadata` is a
         # directory describing it. No gpg key is involved (bootstrap binaries are

stackinator/recipe.py

@@ -176,12 +176,19 @@ def __init__(self, args):
                 )
                 raise RuntimeError("Ivalid default-view in the recipe.")
 
+        # determine the version of spack being used: it is inferred (best effort)
+        # from the spack commit in config.yaml, defaulting to the latest supported
+        # version when it cannot be determined. This must precede the mirror
+        # configuration, which gates the concretizer cache on the spack version.
+        self.spack_version = self.find_spack_version(args.develop)
+
         # resolve the mirror configuration provided with --mirror. --cache is the
         # legacy path.
         self._logger.debug("Configuring mirrors.")
         self.mirrors = mirror.Mirrors(
             self.system_config_path,
             self.mount,
+            self.spack_version,
             pathlib.Path(args.mirror) if args.mirror else None,
             pathlib.Path(args.cache) if args.cache else None,
         )
@@ -198,11 +205,6 @@ def __init__(self, args):
         else:
             self._logger.debug("no pre install hook provided")
 
-        # determine the version of spack being used:
-        # currently this just returns 1.0... develop is ignored
-        # --develop flag will imply the next release of spack after 1.0 is supported properly
-        self.spack_version = self.find_spack_version(args.develop)
-
     # Returns:
     #   Path: if the recipe contains a spack package repository
     #   None: if there is the recipe contains no repo
@@ -276,10 +278,30 @@ def config(self, config_path):
     def with_modules(self) -> bool:
         return self.modules is not None
 
-    # In Stackinator 6 we replaced logic required to determine the
-    # pre 1.0 Spack version.
+    # Make a best-effort determination of the "major.minor" version of spack being
+    # used, inferred from the spack commit in config.yaml. This is only a hint: the
+    # commit can be an arbitrary branch/tag/sha, so when the version cannot be pinned
+    # we default to the latest supported version ("1.1"). Returns a "major.minor"
+    # string (e.g. "1.0", "1.1").
     def find_spack_version(self, develop):
-        return "1.0"
+        # the latest supported version, used when the version cannot be determined
+        # (an explicit --develop, the default branch, or an unrecognised commit).
+        default = "1.1"
+
+        if develop:
+            return default
+
+        commit = self.config["spack"]["commit"]
+        if commit is None or commit in ("develop", "main"):
+            return default
+
+        # match a release branch/tag (releases/v1.0, v1.1, v1.1.2) or a bare "1.0",
+        # and extract the major.minor version.
+        match = re.search(r"v?(\d+)\.(\d+)(?:\.\d+)?", commit)
+        if match:
+            return f"{match.group(1)}.{match.group(2)}"
+
+        return default
 
     @property
     def default_view(self):

stackinator/schema/mirror.json

@@ -56,7 +56,7 @@
             "additionalProperties": false,
             "required": ["path"]
         },
-        "misccache": {
+        "concretizer": {
             "type": "object",
             "properties": {
                 "description": {"type": "string", "default": ""},

…ystems/mirror-bad-misccache/mirrors.yaml …tems/mirror-bad-concretizer/mirrors.yamlunittests/data/systems/mirror-bad-misccache/mirrors.yaml renamed to unittests/data/systems/mirror-bad-concretizer/mirrors.yaml unittests/data/systems/mirror-bad-misccache/mirrors.yaml renamed to unittests/data/systems/mirror-bad-concretizer/mirrors.yaml

@@ -1,2 +1,2 @@
-misccache:
+concretizer:
   path: relative/not/absolute