make mirrors a cli argument, not part of system config

Author: bcumming

docs/build-caches.md

@@ -1,7 +1,14 @@
 # Mirrors and Build Caches
 
 Spack can use *mirrors* and *caches* to speed up image builds and to build on systems with limited or no internet access.
-They are configured in a single `mirrors.yaml` file in the [system configuration](cluster-config.md).
+They are configured in a single `mirrors.yaml` file that you supply on the command line:
+
+```bash
+stack-config -b $build -r $recipe -s $system --mirror mirrors.yaml
+```
+
+The file is not part of the [system configuration](cluster-config.md): mirror locations are usually specific to the person running the build, so each invocation provides its own.
+Paths inside the file (such as relative gpg key paths) are resolved relative to the directory containing the `mirrors.yaml`, so a self-contained mirror directory (the `mirrors.yaml` plus its keys) can be moved around freely.
 
 A `mirrors.yaml` can describe four kinds of entry, each optional and each documented below:
 

docs/cluster-config.md

@@ -93,10 +93,10 @@ packages:
         version: ["git.59b6de6a91d9637809677c50cc48b607a91a9acb=main"]
     ```
 
-### Configuring Spack mirrors: `mirrors.yaml`
+### Configuring Spack mirrors
 
-An optional `mirrors.yaml` connects Spack to local mirrors and caches, to speed up builds and to build on air-gapped systems.
-It can configure a build cache, a bootstrap mirror, read-only source mirrors, and a writable source cache.
+Mirrors and caches are **not** part of the system configuration.
+They are supplied per-invocation with `stack-config --mirror <file>`, because the locations involved (build caches, source caches) are often specific to the user running the build and may not be accessible to everyone using a system.
 
 See [Mirrors and Build Caches](build-caches.md) for the full reference and examples.
 

docs/configuring.md

@@ -16,7 +16,8 @@ The following flags are required:
 
 The following flags are optional:
 
-* `-c/--cache`: configure the [build cache](build-caches.md).
+* `--mirror`: path to a [mirrors.yaml](build-caches.md) file configuring build caches and mirrors.
+* `-c/--cache`: legacy build cache configuration file (deprecated; use `--mirror`).
 * `-m/--mount`: override the [mount point](installing.md) where the stack will be installed.
 * `--version`: print the stackinator version.
 * `-h/--help`: print help message.

memory.md

@@ -74,6 +74,7 @@ def log_header(args):
     root_logger.info(f"  system     : {args.system}")
     mount = args.mount or "default"
     root_logger.info(f"  mount      : {mount}")
+    root_logger.info(f"  mirror     : {args.mirror}")
     root_logger.info(f"  build cache: {args.cache}")
     root_logger.info(f"  develop    : {args.develop}")
 
@@ -99,12 +100,18 @@ def make_argparser():
     parser.add_argument(
         "-m", "--mount", required=False, type=str, help="The mount point where the environment will be located."
     )
+    parser.add_argument(
+        "--mirror",
+        required=False,
+        type=str,
+        help="Path to a mirrors.yaml file describing build caches and mirrors.",
+    )
     parser.add_argument(
         "-c",
         "--cache",
         required=False,
         type=str,
-        help="Buildcache location or name (from system config's mirrors.yaml).",
+        help="Legacy build cache configuration file (deprecated; use --mirror).",
     )
     parser.add_argument("--develop", action="store_true", required=False)
 

stackinator/main.py

@@ -65,32 +65,48 @@ def __init__(
         self,
         system_config_root: pathlib.Path,
         mount_path: pathlib.Path,
+        mirror_file: Optional[pathlib.Path] = None,
         cmdline_cache: Optional[pathlib.Path] = None,
     ):
         """Load and fully resolve the mirror configuration.
 
-        Inputs are the system config's mirrors.yaml, the recipe mount path (used to
-        make a build cache mount-specific), and an optional legacy cache.yaml passed
-        on the command line (--cache).
+        Mirrors are supplied with the --mirror command line option (mirror_file).
+        mount_path is the recipe mount point (used to make a build cache mount-specific).
+        cmdline_cache is an optional legacy cache.yaml passed on the command line (--cache).
+
+        Relative paths in the mirror file (e.g. gpg keys) are resolved relative to the
+        directory containing the mirror file.
         """
 
         self._logger = root_logger
-        self._system_config_root = system_config_root
         self._mount_path = mount_path
+        self._mirror_dir = mirror_file.parent if mirror_file is not None else None
 
         self.buildcache: Optional[Dict] = None
         self.bootstrap: Optional[Dict] = None
         self.source_mirrors: Dict[str, Dict] = {}
         self.source_cache: Optional[Dict] = None
 
-        # Load and schema-validate the system mirrors.yaml (absent file is fine).
-        mirrors_path = system_config_root / "mirrors.yaml"
-        if mirrors_path.exists():
+        # The mirror configuration is supplied with --mirror, not the system
+        # configuration. Reject a mirrors.yaml in the system config so it is not
+        # silently ignored.
+        if (system_config_root / "mirrors.yaml").exists():
+            raise MirrorError(
+                "A 'mirrors.yaml' in the system configuration is not supported.\n"
+                "Provide the mirror configuration with the '--mirror' command line option."
+            )
+
+        # Load and schema-validate the mirror file given on the command line. If none
+        # was given there are no mirrors; an empty config still validates (and picks up
+        # schema defaults such as an empty sourcemirror map).
+        if mirror_file is not None:
+            if not mirror_file.is_file():
+                raise MirrorError(f"The mirror configuration file '{mirror_file}' does not exist.")
             try:
-                with mirrors_path.open() as fid:
+                with mirror_file.open() as fid:
                     raw_mirrors = yaml.load(fid, Loader=yaml.SafeLoader)
             except (OSError, PermissionError) as err:
-                raise MirrorError(f"Could not open/read mirrors.yaml file.\n{err}")
+                raise MirrorError(f"Could not open/read mirror file '{mirror_file}'.\n{err}")
         else:
             raw_mirrors = {}
 
@@ -241,15 +257,15 @@ def _validate_url(self, url: str, name: str):
     def _read_key(self, key: str, name: str) -> bytes:
         """Resolve a key (a file path or base64 blob) to validated gpg key bytes.
 
-        A key is either a path - absolute, or relative to the system config - or a
-        base64-encoded blob inlined in mirrors.yaml. The resulting bytes are checked
-        to be genuine gpg key material before being accepted.
+        A key is either a path - absolute, or relative to the mirror file's directory
+        - or a base64-encoded blob inlined in the mirror file. The resulting bytes are
+        checked to be genuine gpg key material before being accepted.
         """
 
-        # if it is a path (absolute, or relative to the system config), read it
+        # if it is a path (absolute, or relative to the mirror file), read it
         path = pathlib.Path(os.path.expandvars(key))
         if not path.is_absolute():
-            path = self._system_config_root / path
+            path = self._mirror_dir / path
 
         if path.is_file():
             binary_key = path.read_bytes()

stackinator/mirror.py

@@ -176,12 +176,13 @@ def __init__(self, args):
                 )
                 raise RuntimeError("Ivalid default-view in the recipe.")
 
-        # load the optional mirrors.yaml from system config, and add any additional
-        # mirrors specified on the command line.
+        # resolve the mirror configuration provided with --mirror. --cache is the
+        # legacy path.
         self._logger.debug("Configuring mirrors.")
         self.mirrors = mirror.Mirrors(
             self.system_config_path,
             self.mount,
+            pathlib.Path(args.mirror) if args.mirror else None,
             pathlib.Path(args.cache) if args.cache else None,
         )