Multiple package repos in config.yaml (#294)

Extends config.yaml to accept not only
```yaml
...
    packages:
        repo: https://github.com/foo/spack-packages.git
        commit: abcdef
```
but also multiple repos:
```yaml
...
    packages:
        myrepo:
            repo: https://github.com/bar/other-packages.git
            commit: fedcba
            path: path/to/repo
        builtin:
            repo: https://github.com/foo/spack-packages.git
            commit: abcdef
```

Order of repos is significant in the same way it's significant in spack
config files
(https://spack.readthedocs.io/en/latest/repositories.html#search-order-and-overriding-packages).
Packages from repos higher up on the list take precedence.

I made a small change so that there are also two implicit repos
(currently on main there's one: `alps`). A repo `recipe` exists if and
only if there are packages in the recipe. `alps` still exists, but
doesn't have the recipe packages. The example above would produce the
following list of repos:
- `recipe`
- `alps`
- `myrepo`
- `builtin`

Currently there's no enforcement that the repo entry has the same name
as the repo namespace (similar to spack itself, if I understand it
correctly). Is this ok?

There's an optional `path` property that can be used to point to a
non-default subdirectory within the cloned git repo. For spack-packages
this is always in `repos/spack_repo/builtin`, so the default is
`repos/spack_repo/{name}`. In spack-packages there is an index file
https://github.com/spack/spack-packages/blob/develop/spack-repo-index.yaml
which _could_ be used to locate all the package repos within a git repo.
Spack uses this when it clones package repos through git. For now I
decided it's overkill to support that. Thoughts?

Author: msimberg

.github/workflows/main.yaml

@@ -23,6 +23,11 @@ jobs:
         done
         exit $errors
     - name: Install Dependencies
-      run: uv sync --group dev
+      run: |
+        uv sync --group dev
+        sudo apt-get install -y jq
     - name: Run Unit Tests
       run: uv run pytest
+    - name: Run Envvars Test
+      working-directory: unittests
+      run: uv run bash ./test-envvars.sh

docs/cluster-config.md

@@ -129,8 +129,8 @@ If custom package definitions are provided for the same package in more than one
 
 The following precedence is applied, where 1 has higher precedence than 2 or 3:
 
-1. packages defined in the (optional) `repo` path in the [recipe](recipes.md#custom-spack-packages)
-2. packages defined in the (optional) site repo(s) defined in the `repo/repos.yaml` file of cluster configuration (documented here)
-3. packages provided by Spack (in the `var/spack/repos/builtin` path)
+1. packages defined in the (optional) `repo` path in the [recipe][custom-spack-packages]
+2. packages defined in the (optional) site repo(s) defined in the `repo/repos.yaml` file of the cluster configuration (documented here)
+3. packages defined in the package repositories configured in `config.yaml:spack:packages`, in the order specified (which typically includes `builtin`)
 
-As of Stackinator v4, the definitions of some custom repositories (mainly CSCS' custom cray-mpich and its dependencies) was removed from Stackinator, and moved to the the site configuration
+As of Stackinator v4, the definitions of some custom repositories (mainly CSCS' custom cray-mpich and its dependencies) was removed from Stackinator, and moved to the site configuration.

docs/recipes.md

@@ -40,6 +40,37 @@ version: 2
 * `version`:  _default = 1_ the version of the uenv recipe (see below)
 * `modules`: (_deprecated_) _optional_ enable/disable module file generation.
 
+It's possible to configure multiple package repositories for the uenv build by providing a dictionary of spack repositories. For example:
+
+```yaml title="config.yaml"
+name: prgenv-gnu
+store: /user-environment
+spack:
+    repo: https://github.com/spack/spack.git
+    commit: releases/v1.0
+    packages:
+        builtin:
+            repo: https://github.com/spack/spack-packages.git
+            commit: develop
+        foo:
+            repo: https://github.com/foo/spack-packages.git
+            commit: v13.2
+            path: repos/spack_repo/foo
+version: 2
+```
+
+The `path` entry is optional and defaults to `repos/spack_repo/${name}`, where the dictionary key is the `name`.
+For the upstream spack-packages repository, the default value can be used.
+
+!!! info
+   The order of package repositories is significant.
+   stackinator follows the same semantics as spack itself, where package repositories further up in the list take precedence over ones later in the list.
+   Refer to the [spack documentation](https://spack.readthedocs.io/en/latest/repositories.html#search-order-and-overriding-packages) for more information.
+
+!!! info
+    `recipe` and `alps` are reserved repository names for internal stackinator use and can't be used for user-specified package repositories.
+    The `recipe` and `alps` repositories have higher precedence than repositories configured in `config.yaml` (see [custom spack packages][ref-custom-spack-packages] for more details).
+
 !!! note "uenv recipe versions"
     Stackinator 6 introduces breaking changes to the uenv recipe format, introduced to support Spack v1.0.
 
@@ -487,9 +518,10 @@ Modules are generated for the installed compilers and packages by spack.
     - `modules:default:arch_folder` defaults to `false`. If set to `true` an error is raised, as Stackinator does not support this feature;
     - `modules:default:roots:tcl` is ignored, as Stackinator automatically configures the module root to be inside the uenv mount point.
 
+[](){#ref-custom-spack-packages}
 ## Custom Spack Packages
 
-An optional package repository can be added to a recipe to provide new or customized Spack packages in addition to Spack's `builtin` package repository, if a `repo` path is provided in the recipe.
+An optional package repository can be added to a recipe to provide new or customized Spack packages in addition to the package repositories configured in `config.yaml`, if a `repo` path is provided in the recipe.
 
 For example, the following `repo` path will add custom package definitions for the `hdf5` and `nvhpc` packages:
 
@@ -502,6 +534,8 @@ repo
       └─ package.py
 ```
 
+Packages provided together with a recipe will be installed in a separate `recipe` namespace.
+
 Additional custom packages can be provided as part of the cluster configuration, as well as additional site packages.
 These packages are all optional, and will be installed together in a single Spack package repository that is made available to downstream users of the generated uenv stack.
 See the documentation for [cluster configuration](cluster-config.md) for more detail.
@@ -523,12 +557,13 @@ See the documentation for [cluster configuration](cluster-config.md) for more de
     In the above case, the package `fmt` is backported from `origin/develop` into the `stackinator-recipe`.
 
 !!! alps
-    All packages are installed under a single spack package repository called `alps`.
+    All cluster configuration packages are installed under a single spack package repository called `alps`.
     The CSCS configurations in [github.com/eth-cscs/alps-cluster-config](https://github.com/eth-cscs/alps-cluster-config) provides a site configuration that defines cray-mpich, its dependencies, and the most up to date versions of cuda, nvhpc etc to all clusters on Alps.
+    These site packages are installed under the `alps` Spack package repository namespace.
 
 !!! warning
     Unlike Spack package repositories, any `repos.yaml` file in the `repo` path will be ignored.
-    This is because the provided packages are added to the `alps` namespace.
+    This is because the provided packages are installed in the `recipe` namespace.
 
 ## Post install configuration
 

stackinator/builder.py

@@ -13,6 +13,12 @@
 
 from . import VERSION, cache, root_logger, spack_util
 
+_REPO_YAML = """\
+repo:
+  namespace: {namespace}
+  api: v2.0
+"""
+
 
 def install(src, dst, *, ignore=None, symlinks=False):
     """Call shutil.copytree or shutil.copy2. copy2 is used if `src` is not a directory.
@@ -191,26 +197,16 @@ def generate(self, recipe):
 
         spack_git_commit_result = self._git_clone("spack", spack_repo, spack_commit, spack_path)
 
-        # Clone the spack-packages repository and check out commit if one was given
-        spack_packages = spack["packages"]
-        spack_packages_repo = spack_packages["repo"]
-        spack_packages_commit = spack_packages["commit"]
-        spack_packages_path = self.path / "spack-packages"
-
-        spack_packages_git_commit_result = self._git_clone(
-            "spack-packages",
-            spack_packages_repo,
-            spack_packages_commit,
-            spack_packages_path,
-        )
+        package_repos = recipe.spack_package_repos
+        for pkg_repo in package_repos:
+            pkg_repo["path"] = self.path / "repos" / pkg_repo["name"]
+            pkg_repo["commit"] = self._git_clone(pkg_repo["name"], pkg_repo["url"], pkg_repo["ref"], pkg_repo["path"])
 
         spack_meta = {
             "url": spack_repo,
             "ref": spack_commit,
             "commit": spack_git_commit_result,
-            "packages_url": spack_packages_repo,
-            "packages_ref": spack_packages_commit,
-            "packages_commit": spack_packages_git_commit_result,
+            "packages": package_repos,
         }
 
         # load the jinja templating environment
@@ -331,16 +327,12 @@ def generate(self, recipe):
         # 2. cluster-config/repos.yaml
         #   - if the repos.yaml file exists it will contain a list of relative paths
         #     to search for package
-        # 1. builtin repo
+        # 1. package repos from config.yaml in the order specified (typically
+        #    only spack-packages builtin repo)
 
-        # Build a list of repos with packages to install.
+        # Build a list of repos with packages to install from system config and recipe.
         repos = []
 
-        # check for a repo in the recipe
-        if recipe.spack_repo is not None:
-            self._logger.debug(f"adding recipe spack package repo: {recipe.spack_repo}")
-            repos.append(recipe.spack_repo)
-
         # look for repos.yaml file in the system configuration
         repo_yaml = recipe.system_config_path / "repos.yaml"
         if repo_yaml.exists() and repo_yaml.is_file():
@@ -361,7 +353,7 @@ def generate(self, recipe):
                     self._logger.error(f"{repo_path} from {repo_yaml} is not a spack package repository")
                     raise RuntimeError("invalid system-provided package repository")
 
-        self._logger.debug(f"full list of spack package repo: {repos}")
+        self._logger.debug(f"full list of system spack package repos: {repos}")
 
         # Delete the store/repo path, if it already exists.
         # Do this so that incremental builds (though not officially supported) won't break if a repo is updated.
@@ -378,53 +370,82 @@ def generate(self, recipe):
         self._logger.debug(f"created the repo packages path {pkg_dst}")
 
         # create the repository step 2: create the repo.yaml file that
-        # configures alps and builtin repos
+        # configures the alps repo
         with (repo_dst / "repo.yaml").open("w") as f:
-            f.write(
-                """\
-repo:
-  namespace: alps
-  api: v2.0
-"""
-            )
+            f.write(_REPO_YAML.format(namespace="alps"))
+
+        # If the recipe provides a package repo, install it as a separate
+        # "recipe" repo in the store with highest precedence.
+        has_recipe_repo = recipe.spack_repo is not None
+        if has_recipe_repo:
+            recipe_dst = repos_path / "recipe"
+            self._logger.debug(f"creating the recipe spack repo in {recipe_dst}")
+            if recipe_dst.exists():
+                self._logger.debug(f"{recipe_dst} exists ... deleting")
+                shutil.rmtree(recipe_dst)
+
+            recipe_pkg_dst = recipe_dst / "packages"
+            recipe_pkg_dst.mkdir(mode=0o755, parents=True)
+
+            with (recipe_dst / "repo.yaml").open("w") as f:
+                f.write(_REPO_YAML.format(namespace="recipe"))
+
+            packages_path = recipe.spack_repo / "packages"
+            for pkg_path in packages_path.iterdir():
+                dst = recipe_pkg_dst / pkg_path.name
+                if pkg_path.is_dir():
+                    self._logger.debug(f"  installing recipe package {pkg_path} to {recipe_pkg_dst}")
+                    install(pkg_path, dst)
 
         # create the repository step 2: create the repos.yaml file in build_path/config
         repos_yaml_template = jinja_env.get_template("repos.yaml")
         with (config_path / "repos.yaml").open("w") as f:
             repo_path = recipe.mount / "repos" / "spack_repo" / "alps"
-            builtin_repo_path = recipe.mount / "repos" / "spack_repo" / "builtin"
+            recipe_repo_path = recipe.mount / "repos" / "spack_repo" / "recipe"
+            package_repos = [
+                {
+                    "name": pkg_repo["name"],
+                    "path": (recipe.mount / "repos" / "spack_repo" / pkg_repo["name"]).as_posix(),
+                }
+                for pkg_repo in spack_meta["packages"]
+            ]
             f.write(
                 repos_yaml_template.render(
                     repo_path=repo_path.as_posix(),
-                    builtin_repo_path=builtin_repo_path.as_posix(),
+                    package_repos=package_repos,
+                    recipe_repo_path=recipe_repo_path.as_posix(),
+                    has_recipe_repo=has_recipe_repo,
                     verbose=False,
                 )
             )
             f.write("\n")
 
-        # Iterate over the source repositories copying their contents to the consolidated repo in the uenv.
-        # Do overwrite packages that have been copied from an earlier source repo, enforcing a descending
-        # order of precidence.
-        if len(repos) > 0:
-            for repo_src in repos:
-                self._logger.debug(f"installing repo {repo_src}")
-                packages_path = repo_src / "packages"
-                for pkg_path in packages_path.iterdir():
-                    dst = pkg_dst / pkg_path.name
-                    if pkg_path.is_dir() and not dst.exists():
-                        self._logger.debug(f"  installing package {pkg_path} to {pkg_dst}")
-                        install(pkg_path, dst)
-                    elif dst.exists():
-                        self._logger.debug(f"  NOT installing package {pkg_path}")
-
-        # Copy the builtin repo to store, delete if it already exists.
-        spack_packages_builtin_path = spack_packages_path / "repos" / "spack_repo" / "builtin"
-        spack_packages_store_path = store_path / "repos" / "spack_repo" / "builtin"
-        self._logger.debug(f"copying builtin repo from {spack_packages_builtin_path} to {spack_packages_store_path}")
-        if spack_packages_store_path.exists():
-            self._logger.debug(f"{spack_packages_store_path} exists ... deleting")
-            shutil.rmtree(spack_packages_store_path)
-        install(spack_packages_builtin_path, spack_packages_store_path)
+        # Iterate over the alps and recipe repositories copying their contents
+        # to the final repo locations. Because of the order of repos in the
+        # repos.yaml config file, recipe packages have precedence.
+        for repo_src in repos:
+            self._logger.debug(f"installing repo {repo_src}")
+            packages_path = repo_src / "packages"
+            for pkg_path in packages_path.iterdir():
+                dst = pkg_dst / pkg_path.name
+                if pkg_path.is_dir() and not dst.exists():
+                    self._logger.debug(f"  installing package {pkg_path} to {pkg_dst}")
+                    install(pkg_path, dst)
+                elif dst.exists():
+                    self._logger.debug(f"  NOT installing package {pkg_path}")
+
+        # Copy all package repos defined in config.yaml to their final repo
+        # locations.
+        for pkg_repo in spack_meta["packages"]:
+            clone_path = pkg_repo["path"]
+            name = pkg_repo["name"]
+            src_path = clone_path / pkg_repo["repo_path"]
+            dst_path = store_path / "repos" / "spack_repo" / name
+            self._logger.debug(f"copying repo '{name}' from {src_path} to {dst_path}")
+            if dst_path.exists():
+                self._logger.debug(f"{dst_path} exists ... deleting")
+                shutil.rmtree(dst_path)
+            install(src_path, dst_path)
 
         # Generate the makefile and spack.yaml files that describe the compilers
         compiler_files = recipe.compiler_files

stackinator/etc/envvars.py

@@ -621,12 +621,27 @@ def meta_impl(args):
 
     if args.spack is not None:
         spack_url, spack_ref, spack_commit = args.spack.split(",")
-        spack_packages_url = None
-        spack_packages_ref = None
-        spack_packages_commit = None
-        if args.spack_packages is not None:
-            spack_packages_url, spack_packages_ref, spack_packages_commit = args.spack_packages.split(",")
         spack_path = f"{args.mount}/config".replace("//", "/")
+        scalar_vars = {
+            "UENV_SPACK_CONFIG_PATH": spack_path,
+            "UENV_SPACK_URL": spack_url,
+            "UENV_SPACK_REF": spack_ref,
+            "UENV_SPACK_COMMIT": spack_commit,
+        }
+        if args.spack_package_repo:
+            repo_names = []
+            for entry in args.spack_package_repo:
+                name, url, ref, commit = entry.split(",")
+                repo_names.append(name)
+                name_upper = name.upper().replace("-", "_")
+                scalar_vars[f"UENV_PACKAGE_REPO_{name_upper}_URL"] = url
+                scalar_vars[f"UENV_PACKAGE_REPO_{name_upper}_REF"] = ref
+                scalar_vars[f"UENV_PACKAGE_REPO_{name_upper}_COMMIT"] = commit
+                if name == "builtin":
+                    scalar_vars["UENV_SPACK_PACKAGES_URL"] = url
+                    scalar_vars["UENV_SPACK_PACKAGES_REF"] = ref
+                    scalar_vars["UENV_SPACK_PACKAGES_COMMIT"] = commit
+            scalar_vars["UENV_PACKAGE_REPOS"] = ",".join(repo_names)
         meta["views"]["spack"] = {
             "activate": "/dev/null",
             "description": "configure spack upstream",
@@ -636,15 +651,7 @@ def meta_impl(args):
                 "type": "augment",
                 "values": {
                     "list": {},
-                    "scalar": {
-                        "UENV_SPACK_CONFIG_PATH": spack_path,
-                        "UENV_SPACK_URL": spack_url,
-                        "UENV_SPACK_REF": spack_ref,
-                        "UENV_SPACK_COMMIT": spack_commit,
-                        "UENV_SPACK_PACKAGES_URL": spack_packages_url,
-                        "UENV_SPACK_PACKAGES_REF": spack_packages_ref,
-                        "UENV_SPACK_PACKAGES_COMMIT": spack_packages_commit,
-                    },
+                    "scalar": scalar_vars,
                 },
             },
         }
@@ -686,9 +693,11 @@ def meta_impl(args):
         default=None,
     )
     uenv_parser.add_argument(
-        "--spack-packages",
-        help='configure spack-packages repository metadata. Format is "spack_url,git_ref,git_commit"',
+        "--spack-package-repo",
+        help="configure spack package repository metadata. "
+        'Format is "name,spack_url,git_ref,git_commit". Can be repeated.',
         type=str,
+        action="append",
         default=None,
     )
 

stackinator/recipe.py

@@ -213,6 +213,39 @@ def spack_repo(self):
             return repo_path
         return None
 
+    _RESERVED_REPO_NAMES = {"alps", "recipe"}
+
+    @property
+    def spack_package_repos(self):
+        packages = self.config["spack"]["packages"]
+        if isinstance(packages.get("repo"), str):
+            return [
+                {
+                    "name": "builtin",
+                    "url": packages["repo"],
+                    "ref": packages.get("commit"),
+                    "repo_path": packages.get("path", "repos/spack_repo/builtin"),
+                }
+            ]
+        repos = [
+            {
+                "name": name,
+                "url": val["repo"],
+                "ref": val.get("commit"),
+                "repo_path": val.get("path", f"repos/spack_repo/{name}"),
+            }
+            for name, val in packages.items()
+        ]
+        for repo in repos:
+            name = repo["name"]
+            if name in self._RESERVED_REPO_NAMES:
+                raise RuntimeError(
+                    f"The package repo name '{name}' is reserved for stackinator internal use. "
+                    f"Reserved names are: {self._RESERVED_REPO_NAMES}. "
+                    "Choose a different name in config.yaml:spack:packages."
+                )
+        return repos
+
     # Returns:
     #   Path: of the recipe extra path if it exists
     #   None: if there is no user-provided extra path in the recipe