diff --git a/.github/workflows/main.yaml b/.github/workflows/main.yaml index cb2fd33f..8b026f58 100644 --- a/.github/workflows/main.yaml +++ b/.github/workflows/main.yaml @@ -23,6 +23,11 @@ jobs: done exit $errors - name: Install Dependencies - run: uv sync --group dev + run: | + uv sync --group dev + sudo apt-get install -y jq - name: Run Unit Tests run: uv run pytest + - name: Run Envvars Test + working-directory: unittests + run: uv run bash ./test-envvars.sh diff --git a/docs/cluster-config.md b/docs/cluster-config.md index 2ce62019..bca26f1b 100644 --- a/docs/cluster-config.md +++ b/docs/cluster-config.md @@ -129,8 +129,8 @@ If custom package definitions are provided for the same package in more than one The following precedence is applied, where 1 has higher precedence than 2 or 3: -1. packages defined in the (optional) `repo` path in the [recipe](recipes.md#custom-spack-packages) -2. packages defined in the (optional) site repo(s) defined in the `repo/repos.yaml` file of cluster configuration (documented here) -3. packages provided by Spack (in the `var/spack/repos/builtin` path) +1. packages defined in the (optional) `repo` path in the [recipe][custom-spack-packages] +2. packages defined in the (optional) site repo(s) defined in the `repo/repos.yaml` file of the cluster configuration (documented here) +3. packages defined in the package repositories configured in `config.yaml:spack:packages`, in the order specified (which typically includes `builtin`) -As of Stackinator v4, the definitions of some custom repositories (mainly CSCS' custom cray-mpich and its dependencies) was removed from Stackinator, and moved to the the site configuration +As of Stackinator v4, the definitions of some custom repositories (mainly CSCS' custom cray-mpich and its dependencies) was removed from Stackinator, and moved to the site configuration. diff --git a/docs/recipes.md b/docs/recipes.md index b295cd9b..69d8ab0e 100644 --- a/docs/recipes.md +++ b/docs/recipes.md @@ -40,6 +40,37 @@ version: 2 * `version`: _default = 1_ the version of the uenv recipe (see below) * `modules`: (_deprecated_) _optional_ enable/disable module file generation. +It's possible to configure multiple package repositories for the uenv build by providing a dictionary of spack repositories. For example: + +```yaml title="config.yaml" +name: prgenv-gnu +store: /user-environment +spack: + repo: https://github.com/spack/spack.git + commit: releases/v1.0 + packages: + builtin: + repo: https://github.com/spack/spack-packages.git + commit: develop + foo: + repo: https://github.com/foo/spack-packages.git + commit: v13.2 + path: repos/spack_repo/foo +version: 2 +``` + +The `path` entry is optional and defaults to `repos/spack_repo/${name}`, where the dictionary key is the `name`. +For the upstream spack-packages repository, the default value can be used. + +!!! info + The order of package repositories is significant. + stackinator follows the same semantics as spack itself, where package repositories further up in the list take precedence over ones later in the list. + Refer to the [spack documentation](https://spack.readthedocs.io/en/latest/repositories.html#search-order-and-overriding-packages) for more information. + +!!! info + `recipe` and `alps` are reserved repository names for internal stackinator use and can't be used for user-specified package repositories. + The `recipe` and `alps` repositories have higher precedence than repositories configured in `config.yaml` (see [custom spack packages][ref-custom-spack-packages] for more details). + !!! note "uenv recipe versions" Stackinator 6 introduces breaking changes to the uenv recipe format, introduced to support Spack v1.0. @@ -487,9 +518,10 @@ Modules are generated for the installed compilers and packages by spack. - `modules:default:arch_folder` defaults to `false`. If set to `true` an error is raised, as Stackinator does not support this feature; - `modules:default:roots:tcl` is ignored, as Stackinator automatically configures the module root to be inside the uenv mount point. +[](){#ref-custom-spack-packages} ## Custom Spack Packages -An optional package repository can be added to a recipe to provide new or customized Spack packages in addition to Spack's `builtin` package repository, if a `repo` path is provided in the recipe. +An optional package repository can be added to a recipe to provide new or customized Spack packages in addition to the package repositories configured in `config.yaml`, if a `repo` path is provided in the recipe. For example, the following `repo` path will add custom package definitions for the `hdf5` and `nvhpc` packages: @@ -502,6 +534,8 @@ repo └─ package.py ``` +Packages provided together with a recipe will be installed in a separate `recipe` namespace. + Additional custom packages can be provided as part of the cluster configuration, as well as additional site packages. These packages are all optional, and will be installed together in a single Spack package repository that is made available to downstream users of the generated uenv stack. See the documentation for [cluster configuration](cluster-config.md) for more detail. @@ -523,12 +557,13 @@ See the documentation for [cluster configuration](cluster-config.md) for more de In the above case, the package `fmt` is backported from `origin/develop` into the `stackinator-recipe`. !!! alps - All packages are installed under a single spack package repository called `alps`. + All cluster configuration packages are installed under a single spack package repository called `alps`. The CSCS configurations in [github.com/eth-cscs/alps-cluster-config](https://github.com/eth-cscs/alps-cluster-config) provides a site configuration that defines cray-mpich, its dependencies, and the most up to date versions of cuda, nvhpc etc to all clusters on Alps. + These site packages are installed under the `alps` Spack package repository namespace. !!! warning Unlike Spack package repositories, any `repos.yaml` file in the `repo` path will be ignored. - This is because the provided packages are added to the `alps` namespace. + This is because the provided packages are installed in the `recipe` namespace. ## Post install configuration diff --git a/stackinator/builder.py b/stackinator/builder.py index ae4997a1..32dd4d8b 100644 --- a/stackinator/builder.py +++ b/stackinator/builder.py @@ -13,6 +13,12 @@ from . import VERSION, cache, root_logger, spack_util +_REPO_YAML = """\ +repo: + namespace: {namespace} + api: v2.0 +""" + def install(src, dst, *, ignore=None, symlinks=False): """Call shutil.copytree or shutil.copy2. copy2 is used if `src` is not a directory. @@ -191,26 +197,16 @@ def generate(self, recipe): spack_git_commit_result = self._git_clone("spack", spack_repo, spack_commit, spack_path) - # Clone the spack-packages repository and check out commit if one was given - spack_packages = spack["packages"] - spack_packages_repo = spack_packages["repo"] - spack_packages_commit = spack_packages["commit"] - spack_packages_path = self.path / "spack-packages" - - spack_packages_git_commit_result = self._git_clone( - "spack-packages", - spack_packages_repo, - spack_packages_commit, - spack_packages_path, - ) + package_repos = recipe.spack_package_repos + for pkg_repo in package_repos: + pkg_repo["path"] = self.path / "repos" / pkg_repo["name"] + pkg_repo["commit"] = self._git_clone(pkg_repo["name"], pkg_repo["url"], pkg_repo["ref"], pkg_repo["path"]) spack_meta = { "url": spack_repo, "ref": spack_commit, "commit": spack_git_commit_result, - "packages_url": spack_packages_repo, - "packages_ref": spack_packages_commit, - "packages_commit": spack_packages_git_commit_result, + "packages": package_repos, } # load the jinja templating environment @@ -331,16 +327,12 @@ def generate(self, recipe): # 2. cluster-config/repos.yaml # - if the repos.yaml file exists it will contain a list of relative paths # to search for package - # 1. builtin repo + # 1. package repos from config.yaml in the order specified (typically + # only spack-packages builtin repo) - # Build a list of repos with packages to install. + # Build a list of repos with packages to install from system config and recipe. repos = [] - # check for a repo in the recipe - if recipe.spack_repo is not None: - self._logger.debug(f"adding recipe spack package repo: {recipe.spack_repo}") - repos.append(recipe.spack_repo) - # look for repos.yaml file in the system configuration repo_yaml = recipe.system_config_path / "repos.yaml" if repo_yaml.exists() and repo_yaml.is_file(): @@ -361,7 +353,7 @@ def generate(self, recipe): self._logger.error(f"{repo_path} from {repo_yaml} is not a spack package repository") raise RuntimeError("invalid system-provided package repository") - self._logger.debug(f"full list of spack package repo: {repos}") + self._logger.debug(f"full list of system spack package repos: {repos}") # Delete the store/repo path, if it already exists. # Do this so that incremental builds (though not officially supported) won't break if a repo is updated. @@ -378,53 +370,82 @@ def generate(self, recipe): self._logger.debug(f"created the repo packages path {pkg_dst}") # create the repository step 2: create the repo.yaml file that - # configures alps and builtin repos + # configures the alps repo with (repo_dst / "repo.yaml").open("w") as f: - f.write( - """\ -repo: - namespace: alps - api: v2.0 -""" - ) + f.write(_REPO_YAML.format(namespace="alps")) + + # If the recipe provides a package repo, install it as a separate + # "recipe" repo in the store with highest precedence. + has_recipe_repo = recipe.spack_repo is not None + if has_recipe_repo: + recipe_dst = repos_path / "recipe" + self._logger.debug(f"creating the recipe spack repo in {recipe_dst}") + if recipe_dst.exists(): + self._logger.debug(f"{recipe_dst} exists ... deleting") + shutil.rmtree(recipe_dst) + + recipe_pkg_dst = recipe_dst / "packages" + recipe_pkg_dst.mkdir(mode=0o755, parents=True) + + with (recipe_dst / "repo.yaml").open("w") as f: + f.write(_REPO_YAML.format(namespace="recipe")) + + packages_path = recipe.spack_repo / "packages" + for pkg_path in packages_path.iterdir(): + dst = recipe_pkg_dst / pkg_path.name + if pkg_path.is_dir(): + self._logger.debug(f" installing recipe package {pkg_path} to {recipe_pkg_dst}") + install(pkg_path, dst) # create the repository step 2: create the repos.yaml file in build_path/config repos_yaml_template = jinja_env.get_template("repos.yaml") with (config_path / "repos.yaml").open("w") as f: repo_path = recipe.mount / "repos" / "spack_repo" / "alps" - builtin_repo_path = recipe.mount / "repos" / "spack_repo" / "builtin" + recipe_repo_path = recipe.mount / "repos" / "spack_repo" / "recipe" + package_repos = [ + { + "name": pkg_repo["name"], + "path": (recipe.mount / "repos" / "spack_repo" / pkg_repo["name"]).as_posix(), + } + for pkg_repo in spack_meta["packages"] + ] f.write( repos_yaml_template.render( repo_path=repo_path.as_posix(), - builtin_repo_path=builtin_repo_path.as_posix(), + package_repos=package_repos, + recipe_repo_path=recipe_repo_path.as_posix(), + has_recipe_repo=has_recipe_repo, verbose=False, ) ) f.write("\n") - # Iterate over the source repositories copying their contents to the consolidated repo in the uenv. - # Do overwrite packages that have been copied from an earlier source repo, enforcing a descending - # order of precidence. - if len(repos) > 0: - for repo_src in repos: - self._logger.debug(f"installing repo {repo_src}") - packages_path = repo_src / "packages" - for pkg_path in packages_path.iterdir(): - dst = pkg_dst / pkg_path.name - if pkg_path.is_dir() and not dst.exists(): - self._logger.debug(f" installing package {pkg_path} to {pkg_dst}") - install(pkg_path, dst) - elif dst.exists(): - self._logger.debug(f" NOT installing package {pkg_path}") - - # Copy the builtin repo to store, delete if it already exists. - spack_packages_builtin_path = spack_packages_path / "repos" / "spack_repo" / "builtin" - spack_packages_store_path = store_path / "repos" / "spack_repo" / "builtin" - self._logger.debug(f"copying builtin repo from {spack_packages_builtin_path} to {spack_packages_store_path}") - if spack_packages_store_path.exists(): - self._logger.debug(f"{spack_packages_store_path} exists ... deleting") - shutil.rmtree(spack_packages_store_path) - install(spack_packages_builtin_path, spack_packages_store_path) + # Iterate over the alps and recipe repositories copying their contents + # to the final repo locations. Because of the order of repos in the + # repos.yaml config file, recipe packages have precedence. + for repo_src in repos: + self._logger.debug(f"installing repo {repo_src}") + packages_path = repo_src / "packages" + for pkg_path in packages_path.iterdir(): + dst = pkg_dst / pkg_path.name + if pkg_path.is_dir() and not dst.exists(): + self._logger.debug(f" installing package {pkg_path} to {pkg_dst}") + install(pkg_path, dst) + elif dst.exists(): + self._logger.debug(f" NOT installing package {pkg_path}") + + # Copy all package repos defined in config.yaml to their final repo + # locations. + for pkg_repo in spack_meta["packages"]: + clone_path = pkg_repo["path"] + name = pkg_repo["name"] + src_path = clone_path / pkg_repo["repo_path"] + dst_path = store_path / "repos" / "spack_repo" / name + self._logger.debug(f"copying repo '{name}' from {src_path} to {dst_path}") + if dst_path.exists(): + self._logger.debug(f"{dst_path} exists ... deleting") + shutil.rmtree(dst_path) + install(src_path, dst_path) # Generate the makefile and spack.yaml files that describe the compilers compiler_files = recipe.compiler_files diff --git a/stackinator/etc/envvars.py b/stackinator/etc/envvars.py index a20fabce..fb4c17e1 100755 --- a/stackinator/etc/envvars.py +++ b/stackinator/etc/envvars.py @@ -621,12 +621,27 @@ def meta_impl(args): if args.spack is not None: spack_url, spack_ref, spack_commit = args.spack.split(",") - spack_packages_url = None - spack_packages_ref = None - spack_packages_commit = None - if args.spack_packages is not None: - spack_packages_url, spack_packages_ref, spack_packages_commit = args.spack_packages.split(",") spack_path = f"{args.mount}/config".replace("//", "/") + scalar_vars = { + "UENV_SPACK_CONFIG_PATH": spack_path, + "UENV_SPACK_URL": spack_url, + "UENV_SPACK_REF": spack_ref, + "UENV_SPACK_COMMIT": spack_commit, + } + if args.spack_package_repo: + repo_names = [] + for entry in args.spack_package_repo: + name, url, ref, commit = entry.split(",") + repo_names.append(name) + name_upper = name.upper().replace("-", "_") + scalar_vars[f"UENV_PACKAGE_REPO_{name_upper}_URL"] = url + scalar_vars[f"UENV_PACKAGE_REPO_{name_upper}_REF"] = ref + scalar_vars[f"UENV_PACKAGE_REPO_{name_upper}_COMMIT"] = commit + if name == "builtin": + scalar_vars["UENV_SPACK_PACKAGES_URL"] = url + scalar_vars["UENV_SPACK_PACKAGES_REF"] = ref + scalar_vars["UENV_SPACK_PACKAGES_COMMIT"] = commit + scalar_vars["UENV_PACKAGE_REPOS"] = ",".join(repo_names) meta["views"]["spack"] = { "activate": "/dev/null", "description": "configure spack upstream", @@ -636,15 +651,7 @@ def meta_impl(args): "type": "augment", "values": { "list": {}, - "scalar": { - "UENV_SPACK_CONFIG_PATH": spack_path, - "UENV_SPACK_URL": spack_url, - "UENV_SPACK_REF": spack_ref, - "UENV_SPACK_COMMIT": spack_commit, - "UENV_SPACK_PACKAGES_URL": spack_packages_url, - "UENV_SPACK_PACKAGES_REF": spack_packages_ref, - "UENV_SPACK_PACKAGES_COMMIT": spack_packages_commit, - }, + "scalar": scalar_vars, }, }, } @@ -686,9 +693,11 @@ def meta_impl(args): default=None, ) uenv_parser.add_argument( - "--spack-packages", - help='configure spack-packages repository metadata. Format is "spack_url,git_ref,git_commit"', + "--spack-package-repo", + help="configure spack package repository metadata. " + 'Format is "name,spack_url,git_ref,git_commit". Can be repeated.', type=str, + action="append", default=None, ) diff --git a/stackinator/recipe.py b/stackinator/recipe.py index 30bf8b58..607ae6db 100644 --- a/stackinator/recipe.py +++ b/stackinator/recipe.py @@ -213,6 +213,39 @@ def spack_repo(self): return repo_path return None + _RESERVED_REPO_NAMES = {"alps", "recipe"} + + @property + def spack_package_repos(self): + packages = self.config["spack"]["packages"] + if isinstance(packages.get("repo"), str): + return [ + { + "name": "builtin", + "url": packages["repo"], + "ref": packages.get("commit"), + "repo_path": packages.get("path", "repos/spack_repo/builtin"), + } + ] + repos = [ + { + "name": name, + "url": val["repo"], + "ref": val.get("commit"), + "repo_path": val.get("path", f"repos/spack_repo/{name}"), + } + for name, val in packages.items() + ] + for repo in repos: + name = repo["name"] + if name in self._RESERVED_REPO_NAMES: + raise RuntimeError( + f"The package repo name '{name}' is reserved for stackinator internal use. " + f"Reserved names are: {self._RESERVED_REPO_NAMES}. " + "Choose a different name in config.yaml:spack:packages." + ) + return repos + # Returns: # Path: of the recipe extra path if it exists # None: if there is no user-provided extra path in the recipe diff --git a/stackinator/schema/config.json b/stackinator/schema/config.json index d9de503e..6a9799a1 100644 --- a/stackinator/schema/config.json +++ b/stackinator/schema/config.json @@ -4,6 +4,24 @@ "type" : "object", "additionalProperties": false, "required": ["name", "spack"], + "defs": { + "repo_def": { + "type": "object", + "additionalProperties": false, + "required": ["repo", "commit"], + "properties": { + "repo": { + "type": "string" + }, + "commit": { + "type": "string" + }, + "path": { + "type": "string" + } + } + } + }, "properties" : { "name" : { "type": "string" @@ -28,21 +46,21 @@ "default": null }, "packages": { - "type": "object", - "additionalProperties": false, - "required": ["repo"], - "properties" : { - "repo": { - "type": "string" + "oneOf": [ + { + "$ref": "#/defs/repo_def" }, - "commit": { - "oneOf": [ - {"type" : "string"}, - {"type" : "null"} - ], - "default": null + { + "type": "object", + "minProperties": 1, + "patternProperties": { + "^\\w[\\w-]*$": { + "$ref": "#/defs/repo_def" + } + }, + "additionalProperties": false } - } + ] } } }, diff --git a/stackinator/templates/Makefile b/stackinator/templates/Makefile index 10a0ea58..b5e8442b 100644 --- a/stackinator/templates/Makefile +++ b/stackinator/templates/Makefile @@ -59,7 +59,7 @@ modules-done: environments generate-config env-meta: generate-config environments{% if modules %} modules-done{% endif %} - $(SANDBOX) $(BUILD_ROOT)/envvars.py uenv {% if modules %}--modules{% endif %} --spack='{{ spack_meta.url }},{{ spack_meta.ref }},{{ spack_meta.commit }}' --spack-packages='{{ spack_meta.packages_url }},{{ spack_meta.packages_ref }},{{ spack_meta.packages_commit }}' $(STORE) + $(SANDBOX) $(BUILD_ROOT)/envvars.py uenv {% if modules %}--modules{% endif %} --spack='{{ spack_meta.url }},{{ spack_meta.ref }},{{ spack_meta.commit }}'{% for pkg_repo in spack_meta.packages %} --spack-package-repo='{{ pkg_repo.name }},{{ pkg_repo.url }},{{ pkg_repo.ref }},{{ pkg_repo.commit }}'{% endfor %} $(STORE) touch env-meta post-install: env-meta diff --git a/stackinator/templates/repos.yaml b/stackinator/templates/repos.yaml index 3cf34f26..18f2e318 100644 --- a/stackinator/templates/repos.yaml +++ b/stackinator/templates/repos.yaml @@ -1,3 +1,8 @@ repos: +{% if has_recipe_repo %} + recipe: {{ recipe_repo_path }} +{% endif %} alps: {{ repo_path }} - builtin: {{ builtin_repo_path }} +{% for repo in package_repos %} + {{ repo.name }}: {{ repo.path }} +{% endfor %} diff --git a/unittests/data/arbor-uenv/meta/env.json.in b/unittests/data/arbor-uenv/meta/env.json.in index 25086e88..fa9e9b41 100644 --- a/unittests/data/arbor-uenv/meta/env.json.in +++ b/unittests/data/arbor-uenv/meta/env.json.in @@ -9,11 +9,13 @@ "arbor": { "activate": "@@mount@@/env/arbor/activate.sh", "description": "", + "recipe_variables": {"scalar": {}, "list": {}}, "root": "@@mount@@/env/arbor" }, "develop": { "activate": "@@mount@@/env/develop/activate.sh", "description": "", + "recipe_variables": {"scalar": {}, "list": {}}, "root": "@@mount@@/env/develop" } } diff --git a/unittests/recipes/base-nvgpu/config.yaml b/unittests/recipes/base-nvgpu/config.yaml index ccdd1f7c..745cb8f9 100644 --- a/unittests/recipes/base-nvgpu/config.yaml +++ b/unittests/recipes/base-nvgpu/config.yaml @@ -5,4 +5,5 @@ spack: commit: 6408b51 packages: repo: https://github.com/spack/spack-packages.git + commit: 18b4066 version: 2 diff --git a/unittests/recipes/cache/config.yaml b/unittests/recipes/cache/config.yaml index a3c01cf7..d1d9aedd 100644 --- a/unittests/recipes/cache/config.yaml +++ b/unittests/recipes/cache/config.yaml @@ -5,4 +5,5 @@ spack: commit: 6408b51 packages: repo: https://github.com/spack/spack-packages.git + commit: 18b4066 version: 2 diff --git a/unittests/recipes/host-recipe/config.yaml b/unittests/recipes/host-recipe/config.yaml index 780cf031..afa61217 100644 --- a/unittests/recipes/host-recipe/config.yaml +++ b/unittests/recipes/host-recipe/config.yaml @@ -6,4 +6,5 @@ spack: repo: https://github.com/spack/spack.git packages: repo: https://github.com/spack/spack-packages.git + commit: releases/v0.23 version: 2 diff --git a/unittests/recipes/with-multi-repos/compilers.yaml b/unittests/recipes/with-multi-repos/compilers.yaml new file mode 100644 index 00000000..bce71846 --- /dev/null +++ b/unittests/recipes/with-multi-repos/compilers.yaml @@ -0,0 +1,2 @@ +gcc: + version: "11" diff --git a/unittests/recipes/with-multi-repos/config.yaml b/unittests/recipes/with-multi-repos/config.yaml new file mode 100644 index 00000000..b547288a --- /dev/null +++ b/unittests/recipes/with-multi-repos/config.yaml @@ -0,0 +1,14 @@ +name: with-multi-repos +store: '/user-environment' +spack: + repo: https://github.com/spack/spack.git + commit: v21.0 + packages: + my-packages: + repo: https://github.com/example/spack-packages.git + commit: v1.0 + path: custom/path/to/packages + other-packages: + repo: https://github.com/example/other-packages.git + commit: v2.0 +version: 2 diff --git a/unittests/recipes/with-multi-repos/environments.yaml b/unittests/recipes/with-multi-repos/environments.yaml new file mode 100644 index 00000000..33735ee2 --- /dev/null +++ b/unittests/recipes/with-multi-repos/environments.yaml @@ -0,0 +1,13 @@ +gcc-env: + compiler: [gcc] + unify: true + specs: + - osu-micro-benchmarks@5.9 + - hdf5 +mpi + network: + mpi: cray-mpich +tools: + compiler: [gcc] + unify: true + specs: + - cmake diff --git a/unittests/recipes/with-repo/config.yaml b/unittests/recipes/with-repo/config.yaml index a3cb2940..e9a6f54a 100644 --- a/unittests/recipes/with-repo/config.yaml +++ b/unittests/recipes/with-repo/config.yaml @@ -5,4 +5,5 @@ spack: commit: v21.0 packages: repo: https://github.com/spack/spack-packages.git + commit: v21.0 version: 2 diff --git a/unittests/test-envvars.sh b/unittests/test-envvars.sh index 97cba66b..fe66c59f 100755 --- a/unittests/test-envvars.sh +++ b/unittests/test-envvars.sh @@ -36,7 +36,7 @@ find $scratch_path -name env.json echo "===== running final meta data stage ${mount_path}" -../stackinator/etc/envvars.py uenv ${mount_path}/ --modules --spack="https://github.com/spack/spack.git,releases/v0.20" --spack-packages="https://github.com/spack/spack.git,develop" +../stackinator/etc/envvars.py uenv ${mount_path}/ --modules --spack="https://github.com/spack/spack.git,releases/v0.20,abc123" --spack-package-repo="builtin,https://github.com/spack/spack-packages.git,develop,abc123" echo echo "===== develop" @@ -58,3 +58,38 @@ echo "===== modules view" echo cat ${meta_path} | jq .views.modules +echo +echo "===== test UENV_PACKAGE_REPOS with multiple package repos" +rm -rf ${mount_path} +mkdir -p ${scratch_path} +cp -R ${input_path} ${mount_path} +if [[ "$OSTYPE" == "linux-gnu"* ]]; then + sed -i "s|@@mount@@|${mount_path}|g" ${meta_in_path} +else + sed -i '' "s|@@mount@@|${mount_path}|g" ${meta_in_path} +fi + +../stackinator/etc/envvars.py view ${mount_path}/env/arbor /dev/shm/bcumming/arbor +../stackinator/etc/envvars.py view --prefix_paths="LD_LIBRARY_PATH=lib:lib64" ${mount_path}/env/develop /dev/shm/bcumming/arbor + +../stackinator/etc/envvars.py uenv ${mount_path}/ \ + --spack="https://github.com/spack/spack.git,releases/v0.21,abc123def" \ + --spack-package-repo="my-packages,https://github.com/example/spack-packages.git,v1.0,abc123" \ + --spack-package-repo="other-packages,https://github.com/example/other-packages.git,main,def456" + +UENV_PACKAGE_REPOS=$(cat ${meta_path} | jq -r '.views.spack.env.values.scalar.UENV_PACKAGE_REPOS') +echo "UENV_PACKAGE_REPOS=$UENV_PACKAGE_REPOS" +if [[ "$UENV_PACKAGE_REPOS" != "my-packages,other-packages" ]]; then + echo "FAIL: expected my-packages,other-packages, got '${UENV_PACKAGE_REPOS}'" + exit 1 +fi + +UENV_PACKAGE_REPO_MY_PACKAGES_URL=$(cat ${meta_path} | jq -r '.views.spack.env.values.scalar.UENV_PACKAGE_REPO_MY_PACKAGES_URL') +echo "UENV_PACKAGE_REPO_MY_PACKAGES_URL=$UENV_PACKAGE_REPO_MY_PACKAGES_URL" +if [[ "$UENV_PACKAGE_REPO_MY_PACKAGES_URL" != "https://github.com/example/spack-packages.git" ]]; then + echo "FAIL: expected https://github.com/example/spack-packages.git, got '${UENV_PACKAGE_REPO_MY_PACKAGES_URL}'" + exit 1 +fi + +echo "PASSED" + diff --git a/unittests/test_schema.py b/unittests/test_schema.py index 607f80e6..51a320f9 100644 --- a/unittests/test_schema.py +++ b/unittests/test_schema.py @@ -20,7 +20,7 @@ def yaml_path(test_path): return test_path / "yaml" -@pytest.fixture(params=["host-recipe", "base-nvgpu", "cache", "with-repo"]) +@pytest.fixture(params=["host-recipe", "base-nvgpu", "cache", "with-repo", "with-multi-repos"]) def recipe(request): return request.param @@ -37,10 +37,9 @@ def test_config_yaml(yaml_path): schema.ConfigValidator.validate(raw) assert raw["store"] == "/user-environment" assert raw["spack"]["commit"] is None - assert raw["spack"]["packages"]["commit"] is None assert raw["description"] is None - # no spack:commit + # single repo format with packages commit config = dedent(""" version: 2 name: env-without-spack-commit @@ -56,27 +55,22 @@ def test_config_yaml(yaml_path): ) schema.ConfigValidator.validate(raw) assert raw["spack"]["commit"] is None - assert raw["spack"]["packages"]["commit"] is not None + assert raw["spack"]["packages"]["commit"] == "develop-packages" assert raw["description"] is None - # no spack:packages:commit - config = dedent(""" - version: 2 - name: env-without-spack-packages-commit - spack: - repo: https://github.com/spack/spack.git - commit: develop - packages: + # single repo format missing packages commit should fail + with pytest.raises(Exception): + config = dedent(""" + version: 2 + name: env-no-pkg-commit + spack: repo: https://github.com/spack/spack.git - """) - raw = yaml.load( - config, - Loader=yaml.Loader, - ) - schema.ConfigValidator.validate(raw) - assert raw["spack"]["commit"] == "develop" - assert raw["spack"]["packages"]["commit"] is None - assert raw["description"] is None + commit: develop + packages: + repo: https://github.com/spack/spack.git + """) + raw = yaml.load(config, Loader=yaml.Loader) + schema.ConfigValidator.validate(raw) # full config with open(yaml_path / "config.full.yaml") as fid: @@ -98,6 +92,99 @@ def test_config_yaml(yaml_path): raw = yaml.load(config, Loader=yaml.Loader) schema.ConfigValidator.validate(raw) + # map format: single entry + config = dedent(""" + version: 2 + name: map-single + spack: + repo: https://github.com/spack/spack.git + packages: + my-packages: + repo: https://github.com/example/spack-packages.git + commit: v1.0 + """) + raw = yaml.load(config, Loader=yaml.Loader) + schema.ConfigValidator.validate(raw) + assert "my-packages" in raw["spack"]["packages"] + assert raw["spack"]["packages"]["my-packages"]["repo"] == "https://github.com/example/spack-packages.git" + assert raw["spack"]["packages"]["my-packages"]["commit"] == "v1.0" + + # map format: multiple entries with commits + config = dedent(""" + version: 2 + name: map-multi + spack: + repo: https://github.com/spack/spack.git + packages: + my-packages: + repo: https://github.com/example/spack-packages.git + commit: v1.0 + other-packages: + repo: https://github.com/example/other-packages.git + commit: v2.0 + """) + raw = yaml.load(config, Loader=yaml.Loader) + schema.ConfigValidator.validate(raw) + assert raw["spack"]["packages"]["my-packages"]["commit"] == "v1.0" + assert raw["spack"]["packages"]["other-packages"]["commit"] == "v2.0" + + # map format: empty map should fail + with pytest.raises(Exception): + config = dedent(""" + version: 2 + name: map-empty + spack: + repo: https://github.com/spack/spack.git + packages: {} + """) + raw = yaml.load(config, Loader=yaml.Loader) + schema.ConfigValidator.validate(raw) + + # map format: entry missing repo should fail + with pytest.raises(Exception): + config = dedent(""" + version: 2 + name: map-no-repo + spack: + repo: https://github.com/spack/spack.git + packages: + my-packages: + commit: v1.0 + """) + raw = yaml.load(config, Loader=yaml.Loader) + schema.ConfigValidator.validate(raw) + + # map format: custom path + config = dedent(""" + version: 2 + name: map-custom-path + spack: + repo: https://github.com/spack/spack.git + packages: + my-packages: + repo: https://github.com/example/spack-packages.git + commit: v1.0 + path: custom/repo/location + """) + raw = yaml.load(config, Loader=yaml.Loader) + schema.ConfigValidator.validate(raw) + assert raw["spack"]["packages"]["my-packages"]["path"] == "custom/repo/location" + + # map format: no path (default behavior) + config = dedent(""" + version: 2 + name: map-no-path + spack: + repo: https://github.com/spack/spack.git + packages: + my-packages: + repo: https://github.com/example/spack-packages.git + commit: v2.0 + """) + raw = yaml.load(config, Loader=yaml.Loader) + schema.ConfigValidator.validate(raw) + assert "path" not in raw["spack"]["packages"]["my-packages"] + def test_recipe_config_yaml(recipe_path): # validate the config.yaml in the test recipes diff --git a/unittests/yaml/config.defaults.yaml b/unittests/yaml/config.defaults.yaml index 7197272a..cfd68aa0 100644 --- a/unittests/yaml/config.defaults.yaml +++ b/unittests/yaml/config.defaults.yaml @@ -7,7 +7,6 @@ spack: #commit: 6408b51 packages: repo: https://github.com/spack/spack-packages.git - # default: None == no `git checkout` command - #commit: 6408b51 + commit: 6408b51 #modules: True version: 2