Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 23 additions & 21 deletions tutorial_basics.rst
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,8 @@ Let's go ahead and install ``gmake``,

You will see Spack installed ``gmake``, ``gcc-runtime``, and ``glibc``.
The ``glibc`` and ``gcc-runtime`` packages are automatically tracked by Spack to manage consistency requirements among compiler runtimes.
These do not represent separate installs from source, but represent aspects of the compiler Spack used for the install.
For the rest of this section, we will ignore these components and focus on the rest of the install.
They do not represent separate software builds from source, but are records of the system's compiler runtime components Spack used for the install.
For the rest of this section, we will ignore these components and focus on the packages explicitly installed.

Spack can install software either from source or from a binary cache.
Packages in the binary cache are signed with GPG for security.
Expand All @@ -89,14 +89,14 @@ To be able to install from the binary cache, we will need to configure Spack wit
You'll learn more about configuring Spack later in the tutorial, but for now you will be able to install the rest of the packages in the tutorial from a binary cache using the same ``spack install`` command.
By default this will install the binary cached version if it exists and fall back on installing from source if it does not.

Spack's spec syntax is the interface by which we can request specific configurations of the package.
Spack's "spec" syntax is the interface by which we can request specific configurations of a package.
The ``%`` sigil is used to specify compilers.

.. literalinclude:: outputs/basics/zlib-clang.out
:language: console

Note that this installation is located separately from the previous one.
We will discuss this in more detail later, but this is part of what allows Spack to support arbitrarily versioned software.
We will discuss this in more detail later, but this is part of what allows Spack to support many versions of software packages.

You can check for particular versions before requesting them.
We will use the ``spack versions`` command to see the available versions, and then install a different version of ``zlib-ng``.
Expand All @@ -115,7 +115,7 @@ The ``@`` sigil is used to specify versions, both of packages and of compilers.
The spec syntax also includes compiler flags.
Spack accepts ``cppflags``, ``cflags``, ``cxxflags``, ``fflags``, ``ldflags``, and ``ldlibs`` parameters.
The values of these fields must be quoted on the command line if they include spaces.
These values are injected into the compile line automatically by the Spack compiler wrappers.
These values are injected into the compilation commands automatically by the Spack compiler wrappers.

.. literalinclude:: outputs/basics/zlib-O3.out
:language: console
Expand All @@ -134,7 +134,7 @@ Spack generates a hash for each spec.
This hash is a function of the full provenance of the package, so any change to the spec affects the hash.
Spack uses this value to compare specs and to generate unique installation directories for every combinatorial version.
As we move into more complicated packages with software dependencies, we can see that Spack reuses existing packages to satisfy a dependency.
By default, Spack tries hard to reuse existing installations as dependencies, either from a local store or from configured remote buildcaches.
By default, Spack tries hard to reuse existing installations as dependencies, either from a local store or from configured remote binary caches.
This minimizes unwanted rebuilds of common dependencies, in particular if you update Spack frequently.

.. literalinclude:: outputs/basics/tcl.out
Expand Down Expand Up @@ -165,7 +165,7 @@ Note that each package has a top-level entry, even if it also appears as a depen

Let's move on to slightly more complicated packages.
HDF5 is a good example of a more complicated package, with an MPI dependency.
If we install it "out of the box," it will build with OpenMPI.
If we install it with default settings it will build with OpenMPI.

.. literalinclude:: outputs/basics/hdf5.out
:language: console
Expand All @@ -181,14 +181,14 @@ Here we can install HDF5 without MPI support.
:language: console

We might also want to install HDF5 with a different MPI implementation.
While MPI is not a package itself, packages can depend on abstract interfaces like MPI.
Spack handles these through "virtual dependencies." A package, such as HDF5, can depend on the MPI interface.
Other packages (``openmpi``, ``mpich``, ``mvapich2``, etc.) provide the MPI interface.
Any of these providers can be requested for an MPI dependency.
For example, we can build HDF5 with MPI support provided by MPICH by specifying a dependency on ``mpich``.
While ``mpi`` itself is a virtual package representing an interface, other packages can depend on such abstract interfaces.
Spack handles these through "virtual dependencies." A package, such as HDF5, can depend on the ``mpi`` virtual package (the interface).
Actual MPI implementation packages (like ``openmpi``, ``mpich``, ``mvapich2``, etc.) provide the MPI interface.
Any of these providers can be requested to satisfy an MPI dependency.
For example, we can build HDF5 with MPI support provided by MPICH by specifying a dependency on ``mpich`` (e.g., ``hdf5 ^mpich``).
Spack also supports versioning of virtual dependencies.
A package can depend on the MPI interface at version 3, and provider packages specify what version of the interface *they* provide.
The partial spec ``^mpi@3`` can be satisfied by any of several providers.
A package can depend on the MPI interface at version 3 (e.g., ``hdf5 ^mpi@3``), and provider packages specify what version of the interface *they* provide.
The partial spec ``^mpi@3`` can be satisfied by any of several MPI implementation packages that provide MPI version 3.

.. literalinclude:: outputs/basics/hdf5-hl-mpi.out
:language: console
Expand All @@ -212,7 +212,7 @@ Now let's look at an even more complicated package.
:language: console

Now we're starting to see the power of Spack.
Trilinos in its default configuration has 23 top level dependencies, many of which have dependencies of their own.
Trilinos in its default configuration has 23 direct dependencies, many of which have dependencies of their own.
Installing more complex packages can take days or weeks even for an experienced user.
Although we've done a binary installation for the tutorial, a source installation of Trilinos using Spack takes about 3 hours (depending on the system), but only 20 seconds of programmer time.

Expand Down Expand Up @@ -269,16 +269,18 @@ We can uninstall packages by spec using the same syntax as install.

We can also uninstall packages by referring only to their hash.

We can use either ``-f`` (force) or ``-R`` (remove dependents as well) to remove packages that are required by another installed package.
We can use either the ``--force`` (or ``-f``) flag or the ``--dependents`` (or ``-R``) flag to remove packages that are required by another installed package.
Use ``--force`` to remove just the specified package, leaving dependents broken.
Use ``--dependents`` to remove the specified package and all of its dependents.

.. literalinclude:: outputs/basics/uninstall-needed.out
:language: console

.. literalinclude:: outputs/basics/uninstall-r-needed.out
:language: console

Spack will not uninstall packages that are not sufficiently specified.
The ``-a`` (all) flag can be used to uninstall multiple packages at once.
Spack will not uninstall packages that are not sufficiently specified specified (i.e., if the spec is ambiguous and matches multiple installed packages).
The ``--all`` (or ``-a``) flag can be used to uninstall all packages matching an ambiguous spec.

.. literalinclude:: outputs/basics/uninstall-ambiguous.out
:language: console
Expand Down Expand Up @@ -314,13 +316,13 @@ Customizing Compilers
---------------------

Spack manages a list of available compilers on the system, detected automatically from the user's ``PATH`` variable.
The ``spack compilers`` command is an alias for the command ``spack compiler list``.
The ``spack compilers`` command is an alias for ``spack compiler list``.

.. literalinclude:: outputs/basics/compilers.out
:language: console

The compilers are maintained in a YAML file.
Later in the tutorial you will learn how to configure compilers by hand for special cases.
The compilers are maintained in a YAML file (``compilers.yaml``).
Later in the tutorial, you will learn how to configure compilers by hand for special cases.
Spack also has tools to add compilers, and compilers built with Spack can be added to the configuration.

.. literalinclude:: outputs/basics/install-gcc-12.1.0.out
Expand Down
53 changes: 25 additions & 28 deletions tutorial_binary_cache.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,10 @@
Binary Caches Tutorial
==================================

In this section of the tutorial you will learn how to share Spack built binaries across machines and users using build caches.
In this section of the tutorial, you will learn how to share Spack-built binaries across machines and users using build caches.

We will explore a few concepts that apply to all types of build caches, but the focus is primarily on **OCI container registries** like Docker Hub or Github Packages as a storage backend for binary caches.
Spack supports a range of storage backends, like an ordinary filesystem, S3, and Google Cloud Storage, but OCI build caches have a few interesting properties that make them worth exploring more in depth.
We will explore a few concepts that apply to all types of build caches, but the focus is primarily on **OCI container registries** (like Docker Hub or Github Packages) as a storage backend for binary caches.
Spack supports a range of storage backends, such as an ordinary filesystem, Amazon S3, and Google Cloud Storage, but OCI build caches have a few interesting properties that make them worth exploring more in-depth.

Before we configure a build cache, let's install the ``julia`` package, which is an interesting example because it has some non-trivial dependencies like ``llvm``, and features an interactive REPL that we can use to verify that the installation works.

Expand Down Expand Up @@ -42,20 +42,22 @@ Setting up an OCI build cache on GitHub Packages

For this tutorial we will be using GitHub Packages as an OCI registry, since most people have a GitHub account and it's easy to use.

First go to `<https://github.com/settings/tokens>`_ to generate a Personal access token with ``write:packages`` permissions.
First, go to `<https://github.com/settings/tokens>`_ to generate a Personal Access Token (classic) with ``write:packages`` permissions.
Copy this token.

Next, we will add this token to the mirror config section of the Spack environment:
Next, we will add this token to the mirror configuration section for the Spack environment.
Replace `<your-github-username>` with your GitHub username and `<your-github-username-or-org>` with your GitHub username or an organization where you have permission to create packages.
The build cache name, `buildcache-${USER}-${HOSTNAME}`, is a suggestion; you can choose your own.

.. code-block:: console

$ export MY_OCI_TOKEN=<token>
$ export MY_OCI_TOKEN=<paste-your-token-here>
$ spack -e . mirror add \
--oci-username <user> \
--oci-username <your-github-username> \
--oci-password-variable MY_OCI_TOKEN \
--unsigned \
my-mirror \
oci://ghcr.io/<github_user>/buildcache-${USER}-${HOSTNAME}
oci://ghcr.io/<your-github-username-or-org>/buildcache-${USER}-${HOSTNAME}


.. note ::
Expand Down Expand Up @@ -101,7 +103,7 @@ which outputs
...
==> Pushed julia@1.9.3/dfzhutf to ghcr.io/<user>/buildcache-<user>-<host>:julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl.spack

The location of the pushed package
The location of the pushed package, when referred to as a OCI image, will be:

.. code-block:: text

Expand Down Expand Up @@ -142,7 +144,7 @@ The easiest way to do this is to override the ``mirrors`` config section in the
secret_variable: MY_OCI_TOKEN
signed: false

An "overwrite install" should be enough to show that the build cache is used:
An "overwrite install" should be enough to show that the build cache is used (output will vary based on your specific configuration):

.. code-block:: console

Expand Down Expand Up @@ -177,10 +179,9 @@ For convenience you can also run ``spack buildcache push --update-index ...`` to

.. note::

As of Spack 0.22, build caches can be used across different Linux distros. The concretizer
will reuse specs that have a host compatible ``libc`` dependency (e.g. ``glibc`` or ``musl``).
For packages compiled with ``gcc`` (and a few others), users do not have to install compilers
first, as the build cache contains the compiler runtime libraries as a separate package.
As of Spack 0.22, build caches can be used across different Linux distros.
The concretizer will reuse specs that have a host-compatible ``libc`` dependency (e.g. ``glibc`` or ``musl``).
For packages compiled with ``gcc`` (and a few other compilers), users do not have to install compilers first, as the build cache contains the compiler runtime libraries as a separate package dependency.

After an index is created, it's possible to list the available packages in the build cache:

Expand Down Expand Up @@ -255,7 +256,7 @@ Let's add a simple text editor like ``vim`` to our previous environment next to

$ spack -e . install --add vim

This time we push to the OCI registry, but also pass ``--tag julia-and-vim`` to instruct Spack to create an image for the environment as a whole, with a human-readable tag:
This time, when we push to the OCI registry, we also pass ``--tag julia-and-vim`` to instruct Spack to create an additional image tag for the environment as a whole, with a more human-readable name:


.. code-block:: console
Expand Down Expand Up @@ -291,12 +292,8 @@ For those familiar with ``Dockerfile`` syntax, it would structurally look like t

This approach is still valid, and the ``spack containerize`` command continues to exist, but it has a few downsides:

* When ``RUN spack -e /root/env install`` fails, ``docker`` will not cache the layer, meaning
that all dependencies that did install successfully are lost. Troubleshooting the build
typically means starting from scratch in ``docker run`` or on the host system.
* In certain CI environments, it is not possible to use ``docker build``. For example, the
CI script itself may already run in a docker container, and running ``docker build`` *safely*
inside a container is tricky.
* When ``RUN spack -e /root/env install`` fails, ``docker`` will not cache the layer, meaning that all dependencies that did install successfully are lost. Troubleshooting the build typically means starting from scratch either within a ``docker run`` session or on the host system.
* In certain CI environments, it is not possible to use ``docker build`` directly. For example, the CI script itself may already run in a Docker container, and running ``docker build`` *safely* inside a container (Docker-in-Docker) is tricky.

The takeaway is that Spack decouples the steps that ``docker build`` combines: build isolation, running the build, and creating an image.
You can run ``spack install`` on your host machine or in a container, and run ``spack buildcache push`` separately to create an image.
Expand All @@ -307,15 +304,15 @@ Relocation

Spack is different from many package managers in that it lets users choose where to install packages.
This makes Spack very flexible, as users can install packages in their home directory and do not need root privileges.
The downside is that sharing binaries is more complicated, as binaries may contain hard-coded, absolute paths to machine specific locations, which have to be adjusted when binaries are installed on a different machine.
The downside is that sharing binaries is more complicated, as binaries may contain hard-coded, absolute paths to machine specific locations, which have to be adjusted when these binaries are installed on a different machine or in a different path.

Fortunately Spack handles this automatically upon install from a binary cache.
But when you build binaries that are intended to be shared, there is one thing you have to keep in mind: Spack can relocate hard-coded paths in binaries *provided that the target prefix is shorter than the prefix used during the build*.

The reason is that binaries typically embed these absolute paths in string tables, which is a list of null terminated strings, to which the program stores offsets.
That means we can only modify strings in-place, and if the new path is longer than the old one, we would overwrite the next string in the table.

To maximize the chances of successful relocation, you should build your binaries in a relative long path.
To maximize the chances of successful relocation, you should build your binaries in a relatively long path.
Fortunately Spack can automatically pad paths to make them longer, using the following command:

.. code-block:: console
Expand All @@ -327,9 +324,9 @@ Using build caches in CI
------------------------

Build caches are a great way to speed up CI pipelines.
Both GitHub Actions and Gitlab CI support container registries, and this tutorial should give you a good starting point to leverage them.
Both GitHub Actions and GitLab CI support container registries, and this tutorial should give you a good starting point to leverage them.

Spack also provides a basic GitHub Action to already provide you with a binary cache:
Spack also provides a basic GitHub Action that already provides you with a binary cache:

.. code-block:: yaml

Expand All @@ -349,6 +346,6 @@ Summary

In this tutorial we have created a build cache on top of an OCI registry, which can be used

* to ``spack install julia vim`` on machines without source builds
* to automatically create container images for individual packages while pushing to the cache
* to create container images for multiple packages at once
* to run ``spack install julia vim`` on machines and have Spack fetch pre-built binaries instead of building from source.
* to automatically create container images for individual packages when pushing to the cache.
* to create container images for entire Spack environments (multiple packages) at once.
Loading