You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: tutorial_basics.rst
+23-21Lines changed: 23 additions & 21 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -75,8 +75,8 @@ Let's go ahead and install ``gmake``,
75
75
76
76
You will see Spack installed ``gmake``, ``gcc-runtime``, and ``glibc``.
77
77
The ``glibc`` and ``gcc-runtime`` packages are automatically tracked by Spack to manage consistency requirements among compiler runtimes.
78
-
These do not represent separate installs from source, but represent aspects of the compiler Spack used for the install.
79
-
For the rest of this section, we will ignore these components and focus on the rest of the install.
78
+
They do not represent separate software builds from source, but are records of the system's compiler runtime components Spack used for the install.
79
+
For the rest of this section, we will ignore these components and focus on the packages explicitly installed.
80
80
81
81
Spack can install software either from source or from a binary cache.
82
82
Packages in the binary cache are signed with GPG for security.
@@ -89,14 +89,14 @@ To be able to install from the binary cache, we will need to configure Spack wit
89
89
You'll learn more about configuring Spack later in the tutorial, but for now you will be able to install the rest of the packages in the tutorial from a binary cache using the same ``spack install`` command.
90
90
By default this will install the binary cached version if it exists and fall back on installing from source if it does not.
91
91
92
-
Spack's spec syntax is the interface by which we can request specific configurations of the package.
92
+
Spack's "spec" syntax is the interface by which we can request specific configurations of a package.
93
93
The ``%`` sigil is used to specify compilers.
94
94
95
95
.. literalinclude:: outputs/basics/zlib-clang.out
96
96
:language: console
97
97
98
98
Note that this installation is located separately from the previous one.
99
-
We will discuss this in more detail later, but this is part of what allows Spack to support arbitrarily versioned software.
99
+
We will discuss this in more detail later, but this is part of what allows Spack to support many versions of software packages.
100
100
101
101
You can check for particular versions before requesting them.
102
102
We will use the ``spack versions`` command to see the available versions, and then install a different version of ``zlib-ng``.
@@ -115,7 +115,7 @@ The ``@`` sigil is used to specify versions, both of packages and of compilers.
115
115
The spec syntax also includes compiler flags.
116
116
Spack accepts ``cppflags``, ``cflags``, ``cxxflags``, ``fflags``, ``ldflags``, and ``ldlibs`` parameters.
117
117
The values of these fields must be quoted on the command line if they include spaces.
118
-
These values are injected into the compile line automatically by the Spack compiler wrappers.
118
+
These values are injected into the compilation commands automatically by the Spack compiler wrappers.
119
119
120
120
.. literalinclude:: outputs/basics/zlib-O3.out
121
121
:language: console
@@ -134,7 +134,7 @@ Spack generates a hash for each spec.
134
134
This hash is a function of the full provenance of the package, so any change to the spec affects the hash.
135
135
Spack uses this value to compare specs and to generate unique installation directories for every combinatorial version.
136
136
As we move into more complicated packages with software dependencies, we can see that Spack reuses existing packages to satisfy a dependency.
137
-
By default, Spack tries hard to reuse existing installations as dependencies, either from a local store or from configured remote buildcaches.
137
+
By default, Spack tries hard to reuse existing installations as dependencies, either from a local store or from configured remote binary caches.
138
138
This minimizes unwanted rebuilds of common dependencies, in particular if you update Spack frequently.
139
139
140
140
.. literalinclude:: outputs/basics/tcl.out
@@ -165,7 +165,7 @@ Note that each package has a top-level entry, even if it also appears as a depen
165
165
166
166
Let's move on to slightly more complicated packages.
167
167
HDF5 is a good example of a more complicated package, with an MPI dependency.
168
-
If we install it "out of the box," it will build with OpenMPI.
168
+
If we install it with default settings it will build with OpenMPI.
169
169
170
170
.. literalinclude:: outputs/basics/hdf5.out
171
171
:language: console
@@ -181,14 +181,14 @@ Here we can install HDF5 without MPI support.
181
181
:language: console
182
182
183
183
We might also want to install HDF5 with a different MPI implementation.
184
-
While MPI is not a package itself, packages can depend on abstract interfaces like MPI.
185
-
Spack handles these through "virtual dependencies." A package, such as HDF5, can depend on the MPI interface.
186
-
Other packages (``openmpi``, ``mpich``, ``mvapich2``, etc.) provide the MPI interface.
187
-
Any of these providers can be requested for an MPI dependency.
188
-
For example, we can build HDF5 with MPI support provided by MPICH by specifying a dependency on ``mpich``.
184
+
While ``mpi`` itself is a virtual package representing an interface, other packages can depend on such abstract interfaces.
185
+
Spack handles these through "virtual dependencies." A package, such as HDF5, can depend on the ``mpi`` virtual package (the interface).
186
+
Actual MPI implementation packages (like ``openmpi``, ``mpich``, ``mvapich2``, etc.) provide the MPI interface.
187
+
Any of these providers can be requested to satisfy an MPI dependency.
188
+
For example, we can build HDF5 with MPI support provided by MPICH by specifying a dependency on ``mpich`` (e.g., ``hdf5 ^mpich``).
189
189
Spack also supports versioning of virtual dependencies.
190
-
A package can depend on the MPI interface at version 3, and provider packages specify what version of the interface *they* provide.
191
-
The partial spec ``^mpi@3`` can be satisfied by any of several providers.
190
+
A package can depend on the MPI interface at version 3 (e.g., ``hdf5 ^mpi@3``), and provider packages specify what version of the interface *they* provide.
191
+
The partial spec ``^mpi@3`` can be satisfied by any of several MPI implementation packages that provide MPI version 3.
@@ -212,7 +212,7 @@ Now let's look at an even more complicated package.
212
212
:language: console
213
213
214
214
Now we're starting to see the power of Spack.
215
-
Trilinos in its default configuration has 23 top level dependencies, many of which have dependencies of their own.
215
+
Trilinos in its default configuration has 23 direct dependencies, many of which have dependencies of their own.
216
216
Installing more complex packages can take days or weeks even for an experienced user.
217
217
Although we've done a binary installation for the tutorial, a source installation of Trilinos using Spack takes about 3 hours (depending on the system), but only 20 seconds of programmer time.
218
218
@@ -269,16 +269,18 @@ We can uninstall packages by spec using the same syntax as install.
269
269
270
270
We can also uninstall packages by referring only to their hash.
271
271
272
-
We can use either ``-f`` (force) or ``-R`` (remove dependents as well) to remove packages that are required by another installed package.
272
+
We can use either the ``--force`` (or ``-f``) flag or the ``--dependents`` (or ``-R``) flag to remove packages that are required by another installed package.
273
+
Use ``--force`` to remove just the specified package, leaving dependents broken.
274
+
Use ``--dependents`` to remove the specified package and all of its dependents.
Spack will not uninstall packages that are not sufficiently specified.
281
-
The ``-a`` (all) flag can be used to uninstall multiple packages at once.
282
+
Spack will not uninstall packages that are not sufficiently specified specified (i.e., if the spec is ambiguous and matches multiple installed packages).
283
+
The ``--all`` (or ``-a``) flag can be used to uninstall all packages matching an ambiguous spec.
Copy file name to clipboardExpand all lines: tutorial_binary_cache.rst
+25-28Lines changed: 25 additions & 28 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,10 +11,10 @@
11
11
Binary Caches Tutorial
12
12
==================================
13
13
14
-
In this section of the tutorial you will learn how to share Spackbuilt binaries across machines and users using build caches.
14
+
In this section of the tutorial, you will learn how to share Spack-built binaries across machines and users using build caches.
15
15
16
-
We will explore a few concepts that apply to all types of build caches, but the focus is primarily on **OCI container registries** like Docker Hub or Github Packages as a storage backend for binary caches.
17
-
Spack supports a range of storage backends, like an ordinary filesystem, S3, and Google Cloud Storage, but OCI build caches have a few interesting properties that make them worth exploring more indepth.
16
+
We will explore a few concepts that apply to all types of build caches, but the focus is primarily on **OCI container registries** (like Docker Hub or Github Packages) as a storage backend for binary caches.
17
+
Spack supports a range of storage backends, such as an ordinary filesystem, Amazon S3, and Google Cloud Storage, but OCI build caches have a few interesting properties that make them worth exploring more in-depth.
18
18
19
19
Before we configure a build cache, let's install the ``julia`` package, which is an interesting example because it has some non-trivial dependencies like ``llvm``, and features an interactive REPL that we can use to verify that the installation works.
20
20
@@ -42,20 +42,22 @@ Setting up an OCI build cache on GitHub Packages
42
42
43
43
For this tutorial we will be using GitHub Packages as an OCI registry, since most people have a GitHub account and it's easy to use.
44
44
45
-
First go to `<https://github.com/settings/tokens>`_ to generate a Personal access token with ``write:packages`` permissions.
45
+
First, go to `<https://github.com/settings/tokens>`_ to generate a Personal Access Token (classic) with ``write:packages`` permissions.
46
46
Copy this token.
47
47
48
-
Next, we will add this token to the mirror config section of the Spack environment:
48
+
Next, we will add this token to the mirror configuration section for the Spack environment.
49
+
Replace `<your-github-username>` with your GitHub username and `<your-github-username-or-org>` with your GitHub username or an organization where you have permission to create packages.
50
+
The build cache name, `buildcache-${USER}-${HOSTNAME}`, is a suggestion; you can choose your own.
==> Pushed julia@1.9.3/dfzhutf to ghcr.io/<user>/buildcache-<user>-<host>:julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl.spack
103
105
104
-
The location of the pushed package
106
+
The location of the pushed package, when referred to as a OCI image, will be:
105
107
106
108
.. code-block:: text
107
109
@@ -142,7 +144,7 @@ The easiest way to do this is to override the ``mirrors`` config section in the
142
144
secret_variable: MY_OCI_TOKEN
143
145
signed: false
144
146
145
-
An "overwrite install" should be enough to show that the build cache is used:
147
+
An "overwrite install" should be enough to show that the build cache is used (output will vary based on your specific configuration):
146
148
147
149
.. code-block:: console
148
150
@@ -177,10 +179,9 @@ For convenience you can also run ``spack buildcache push --update-index ...`` to
177
179
178
180
.. note::
179
181
180
-
As of Spack 0.22, build caches can be used across different Linux distros. The concretizer
181
-
will reuse specs that have a host compatible ``libc`` dependency (e.g. ``glibc`` or ``musl``).
182
-
For packages compiled with ``gcc`` (and a few others), users do not have to install compilers
183
-
first, as the build cache contains the compiler runtime libraries as a separate package.
182
+
As of Spack 0.22, build caches can be used across different Linux distros.
183
+
The concretizer will reuse specs that have a host-compatible ``libc`` dependency (e.g. ``glibc`` or ``musl``).
184
+
For packages compiled with ``gcc`` (and a few other compilers), users do not have to install compilers first, as the build cache contains the compiler runtime libraries as a separate package dependency.
184
185
185
186
After an index is created, it's possible to list the available packages in the build cache:
186
187
@@ -255,7 +256,7 @@ Let's add a simple text editor like ``vim`` to our previous environment next to
255
256
256
257
$ spack -e . install --add vim
257
258
258
-
This timewe push to the OCI registry, but also pass ``--tag julia-and-vim`` to instruct Spack to create an image for the environment as a whole, with a human-readable tag:
259
+
This time, when we push to the OCI registry, we also pass ``--tag julia-and-vim`` to instruct Spack to create an additional image tag for the environment as a whole, with a more human-readable name:
259
260
260
261
261
262
.. code-block:: console
@@ -291,12 +292,8 @@ For those familiar with ``Dockerfile`` syntax, it would structurally look like t
291
292
292
293
This approach is still valid, and the ``spack containerize`` command continues to exist, but it has a few downsides:
293
294
294
-
* When ``RUN spack -e /root/env install`` fails, ``docker`` will not cache the layer, meaning
295
-
that all dependencies that did install successfully are lost. Troubleshooting the build
296
-
typically means starting from scratch in ``docker run`` or on the host system.
297
-
* In certain CI environments, it is not possible to use ``docker build``. For example, the
298
-
CI script itself may already run in a docker container, and running ``docker build`` *safely*
299
-
inside a container is tricky.
295
+
* When ``RUN spack -e /root/env install`` fails, ``docker`` will not cache the layer, meaning that all dependencies that did install successfully are lost. Troubleshooting the build typically means starting from scratch either within a ``docker run`` session or on the host system.
296
+
* In certain CI environments, it is not possible to use ``docker build`` directly. For example, the CI script itself may already run in a Docker container, and running ``docker build`` *safely* inside a container (Docker-in-Docker) is tricky.
300
297
301
298
The takeaway is that Spack decouples the steps that ``docker build`` combines: build isolation, running the build, and creating an image.
302
299
You can run ``spack install`` on your host machine or in a container, and run ``spack buildcache push`` separately to create an image.
@@ -307,15 +304,15 @@ Relocation
307
304
308
305
Spack is different from many package managers in that it lets users choose where to install packages.
309
306
This makes Spack very flexible, as users can install packages in their home directory and do not need root privileges.
310
-
The downside is that sharing binaries is more complicated, as binaries may contain hard-coded, absolute paths to machine specific locations, which have to be adjusted when binaries are installed on a different machine.
307
+
The downside is that sharing binaries is more complicated, as binaries may contain hard-coded, absolute paths to machine specific locations, which have to be adjusted when these binaries are installed on a different machine or in a different path.
311
308
312
309
Fortunately Spack handles this automatically upon install from a binary cache.
313
310
But when you build binaries that are intended to be shared, there is one thing you have to keep in mind: Spack can relocate hard-coded paths in binaries *provided that the target prefix is shorter than the prefix used during the build*.
314
311
315
312
The reason is that binaries typically embed these absolute paths in string tables, which is a list of null terminated strings, to which the program stores offsets.
316
313
That means we can only modify strings in-place, and if the new path is longer than the old one, we would overwrite the next string in the table.
317
314
318
-
To maximize the chances of successful relocation, you should build your binaries in a relative long path.
315
+
To maximize the chances of successful relocation, you should build your binaries in a relatively long path.
319
316
Fortunately Spack can automatically pad paths to make them longer, using the following command:
320
317
321
318
.. code-block:: console
@@ -327,9 +324,9 @@ Using build caches in CI
327
324
------------------------
328
325
329
326
Build caches are a great way to speed up CI pipelines.
330
-
Both GitHub Actions and Gitlab CI support container registries, and this tutorial should give you a good starting point to leverage them.
327
+
Both GitHub Actions and GitLab CI support container registries, and this tutorial should give you a good starting point to leverage them.
331
328
332
-
Spack also provides a basic GitHub Action to already provide you with a binary cache:
329
+
Spack also provides a basic GitHub Action that already provides you with a binary cache:
333
330
334
331
.. code-block:: yaml
335
332
@@ -349,6 +346,6 @@ Summary
349
346
350
347
In this tutorial we have created a build cache on top of an OCI registry, which can be used
351
348
352
-
* to ``spack install julia vim`` on machines without source builds
353
-
* to automatically create container images for individual packages while pushing to the cache
354
-
* to create container images for multiple packages at once
349
+
* to run ``spack install julia vim`` on machines and have Spack fetch pre-built binaries instead of building from source.
350
+
* to automatically create container images for individual packages when pushing to the cache.
351
+
* to create container images for entire Spack environments (multiple packages) at once.
0 commit comments