Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 27 additions & 28 deletions tutorial_packaging.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Package Creation Tutorial
=========================

This tutorial walks you through the steps for creating and debugging a simple Spack package.
We will develop and debug a package using an iterative approach in order to gain more experience with additional Spack commands.
We will develop and debug a package using an iterative approach to gain more experience with additional Spack commands.
For consistency, we will create the package for ``mpileaks`` (https://github.com/LLNL/mpileaks), which is an MPI debugging tool.

------------------------
Expand Down Expand Up @@ -61,7 +61,7 @@ Spack's *create* command builds a new package from a template by taking the loca

* fetch the code;
* create a package skeleton; and
* open the file up in your editor of choice.
* open the file in your editor of choice.

.. note::

Expand All @@ -81,7 +81,7 @@ Your ``package.py`` file should reside in the ``tutorial-mpileaks`` subdirectory

Take a moment to look over the file.

As we can see from the skeleton contents, shown below, the Spack template:
As we can see from the skeleton contents, the Spack template:

* provides instructions for how to contribute your package to
the Spack repository;
Expand All @@ -104,7 +104,7 @@ As we can see from the skeleton contents, shown below, the Spack template:
names for those people who are willing to be notified when a change
is made to the package. This information is useful for developers who
maintain a Spack package for their own software and/or rely on software
maintained by other people.
maintained by others.

Since we are providing a ``url``, we can confirm the checksum, or ``sha256`` calculation.
Exit your editor to return to the command line and use the ``spack checksum`` command:
Expand All @@ -125,10 +125,10 @@ For the moment, though, let's see what Spack does with the skeleton by trying to
.. literalinclude:: outputs/packaging/install-mpileaks-1.out
:language: console

It clearly did not build.
The build was unsuccessful.
The error indicates ``configure`` is unable to find the installation location of a dependency.

So let's start to customize the package for our software.
Let's start to customize the package for our software.

----------------------------
Adding Package Documentation
Expand Down Expand Up @@ -168,7 +168,7 @@ The resulting package should contain the following information:
:emphasize-lines: 5,7,10,12

At this point we've only updated key documentation within the package.
It won't help us build the software but the information is now available for review.
It won't help us build the software; however, the information is now available for review.

Let's enter the ``spack info`` command for the package:

Expand Down Expand Up @@ -219,7 +219,7 @@ The ``mpileaks`` software relies on three third-party libraries:

.. note::

Luckily, all of these dependencies are built-in packages in Spack;
Fortunately, all of these dependencies are built-in packages in Spack;
otherwise, we would have to create packages for them as well.

Bring mpileaks' ``package.py`` file back up in your ``$EDITOR`` with the ``spack edit`` command:
Expand Down Expand Up @@ -265,16 +265,15 @@ It found that:
* ``adept-utils`` is a concrete dependency; and
* ``callpath`` is a concrete dependency.

**But** we are still not able to build the package.
We are still not able to build the package.

------------------------
Debugging Package Builds
------------------------

Our ``tutorial-mpileaks`` package is still not building due to the ``adept-utils`` package's ``configure`` error.
Experienced Autotools developers will likely already see the problem and its solution.

But let's take this opportunity to use Spack features to investigate the problem.
Let's take this opportunity to use Spack features to investigate the problem.
Our options for proceeding are:

* review the build log; and
Expand All @@ -301,7 +300,7 @@ Most importantly, the last line is very clear: the installation path of the ``ad
information to not get picked up. Some software, like ``mpileaks``,
requires the paths to be explicitly provided on the command line.

So let's investigate further from the staged build directory.
Let's investigate further from the staged build directory.

~~~~~~~~~~~~~~~~~
Building Manually
Expand All @@ -316,7 +315,7 @@ Let's move to the build directory using the ``spack cd`` command:
$ spack cd tutorial-mpileaks

You should now be in the appropriate stage directory since this command moves us into the working directory of the last attempted build.
If not, you can ``cd`` into the directory above that contained the ``spack-build-out.txt`` file then into it's ``spack-src`` subdirectory.
If not, you can ``cd`` into the directory above that contained the ``spack-build-out.txt`` file then into its ``spack-src`` subdirectory.

Now let's ensure the environment is properly set up using the ``spack build-env`` command:

Expand Down Expand Up @@ -350,7 +349,7 @@ Note that you can specify the paths for the two concrete dependencies with the f
* ``--with-adept-utils=PATH``
* ``--with-callpath=PATH``

So let's leave the spawned shell and return to the Spack repository directory:
Let's leave the spawned shell and return to the Spack repository directory:

.. code-block:: console

Expand Down Expand Up @@ -395,9 +394,9 @@ Now let's try the build again:

Success!

All we needed to do was add the path arguments for the two concrete packages for configure to perform a simple, no frills build.
All we needed to do was add the path arguments for the two concrete packages for configure to perform a simple, simple build.

But is that all we can do to help other users build our software?
Is that all we can do to help other users build our software?

---------------
Adding Variants
Expand All @@ -407,7 +406,7 @@ What if we want to expose the software's optional features in the package?
We can do this by adding build-time options using package *variants*.

Recall from configure's help output for ``tutorial-mpileaks`` that the software has several optional features and packages that we could support in Spack.
Two stand out for tutorial purposes because they both take integers, as opposed to simply allowing them to be enabled or disabled.
Two stand out for tutorial purposes because they both take integers, as opposed to allowing them to be enabled or disabled.

.. literalinclude:: outputs/packaging/configure-build-options.out
:language: console
Expand All @@ -423,8 +422,8 @@ Supporting this optional feature will require two changes to the package:
* change the configure options to use the value.

Let's add the variant to expect an ``int`` value with a default of ``0``.
Defaulting to ``0`` effectively disables the option.
Also change ``configure_args`` to retrieve the value and add the corresponding configure arguments when a non-zero value is provided by the user.
Setting the default to ``0`` effectively disables the option.
Change ``configure_args`` to retrieve the value and add the corresponding configure arguments when a non-zero value is provided by the user.

Bring mpileaks' ``package.py`` file back up in your ``$EDITOR`` with the ``spack edit`` command:

Expand Down Expand Up @@ -461,7 +460,7 @@ If we look at a successful installation, we can see that the following directori
* lib
* share

So let's add a simple sanity check to ensure they are present, BUT let's enter a typo to see what happens:
So let's add a simple sanity check to ensure they are present, but let's enter a typo to see what happens:

.. literalinclude:: tutorial/examples/packaging/5.package.py
:caption: tutorial-mpileaks/package.py (from tutorial/examples/packaging/5.package.py)
Expand Down Expand Up @@ -489,8 +488,8 @@ Installing again we can see we've fixed the problem.
.. literalinclude:: outputs/packaging/install-mpileaks-6.out
:language: console

This is just scratching the surface of testing an installation.
We could leverage the examples from this package to add post-install phase tests and/or stand-lone tests.
This only scratches the surface of testing an installation.
We could leverage the examples from this package to add post-install phase tests and/or stand-alone tests.
Refer to the links at the bottom for more information on checking an installation.


Expand All @@ -501,7 +500,7 @@ Querying the Spec Object
As packages evolve and are ported to different systems, build recipes often need to change as well.
This is where the package's ``Spec`` comes in.

So far we've looked at getting the paths for dependencies and values of variants from the ``Spec`` but there is more.
Previously, we've looked at getting the paths for dependencies and values of variants from the ``Spec``; however, there is more to consider.
The package's ``self.spec``, property allows you to query information about the package build, such as:

* how a package's dependencies were built;
Expand Down Expand Up @@ -576,8 +575,8 @@ Multiple Build Systems
----------------------

There are cases where software actively supports two build systems, or changes build systems as it evolves, or needs different build systems on different platforms.
Spack allows you to write a single, neat recipe for these cases too.
It will only require a slight change in the recipe's structure compared to what we have seen so far.
Spack also allows you to write a single, concise recipe for these cases.
It will require only a slight change in the recipe's structure compared to what we have seen so far.

Let's take ``uncrustify``, a source code beautifier, as an example.
This software used to build with Autotools until version 0.63, and then switched build systems to CMake at version 0.64.
Expand Down Expand Up @@ -605,7 +604,7 @@ We also need to explicitly specify the ``build_system`` directive, and add condi
with when("build_system=cmake"):
depends_on("cmake@3.18:", type="build")

We didn't mention it so far, but each spec has a ``build_system`` variant that specifies the build system it uses.
We haven't mentioned previously, but each spec has a ``build_system`` variant that specifies the build system it uses.
In most cases that variant has a single allowed value, inherited from the corresponding base package - so, usually, you don't have to think about it.

When your package supports more than one build system though, you have to explicitly declare which ones are allowed and under which conditions.
Expand All @@ -632,7 +631,7 @@ Depending on the ``spec``, and more specifically on the value of the ``build_sys
Cleaning Up
-----------

Before leaving this tutorial, let's ensure what we have done does not interfere with your Spack instance or future sections of the tutorial.
Before leaving this tutorial, let's ensure that our work does not interfere with your Spack instance or future sections of the tutorial.
Undo the work we've done here by entering the following commands:

.. literalinclude:: outputs/packaging/cleanup.out
Expand All @@ -643,7 +642,7 @@ More information
--------------------

This tutorial module only scratches the surface of defining Spack package recipes.
The `Packaging Guide <https://spack.readthedocs.io/en/latest/packaging_guide.html#>`_ more thoroughly covers packaging topics.
The `Packaging Guide <https://spack.readthedocs.io/en/latest/packaging_guide.html#>`_ covers packaging topics more thoroughly.

Additional information on key topics can be found at the links below.

Expand Down
28 changes: 14 additions & 14 deletions tutorial_scripting.rst
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ We'll give you enough information to start writing your own scripts and to find
Scripting with ``spack find``
-----------------------------

So far, the output we've seen from ``spack find`` has been for human consumption.
The output we've seen from ``spack find`` has been for human consumption.
But you can take advantage of some advanced options of the command to generate machine-readable output suitable for piping to a script.

^^^^^^^^^^^^^^^^^^^^^^^
Expand All @@ -35,7 +35,7 @@ By default, we display them with some default attributes, like the ``@version``

The ``--format`` argument allows you to display the specs however you choose, using custom format strings.
Format strings let you specify the names of particular *parts* of the specs you want displayed.
Let's see the first option in action.
Let's examine the first option.

Suppose you only want to display the *name*, *version*, and first ten (10) characters of the *hash* for every package installed in your Spack instance.
You can generate that output with the following command:
Expand Down Expand Up @@ -73,7 +73,7 @@ What if we need to perform more advanced queries?

Spack provides the ``spack python`` command to launch a python interpreter with Spack's python modules available to import.
It uses the underlying python for the rest of its commands.
So you can write scripts to:
You can write scripts to:

- run Spack commands;
- explore abstract and concretized specs; and
Expand All @@ -85,13 +85,13 @@ Let's launch a Spack-aware python interpreter by entering:
:language: console
:emphasize-lines: 1,5

Since we are in a python interpreter, use ``exit()`` to end the session and return to the terminal.
As we are in a Python interpreter, use ``exit()`` to end the session and return to the terminal.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Accessing the ``Spec`` object
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Now let's take a look at the internal representation of the Spack ``Spec``.
Let's take a look at the internal representation of the Spack ``Spec``.
As you already know, specs can be either *abstract* or *concrete*.
The specs you've seen in ``package.py`` files (e.g., in the ``install()`` method) have been *concrete*, or fully specified.
The specs you've typed on the command line have been *abstract*.
Expand All @@ -110,7 +110,7 @@ Notice that there are ``Spec`` properties and methods that are not accessible to
- there are no associated ``versions``; and
- the spec's operating system is ``None``.

Now, without exiting the interpreter, let's concretize the spec and try again:
Without exiting the interpreter, let's concretize the spec and try again:

.. literalinclude:: outputs/scripting/spack-python-concrete.out
:language: console
Expand Down Expand Up @@ -157,7 +157,7 @@ We will use the ``spack.cmd.display_specs`` for output to achieve the same print

Now we have a powerful query not available through ``spack find``.

Let's exit the interpreter to take us back to the command line:
Exit the interpreter to return to the command line:

.. code-block:: console

Expand All @@ -177,15 +177,15 @@ Open a file called ``find_exclude.py`` in your preferred editor and add the foll
.. literalinclude:: outputs/scripting/0.find_exclude.py.example
:language: python

Notice we added importing and using the system package (``sys``) to access the first and second command line arguments.
We added importing and using the system package (``sys``) to access the first and second command line arguments.

Now we can run our new script by entering the following:

.. literalinclude:: outputs/scripting/find-exclude-1.out
:language: console
:emphasize-lines: 1

This is *great* for us, as long as we remember to use Spack's ``python`` command to run it.
This is beneficial for us, as long as we remember to use Spack's ``python`` command to run it.

-------------------------------------
Using the ``spack-python`` executable
Expand All @@ -194,15 +194,15 @@ Using the ``spack-python`` executable
What if we want to make our script available for others to use without the hassle of having to remember to use ``spack python``?

We can take advantage of the shebang line typically added as the first line of python executable files.
But there is a catch, as we will soon see.
There is a catch, as we will soon see.

Open the ``find_exclude.py`` script we created above in your preferred editor and add the shebang line with ``spack python`` as the arguments to ``env``:

.. literalinclude:: outputs/scripting/1.find_exclude.py.example
:language: python
:emphasize-lines: 1

Then exit our editor and add execute permissions to the script before running it as follows:
Exit our editor and add execute permissions to the script before running it as follows:

.. literalinclude:: outputs/scripting/find-exclude-2.out
:language: console
Expand All @@ -217,16 +217,16 @@ Bring up the file in your editor again and change the ``env`` argument to ``spac
:language: python
:emphasize-lines: 1

Exit your editor and let's run the script again:
Exit your editor and run the script again:

.. literalinclude:: outputs/scripting/find-exclude-3.out
:language: console
:emphasize-lines: 1

Congratulations! It will now work on any system with Spack installed.
It will now work on any system with Spack installed.

You now have the basic tools to create your own custom Spack queries and prototype ideas.
We hope one day you'll contribute them back to Spack.
We encourage you to contribute them back to Spack in the future.

.. LocalWords: LLC Spack's APIs hdf zlib literalinclude json uniq jq
.. LocalWords: docs concretized REPL API SpecError spec's py ubuntu
Expand Down
Loading