Skip to content

Commit 222afe6

Browse files
committed
scripting changes
1 parent 15342f4 commit 222afe6

1 file changed

Lines changed: 70 additions & 65 deletions

File tree

tutorial_scripting.rst

Lines changed: 70 additions & 65 deletions
Original file line numberDiff line numberDiff line change
@@ -11,49 +11,51 @@
1111
Scripting with Spack
1212
====================
1313

14-
This tutorial introduces advanced Spack features related to scripting.
15-
Specifically, we will show you how to write scripts using ``spack find`` and ``spack python``.
16-
Earlier sections of the tutorial demonstrated using ``spack find`` to list and search installed packages.
17-
The ``spack python`` command gives you access to all of Spack's `internal APIs <https://spack.readthedocs.io/en/latest/spack.html>`_, allowing you to write more complex queries, for example.
14+
This tutorial introduces advanced Spack features related to scripting.
15+
Specifically, we'll show how to write scripts using ``spack find`` and ``spack python``.
1816

19-
Since Spack has an extensive API, we'll only scratch the surface here.
20-
We'll give you enough information to start writing your own scripts and to find what you need, with a little digging.
17+
Earlier sections demonstrated using ``spack find`` to list and search installed packages.
18+
The ``spack python`` command provides access to all of Spack's `internal APIs <https://spack.readthedocs.io/en/latest/spack.html>`_, allowing you to write more complex queries.
19+
20+
Since Spack has an extensive API, we'll only scratch the surface here.
21+
Our goal is to give you enough information to start writing your own scripts and to help you discover what you need—with a little digging.
2122

2223
-----------------------------
2324
Scripting with ``spack find``
2425
-----------------------------
2526

26-
So far, the output we've seen from ``spack find`` has been for human consumption.
27-
But you can take advantage of some advanced options of the command to generate machine-readable output suitable for piping to a script.
27+
So far, the output we've seen from ``spack find`` has been intended for human consumption.
28+
However, the command also provides options for generating machine-readable output that can be piped into scripts.
2829

2930
^^^^^^^^^^^^^^^^^^^^^^^
3031
``spack find --format``
3132
^^^^^^^^^^^^^^^^^^^^^^^
3233

33-
The main job of ``spack find`` is to show the user a bunch of concrete specs that correspond to installed packages.
34-
By default, we display them with some default attributes, like the ``@version`` suffix you're used to seeing in the output.
34+
The main purpose of ``spack find`` is to display information about concrete specs corresponding to installed packages.
35+
By default, it shows these specs with a set of standard attributes, such as the familiar ``@version`` suffix.
36+
37+
The ``--format`` argument allows you to customize the output string for each found package.
38+
Format strings let you specify which *parts* of each spec you want to display.
3539

36-
The ``--format`` argument allows you to display the specs however you choose, using custom format strings.
37-
Format strings let you specify the names of particular *parts* of the specs you want displayed.
38-
Let's see the first option in action.
40+
Let's see this option in action.
3941

40-
Suppose you only want to display the *name*, *version*, and first ten (10) characters of the *hash* for every package installed in your Spack instance.
42+
Suppose you only want to display the *name*, *version*, and the first ten (10) characters of the *hash* for every package installed in your Spack instance.
4143
You can generate that output with the following command:
4244

4345
.. literalinclude:: outputs/scripting/find-format.out
4446
:language: console
4547
:emphasize-lines: 1
4648

47-
Note that ``name``, ``version``, and ``hash`` are attributes of Spack's internal ``Spec`` object and enclosing them in braces ensures they are output according to your format string.
49+
Note that ``name``, ``version``, and ``hash`` are attributes of Spack's internal ``Spec`` object, and enclosing them in braces ensures they are rendered according to your format string.
4850

49-
Using ``spack find --format`` allows you to retrieve just the information you need to do things like pipe the output to typical UNIX command-line tools like ``sort`` or ``uniq``.
51+
Using ``spack find --format`` allows you to retrieve only the information you need—making it easy to pipe the output into standard UNIX command-line tools like ``sort`` or ``uniq``.
5052

5153
^^^^^^^^^^^^^^^^^^^^^
5254
``spack find --json``
5355
^^^^^^^^^^^^^^^^^^^^^
5456

55-
Alternatively, you can get a serialized version of Spec objects in the `JSON` format using the ``--json`` option.
56-
For example, you can get attributes for all installations of ``zlib-ng`` by entering:
57+
Alternatively, you can get a serialized version of ``Spec`` objects in `JSON` format using the ``--json`` option.
58+
For example, to retrieve attributes for all installations of ``zlib-ng``, you can run:
5759

5860
.. literalinclude:: outputs/scripting/find-json.out
5961
:language: console
@@ -64,51 +66,51 @@ You can pipe its output to JSON filtering tools like ``jq`` to extract just the
6466

6567
Check out the `basic usage docs <https://spack.readthedocs.io/en/latest/basic_usage.html#machine-readable-output>`_ for more examples.
6668

67-
6869
----------------------------------------
6970
Introducing the ``spack python`` command
7071
----------------------------------------
7172

7273
What if we need to perform more advanced queries?
7374

74-
Spack provides the ``spack python`` command to launch a python interpreter with Spack's python modules available to import.
75-
It uses the underlying python for the rest of its commands.
76-
So you can write scripts to:
75+
Spack provides the ``spack python`` command, which launches a Python interpreter with Spack's modules available for import.
76+
This interpreter uses the same underlying Python environment that Spack uses for its other commands.
77+
78+
Using this interface, you can write scripts to:
7779

78-
- run Spack commands;
79-
- explore abstract and concretized specs; and
80+
- run Spack commands;
81+
- explore abstract and concretized specs; and
8082
- directly access other internal components of Spack.
8183

82-
Let's launch a Spack-aware python interpreter by entering:
84+
Let's launch a Spack-aware Python interpreter by entering:
8385

8486
.. literalinclude:: outputs/scripting/spack-python-1.out
8587
:language: console
8688
:emphasize-lines: 1,5
8789

88-
Since we are in a python interpreter, use ``exit()`` to end the session and return to the terminal.
90+
Since we are in a Python interpreter, use ``exit()`` to end the session and return to the terminal.
8991

9092
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
9193
Accessing the ``Spec`` object
9294
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
9395

94-
Now let's take a look at the internal representation of the Spack ``Spec``.
95-
As you already know, specs can be either *abstract* or *concrete*.
96-
The specs you've seen in ``package.py`` files (e.g., in the ``install()`` method) have been *concrete*, or fully specified.
97-
The specs you've typed on the command line have been *abstract*.
98-
Understanding the differences between the two types is key to using Spack's internal API.
96+
Now let's take a look at the internal representation of the Spack ``Spec``.
97+
As you may already know, specs can be either *abstract* or *concrete*.
98+
The specs you've seen in ``package.py`` files (e.g., in the ``install()`` method) have been *concrete*—fully specified with resolved dependencies, compilers, and variants.
99+
In contrast, the specs you've typed on the command line have been *abstract*.
99100

100-
Let's open another python interpreter with ``spack python``, instantiate the ``zlib`` spec, and check a few properties of an abstract spec:
101+
Understanding the difference between these two forms is key to effectively using Spack's internal API.
102+
103+
Let's open another Python interpreter using ``spack python``, instantiate the ``zlib`` spec, and inspect a few properties of an abstract spec:
101104

102105
.. literalinclude:: outputs/scripting/spack-python-abstract.out
103106
:language: console
104107
:emphasize-lines: 1-3,5,11,13
105108

106-
Notice that there are ``Spec`` properties and methods that are not accessible to abstract specs; specifically:
109+
Notice that certain ``Spec`` properties and methods are not accessible on abstract specs:
107110

108-
- an exception -- ``SpecError`` -- is raised if we try to access its
109-
``version``;
110-
- there are no associated ``versions``; and
111-
- the spec's operating system is ``None``.
111+
- An exception—``SpecError``—is raised if you try to access the ``version``.
112+
- There are no associated ``versions``.
113+
- The spec's operating system is ``None``.
112114

113115
Now, without exiting the interpreter, let's concretize the spec and try again:
114116

@@ -119,10 +121,10 @@ Now, without exiting the interpreter, let's concretize the spec and try again:
119121
Notice that the concretized spec now:
120122

121123
- has a ``version``;
122-
- has a single entry in its ``versions`` list; and
123-
- the operating system is now ``ubuntu22.04``.
124+
- includes a single entry in its ``versions`` list; and
125+
- has ``ubuntu22.04`` as its operating system.
124126

125-
It is not necessary to store the intermediate abstract spec -- you can use the ``.concretized()`` method as shorthand:
127+
It's not necessary to store the intermediate abstract specyou can use the ``.concretized()`` method as a shorthand:
126128

127129
.. literalinclude:: outputs/scripting/spack-python-sans-intermediate.out
128130
:language: console
@@ -132,24 +134,26 @@ It is not necessary to store the intermediate abstract spec -- you can use the `
132134
Querying the Spack database
133135
^^^^^^^^^^^^^^^^^^^^^^^^^^^
134136

135-
Even more powerful queries are available when we look at the information stored in the Spack database.
136-
The ``Database`` object in Spack is in the ``spack.store.STORE.db`` variable.
137-
We'll interact with it mainly through the ``query()`` method.
138-
Let's see the documentation available for ``query()`` using python's built-in ``help()`` function:
137+
More powerful queries become available when interacting with Spack's installation database.
138+
The ``Database`` object is accessible via the ``spack.store.STORE.db`` variable.
139+
We'll primarily interact with it through the ``query()`` method.
140+
Let's view the documentation for ``query()`` using Python's built-in ``help()`` function:
141+
139142

140143
.. literalinclude:: outputs/scripting/spack-python-db-query-help.out
141144
:language: console
142145
:emphasize-lines: 1-2,9-13
143146

144147
We will primarily make use of the ``query_spec`` argument.
145148

146-
Recall that queries using the ``spack find`` command are limited to queries of attributes with matching values, not values they do *not* have.
147-
In other words, we cannot use the ``spack find`` command for all packages that *do not* satisfy a certain criterion.
149+
Recall that queries using the ``spack find`` command are primarily intended to *match* packages based on specified criteria.
150+
However, it's more difficult to use ``spack find`` for queries that involve *excluding* packages based on complex logic (e.g., “does *not* depend on X”).
151+
152+
Using the Python interface, we *can* write these more advanced queries.
153+
For example, let's find all packages that were compiled with ``gcc`` but do **not** depend on ``mpich``.
154+
155+
We'll use ``spack.cmd.display_specs`` to print the results, replicating the display behavior of the ``spack find`` command:
148156

149-
We *can* use the python interface to write these types of queries.
150-
For example, let's find all packages that were compiled with ``gcc`` but do not depend on ``mpich``.
151-
We can do this by using custom python code and Spack database queries.
152-
We will use the ``spack.cmd.display_specs`` for output to achieve the same printing functionality as the ``spack find`` command:
153157

154158
.. literalinclude:: outputs/scripting/spack-python-db-query-exclude.out
155159
:language: console
@@ -169,15 +173,15 @@ before generalizing the functionality for reuse.
169173
Using scripts
170174
^^^^^^^^^^^^^
171175

172-
Now let's parameterize our script to accept arguments on the command line.
173-
With a few generalizations to use the include and exclude specs as arguments, we can create a powerful, general-purpose query script.
176+
Now let's parameterize our script to accept command-line arguments.
177+
With a few generalizations, we can use the include and exclude specs as arguments to create a powerful, general-purpose query script.
174178

175-
Open a file called ``find_exclude.py`` in your preferred editor and add the following code:
179+
Open a file named ``find_exclude.py`` in your preferred editor and add the following code:
176180

177181
.. literalinclude:: outputs/scripting/0.find_exclude.py.example
178182
:language: python
179183

180-
Notice we added importing and using the system package (``sys``) to access the first and second command line arguments.
184+
Notice that we've imported the ``sys`` module and used it to access the first and second command-line arguments.
181185

182186
Now we can run our new script by entering the following:
183187

@@ -191,12 +195,12 @@ This is *great* for us, as long as we remember to use Spack's ``python`` command
191195
Using the ``spack-python`` executable
192196
-------------------------------------
193197

194-
What if we want to make our script available for others to use without the hassle of having to remember to use ``spack python``?
198+
What if we want to make our script available for others to usewithout requiring them to remember to run it with ``spack python``?
195199

196-
We can take advantage of the shebang line typically added as the first line of python executable files.
197-
But there is a catch, as we will soon see.
200+
We can take advantage of a shebang line, typically included as the first line in executable Python scripts.
201+
However, there's a catch, as we'll see shortly.
198202

199-
Open the ``find_exclude.py`` script we created above in your preferred editor and add the shebang line with ``spack python`` as the arguments to ``env``:
203+
Open the ``find_exclude.py`` script you created earlier in your preferred editor, and add a shebang line that invokes ``spack python`` using ``env``:
200204

201205
.. literalinclude:: outputs/scripting/1.find_exclude.py.example
202206
:language: python
@@ -208,25 +212,26 @@ Then exit our editor and add execute permissions to the script before running it
208212
:language: console
209213
:emphasize-lines: 1-2
210214

211-
If you are lucky, it worked on your system, but there is no guarantee.
212-
Some systems only support a single argument on the shebang line (see `here <https://www.in-ulm.de/~mascheck/various/shebang/>`_). ``spack-python``, which is a wrapper script for ``spack python``, solves this issue.
215+
If you're lucky, the script worked on your system—but there's no guarantee.
216+
Some systems only support a single argument on the shebang line (see `here <https://www.in-ulm.de/~mascheck/various/shebang/>`_).
217+
``spack-python``, a wrapper script for ``spack python``, solves this limitation.
213218

214-
Bring up the file in your editor again and change the ``env`` argument to ``spack-python`` as follows:
219+
Open the script in your editor again and change the ``env`` argument to ``spack-python`` as shown below:
215220

216221
.. literalinclude:: outputs/scripting/2.find_exclude.py.example
217222
:language: python
218223
:emphasize-lines: 1
219224

220-
Exit your editor and let's run the script again:
225+
Exit your editor, and let's run the script again:
221226

222227
.. literalinclude:: outputs/scripting/find-exclude-3.out
223228
:language: console
224229
:emphasize-lines: 1
225230

226-
Congratulations! It will now work on any system with Spack installed.
231+
Congratulations! It will now work on any system with Spack installed.
227232

228-
You now have the basic tools to create your own custom Spack queries and prototype ideas.
229-
We hope one day you'll contribute them back to Spack.
233+
You now have the basic tools to write your own custom Spack queries and prototype new ideas.
234+
We hope you'll consider contributing them back to Spack in the future.
230235

231236
.. LocalWords: LLC Spack's APIs hdf zlib literalinclude json uniq jq
232237
.. LocalWords: docs concretized REPL API SpecError spec's py ubuntu

0 commit comments

Comments
 (0)