From e3280ac4b6247ca9f8d9ef76ff947a5a906ca444 Mon Sep 17 00:00:00 2001 From: Robert Young Date: Fri, 30 May 2025 17:20:36 -0400 Subject: [PATCH 01/11] first pass on first page --- docs/hpc/14_tutorial_intro_hpc/01_intro_hpc.mdx | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/docs/hpc/14_tutorial_intro_hpc/01_intro_hpc.mdx b/docs/hpc/14_tutorial_intro_hpc/01_intro_hpc.mdx index 702ef93525..c6552e1af5 100644 --- a/docs/hpc/14_tutorial_intro_hpc/01_intro_hpc.mdx +++ b/docs/hpc/14_tutorial_intro_hpc/01_intro_hpc.mdx @@ -1,3 +1,16 @@ # Introduction to High-Performance Computing -This workshop is an introduction to using high-performance computing systems effectively. We can’t cover every case or give an exhaustive course on parallel programming in just two days’ teaching time. Instead, this workshop is intended to give students a good introduction and overview of the tools available and how to use them effectively. \ No newline at end of file +This tutorial is an introduction to using the Greene high-performance computing systems at NYU effectively. It is not intended to be an exhaustive course on parallel programming. The goal is to give new users of Greene an introduciton and overview of the tools available and how to use them effectively. + +:::warning[Prerequisites] +Command line experience is necessary for this lesson. We recommend the participants to go through our [Introduction to Using the Shell on Greene](../13_tutorial_intro_shell_hpc/01_intro.mdx), if new to the command line (also known as terminal or shell). +::: + +:::note[Objectives] +By the end of this workshop, students will know how to: +- Identify problems a cluster can help solve +- Use the UNIX shell (also known as terminal or command line) to connect to a cluster. +- Transfer files onto a cluster. +- Submit and manage jobs on a cluster using a scheduler. +- Observe the benefits and limitations of parallel execution. +::: \ No newline at end of file From 05bb73597bf46e88d3e92b034e698b61dd57d1ac Mon Sep 17 00:00:00 2001 From: Robert Young Date: Mon, 2 Jun 2025 12:02:23 -0400 Subject: [PATCH 02/11] first pass on 2nd section of tutorial --- .../02_why_use_cluster.mdx | 59 +++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 docs/hpc/14_tutorial_intro_hpc/02_why_use_cluster.mdx diff --git a/docs/hpc/14_tutorial_intro_hpc/02_why_use_cluster.mdx b/docs/hpc/14_tutorial_intro_hpc/02_why_use_cluster.mdx new file mode 100644 index 0000000000..43ca916e45 --- /dev/null +++ b/docs/hpc/14_tutorial_intro_hpc/02_why_use_cluster.mdx @@ -0,0 +1,59 @@ +# Why Use a Cluster? + +:::info[Overview] +Questions +- Why would I be interested in High Performance Computing (HPC)? +- What can I expect to learn from this course? + +Objectives +- Be able to describe what an HPC system is. +- Identify how an HPC system could benefit you. +::: + +## Why Use These Computers? +:::info[What do you need?] +How does computing help you do your research? How could more computing help you do more or better research? +::: + +Frequently, research problems that use computing can outgrow the desktop or laptop computer where they started: + +- A statistics student wants to do cross-validate their model. This involves running the model 1000 times — but each run takes an hour. Running on their laptop will take over a month! +- A genomics researcher has been using small datasets of sequence data, but soon will be receiving a new type of sequencing data that is 10 times as large. It’s already challenging to open the datasets on their computer — analyzing these larger datasets will probably crash it. +- An engineer is using a fluid dynamics package that has an option to run in parallel. So far, they haven’t used this option on their desktop, but in going from 2D to 3D simulations, simulation time has more than tripled and it might be useful to take advantage of that feature. + +In all these cases, what is needed is access to more computers than can be used at the same time. Luckily, large scale computing systems — shared computing resources with lots of computers — are available at many universities, labs, or through national networks. These resources usually have more central processing units(CPUs), CPUs that operate at higher speeds, more memory, more storage, and faster connections with other computer systems. They are frequently called “clusters”, “supercomputers” or resources for “high performance computing” or HPC. In this lesson, we will usually use the terminology of HPC and HPC cluster. + +Using a cluster often has the following advantages for researchers: + +- **Speed**: With many more CPU cores, often with higher performance specs, than a typical laptop or desktop, HPC systems can offer significant speed up. +- **Volume**: Many HPC systems have both the processing memory (RAM) and disk storage to handle very large amounts of data. Terabytes of RAM and petabytes of storage are available for research projects. +- **Efficiency**: Many HPC systems operate a pool of resources that are drawn on by many users. In most cases when the pool is large and diverse enough the resources on the system are used almost constantly. +- **Cost**: Bulk purchasing and government funding mean that the cost to the research community for using these systems in significantly less that it would be otherwise. +- **Convenience**: Maybe your calculations just take a long time to run or are otherwise inconvenient to run on your personal computer. There’s no need to tie up your own computer for hours when you can use someone else’s instead. + +This is how a large-scale compute system like a cluster can help solve problems like those listed at the start of the lesson. + +:::info[Thinking ahead] +How do you think using a large-scale computing system will be different from using your laptop? Please think about some differences you may already know about, and some differences/difficulties you imagine you may run into. +::: + +## On Command Line +Using HPC systems often involves the use of a shell through a command line interface (CLI) and either specialized software or programming techniques. The shell is a program with the special role of having the job of running other programs rather than doing calculations or similar tasks itself. What the user types goes into the shell, which then figures out what commands to run and orders the computer to execute them. (Note that the shell is called “the shell” because it encloses the operating system in order to hide some of its complexity and make it simpler to interact with.) The most popular Unix shell is Bash, the Bourne Again SHell (so-called because it’s derived from a shell written by Stephen Bourne). Bash is the default shell on most modern implementations of Unix and in most packages that provide Unix-like tools for Windows. + +Interacting with the shell is done via a command line interface (CLI) on most HPC systems. In the earliest days of computers, the only way to interact with early computers was to rewire them. From the 1950s to the 1980s most people used line printers. These devices only allowed input and output of the letters, numbers, and punctuation found on a standard keyboard, so programming languages and software interfaces had to be designed around that constraint and text-based interfaces were the way to do this. A typing-based interface is often called a command-line interface, or CLI, to distinguish it from a graphical user interface, or GUI, which most people now use. The heart of a CLI is a read-evaluate-print loop, or REPL: when the user types a command and then presses the Enter (or Return) key, the computer reads it, executes it, and prints its output. The user then types another command, and so on until the user logs off. + +Learning to use Bash or any other shell sometimes feels more like programming than like using a mouse. Commands are terse (often only a couple of characters long), their names are frequently cryptic, and their output is lines of text rather than something visual like a graph. However, using a command line interface can be extremely powerful, and learning how to use one will allow you to reap the benefits described above. + +## The rest of this lesson +The only way to use these types of resources is by learning to use the command line. This introduction to HPC systems has two parts: + +- We will learn to use the UNIX command line (also known as Bash). +- We will use our new Bash skills to connect to and operate a high-performance computing supercomputer. + +The skills we learn here have other uses beyond just HPC: Bash and UNIX skills are used everywhere, be it for web development, running software, or operating servers. It’s become so essential that Microsoft now [ships it as part of Windows](https://apps.microsoft.com/detail/9nblggh4msv6?hl=en-US&gl=US)! Knowing how to use Bash and HPC systems will allow you to operate virtually any modern device. With all of this in mind, let’s connect to a cluster and get started! + +:::tip[Key Points] +- High Performance Computing (HPC) typically involves connecting to very large computing systems elsewhere in the world. +- These HPC systems can be used to do work that would either be impossible or much slower or smaller systems. +- The standard method of interacting with such systems is via a command line interface such as Bash. +::: From 2b218a109b40bdeefc4bc05b38711e2e3f220fa5 Mon Sep 17 00:00:00 2001 From: Robert Young Date: Wed, 4 Jun 2025 10:51:08 -0400 Subject: [PATCH 03/11] removed a tutorial already covered and added first pass on exploring remote resources --- .../02_connecting_to_hpc.mdx | 79 +- .../static/connect-to-remote.svg | 25608 ++++++++++++++++ .../03_exploring_remote_resources.mdx | 251 + .../static/node_anatomy.png | Bin 0 -> 65335 bytes 4 files changed, 25934 insertions(+), 4 deletions(-) create mode 100644 docs/hpc/13_tutorial_intro_shell_hpc/static/connect-to-remote.svg create mode 100644 docs/hpc/14_tutorial_intro_hpc/03_exploring_remote_resources.mdx create mode 100644 docs/hpc/14_tutorial_intro_hpc/static/node_anatomy.png diff --git a/docs/hpc/13_tutorial_intro_shell_hpc/02_connecting_to_hpc.mdx b/docs/hpc/13_tutorial_intro_shell_hpc/02_connecting_to_hpc.mdx index 28e5bdd1a6..773d2b083d 100644 --- a/docs/hpc/13_tutorial_intro_shell_hpc/02_connecting_to_hpc.mdx +++ b/docs/hpc/13_tutorial_intro_shell_hpc/02_connecting_to_hpc.mdx @@ -14,6 +14,17 @@ Objectives: To access the Greene HPC cluster, you must be connected to the NYU network. If you are physically on campus and connected via a wired connection in your office or through NYU’s WiFi, you can directly SSH into the clusters without any additional steps. However, if you are off-campus or working remotely, connecting through the NYU VPN or using the gateway servers is required to establish a secure connection to the HPC systems. ::: +## Secure Connections +The first step in using a cluster is to establish a connection from your computer to the cluster. When we are sitting at a computer (or standing, or holding it in our hands or on our wrists), we have come to expect a visual display with icons, widgets, and perhaps some windows or applications: a *graphical user interface*, or GUI. Since computer clusters are remote resources that we connect to over slow or intermittent interfaces (WiFi and VPNs especially), it is more practical to use a *command-line interface*, or CLI, to send commands as plain-text. If a command returns output, it is printed as plain text as well. The commands we run today will not open a window to show graphical results. + +If you have ever opened the Windows Command Prompt or macOS Terminal, you have seen a CLI. This is the CLI on your *local machine*. The only leap to be made here is to open a CLI on a *remote machine*, while taking some precautions so that other folks on the network can’t see (or change) the commands you’re running or the results the remote machine sends back. We will use the Secure SHell protocol (or SSH) to open an encrypted network connection between two machines, allowing you to send & receive text and data without having to worry about prying eyes. + +![connect-to-remote](./static/connect-to-remote.svg) + +SSH clients are usually command-line tools, where you provide the remote machine address as the only required argument. If your username on the remote system differs from what you use locally, you must provide that as well. If your SSH client has a graphical front-end, such as PuTTY or MobaXterm, you will set these arguments before clicking “connect.” From the terminal, you’ll write something like ssh userName@hostname, where the argument is just like an email address: the “@” symbol is used to separate the personal ID from the address of the remote machine. + +When logging in to a laptop, tablet, or other personal device, a username, password, or pattern are normally required to prevent unauthorized access. In these situations, the likelihood of somebody else intercepting your password is low, since logging your keystrokes requires a malicious exploit or physical access. For systems like log-1 running an SSH server, anybody on the network can log in, or try to. Since usernames are often public or easy to guess, your password is often the weakest link in the security chain. Many clusters therefore forbid password-based login, requiring instead that you generate and configure a public-private key pair with a much stronger password. Even though Greene does not require the use SSH keys to login, please consider using the instructions below to use them. It will make for quicker and more secure connections with Greene. + ## Remote Connections with the NYU VPN & HPC Gateway Server If you are connecting from a remote location that is not on the NYU network (your home for example), you have two options: @@ -150,13 +161,13 @@ The link above will automatically search for the Open OnDemand site data and coo Once done, try navigating again to [https://ood.hpc.nyu.edu](https://ood.hpc.nyu.edu) and the site should load. For other issues please email [hpc@nyu.edu](mailto:hpc@nyu.edu). -## Creating an SSH key (optional) +## SSH keys (optional) SSH keys are an alternative method for authentication to obtain access to remote computing systems. They can also be used for authentication when transferring files or for accessing version control systems. In this section you will create a pair of SSH keys, a private key which you keep on your own computer and a public key which is placed on the remote HPC system that you will log into. -### Windows +### Creating SSH keys on Windows We mentioned methods for creating SSH keys using some of the [Windows SSH options above](#windows). -### Linux, Mac and Windows Subsystem for Linux +### Creating SSH keys on Linux, Mac and Windows Subsystem for Linux Once you have opened a terminal check for existing SSH keys and filenames since existing SSH keys could be overwritten by the following command if the filename is the same. If you already have a key with the name given after the `-f` option you will need to change the filename to keep from losing your existing file. ```bash @@ -188,6 +199,55 @@ Protect this key as if it unlocks your front door. In many ways, it does. For more information on SSH security and some of the flags set here, an excellent resource is [Secure Secure Shell](https://blog.stribik.technology/2015/01/04/secure-secure-shell.html). ::: +### SSH Agent for Easier Key Handling +An SSH key is only as strong as the password used to unlock it, but on the other hand, typing out a complex password every time you connect to a machine is tedious and gets old very fast. This is where the SSH Agent comes in. + +Using an SSH Agent, you can type your password for the private key once, then have the Agent remember it for some number of hours or until you log off. Unless some nefarious actor has physical access to your machine, this keeps the password safe, and removes the tedium of entering the password multiple times. + +Just remember your password, because once it expires in the Agent, you have to type it in again. + +#### SSH Agents on Linux, macOS, and Windows +Open your terminal application and check if an agent is running: +```bash +[user@laptop ~]$ ssh-add -l +``` +If you get an error like this one, +:::danger[Error] +Error connecting to agent: No such file or directory +::: +… then you need to launch the agent as follows: +```bash +[user@laptop ~]$ eval $(ssh-agent) +``` +:::note[What’s in a $(...)?] +The syntax of this SSH Agent command is unusual, based on what we’ve seen in the UNIX Shell tutorial. This is because the ssh-agent command creates opens a connection that only you have access to, and prints a series of shell commands that can be used to reach it – but *does not execute them*! +```bash +[user@laptop ~]$ ssh-agent +SSH_AUTH_SOCK=/tmp/ssh-Zvvga2Y8kQZN/agent.131521; +export SSH_AUTH_SOCK; +SSH_AGENT_PID=131522; +export SSH_AGENT_PID; +echo Agent pid 131522; +``` +The `eval` command interprets this text output as commands and allows you to access the SSH Agent connection you just created. + +You could run each line of the `ssh-agent` output yourself, and achieve the same result. Using `eval` just makes this easier. +::: +- Otherwise, your agent is already running: don’t mess with it. + +Add your key to the agent, with session expiration after 8 hours: +```bash +[user@laptop ~]$ ssh-add -t 8h ~/.ssh/id_ed25519 +Enter passphrase for .ssh/id_ed25519: +Identity added: .ssh/id_ed25519 +Lifetime set to 86400 seconds +``` +For the duration (8 hours), whenever you use that key, the SSH Agent will provide the key on your behalf without you having to type a single keystroke. + +#### SSH Agent on PuTTY +If you are using PuTTY on Windows, download and use `pageant` as the SSH agent. See the [PuTTY documentation](./02_connecting_to_hpc.mdx#opening-a-terminal). + + ## Modifying your .ssh/config file Please add the following lines to your `~/.ssh/config` file: ``` @@ -229,9 +289,11 @@ Let’s attempt to connect to the HPC system now: If you'd like to connect without typing your password you'll need to copy your public key file to greene first: ```bash -scp ~/.ssh/id_Greene_rsa.pub @greene.hpc.nyu.edu:/home//.ssh/authorized_keys +scp ~/.ssh/id_Greene_rsa.pub @greene.hpc.nyu.edu:/home/ ``` +You'll need to log in with your password at least once even if you plan to use SSH keys in the future because we'll need to set up your keys. + If you are on NYU WiFi or VPN you can connect directly with: ```bash ssh @greene.hpc.nyu.edu @@ -266,6 +328,15 @@ By looking at the information after *Hostname:* and in the prompt you'll notice If you logged in using PuTTY this will not apply because it does not offer a local terminal. +### Setting up your SSH keys (optional) + +If you copied your SSH keys to your home directory in an earlier step, we'll guide you through setting them up for use now. +1. If you do not have a directory name `.ssh` in your home directory create one now with the command: `mkdir ~/.ssh` + - Remember you can list the hidden (dot) files by running the command: `ls -a` +2. add your key to the list of `authorized_keys` with the command: `cat ~/id_Greene_rsa.pub >> ~/.ssh/authorized_keys` + +That’s all! Disconnect, then try to log back into the remote: if your key and agent have been configured correctly, you should not be prompted for the password. + ### Telling the Difference between the Local Terminal and the Remote Terminal You can see that the prompt has changed after you log into a remote system. Let's take a closer look at the prompt after login: `[NetID@log-1 ~]$` (in this example) tells us that we are logged into the login node `log-1` with the identity `NetID`. diff --git a/docs/hpc/13_tutorial_intro_shell_hpc/static/connect-to-remote.svg b/docs/hpc/13_tutorial_intro_shell_hpc/static/connect-to-remote.svg new file mode 100644 index 0000000000..604d2cca9a --- /dev/null +++ b/docs/hpc/13_tutorial_intro_shell_hpc/static/connect-to-remote.svg @@ -0,0 +1,25608 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Internetornetwork + Cluster + Laptop + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + TM + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/hpc/14_tutorial_intro_hpc/03_exploring_remote_resources.mdx b/docs/hpc/14_tutorial_intro_hpc/03_exploring_remote_resources.mdx new file mode 100644 index 0000000000..26f16f5c06 --- /dev/null +++ b/docs/hpc/14_tutorial_intro_hpc/03_exploring_remote_resources.mdx @@ -0,0 +1,251 @@ +# Exploring Remote Resources +:::info[Overview] +Questions +- How does my local computer compare to the remote systems? +- How does the login node compare to the compute nodes? +- Are all compute nodes alike? + +Objectives +- Survey system resources using nproc, free, and the queuing system +- Compare & contrast resources on the local machine, login node, and worker nodes +- Learn about the various filesystems on the cluster using df +- Find out who else is logged in +- Assess the number of idle and occupied nodes +::: + +## Look Around the Remote System +If you have not already connected to Greene, please do so now: +```bash +[NetID@glogin-1 ~]$ ssh NetID@greene.hpc.nyu.edu +``` +Take a look at your home directory on the remote system: +```bash +[NetID@log-1 ~]$ ls +``` + +
+ +:::info[What’s different between your machine and the remote?] +Open a second terminal window on your local computer and run the `ls` command (without logging in to Greene). What differences do you see?
+**[Click for Solution]** +::: + +:::tip[Solution] +You would likely see something more like this: +```bash +[user@laptop ~]$ ls +Applications Documents Library Music Public +Desktop Downloads Movies Pictures +``` +The remote computer’s home directory shares almost nothing in common with the local computer: they are completely separate systems! +::: +
+ +Most high-performance computing systems run the Linux operating system, which is built around the [UNIX Filesystem Hierarchy Standard](https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard). Instead of having a separate root for each hard drive or storage medium, all files and devices are anchored to the “root” directory, which is `/`: +```bash +[NetID@log-1 ~]$ ls / +afs bin@ dev gpfs lib@ media mnt opt root sbin@ share state tmp var +archive boot etc home lib64@ misc net proc run scratch srv sys usr vast +``` +The `/home/NetID` directory is the one where we generally want to keep all of our files. Other folders on a UNIX OS contain system files and change as you install new software or upgrade your OS. + +:::tip[Using HPC filesystems] +On Geene, you have a number of places where you can store your files. These differ in both the amount of space allocated and whether or not they are backed up. + +- **Home** – data stored here is available throughout the HPC system, and often backed up periodically. Please note the limit on the number of files (inodes) which can get used up easily. Use the `myquota` command to ensure that you are not running out of inodes! +- **Scratch** – used for temporary file storage while running jobs. It is not backed up and files that are unused for over 60 days are purged. It should not be used for long term storage. +- **Vast** – flash-drive based system that is optimal for workloads with high I/O rates. Like the `scratch` storage, it is also not backed up and files that are unused for over 60 days are purged. +- **Research Project Space (RPS)** – provides data storage space for research projects that is easily shared amongst collaborators, backed up, and not subject to the old file purging policy. Note that it is a paid service. +- **Archive** – provides a space for long-term storage of research output. It is only accessible from the login nodes, so it is unaccessible by running jobs. +::: + +## Nodes +Individual computers that compose a cluster are typically called *nodes* (although you will also hear people call them *servers*, *computers* and *machines*). On a cluster, there are different types of nodes for different types of tasks. The node where you are right now is called the *login node*, *head node*, *landing pad*, or *submit node*. A login node serves as an access point to the cluster. + +As a gateway, the login node should not be used for time-consuming or resource-intensive tasks. It is well suited for uploading and downloading small files, setting up software, and running tests. Generally speaking, in these lessons, we will avoid running jobs on the login node. + +Who else is logged in to the login node? +```bash +[NYUNetID@log-1 ~]$ who +``` +This may show only your user ID, but there are likely several other people (including fellow learners) connected right now. + +:::tip[Dedicated Transfer Nodes] +If you want to transfer larger amounts of data to or from the cluster, Greene offers dedicated nodes for data transfers only. The motivation for this lies in the fact that larger data transfers should not obstruct operation of the login node for anybody else. As a rule of thumb, consider all transfers of a volume larger than 500 MB to 1 GB as large. But these numbers change, e.g., depending on the network connection of yourself and of your cluster or other factors. +::: + +The real work on a cluster gets done by the *compute* (or *worker*) nodes. compute nodes come in many shapes and sizes, but generally are dedicated to long or hard tasks that require a lot of computational resources. + +All interaction with the compute nodes is handled by a specialized piece of software called a scheduler (the scheduler used in this lesson is called Slurm). We’ll learn more about how to use the scheduler to submit jobs next, but for now, it can also tell us more information about the compute nodes. + +For example, we can view all of the compute nodes by running the command `sinfo`. +```bash +[NetID@log-1 ~]$ sinfo +PARTITION AVAIL TIMELIMIT NODES STATE NODELIST +cs up infinite 1 drain* cs524 +cs up infinite 1 drng cs477 +cs up infinite 3 drain cs[001,478,523] +cs up infinite 117 mix cs[005-009,014,023,027-028,030,093-104,109-115,124,127-128,132-133,135-148,151,155-156,159-168,170-178,181-192,194-214,274,305-306,322,338-340,346,400,436,442,444-445,488] +cs up infinite 1 resv cs402 +cs up infinite 400 alloc cs[003-004,010-013,015-022,024-026,029,031-092,105-108,116-123,125-126,129-131,134,149-150,152-154,157-158,169,179-180,193,215-273,275-304,307-321,323-337,341-345,347-399,401,403-435,437-441,443,446-476,479-487,489-522] +cm up infinite 37 mix cm[001-003,005,010,013-044] +cm up infinite 7 alloc cm[004,006-009,011-012] +cl up infinite 2 mix cl[003-004] +cl up infinite 2 alloc cl[001-002] +v100 up infinite 4 resv gv[001-004] +v100 up infinite 6 mix gv[005-010] +rtx8000 up infinite 1 resv gr030 +rtx8000 up infinite 47 mix gr[001-004,007-008,010-027,029,031,034-042,049-060] +rtx8000 up infinite 12 alloc gr[005-006,009,028,032-033,043-048] +a100_1 up infinite 9 mix ga[001-009] +a100_2 up infinite 1 drng ga037 +a100_2 up infinite 18 mix ga[010-024,041-043] +a100_2 up infinite 15 alloc ga[025-036,038-040] +cds_rtx_d up infinite 1 drain gr073 +cds_rtx_d up infinite 19 mix gr[054-072] +cds_rtx_a up infinite 1 drain gr073 +cds_rtx_a up infinite 1 resv gr030 +cds_rtx_a up infinite 59 mix gr[001-004,007-008,010-027,029,031,034-042,049-072] +cds_rtx_a up infinite 12 alloc gr[005-006,009,028,032-033,043-048] +cilvr_a100 up infinite 9 mix ga[001-009] +cilvr_a100_1 up infinite 8 mix ga[001-008] +tandon_a100_1 up infinite 1 mix ga009 +cds_a100_2 up infinite 6 mix ga[010-012,041-043] +tandon_a100_2 up infinite 12 mix ga[013-024] +tandon_a100_2 up infinite 2 alloc ga[025-026] +chemistry_a100_2 up infinite 1 drng ga037 +chemistry_a100_2 up infinite 13 alloc ga[027-036,038-040] +stake_a100_1 up infinite 9 mix ga[001-009] +stake_a100_2 up infinite 1 drng ga037 +stake_a100_2 up infinite 18 mix ga[010-024,041-043] +stake_a100_2 up infinite 15 alloc ga[025-036,038-040] +tandon_h100_1 up infinite 15 mix gh[001-015] +stake_h100_1 up infinite 15 mix gh[001-015] +h100_1 up infinite 15 mix gh[001-015] +cpu_a100_1 up infinite 9 mix ga[001-009] +cpu_a100_2 up infinite 1 drng ga037 +cpu_a100_2 up infinite 18 mix ga[010-024,041-043] +cpu_a100_2 up infinite 15 alloc ga[025-036,038-040] +cpu_gpu up infinite 1 drain gr073 +cpu_gpu up infinite 5 resv gr030,gv[001-004] +cpu_gpu up infinite 65 mix gr[001-004,007-008,010-027,029,031,034-042,049-072],gv[005-010] +cpu_gpu up infinite 12 alloc gr[005-006,009,028,032-033,043-048] +mi50 up infinite 1 resv gm011 +mi50 up infinite 16 mix gm[001-008,012,014-020] +mi50 up infinite 3 idle gm[009-010,013] +mi100 up infinite 3 mix gm[021-023] +mi250 up infinite 1 resv gm025 +mi250 up infinite 1 mix gm024 +gpu_misc_v100 up infinite 1 mix gv012 +chem_cpu0 up infinite 1 mix cs488 +chem_cpu0 up infinite 22 alloc cs[489-510] +xwang up infinite 10 alloc cs[511-520] +short up infinite 38 mix cm[001-003,005,010,013-045] +short up infinite 7 alloc cm[004,006-009,011-012] +short up infinite 5 idle cm[046-050] +``` +A lot of the nodes are busy running work for other users: we are not alone here! + +There are also specialized machines used for managing disk storage, user authentication, and other infrastructure-related tasks. Although we do not typically logon to or interact with these machines directly, they enable a number of key features like ensuring our user account and files are available throughout the HPC system. + +## What’s in a Node? +All of the nodes in an HPC system have the same components as your own laptop or desktop: *CPUs* (sometimes also called *processors* or *cores*), *memory* (or *RAM*), and *disk* space. CPUs are a computer’s tool for actually running programs and calculations. Information about a current task is stored in the computer’s memory. Disk refers to all storage that can be accessed like a file system. This is generally storage that can hold data permanently, i.e. data is still there even if the computer has been restarted. While this storage can be local (a hard drive installed inside of it), it is more common for nodes to connect to a shared, remote fileserver or cluster of servers. + +![node anatomy](./static/node_anatomy.png) + +
+ +:::info[Explore Your Computer] +Try to find out the number of CPUs and amount of memory available on your personal computer. + +Note that, if you’re logged in to the remote computer cluster, you need to log out first. To do so, type `Ctrl`+`d` or `exit`: +```bash +[NYUNetID@log-1 ~]$ exit +[user@laptop ~]$ +``` +**[Click for Solution]** +::: + +:::tip[Solution] +There are several ways to do this. Most operating systems have a graphical system monitor, like the Windows Task Manager.
+More detailed information can be found on the command line: + +- Run system utilities: +```bash +[user@laptop ~]$ nproc --all +[user@laptop ~]$ free -m +``` +- Read from `/proc`: +```bash +[user@laptop ~]$ cat /proc/cpuinfo +[user@laptop ~]$ cat /proc/meminfo +``` +- Run system monitor +```bash +[user@laptop ~]$ htop +``` +::: +
+ +
+ +:::info[Explore the Login Node] +Now compare the resources of your computer with those of the login node.
+**[Click for Solution]** +::: + +:::tip[Solution] +```bash +[user@laptop ~]$ ssh NetID@greene.hpc.nyu.edu +[NetID@log-1 ~]$ nproc --all +[NetID@log-1 ~]$ free -m +``` +You can get more information about the processors using lscpu, and a lot of detail about the memory by reading the file `/proc/meminfo`: +```bash +[NetID@log-1 ~]$ less /proc/meminfo +``` +You can also explore the available filesystems using `df` to show disk free space. The `-h` flag renders the sizes in a human-friendly format, i.e., GB instead of B. The type flag `-T` shows what kind of filesystem each resource is. +```bash +[NYUNetID@log-1 ~]$ df -Th +``` +::: +:::note[Different results from `df`] +- The local filesystems (ext, tmp, xfs, zfs) will depend on whether you’re on the same login node (or compute node, later on). +- Networked filesystems (beegfs, cifs, gpfs, nfs, pvfs) will be similar – but may include NetID, depending on how it is [mounted](https://en.wikipedia.org/wiki/Mount_(computing)). +::: +:::note[Shared Filesystems] +This is an important point to remember: files saved on one node (computer) are often available everywhere on the cluster! +::: +
+ +:::info[Explore a Worker Node] +Finally, let’s look at the resources available on the worker nodes where your jobs will actually run. Try running this command to see the name, CPUs and memory available on a worker node: +```bash + sinfo --node cs012 -o "%n %c %m" +``` +::: + +
+ +:::info[Compare Your Computer, the Login Node and the Compute Node] +Compare your laptop’s number of processors and memory with the numbers you see on the cluster login node and compute node. What implications do you think the differences might have on running your research work on the different systems and nodes?
+**[Click for Solution]** +::: + +:::tip[Solution] +Compute nodes have substantially *more memory (RAM)* installed than a personal computer. More, faster memory is key for large or *complex numerical tasks*. +::: +
+ +:::tip[Differences Between Nodes] +Many HPC clusters have a variety of nodes optimized for particular workloads. Some nodes may have larger amount of memory, or specialized resources such as Graphics Processing Units (GPUs or “video cards”). +::: + +With all of this in mind, we will now cover how to talk to the cluster’s scheduler, and use it to start running our scripts and programs! + +:::tip[Key Points] +- An HPC system is a set of networked machines. +- HPC systems typically provide login nodes and a set of compute nodes. +- The resources found on independent (worker) nodes can vary in volume and type (amount of RAM, processor architecture, availability of network mounted filesystems, etc.). +- Files saved on shared storage are available on all nodes. +- The login node is a shared machine: be considerate of other users. diff --git a/docs/hpc/14_tutorial_intro_hpc/static/node_anatomy.png b/docs/hpc/14_tutorial_intro_hpc/static/node_anatomy.png new file mode 100644 index 0000000000000000000000000000000000000000..b9675ddac6920f2240b30d7b6025bc5cc15403f0 GIT binary patch literal 65335 zcmeGEg;$ho`vweC0^-o1Lk@_*5YpX>G*XfxB_+}wk|H4ul1fSn$Pkj!HA9y)(n$Bv z_1@!lKl^$1@B0J3^{#iVafuFd-`9OzXB_8o9_PGQQ<1~Rqr^i)L&H~)f1-hghUtTb zhB1hX4gAjoKMHZ+FLY-OxyNWF{nYEgKW;n7>p7#L-6KK$LAT5p20kK%rtsvErU&{~ znpH84j91g0(z3Gs#$ACb?g`_Uy*%u+#=7zE6ha?sKb0`ciTy19KHfeyE;i0P>i&xK z=Ld=s4?x7cA8sN@jCpQzdFTdrY_%BI?_M;Tjx}Yx^c`RMmHMW?;lw3lN1`ciL(|>W z)phjQkz13!mTLylmU1)Jzh1AE=Hmt=Z!XIZEA=&|DeGP`8DsppU+Nb`wBr zF<x|qiO!@C6qJEB)IQiet1D7k{ zMyK-Ow|k+}_wPNiLDmxf&r<@IkczxLu~y``RQLLSb_cbF>;a$uyRFL2vPO|q<66A` zzAu%}4(7ivp=Js^{_B~Vl8^u1uZcBh$iE()D}a);bVz5Z({=RU`_(L)aq;g*2e!Pq zl=a8<;eaIUF}H#2Kidd>7@)*aoAb@wxh0I=IIRxczpmAN-#wT6ygyx>O5$w!gTvND z+1`*GBuG{AiqFfv|J$_3^go-0ts)h+YMx|pcQ47v*Ap!1>kSsL(XCkwW5Z6*8UU_N zD0#^>KaeU+s*)i=BXO}+vBpw*w6ViSb>P09nV z{V%#|$(~AK(})}$Z%?ermkyo`v99n6;zu_g!0T?=FS_V^!| zp5l!gzOki#$MX2N&JM|q*TO#cX1C$q@y<`Gv9J2~qNs&QI-K2ux==0RisnVAOSDu?= zUmZzS_TdmhkHw+Nud43VJ}#~luIz_qdi)jM--vm)t7n{)B9?RwOEzWa8htKWqi94I zLrJqqT~tK>u@tTV@7u>p!%Orkdy9rWSIb^S-&5 zBUBYX(xflfsTpE6-Y9U<7}$FsCjHO4p;2hEv=xunOhJh03AQMo{E{c1hN6`7GM`;s zHK@e)2|_GRUco+_!S+_tc#hXnT@;IF#NQmZM}6dczH0YhW*0*;KXU!uZ98lixo-## zh8XT%=UO!wa9|97IXVHsqalRs`#D!z-KuB|G?PnOf;RVUvc=3{4%w-QQQGU^V<2dqbmF(j8|$BClN3I{hNOG^ z&p9mK&J4*FNjWNCoX-5orkKRkKL0_*HV|g;c12N6wA+|D=jmu?8oSkkEF>uZzSDZu z;(y)L|Ms$~RJVCybnf(7QnL=26}@sn?Qs9>CxhA(s=Ic`FB*7d7W;1h9i2KzV>t5E zM4s<6?vvv@rnBX&vJ(8hL7e$BEZKZ_L)0g5J+)rf`)nm?^%{0Gy2-;)_P3S#-}Wy1 z9Cm`l&bCZ3+9jHgZVn-Y*rnCAJ&oEjd6(3u)xY+M$6T@IV&{7sxy}n>*+|NjsOMF* zD+m7xy)Hq(@n!g}T1OS>z@lzi1Y_PhG~#yz5V6TkxpcyQlI+XP^IGH}%11GxX47d>mNmPnIjt3>uOj7rXdL+b!ugy?1#qc59n(uiS;XjGF+5fiuF8z;km3a4c8O;7cW+WN9i^ch!b3R%I zBKx#DS3!Am)ru{!-vqOH+QY9oLm^o>nuhVqop~t|RNh}jzdZYonfwSr>t-g%-d&28 zj5?2vY&`#FpU&SRlm&ZrKgj(3+VFv=;dcU0rDGf9%8!pOPa>bZ;(L6zvNe@s+ve$i z5_hI17E(f;;i^FMmaB=P0^R7BT%DdZhHus?QsvA`s_dUKhJG?6d|Jm$5@BE<={;Bv zQgH9zabq(2`S;#r>S7H0sEO?l$A&!&f@#ZOY_RcYPn>Mj9_wqA!zpa?siy~WYvL35 z&are#$?t~r>G1p$0(VH|UFoVuu6C0884K{xk7!OZj`fv`+e@x^zkeu?lQrV4c{@iF z-&pw1sgfy4DfPIFT8P9&iSh-w;p*nwJ^k{!+}&!ERXFIMU;DF7Qd%hd^17#X=D*z= z_46P$fP-wMoJjt*yT4aezzV#fWhI%FF)>k-yQH79;RR;lov@ z|AP~$03=t5-lg!5xBRvBnT5a`T9qp=?EZcGSAhUj6rdA+|L%{rBx{ ziNIsBl+0c4n}KXYyO-f}tf!*mP{pK1fRXgVNuk-u%j%>vC5E@>x$t?QQY-|$E4lTu zXiRq5{su={9{i(GDP(^2Tvmke~W%XNA3t?;}q1#+Qe5uKxg*osWxR7!M z;E_VLFABK|RLBh~@H4y*6oecXpPcTs5w?|?!o8dxe0!#xrdoSDb1M_g!1rv$@y913 zb3Xee)rvco1L=`YZUESUI%^1!Cw@v-vc%XXZJroJ3kOLZPjU>?zX5$S-pVyJFZi@& zjxXYIXtOGje_<#%^{Ac(m8{T!!y%s`B%l@|BA^um3)szoX+4)?g$|p6T`>Mh`|eiu z(Tbw7Kch;o)Qs=Nw#mkiFDPIYDN)4kT8<|dMSUBH`UC)cbV2qa>8*u&o--?m!pe!H z(z%3yh)ceRcV?~6#c`>5Pa*&UIiILfH|kJt`$NRN zTxt&cMcdxpCT*1_6ej~P)H8WeB%b9s5TA^Vfu-U{jK@67_Sl)O=Ct`)$6tTGQRMiG z{`;0W;+`WmliFj92#;tAUZ&n;ZU%PUQo4Su%*cLH8;H<$Zfo$n1RadxbjCsvZU zf$n`-0?Hn)W)Ii-#*3Xh5Wh=DlieRq2HLQ_h&QWg?ycdRjT04tAHhEC6=|q6wwnG1 z&o(5=oz?Oc29ug6fame-S_Nl47{snmmrUSMH2W=xwuyc$uK+Zks{|dx_dRJ$DyiVR z51*4I(35=@qsCkPpejKz>$UmS;qr990BhBeX4MMAoS}KRwR+B1)MmQs;fy1kW@tjU z+)RN_>Hx|XOuI8v>#RNoHalDOyIyVXzDv`yRA)Z$!L50%WAFP)qW+@jW%xbLs6Zo6 ze7A(O_dNiL9TZ0`&0@f*qVJPn$koLO97l>%lIV`?ZjWmUni6Y@XB#mORdcVAVbn_6 zCjnc$5CWPoMSH_>$3`YK0Sqyngbw%wY6u2+OM(KrzE^<7+NXW?lgJ%RetnpfoM`Uxo)HE7GVB2m zXym6>VI5E?;S;+{H>SZNEHy7(yu{TlOhu0J#QgHiFWBsB zORlSnu~PLtqk9x6Bx9Lb~s59dzxA^#J93dqqT6#KM^H`vl-z{y&ew@LfJf;+lcERjLULLTl~OW!u`cP8`W?1(IlRZN zKU$Z(zuD!rpRGS#Eura|p-7R6!csH-=Iwd3F}lBWOW4p3p?4^JbauE3=iJ=CM{=2z znRZ(vOhnv*??hWm6Tct2oAOskAfbD5_SmJoq@OZ<|-D zCjv)>T`5Dr6INv0V$KeT>aWi?+rC+iv>LeoB)>6pkI_5hBZbDJbrar5cO_kn73)2_ zoo&B9oV!o>VYlIA_Ed)?2;;Cr**FG^i&~fUbm#kS=S?eH_>i3I6!&Yc^Oy$5rOu$q z3QHZg=AbvUyW9ts$V&*+cDOeen>*q+y^#3|Ft9MD*+s7ij{}IsvwW4v3L+gtPxEkI z?+L8#M`*dhuoHYL0rqpJ_(D$>=<4bIQg$)-2;4s$_0VXtn7ae}?r3cxgs{i4oU9Dy z>0Bm{y~p~JxwFee=vcoc}+V1zq`6w<6$<6?Zx-f>! z%P&R0uZKsZREb+us!5WqS}p(tTrQ_&1go(d#Z$WeJnpt{+m{)zq(@_B?NFQ1CvoDS zCYEP;nlEDt)YEFl_q9|ac2Om=Itd(wT%@dk<7a#N%v+s^mKyBqAuH`EG?gMnjyq_n zF14?v=C7eA-Op~VwPw!2fMxbVOJf?)d0td&QXjGM zZ*m}jo~U$45X(bi^lwn{H$LfB0iN*xH6Jn7oxHPlP|37+wwi$>red;Lk@{KwcZ#l` z$xR}#OI@iLkO*JBH`K_=mQ@vQrqyXp-qjRw#Q7%Whib^MC8Pf9O|i|?#H7tvk^|Nv z_P<>>U=~IHU~k{7=VxHo$0t&1e$m#7c`WAvR40UmcF@n?I2)n4E)%6D3*|i=RGOZf zZo|wO-owmBi4{&&90&aI5^NLlh?Ve0~!HT+g0Sy3!$Za%O(omT| zQ^jp(hQe)oiaY^_?DlsjA~jKkE-HZk(5g7+yQSxil;+J24}FjfkQ%e^*- z@W+cenPdLc;Q`60`Qwt*UhmYQf8I**GRm;!fz>4VGgtP zoJZhtBvzJ z0-#Kl4e-L}#H{;{7AIV|DE@_jMuaTEd7w8wza?jecvu_i@@-sdI`mLAO@tgM`tK$T z1Eo}Zy;-PhdFP989*x=eq|C<&qS4l_zk!D^TQLbT6pSB#;} zMThC(#)klqD&)(v{9wTE;xpfIU|_vspyqV}GEdm_>+jkO=OQ}TVB5Z9An7lHsC!zm zziaZ{&FLcShGODg9B+pQGgfrPYX!e9auV#SavFNg?9rTl9!)LG)0-hF89~9D=XCVN zIlw9-5Qi|h;mt|s>tRbP&P1={aU(|4cZj>#;B9f!&Ik{V6U~FQZ>D#N&xVhg`}|xQIUflwwM;27@WeGh-zcn`%W> zTq`Tl>mDRE4I8VCi){j%{_N-O_!C!6Ndq|^S?}ZIVIg)x^eK6sJdH0u%YC%>b5tAp z+3suJ3|DIA+32~>Wc>lxpv30+Mu_A@(U}%S<}q~DkliNrf{68B(PJ10y*eNB z93K6j3gr;Ur>^N=ocaDPjQ$X^0sr4e{{MByz}g?X6cZ?*}b^M&RB z%!S?*-UWbkTF8!;z(aX@d^IZIS&ih$=urvSB%9kpn|(_DIAVM#V9zA{vYdBs$sk8m zB_qR_gK2~3_LtzTPpA+(l<53HBq z(S%E#k#_`V9He#YT&&_NyFM`EBzjN%hw;%&@~`~0QDtDRR6-GQv^gH~P`~QkJtJ>! zu`A?=dRhB~bXPk<)MZUg$otg68y@W*mH1Hq?yvqds>mjw&YfB-vLSS63BT`OY=XR+ zySXyrJHFnYyMEYBjmdiqELX9(j1g>@KACR1ns#L2$9xDX1@Ky7hd0fiqeX-O zo9kU8GR}m>*B0`&*6V1a6d>x753VnEqdTb;nUqBIEb`nhbc3s`$4j`ZhI7F1#-t2> z^sQHkvrBF&$Dgj_M}0foq=>C28_2SIvs#PRnanDwp|YP7%Nz*|jaZZi%s(AuogT=v zq3Xt5TXybT_Ih{u$uk7Vd~q|_TaOHOf|m<^M$p}Sq+Y)NBPB&K@oRbXlGrG(>?s^e zJQTwU1NxPJp6%?B(jG^=SSZsAf zqwRE6$4V}=GsiM7*2(^7U8QOnqvU}~&m&q7i~H`ovor~IEM@GL9VW=!RkNN2A5wsL zKhFd)W9zO33eI^udkYfk>DPNWW)!bKN09lX1rW?Ge0WxWOn#pY`~uO@pC+2UB|qk| zo5&f!(>#n=`%t{!$Pq{n4teo`hj;#MOUR|MloLqRkmnp4crf^;iJer=P9ggOd$sJT zMA);l-$V`y62gG6Hg5fJHus!|HTrGnqfUev9xcOuh2_A`Ckn*fr(u*Pg?lV%WRejI z@e&8-lTbwlUN>l7YWUoDcY96RvnI)S4Uh=gGvR|_8(O6qp(sa$E4LbpC>~ST!wQo2 z57>cmoAbg^0f@@I<&b=KX!UXcN!0%PgUH7bJVjoG>;!+sMQqv_%yVj=7(Ew|2qQgPtTvWLcI#FH0HJD?gUp$E+$L z6j*Ou;6%V1--{L(-Q9fFyXDAfLKmDOyBsBA>aRh&$fbC+z`PMrJozX5NWn;eXD@na zF?zUUpW$wLsd_EQ3O7 zct45Iea^?-IEAW5aYct>{+SVVT09^3|wT~oMi|Vo#p$T=kHT`ir_&wS;TV+rj6fSl;I9-aCTOwc}(qniM45nVBjH{HR^?a}=$O%pc z#~|pjA7b)>{kKe)_sNH0epj15n97mZ(V83~f2a*LO8_=)%+xU!Qeji#QH3$!J?Q>S zR%yXY_%$p#tJ4Nusr4H>=oT?U7^4l?2m1NNJyX&Ujj-}ilI}Ql3jw}6St(e<^J*Vi zs44=fO@mbwq20W|pNEP}!8G)pka0%82tA8illk7z?;d21@lNs2hbPyk|!XQmm$>DKS<3*G%${{q;{U}weh7O^z+32&K%^j1=60v*Wet2(XFA;`0})q z%Klx()t2Rg=-io#CdrwNcrY`slU|5MmoirHw9)m7k!(<`3p5eh4~@2YcCV4W8^pj% z4Q=RUzx-u?ZJ{ZvE}s%2-@C>||Ya;Bd|4*em4?P$9~cw9?k|Q7RWm_LZeYY*!sZ6isSfw8z$_KR$Y5 zryHg)#}wt{Kv3R^ro;dj1{KV6A0s5$tm8kWNOVz-Kj~Kpf5BZ5YJ#xI+reCxsTB^f z?KbEB2=z};H)k)6Our8ok-ZEs_Q7cw@Az3HZO~=*&?mI&F7&XJb9^`>~kqt*<`a_37 z?cWEQI13nGcYi=af5vYHQ`vww%Q1k^}FmsP$t>j({MZ^=cgaC?_ zw1*)lfBIdS{Xj`GTI*{4T#!s1Bxlj3W}}X+E2$Lf~%Z+u@o z2%{)%@rW;`Cyg}2$GkHcKQp+xez81+ptnX)1l|YR^qECE$&Wpdt5@o&MAC!1<$QZN zUYnz@#>_g6g*GnzRf2g1HQ#z#rSp2r%Pk14o4%)1ZAYzJtyvArQx!)VBvWP(au7lv zWJ$#kF^M6jXgcOdEfYuVOQz&A7~t-sOCYO9N#T>zVM(rkpxP7d^!{bnJ&_dg90?EM z?$B7c&8=zZ7ijZxi$MB;c~9mc&Btab+(9FX(q?(nPrh(D?HGQLz%tB0&acCN(HmQ6 zEkCYOme&GlasWSl$--;(I}QyaV$%uuY|31pANX@ z&wIsQH(Qmi?0Np3q_luQRXF8D&L*hHfPFWZup4~AF1UnA%)b3tp8p46g+>%a3u5tsTaVZwY@r zrs|?}l^#oexOyGHt}fV^9v;ypzx=UIr-0yV$?DSI%a%js)`>#KVp2u0(N5bM(6JvV zXgd4#$+3#?R;>CPvTBPkitY!zcY6&pvEm4`PoU+{tjk(nt5eC6VutkSygh=f>n4zS zj2?P-tE&R9TF8AR;XP>956l(H+Kd^ktTaBSMd%P1Q#`n8pfRJg7_qWQj=MG}qLD-m2vYYu@na;@AOWYI7-H4ZS4; z@q@e))(o=~aN-RDVX4Ej0LC$SMK99vA! zhM?f-@I-)l9wUTvl8VcepF6506>|p;z>S1d40E!mL2;l?{y6?_IiDEesAn9@%H^{1 zSDLVv7vq2Er2lbH3uqKHy9ue(M0P{9+r^x*by0zQ`bnVU z0eu~+3!dGf?a7LK)v>#rhSV40706|JPZ{i(TM@F?oY6OV?F`Yr zL7BA!Mjo`T3idkba6Kk~URxk7Pc1<%(2 zZeUTF29soKgplGiGkTFCUTVyPGfGcnIn=k84qHif9md&o=@id@F63!USD3+I4fbY7 zAEk>ni{t@yIO2CvnoEswz4`))a^ku3r71a6;s;@1jFaQ7+P!L|@T6`@NPxmuFE zHszBQjymXN$5mvjVD816^*Q;;O~v+^<*B`U92$?5QNH~!i~_Ai_p!?L+|7mG5Rpa( zP$+?TplzOrFCY;Me2B+GBdxs$Wxb-L1%1SiMhG5oZ$?ePZ{7KjGi zQC+MX|8c~23q4-EL8;nheYmDQ!hI{5Uy8*%5wt>ViknL=ZFP=ZAJT2je3_GBZU}XO zhJ)3%vL{|nm`Ih^ljg>V$xG9MQ`m2P@|*QA`$DlQzm5f(j*s+CF@K_9miN~A!Z{VL zjE$XZLE&K^#=~{zEl|_*#XxxJUtq$Iwv_2OF+iop-{|ilJ^Plqr9Qh~u6^=2XF3Ru zFex>;R2W;!r#6K!>n%wuBiMPZgt{pG40s$D+h5V~h=$tm#qme*c2PPp)rO|);}Fyfm5A za4exL^SCOFm71pK_I3*Jjj?D&@!k51oRrn%2j;7fJ4h{`}Gr~8{! zJ6uy9JsM)9an1I2V~Sx$Kj-Yc$7=$1Mk*|0Qkbe?2xr|4k8=<1A=T?67>}HLYXf>ahY980^?i5ImuN*z@u~Ct7;nfnNZNj1iRH2`6vQE zppK~K)apgj^IF2+|pX>KmF-+gcZ&2YS^b#l=L=I>6GD*uXhHegGW)j7 zg#D{iIokvB5%;VPvUvvYdQ1g7%}=vPf3TtXc?)Ah5cw3FSodyM5`EFm+LF-5lb<9k|YDWYc&q&FPgTbYmd88P4`B5X+i;;kIw@nhZ`3wz`OrwPq^UhMa>zXEB z-J#y09VCv0_xf}%Ng?N;kx)kCfSg~B+4lHzhFc6UKuvMRDAg?|ZUb>Oqb+P`I zdDDbjjzVU7Hw;~hpz|p2ic9L>{Fy6C`1ScK0#lh`Lu?Q^ct2jLntZhsjPWh{G&^-r z8w$B%VeN?T)uF&epTfs{%lMr=@&`!oW5|?nZ zBkx}ZkI0B@w0J@TY=PQ60J*Ww#6fd}9>>$BlF_n#CYBrbdqp@UXPYCi>DBVWzY2HI z*Mnjc?E5WirRbq|E%WeI)N#v*!0U<`&Ye5UfSyop`u>}PNRDEOW_U0d%bds`_T%N+ zHXvFe2z@!hLIYRm@uznD%^$0|qi#3Rs^NmG?zH>`Z~9Ax)5l?`h5mMSQCY^(I=8JT z`#tnZ3Ost=H3Ui6OV(%WZncU=yTds=d2Yi#FR+0^*qrD=$H!zf9Wm%YM$N;iqpPgewyT5nc9HzbTX8lVgYej zn@RhQhxMQ~%$bHA9XLb22{CWO6WoIB91l0xgGMJ_jKnevvqGw9;yl9mp+*E`cYc9< zfLFCg#8WQX=eiNMO@-G>&r~Hi9Y>*PTJVJv5EcV~@eDx+3j@jQSd5Q81+EP-98%I4 zuSJ9KNrlTdv3^MK9IzW1*vD1f$*g)dar|CIR|Eq=<2T^%C;*6xPiblWYNB40+k7x( zUtgw?Xl(D&5#>HbHEZTpmKZqBKUcEKHO*6lp3IWAucsO0Is$?-V+>c6AS&wXhRENH zCqAxmvdJWPX?Kd(yZ zV%m8LF43>%pah5keqN%bM>i-?EA0p4Jz?R<4G8KM!?=8m&29yo` zc-^Tjnf{7HUXdtpXubbvED6W_XPTug@GzqVMh5162U?NWI=?jTQ}zD(!pZW0b}pk# z=a*Nl4eG*bSb$H-sFm#Te{+!~94*wAVQ6JvHK8&OAVfn{3|!D}9ln)R!IRP2+;ReR zYdw7qaonwiq8rT((ynpd^rrPmL2$j!!r9@DBqJrB@TPLu@00)2UP{tm=Z@IBoEyrL z4q1FdqUAkN{vgx|)v=5sI=m%RFERPF0M@cUX@9+I1+*-+m7O>B`gO6)!22a6SDHRr z@FGeL&K;T`;i{{=F>f8=-*r(vJz&fxoVHCBvRJyWGdkv0`xk*!h}wUx=Y@Axo_H?N zya!rnGK>T=C5ll8rd#{h*sjeUqdnI==Um9sc0qc-^Pa7Cpaq&87xwtI`8UOfh7INS zpCQ$KTuWc`B$CL@?tx*585E#LyVFS%JO_w279Fv*V|W+Mkhy2Tz>x+<9S-ShQSsR? zw&!*UE3jwOsnNip`Je>N_R|D~7GkFkP;=<#-Y^;oZz1K6d@TK;_^cyuRX-ENr`@~u zyIg{V06m%@OdNuDsOb(bwR<4(cw1`1bG-_SAfvMcqxVM^Zq9&u#ZkL8*r+9P95d-d z#LzBR3`+AF`2d>=>|O9Vn*&rpr!ydNVKY%_t>Xc7-4?Hw3X4j@c-N|by6Niz4V+@$ z14J?8CMkHIIVoQQW-U}qfPUVV%x$^|DIoI0k>&gsUs7IC0nIIr- zCuET;>SD(WCCV(SE35Yc!dR`=INw@%jXdrb+(@A6pp2pwr?fmrwGNkBO7x^6#5THY z=sIcPr_?p^hnz+&+)Th^6q4IwWZk)5={Y&Yvuzg11KBgiXQ*C)ETl)2g72k;D*Q|K zMg-841=K+^;kEY8_7>kqqOK8#krE#Xq+HHL`Ug^J2ji-8D(rP}ZL$m-hF4)YQUjJQy`3_uK-@mBS~}^6fIGzSH75 zT&TYiC8rCh1LBy@cp}ZqhGCosgsSoUKBG>ddaGV_wDatc=`vHf7mHnisVkqVXT7Xb z6-?TbQ^e!ZtMXOSD3eO#Xm63<@?ZK3Y8NhpD5TY8IwwlPK0NL#S4niS6K#q=2e>W# zXL-u+d$eoqbMS;*+wxsuiUtP#J49+m>Jx=zIZNrV#(W)ep=eXRaVW>9&Wqo^kAq@- z2E&AZJR+Q66(?e0Q-sP=DT$=4EQzy#Y5_$h{3O?`=Cj9Vu)|i}W=VMWMle3rB8h6^ zTb|9&Ft1F zAjdByEgECrnwE3Xkc+^nH zi!RH7lg8ViL^PkUYxq@ainzAzT6kZ*`vzlbAmipC;9i9&=rO84xU$R8+?JkeR*!i< zFw$MrLlQQ5Bbo)yBrB!Hg?mO2W%I>k0OhBYKw+E|-SEXQw4p+Ye zF);2aWsAG#57%wmGp(iIb?#j@IXwafiuPiIkv!*M{Hn*#Kb0(m0Z9%?hyHMARk*p~ zXA!?wDQQhNV3j_=6CY*m$i_*L}hs z)PerYs_95&BQ^&KV1`fa4d8jN=h_>dJ)%91z~*f&A~tDQf(JImwoQycx!}UloIY_O zk&UwL)W17L9YpiAy>sivd;8=TJ3ZS%{idbS(O&zEzW`LxnbK%65a4#;Liu~4D2%LS z|0wC^J_h@(mzv2T64*3^o{2}9XnaoWOUpv*^3e;3vt`q47%}%>0%AujaQu>Q1p4@v zgo@R-=hD+Ws9D=LL*k7!aTkie0|f^P5|7ey^bz42mvWQz7H|Pgc=YoMw`-dQV984Z z6AH}P(TCzAKW$;;Pv`v+p zy#f?LD9OL0Ta2^CKsrs{(@%FJJA;{phkui7nM4?1dym8gQ)JmCZ5t>5&O|2*dya8q zC<+y7MgTVm<4+EsmPEHW;nRml(D3@8W)UiU0Kbn4rnUl$cOGh0qm(F&>CY*<&vN!f zeXc5w)k*!)Dp;V?2kqQ}B@~p43T+bm2=dtqu9$M`B(xJV>|@=IfFU`fmqu5LS88Xm z!Eb8MY4>-WKQ9epg)ui*povG6jzn388TJgl_E#~=CODg1E0SibBZ&2n;FEp3TeB2} zn*H1Rka78-Sg(S~7YofCrI;Fm0N{f%aC00+7%g%BN|WGsY7Dw_uagP~FBs*>H&`AX zhxy~Zt)&qkZFx`C9{>Ga05c&FUvSb>-hx){cw20v)3Eza-O?#INI-pc(c~X}EQWr5 z=VYmdH9MofVCtZ+BY_8hW%K2!~!2P&d$KVX)FlQM)FF8?4BUy|7A6-EcK)K1KR zWx8jcDuJlULwBd{tF?k{yfe2rhp>M(G$Ab<6jKLzBi49N8cPsjTz1P099Q{^wx59LU`cZ z6h7dfoCT8z@3ME4lOrG%`dh~`taAR>EfF*)PGT1tYig87wAI+unvc$t6W_je8$59x z0t%N75-DxZyIa86vjkPmiF+&k$P6hL5^T0S+iWf3h8nrI?(HZeXYzmRFd%;N?vs6^ zFl$nMMKv%Uv{&tSQC+4z<1HU(nYYk_1Bt*}Ln*23sb{26gvfK!dFS)prW;@H7|8NB=s!56=@qX@8lDhDUw!3VUU5 z<`YFhTk{h5qJ03+63;GCthI5211!1T@`1Pp?+N!e{|?XGST#2c;Rs0J_RTr$hO#M3 zsy-cGn&th0WoE`I0??(keyyIR)QNlL@Ctf1zWu_w+SOWpoS8K`0=ZXeDa>4f?_FvR zj9%|YPpE5Bpj(^`08+uF5PdZOUD5cgL~ zB{62;qp&sUk9tgoyQfy4Bp7Alh-5wpS)Bh+z4)FO4OD{5(o#-cbol2K<;68_FnO5wJ8)_FquK+%~y6RJ3r(|y%RfbL>5cWfuC?!e!Y$$qp@%VecH!Er1=-H-(|<5F{$ z3Cc`$Vrb!776Y_|0mrzR5u65fe2w-U7o`X_-jK_MTyGKMZP2i%EovYTbkXi?PY0K(En&=gCj^rSHr7RW|R>tZyX&G=NZ$m2X3xZ8l6tzwy_REW#6^ znjs-R7XJw?otR#%!W)V!G<@)OA2r6_=m(g=6oY@n{s~m5>v;w55$(f+;8mLE4cVhx z)bLMeysjRyd+O)lKFd87mF%l6-X&)ERk|}588`K0{v4`D>Jow$xK%HkJt<-qw_VIY z3+FaVCla;{j&|>O&%oI@(#-85>>?5&@fA;vcC!AB&rK^$`M5&ObtMy>%rimk=<~6< zS_5HV*C>gk9Hl3heEZNIWnWZ*9uU$CB?LfO#D>W`VckAx#208{M|4u+* zUD)pw;)&5ambGSrbiC=@V~8)84}BZ*GT)Tia_;1!Td;{u%^4%kv~P`mm;GwldcfNx zZL%KdYTddv+zcd?2rch-l$dJWhy#N9JL{5Krk#q7cL6@jv~sforyM&{0_PmR7R3K_ z1h@i7h?87EcVDINsxhIvLNzvxUIc?ckD5MEYtYUJ# zAoqMy607$Z+hq?BzlO@rqO^BSgFQN4XW@|H`Ph|<-6lz2hv*ZyXLOR#&Ta2nUV|w; zo9#kZekz~Q*RaQmMX|(@aYXlBBHDk9e(|NGFSHgnfH{%Kn=2pa^v4l2YEpJROa2k0uR=ssjYYiS^DXBJ!N{!YiM!BhCfv*KSI;sDJ_{fGL{kp1fI(Sw&CZ zW!YQEMU{%E%+hqVptRY0Kpf%-4>`eI&A73?YjKR4Sv3CygE1T;M4d`fd`DHrM;hA> zcg3mR-bTtvbwiDmUoPjCL=j@D$;pO;)qi0xQppCm+^^ez|5mj(BemE2>Zr(E-DTfqGGIs-uvv2>KJ?1lO@E;sJk1)yvNT!UA^rl-}42^Rkal0TX z@QTRl3myQm7+EYPA7Xh-ED~#o^KFVLb{}BGYmnj2aj|2_kV;i~2&V;v)KN^Ho&w(pa)cU>l@M31%L56mTfGCjw`ar327Ch4 zy!1s+-tWY$U2EAQU7PR~^)Rp$AGfcGJ?=!*W0l1R(LTw_aoFf=c`7YP^wTEZf@t!B zjP#YflusB@?PG3p{caW~?R!aMpnAB z%5;(#!xXjS*-E}gMMn6jFT?O_@{ib=tYB_*?%`3@ZrD4y(|gIyg~-7?PsYZbWPrDeav2D;OvS*^s;Tw{~!<3H9& z!$rm?UD?w8&lAj)J8Ab_d5<;fcn%&|F2>~TSN*0YrLJ%@C7pJSfbUh|+bN*mojM@C zcU|=eM)(y#e)WJ}W54O-^!aMN=i%qo=!tzXP6}v3iL8WgqEwc+Wj0J)rBanKOCdpJ zk%*moy1^K!u4fYztJ5MJNi`a_jVBY?iOKtHpFBznxcjQl@jon=&%UHkR{5kUu# z5QiKPgdwDn4naT~L>U8eC2yTMD8T4e=TL9p8T|Z8@uZs zak&BaDU<0pckQgR&z~4#eEcl4ZYC;cyI^2O6R`&H+S8$-p{zTyegZ@#a&AMz@Q zZH|5gK6lkNwKmLN0MaUT{4bIU9hHXel$zr0oYQJQuboGOT*r54FU`_R*mLFxe=}&; z-|yV~hd*-_K*&#AAc#!{rNK5!vP|E9Mrik-=emZ?TWi+!5sLX?V1Wg`2Ffa%anjLZ zed^1zrYliSY;#Juk_1rHZzJd!{I;6?POT2@9r8BQD?pm7`noRv`f$9i!?NN$hk7K2 z=)TiZRb&ohGJr9R&?^=0T`%&iJzS(xH<3x>GTlm3TbYvr)&A!_=4(!G3uByXH;h*bP zo?nY!+844VN=^=xpS>)8U08nfT>o|1E_xYvm4)|o;^lHKKW1|n0;>RRULza;U4fWx zGW47a!sOB-B+Opi;OpdAP*of(pH@tTA8TcIh*#$1$V1Os!r}}VS zCJOU7sgOLY5CSs(Ov_8EpFbiSJM?J&YrkV2l*NGuL>%Tcr(aiwFMqNTS!pm4XmDtV zXz>1newk+Qr+JWgT{cwue(*zeky;qJ%lK54z%h+6m@LH15OZ6fbV~*7V?-pS5w7^T6<=ww<0U0_n=y|P+c%_iiasljA&5`H;tiKi(EWC@SuxN-{KE*VyK^Kf;_z!!6#*LRh3Un4up$I$ zIGXj7oMF615SaUbd<%^bJqk}!Aie-aqU;X0m;aSED&nf|JHSJ?gyfDj4wDGxXvh`G zpJK{L{}MeZ;0fp_jhPP?<7G!;MMX%^jzC}xs%Ih@3HZXx-iuToERQfMX&99euQMEQ zP~O);^!!H+qMY>$AYzUOZ-mkBV3kdUANeG0nPyT-UT@hZ_?>U!^csQFI^bw8r4NXB zM?9=pLtEFN9_5kVdvq|Et1(54822UgxW%KC#pHt|h@nb%69`@eV+o2((}?Q6!Tcut z1Ud_zpRmef5+hn(IE@y1?-dDd1xtT0>GG1ySnG+a+It*uN$@OjPKNV*fG#*0T)mR> zU;~eMzR)mh#@#@olVx#<1lMRN&jX7*eyheAi7VF^GS}r4BJ6m|M~3v9m5gc^YN~W6 z?zQ!x&MTrK)q%5tg<-%E7y@c)VjwB1I}#vC})R^wQ%SH_yX>5)SKhv zJ#4OZwX>UARHyneyN2V;*X|sr>{C#JI#H`R+m!ZU@CI{oYnDW_1yqsb(G07Q*P%>1)sIyReTFRt46(OQM(bpQzu^jjg%Vq-PR9 zg0OtjB%&Rg#~#F1)SpTyLNd%l;H>q|!MtXI+&EMB(WR}eFFGo>P0tJN+x8$C$puRb zg?@lMo`rxuBvOnW2F;C)MK+Mo;<5;jd2FJdg?l@5J&Z@+Cq6A;0LzE4OlTOlTgIO{ zjr?i+5O7&6KOT|)UXvl$TOGTO_nPhjXDES7g8!+QB5v5N4%no}fe*&qj4Hb2SEIM3 zwN~cTZ;FLnuQ(x-U6K$JIBEyD$w;tn=UrD4gz@QfR(U&FdAr5$Q$8e$Hsy|R->;Tj zmr#J8un=ORSX8L;t)5xB2Y?#NzpsL+xi2CyJ~?!5{yofW!J zNnq20BsVN2U0+ZcZj8$GPBm|ZW9M>;fb}m>hidKNYNEL+)#vm1s7biYVNfrm2lgss zz#1?fFE<~;YJc@GzRZYL5N zlA2eXI3pzLLAYX$W6mViBwwXYp{~eH!~gpCSBw*(hnX%fbIrzY*Yyc$Wvzng3jx&mA*jed+waTz02AFNv&DX%b8ujJ<}koRRcpg zc3uGDT#i`IRRpyl=ZvH<^9f5lx2PVF_n!bh6fFB!1I36TYbd(=FE>Bia z?gWij7d!HdcGs#i3bolsh|{$K6y$0olEFWHNcp_H%b}~U+Gjk7#}YSHc=9` zY(~(HHM1@n7`tB+5AW~oW)Q@G#1iDPGkY#4YP+jpHBImOs2EwbphVCS#2dgss(z)_ zlo-M4`(fPBr4|!!!&(oIb?1EGZ`@l%r^rCpu2FFl>MIm4X<-q%GSlhxq?feR?EW$| zs8s7ToO2GC?E>@&=mQap0Q;7HjXp8nnQ73_@^Cc=c`W)K#dd?_+4#UOU};2Ue&^CH zXCfNqX75BMK^Nx@{Jj}hs32oLFO7>oeMKw-@xzQir|$)U10!m=LxreaHfKD&xW+E& zJ)+n=uW}?l-EbT~_ktFvTIk!{Cloqj*!8p;UnHm+{`_{*5=bm!R9O~jjP&y4m^53C zat}1>#BllG;Gf#RgkC*4zk(sSh0w9<&buIm6L+k z2mg4;Sw z;3c(!MqAQ~wPJourPmZO{7sC8!#aB`fXMvM7Xl4&Zh1nZZ z!eLH83CaDz3mda_W_k9=Jg3A*OVtV|h^%v__b(SFAhyAG9?Sa)S!c3Tj>HU{GunjQ zHC?RR;AzIL@z=x%iFkRj+a~NG(w$B`7U#P{dv>yY$Udd_}ii)>(k( z!o7~*(pD;0=yAmJa*=>Jj+YwC=#hF}9E>6NX>zvHqW2m^O;(SPa)wO^??K#JrDnmh zZOqZ14Yg)ot;@%9HE}>;H3olbwWu!@Awh0BaGU?L3xjI+50lE-F!w@KH&==JQoLiX zo7NWu#Bca*rn*GrO1&Jqja5xruU3ujqTRW%IOqHx8*C}IA)$Z7F(}uRB61HD&1l81 zJhbuTDVJE)YWpPcV01FXIgbReYFNlZh6=E$DSgty13u_Y5~hJeB!F*^@;}kD5zfuK zQrWUK?H^hN_=fYCKI?V=MdjRu$H6;ZX=zEtvbz%;{|Z@h2jI(azTDD4t8USkQ{x@1 zoP;-}tU3)RMMBikdV#3T0_3NH1!`K&G>jS{e1U4KVC4&>7B72Yc%$O`F+Wqurob?Y zEwH$&Y!hwav2kieSDSztbh4Ci`4#dUVP9+$b@(Y(9(YeNoIX+Rd#je~*5BBDVw+s& z?6r2in}eKrj+0sn{PKM&Y+O05JwT8Bhq)f-f9h7N#yD_)creu4yAwM|t;3Mw4pJRg z1MYDZLW$dWyYu80Frd$}^b22JUZkfc#Cp$ZGRxN) zrx{Nf7oj;pV@!?)6Bfa1jV#K5NJIW!6PYYCeI6ewR3)!5(jH-7X8C}#pe$w4eoO*v z-CGrGOb-P`B?MJgRwNbfl8Taj!NIm`tk*6Dps!;GKu5`gVD3tfs#5{Y|81c8KL;4p z91YDNUgD~VPSy&nK629qG(ht)7_~}tKg`IpUmW3xZb|sgxzciTI*{_-Df#KGbBEHl zt|WSreP;H6azWkAs+osqTny+x6KKm>XQHhRs8X-@j zZHD`W@F@8QrM&inRA=T}|H`UdlY_jxfJJO%*ALe**E9L}of&UkiK~G;rs8PEt7Io2PG!Z149WIh zt#iWYB>S$(hb$W2U(pNjo6m1cGOR%>83@mw!!@A7ndTC@j5jlkh3tP;~PR;4n; z`UkNZyKgv@T_I-?QE(gh*os#ocClYm+;b}yyJ_KIHL%> zL$Qd7kQik=Xxj>xnsf**Z6wgu?%E&gXG*td(%qXBO+eTKFN8YO z4#dEMx%lv?n3kZy4B5kuQRxIO;r@8!nleGQ-fx>rqm7vu;e@WmQPzeGK54!K<NpyuOusVEb6eofGphAywHn7p{eLyq zUroQRreQqk&H2V|C>kGDAE{)mo?qH!D*ci70D#!3ucfn~~qf_&51=3}2G4!O+k zFEQm99*XYN{#^~OC=EdiEOY#Cjcq-Hrqr1jhV=h){a@DWBk4?N>tgtL14zl{7_w5` z4B8i4ngfIWP4Wh)S#?}Z=w(Sy@}uB&#}fxJ9;;e|QN%@nWTb!qs`yMby0FER?@iM%)j;P;xvDTBN~ zGf`tLe|FICL!wi$`uPaTrpK9PKtk1{wB*<&6;-_tMOL84QIUrre@@xIi5aetTj9Sg z%=IX7_%2IH^*+^F893VnoYN+wtm9QL(i)3bSZ#IWjW^x92P4hpZ&d0JAlRR6f z)y~k$1%}@Y$M!!~{)pW~0r${ILXW?GCTi zZd^7M!IE4^?WS$e>D@iuC_$DCFS*jLScA;N|MUBLhFjW(KCU!atfs<~tYk-!M`sH& z0VeFGkG&+q9>C&Ej6RjPlVUQ+(4c*prM@>Nu(vWm3`p1xo9rB|ny>|q33}J@3w}am z3avePiMK0>j@lzQPP{2uu|4=LS6Lp!1vb zCyLzWzVi0_fh|;eSUx1kZ6KrC$C7t)n}m_l>n@&46T*J;jlAokN-}_TR$ayR%%n1qr*@Qwe=){dw32{!pe=e?P2$fhd#^B1KHJ`oSPMdCnuT!vurB5&WG=8GsD z<4rtxi}P}CXiq!$8+Gy%LmBpP!|iZ~U>-T%%=Qqz$%T_8K2GS%=My9RRy31+5e#{n ze>tN0CF-VEr);PWE7txFbO%8m{_}9UfD>^kN;*)NkHBLZ7AaV??#Ude-ta(saF9gKi*UI zh~-SdkE>^5&M|@CM7GJG4k=wmu8?a>@|=g;XPSID*vhWh9|+44&bEx!?bIWjzs>4J z1oIzObhIQXZ!4vz_GzBgW*k=JY1L!qs~6%Q*OeU0F<7~(ME$H#5z&`naEwfgDy*N- zPRhs;3R>f-3Pl!1?E8uN={o_Ws?gn!;sua1Ha-z9j8d}WH?>!C3qnz`&l*+}+Za~F z>i%$zIM+|p1*}UDOcWZfJztX(u^uhjoeL!HQ09b<7`JLkK)fHGK){{v?I+a!*L;16;_>rLV78d zIr}vBjn?9<2JF^%@3i!sDoY!8FXU0Ko`mO-%Y^zt*N;C4cNi=9mXm z&W4eC_2YH$Y!2^1v!Gq@7rwMP9yd1?;Rkj<#|+M69JQy z$Y0buE%`r3P}whU?;t1a5BUYD2zv@q!*Gl46*erC2-I73psmOQ>YB#YGl# z$tLMaP)2GIQOBM>Aj*JJ!N;I}P;X4GKZ_N4SW`rtP5mK(!U#N?KeAX&WNCa=tiS8j zrpBd@-@t=p-CLherNFd1Fa~8cQR&^cP;s(KWmR9C%^S-sus)19Z7j~)!{^Ds7w)g& z@ZcvKn#8uo7!1k^8AA7E3y zh5NxhQpVrBWFoh)z!eTj#wJ%@`m;V|AR0B_4D)OPeY<(J%SE-*__spMB^pr1*E9ne zlj~{x;**t$3n`xhP#1Yaj!Su!o-M-bkv<^e zQJQg4y%PdHvD&FMZ^>z-ez9; z4F5Z=+u!pLERwoP6{2DAb;XLYP@K?i7w+8Q+#l>CNFV1(6mp(n)DtT z5((+zZ}&sAQ>zTWzG=jdDQJpEd3!RnOk$ zSZtodrS$uZncD>i`Udi?6A`Cp=?r$pSBs>KB++Q()m>6afQY%4d?DVBzUf_Fxd;cc zq`IV#BT$q1_m83!mMlqYB&CSOF+R`w2jpIq;V2LFv-(Rne_Njh&D-XlM2_`)A-r8S zCVb}n;>bymO5b2?&boa=@VZo&R%6-}E5@e17@I87$kw9_yR!8eQ4tKfd57XX$e%jS zumh#sZZHCFqEUU?#Q!=HSYUHbs-YHjv(X#<+Ef0XOs4cx>*`36R8A#4qnDn$@S0%> zeXO`51PXF3BvKtoPZEcIf{g3UVC?IhWY;_-_itV`NFx}L!QIWv=L5l-e`fukJkv?> ze@pNke%2uRK-FSS{W58(r_ZGSCd-|n|yuvX?U<4Q_wqlRvVdd0L}ih%=#bG0(b(S82z7rc=y4nfIC{TG&jyV z+w58+vMA%9Hvt75UPw7$pd32B?_|PM15Vc}<#Eeut79WXLb~qoS{Ii2N%KEz3kXY; zrSNnOc6MW$$oty+l%&B_>U8&#&}gP+K$PesA02fYUK z2%&BPfz2^}Z-%&N<18R+S&yZF+RA7|+OPk<)MnhbN?7JpH{M-6HlvnLvSa6@>!ndc z4cn%x>D8uhvl|$D`iibiDCE?q2Gv?T=E@tOQ$zjuu5LB1omMsC0}-UO6)E0#%H7?HhsW?B@j-eqt*$jdqRGXgR< z`2M$~!C%;vvD&E~rj_J!zN&Ziq$YnUDCUsE$2P8-{17<0UV|vsgK{2|6hP)zd;sdSNop zBd_yokW#7D2y?ntSWJ2N%?@gFiw;s0QKm?{)$&L8GwwGYFb9w`f*C4b%*QR?a_4&> z>V!cpq;LHAAD!s+<^skFrC=!vD++}jfKL3v6%9uFej7V3)*0{Rb6JVHKHv4cU9ylk zo&ri*N~PRZ)`r88`oE)&?aomc<>)-L&4ympCr;{*%;Lq?{33}iZQ-HaS)QyC7RP(A zr4pf~l8b;~s08%cVyi39x9(Lq%v>C(WzV18ndIFq2V_(ku;fiC1~M!7>g2CSNSv)J zy!!vwXp+=-e516^xof`6!Y$ro7(e6zrP=sx=OS7bMtO#n+{raxkcO6{H!V@|33n64 zGZF;7HpBTEkN@xljhnWazPZb$^0+r6Zo6XI&U1L}mDNd?Oh=SO+iz9q(=8EUm4hf(9y_?uc|*^>x-v8uA7}b z$yFJ#er@c@>BwezSbcaee??%zE?(8~wkxov>5E2=xjnW?kgsqV-&YJN8nLS+GaeOs zQEL}cI)Bmw#J}z-Yt}C~*E&}_u2sN>8;j?_ozUJwaLUX#=yn4{4fb=r!7g~gbiSbD zFn>$(_=)$t8<>Bi0bTtK6c=B}u<4|yiZqz$t_|{VT0_*Yl{N1DT>yl9zF+6QLxeCZ z8i$)^EIH0F#|UD_OIX+-W|x37`h#J0-L-I5dmGbOwO7K44Y?iv2GXby$J`EjRfauT zZ_FIvs>ZL*_@ivMYpl|S*@|PcKbtB}a?rdL=~vr(_Or&qtw~pS_}HyqgDPn4B(kbv zEteg`5r-Y5ArO>=UlC{3T=O_j6X^dKhl1waID(U)@XRw*}* ztE8rARW}mZbZsl1%ec0Z|HY%@W#y3!;&sH-7*>f4ZS%SyQERYeSFRHTOP>Vk>umll=<|9 z1Ky7yw>rI}_&Cs|*1qryd_CpmufNS!h74B2E^P%!A^8VZ@G00^hDzJvD zEeIt(kxPqWC*ERjD3+;Th)*zGtHN8muENxJ7n~1!#~bFvxO6M6u;oh7SQ=&k2lvxN zv&Y|+qskZ73<3L`5HQ;O?$7?w6c=Jnv@2A5JBKY39sT=_)H~;RxfM|`fJDyFT8>tw zB{4k7(MW#}ocEuN-1l+%dvO(?Np&RP2|;7UoR^i072!LC+&xGeSUH$Q)$r8cT{l)V zL%4|h$&;cA8SLKL7ooj$dzUwH9DX@u4LA%e`gT26!@H|q>&PqeEL_`fK`gz*bYhHW zi|la^ySTQ8Wr>2XRoY>f=9pr1T)|-l@`oC{q}%$~8=j$Ee7d79fkPZ}=*lDUQ>=O8uTLQ|9v(m_hzg(c zSlj`W-c8&X!3<%Jf@RI~-i6xoHuuumk>$x*6+d0fpA>)Z;0hLZdpVSv%h&c7tbNC5uS?JTF=tRl^1s14Db5i zw5f*inZ6%s&{aIX&w62v6UazU`6+-|JW1Ew1zT&`A@lQWtehMy7&hsxgJ0tPTobb{3?Q`>I9i3ETECpmHk^LTa*Ar0RBN>C=dvU)5m+J19IGU_op6&sB$(?lcOQQx< zj=7<5{bTB z!56?r67$*rVg)$W@`@P2APV?`Dteh1tVm-N_nF>HX=U^o}uo#u9(1v=|Pkt!k>&fE3n1-2rraSw}9&Hi)q~f@8d{RN?R5jmP zu%t50rf|R#Raa9|JNOgo{p9kRxSp7?*YXv}NcY7HKRtfLbwkv9Y+2SCt;T-~l0j5_rGY6uTu>5;oe%icqRAp7rbp)`v# zFXTLg zr*2NZ%=C}Iq9mNqerBi;d9$7UceCN+$S9c){4|MU!iqVFpf=PnYVGk_`PsPhh;^<< z+4oA`e8i>Qc=pl%rz=-ri5%PJsclKZ)ZAv7MC0 z!wGA&!4a6N(tdSx0h;>bM~#N8Y`WRf0B@^n>Ud*rK;tAr+|QuT^6fvJAN5L|s+L4y zf!F%WPMUIy5yV{_uPQrlaG2Y1ByK&xfqNhzloAjv_mG9oin}?{O{7x7i@t%`AwwNL zw@}fxW%Ih-5)s^&oiZ!NyN+pISSNE^joc6B9^T?ZC^~9;QysccTpct^%Z!@m1*`A&Zt!x9Q{0(Bk6aO%bYA)$_HV95E!1C`_CCOt zCjFpd^YO2HFI}K1vV+>-1G|A~UD~%7cBm<)D=E9@pTfAFcgc1qs($K;+B&wZE9Gff zg4*1`4CIlb`V8>&ft{wNBq)3=lT|!jN=Zmo!v~%~2)DRxkz>yq}wh;TO z=ZEc3l`)8F(0Qo&_kRK>cd^)0s*eH{3;`Lq>Iv#u2(D$oO8a)qy;q4BahD?BQ=YXg z#xy~pyO*Qc!D#;SMj|e`367$Zn6EcWMCC5kgDdZToWNKCtKPR#v2sUQtta4cl`pVePt{VY9 zl|SuPmJ$6*6H9GZXI(s_&bDKyI6SnIN>pCM$RW=Mom~^bGR6-?RG1?a?5etR3n_6} zHav3;m>xzep5-)ZsxGG83UjC+nv2U~2pD zJ@M0#8|7)&plWcLGiSO)JRdq$YleA6KBuJXhv^IxJvPBJDI7k~D26cN%HC}!ks*%8 zzHLN4bF!O7LwbLQK0{o@OT7(!7%O4b@8thTjwjs5iZyiXUF+>t_Fk15X)I&;& z^c7~pF?-U%Rj;_9*Q7E2xYxJ9{~k+GuU-}I4)uW;%B-1E=appMZ6`w!*>?iHX;^u( z`+5HO>j`351g{Y4rS4frj4=5+;?$(nmF;)>RgLPQdAiST$&OM+IoC%2g>cj4Wxr`? zoPbrii-G7^cP*Xdx6WZd7A9S$;$b;d{Rl~2 z^=~{(3u_x88VhFWPVSc_J1c+nTn9JmJ*mdGtMVb!zx1reP^1Og^)x-TzIHxd#i-34 z*CK5y9=Dh~6TBVdcrP?XxH}Cg*8Qv6Nm1m{q8R0~Fk@udZk93!&)Zlpv2zzW2O_26 z5@)A6L2OW-@)l$JXR(mRG~?oDVSR2Z|NTj_>OQqP`5stTD#bjB#-)F)E4S*o{cp`k z1bd8r+JG+{IJr(HqNbw`>bSttxX4fvTmMPd)Tpmk;avB+{>hEk_Ho|bbsgJ~C zeoR?ImP-Fw2hXx%an}{~8To`BCQFBQ7)fDUf(9<)1;sbzBWk}!gfnfXwcpQgI?tv~ zPwoiYGdQl)c6eL4&_(N+^*OGW@)=xs#!WHXLw)P{y6b+pPdUmbe_+Fdb?Av0cUp>_ zf?6+ECIYb-EC21zs#!jsMrAv@j5;-uR zT6#NiU=;8?D)xRLm)xugB-Hp1Sz2)ND0~=LYdG^GT*v8&(Y}wEzZ=JAYf~+Ty0z_U zMl=>=`qqN3PF>i%+u^unqQGo#-U!19GF*Pn%eQwbbm%8&aPy*AW)EARG1;+%7@X|> zrRFQV^91LA-$krca7i}5MW{(NG$`76k4R%>x0E@-FU{kf%HNK^g*zK8T*6J6)g6>X z0U~>+hYrtS25eAQ?+>&mlR)T`{5Qyc8y&R;;1-8oiL5tdO2^gefy(1PDr!J%_axXm z!vrCmnjX)8YEe-9=+)!{-GvfCqob7BeY*v^*^lK{=z_x={*v;vePe zZ-2+;`xUXk^UfY(Q+1yz#RkDq$zl7!aOeDsz2KuAWupBqHQ#cR&Sik$GZY6C$o^a- z10Do=H!@`2&c5K5;o>{s9lrynnUWk0crWvr5IMPEYp~kT2ZXHMX))7xm9GXbG*v-_1fRAwkM&{5qyhe!sfyUzcmX1Eb#@B###9! znuBGVsb{Rp+Is)T_m19@fB%0YJTNZ>ZObL*1bu@@Pin@#iLP16VCTa+49C71ySK|c zw}z6G>E;bLYFbmau2;P&a-~bnRH}|+CE4jv#dFHNxI26?J$s>lq9fWW8f!i?-C7u( z=1f!`7J&Fgz!J}p4fW!fx`D9@gZkd-o8&-_t)?zb{M`J{W`aj1g>82H5a@HER*0*= z_XFQwT;Pg%ruA$F;6^Bj1z%*$4hTdrViVM-W8uNDa|mnNzqGy)J^X8WL_8@+7i+&r zq6Ty1-cR}-rxsco?V@AyY?*HNkiE8m8te&nFQ!=jyC<{}WMT2*?g3vaT=9WSgZe)2 zW*q7QY6}fTkf?L)fS%9uVhH8d%6VuY_YKlF4A^G(7zruCz!=1142}IXn0jO&;(#%* zHRI`^St+GI$_*S!5}177X3LOKW(ly7o1G0=|9zzwcRxsnd`6Nk5zEG0LG4cRwaMj& zBR_XG)>k0i;uRNuqxNROu@I4fg^94Sh@96-R4C54Y70F_{zXl3B`9?2Y~QNjG;CPE{TS|TJ(JWH;Esl3?#afApaT0u-M2YfC6B<|4N2CSX;i8a z^gWGb-(8J<1^u1q&0t(6Zc>~YmJwaD^m`|!k{gge#XaWRAw@Ov9Tka0u+hz<%bH!b zk=55i6&-0L*xm6}Y`KwNQ+}E=HT;G@2q&oFG4XE0(p46@NQ+D5Z{-H+8N8{1nh~my zBgkBF`QKiu<#{#HyuNY{eutf>9Xin{+7dXshrBFhYF!8S?iWp+vRz5=g(^p!vGKkl z<_0k1w8~gdnx7Bp1JERMCnpQxTnR+b#ZOo0;fg3#rKlJ~3Ktf$`?_+6x`?J|fiuV* z#;l>A+y;>!Xj?uo?1DkgFRxAGBoSf|>ewO50o;T5TAZ6Bt}1_!^9JNM;tsWA{koYI6RRV|%7)-w7qo0pm3iDLagK*dY@y zEUhXE(%oLQzq85BD9(LD{5N57qs%>0KP$$%zOdb;U&Ftp*)%cdwL$}KQ6fE z8$;c>=p@!}1;v`eeK!sFg73%J`#MH5<%+9Ew;7k?uP32$w)s^Yj@B~Wb9_LRN2SF~F%sm|^!(KmTw;Ir)>+vD0S*gzU1N;0%#aq2zvA>8U zw38>GjP|TXSil9j;vwt!vOzxK1PB=Kh+aH}-Mdzz#pwDDFJ5}EcI z3t39zR~b_!6n#}XBw}QBEbx z=WUNzjz{i;&$N+Sfc^PQs#!i^u$}e{7ftoM*?nR6{PEc4uewz3_Oj+PXFgOg zG$%HdM)Uq&I-8v@M=OzD_|6iTMEPUty_G30IomJq6VBNDIhF%lVv6qQ7WYx$?ah9u zX}bTm;J+*Y^|r69I8J4tOuG&cIviVZSHH)+6@ASj1e}8J(=rC^NCV@graX@~e)5A7 zD5u_i8NOEHt{(4`jFhmA9Qsm!!aF;trOT)KBtk?ojc2mynwGyMa~O0`x^v>qKnS-3 zTOUW7M?SzNfP_kImvpioX}g5ein`V3Rj?l|=dudU&P|MCr$STHQC+hmr~+WX1tq?X@=-P2iuIk}$WfRZK7i(J^ zRbLP1oEzUg?AEDxODy}@E-kN-BBduS#FF6TbH>IRCx1iD!blXix(=DTgVR<`fAZ2r zsd0d&h;g;r!_q#UZEXkh)MjA~h%xPq@W2o7$xGmLGV*qxAzZv=A2ZG#q|vvoR!oHt zW#V^w$cZk|y^=(3UE;slKR|vZDZ3lx*?HqG_8PO$LA9M)29_(ow!R}@QOIxCi?ma; zqfMOl1a#y5Vj*9X@XVoWccRnNj=!@W9FBT|Ay}^DluE_V?bB@^pM@G4I?V`;S{b6M zwWv#;&}P$rJ~w~kYAOEfu05+G&#-360meyn#p3{FrzC}w!{9Gw(L#2W=y#HM7BVFN zb2VP~awIuqa~5D+{{S?sCeYv2MM#_EYFeOGYP;R8TnmQAB-u?Bjf`+=6b@1crBO$z z#pq=gth4v0Zih8?bOD_`((B>X8WQj!>3&3{c0kKKny|h<$ehdBY#?*k6Hy2J7scnT4U5%D~gxz%xk>CoJlA3_2;dHZYAo6I_5nm0+Lv^JDju%k#wL)@{Fzl7eT z?ASY+avpf;a@hv35v^CwWP_`zsVb=Gr^dTaMvuQuWVZYszofO}|LA}4yYnrjLVnRk ztP!o^XQ<8oGT&DsMOZLR#fzLx4SUGdue)Z$x1z?5a`>Gf4`V4q`8?Xxc`Y80#z-0B z#>gxR$&{?cJW*ZC>S{=Im9E-rV}vSYpmzMMxHZ5R)s83+#kr zddm=}!o=g2*?cSp7ilZClGnSgBX7cg0{~Mdf1D2mQKi75AxzllnIIH$6ZG1n-A^!) zP4M`G<9~jX(SLrlue9|;c+qGH-FW+|=@Vbd(Jn8a`vwSOS1^qB(}2X>79U5NzufGv zWDjsE|CB1%dO4fV%e*43J4-)-}+#!UNkxw+E8Vw@U^kL;cl^giW{=Xl33S{LTJ0Y(10q_7DY9-IiA|+t)EM z=v}dAzM11AtRF%81tWN;MCDKjrHPqBtNkB8w|i^;6CN`Yw5^&_NK%NsiuL zxq)xgizeCz!^8A1*0r11f~jV^^}uAxlU^L&aBoZ$U8csom)AA_jOX0w>=!2I^R7lX(8C-vDsbCFWKx&Ab3Dpq_H>w9ut}`HCsTz7Q}K31)FTcJKuiPv zIi()JrSq(EuM6FZo>~7>*mLG3PwV=EhsJcZy|pCKe*WO=&`$@842J*4$4U5EQ?y_l zn=I=hY7;@M4$;L&)0ajLhyKKBC$q=d0LuAMhRkN3XGXZVF~SrgI;##Xrbi7sJ|$3B zz3ooM=pxSQxpyY}vdm!knVyqVFAtJDaQnKd#^-|Rr$fg&nn~aphHyeL}V9qb14xXlg)5RD@-_6k4{ULQ{e;&=)2tHn@ zL_p^>i9C%4v)(+d_{|rCDsRm(M)XotQEY6?_aC2dv9T%NXYbPW@WRxK19W z?OU<##a64CVBudCMX>&#+a5Hp9s_BCPP|NT$h?Zt8=VMYF$m;BI^bz+n@qD`a6CGS z1c|=nSrT2je)=s0xi=ota&@xP@_9eKxE@8*E$Ri-OZFbf9Wd67_3DJ_1|GeWqI~~# z&LGk10Kne~3>i@0Yy7Ou_B#1| zo`nSgqg3}J-Xe$^T{rz1HBv>24ENOH+LPLOF;VYv-Uy z5g`GuQ2EfCD0#ULPRTK{=vxo}-PK5w$g#<5L50i{frqg(n4V+wJGeq$W^;5Xaew7S z=o9pk26S>@peL8Q`A?T+&2ayHtNyqtQdof{U=V3NCzBmnw*?*3pal*zfCqQ4pO()3 zu#DM=d?7EqjADT9%wr%rimHkdr&~F*7t`enk(OxqWWyjmPZ&Jmg*CQK0(W9OeE*AvEHCjQi;>ui3Lz?-dU{98w)H9l^86bdaI2lM7F; z-WPqmB#Gp~(?zwL_Hi`OSW5Et?5fV(SkyPD_p9Ga9FV%p)xjvNDnXHOHXinUNbvQ+ zboKCq#rOEa@A}=|QwCVf%e|8tyIJoT#3Much8x@LkrMfwn8SusaCt=036`ycdB6H7 zh6rXGU)A+%kA5JuMt#s%LnsuMV_#goo#>AnEJN@4Dnq~{BSz%x^CUXc7UwKyl1KLr z!vL&bJ}TB&V(eW(^!&tuOAo>2zN}PE374`Q)jjO@=^1)@M-gisKG9yVM2UvV-sTR7 zx_8gbCo91*sV!eHX0@bAayXq-HCq>Q^Y2^#y9_rgkI@X7FZ{ z4;K!cdBeTLh?B71Sz-@BGRUTJZZi$suE%k)nR{d;1jWn!caITcFOMP#f$VK6++xgd zKV%3$xHuW8wS({u-Pyq|)~~Db1We7>5Gfbw47D^hJ&2TyA9(MlC^ImZM@ompjv`)- zZc}+SdkzSZIv6me7n~*#7uWoPL~3;Pp=?wPJNcUk@kNM-45i$t zj{3LQ3UBJrTY}-`k6)^Mei@WKjY%`2HIF&`YoD5XG18$6xjvZ*o~n98(T8(v$|Q_4 zm-b&Fb{}j&-B*1!SqMi@>@0`r>_aDf1R@*V#thk=qy2NZXtq~@ zw9z*dSvu&sb|0*6nvUjk`K&1cJXK+s`~$x7AEfV+6PlsdlP?EL!2GgdM<-ya(*~%h z@e-_&@NSuVx~46nzkWr$-SIuD3olCd#&jL#Mr9Y*KXu39Ii>?A$uJRGIp?kkZvQsy5YyyrnDM)$0YlLuwpc&ttf$UEB*>G3uIB-demM8Cxu zB(ib5_^K?C4!dKnwXpea1z<2#dhWK|D)ic}DS&4Jgs>|anpRIcB5B*_&cIXFB_GsO70x^D)^2>B?$|+T-m=?Tnj@bWl${PQ z-1NH6;A^$HcAnl|mVG$LjRQfowF2x%)jIX;v7ClKSMBQObs5;Hy6akAVnYADVCT<5 zU}rC~ovE+k0MB6GS67q7PN%(J-A#S}IEilncNDTmG^*vw5Z*#E}86M9ERM8hVTf|_{dp0=JwRcuHn)mG|!2J{_4 z?g-^@a79!-6CLqZZci;AQ(q?ma-*E0W_~L76H0-7%tECxmW!gh_m@YF>Q6By+Oa+s z|NqC{d;e4Y#(%&Z5;>pDIM%T;D&m;o7?HhGp=8g5tQ?!=mVu zy|VYpeZAG^bARve<8l87-yeDK9@qQ2UgPt_@6x+F;RVpSzv8QG9D$Lxp&H+J2mS& zwwnrFI$+IxOayifZo0dx%j8O^Rx5y%w)* z6`8R^vBms9nm+ebb@}a@!9r-W{Cj8YLylLDkLtK68BHEsjTNf)^C&U@@r6Al_C1(X z{_J52n!_=D^cjJ<<-FTf-`3S9g}q(+fLL0+kska(fIwGzk;@Uzr#x?KwjN!W?%pLe zePz8RP;m+;Mj#nIZbkYQ`T!za+bbd!GGWRKltU{CVO{9Gx@6MltG#j#W=9DBrrgdM zld0fx<&X0WeHl_Ml~9Q8M;c?GI9Ms-7J)7)YIl}3Eidh=ESlv75xpb}9v(}i*d8L; zLYHwm@24y%6g_ck5nD?qzH0kvPt_DCu3ZSH zHGWvT*62#FgF|C!QB6^)XB{lm)=lXhvO7v_OPB^tjyZa%&K`U;G7z zDls_xL|MxyxjyCHLO*n@Y6lqTJqOIhe zb=AhHcPX{RzkgKxOuj5}U&M?vJas>httVMEmFGdBsK@sqrq*L%b$2PIDqrBv;%ijK zR*h1wQ-NAPznwn)UiU{oxVX$rl9f-I)DEXJDY`=+?NT|N(AVDGB+MVh4;&6F{<)e4 zf1SV@>3Prf5UvPVBp+=zOoK;~t*>QHDYnI^(Lk-LsUfJTk$^(~Lxo;d&{tRs$RaI` z$prFo&p6-QH(18|O>s9THBUZzWEoTcmb@on(n*JP3I( z*3ae0>W`Ii55w7_%T)WaoS@v|A@yp8><^#5IhM=12B{%LDzr7L2vvB+)fgsZ#B=PU z@}&AkR_-UrKzfsSd9*!$q~j=to$3^0C~$j+d=h1JZBfICIA2;3KK5>%o29!c?{*-IQX$K{xLd^j8F*RyxE$|Hrt5wmZ< z5z&5L8)Ld!>m@23D=w`CW}-E!qos|)BMmc-((u))gL)Gb1euahE!VVzGn6G=pH}s) zqrA?=n)RWyB#DR7-H2flHkz{2%Z+8Rn8Uo(pxihBPxPeMPlyFAJ5(CN+-c`*HhV99 zOsZE(&$b1j{B!nH2{5ME}_UDSPZ@#4}CxEu-eWl&Mbaj~ez)u_SvpTmzn zeP`5VKZ76}Q>>vv4;ZUaS)Dy4yh64#3+mdW9nf)xUkvfqB<|XwnQ%e6(|{XQ}_x!l2uw3o^mf zS+c$5Mql$fx$FeCB);hm$BM_>RWFq70GG~YT+mLsl*F0rk!NmixoLsnlqX2FnAj-{ zCZ#8Opg3ab`to14OA$1zqb&MOGTmd}6Q~@eZ^sz-S#hycA65ryj$Y8Ut4iNnJNBT? zDsng`jXIitWuOVL2{`n=!T}j0e8k9FgaiL`@0YM?xUvUI12I2t(>T{?uqqk2Avt{{ z6)#KQ>!U!^G^dn(xYr$MN`DQTJP{S%dlTO+p>%ERV00%zOsYaoKyZ(T3bns_rWU9Z zUNTj|skLGIu5!qo5!zGaoT-xE`>gwfSPSnl@r>|W&0GOirir~>@>csm^K`pnlw64- zGlA076aGeiFHZh~TX4h8M9K*+$1>MyAB2wyixBIOomoXt;a!k2@@P*CoZGA_-ui0^_z%c+-PXvYtAZt~FrT80#YQ2=}b za#R3p?`}&Xp;qvoY=~53Xr|-AWlQi|d>iWK!C>5xlvEyFQG8aHL!%ez@aU<@euSSo zcbiR7=7**gvR}vvJS93BzH#3C2W@z(?|IC&-lw(owX*;pZ!TPT5_oW#+H5sdsH~s^f^=krU@}pEX+TT+P zj=pqz-2u6Hy%+=n&kQ^1ebZxLDi8G{QaZx}=O!1rTl?mV%R_I>r1vY`GmURi4DrKV zLA2D3wB1A*GQUMPahxdzD!*;5G3_-LQ1v=9r`-}+B3P-akLy`P?@k0On~QJt2i z#Bsc~s+^qO{y~emh^I%0cvu2UP5B)m124#XQ&OCY>v$&NbWZ7syF)8M**dCBhQ#d} z;BA`9OmMLfm=~Rccc%*6U)fLa9>;T*JOo)!;-dKtH@O32eiljaI@l{N=~D&|G5$Oq!JPiNU!PT!Wm_d8K%IN-uBJIPjBslGu^r9{?NR+5 zd;jNrUgr*r{r58WKF-Zcj!lt?QQM=%pX3Z{V`2#uZD>)~3@CGQt+mQ3=cpu0em z2DyGPPNO7-S)GmTc*Cv1^@E;)eZ>2PwztvVW?pe;kA8BZ$=Y89>bo2M?3l zqx!;oL;50MST1aDEBz|EM^|b({&~#S^|pmGd~K|4>}%h;L(+4k{v(4!X?F(AWvQPVpk{K0;wnxr+QE4&gb zFtia={gZPl4WX<5g4=Nf+s5h0Kg6UHRv3HQS}-GV-*f-lOcg+$N?PRF9%*I&5QRS* zpPV=1y6pP1A;kl}#H}z(yrs(4e|h#ySWz^GKEB7Sp znjXiuMtMaV>5(UIr|(3oxn#EyojFg&eTL$U;NJMYUp#X&X{e0DI(s*A?RmH=@hzIA zLcwDy4M4wwr!6T7!#Bir{SoEodWC>;8=Yk!K z_8~ta`yCpNo!RI&awlCOaH}6O{KNOakgux^FDe*p;5;#>LL!m{y za(#S)^!oJLur>;Bjyl$y4hnsh56C^s^hLgCM+&$`&_m^yWKOI?2rP3tvdqw$$ha{k zatOB@`+k3lkHA-=a=?4p)MpTQFs70O32kYDg|xi=@%(ByO?lEfn1=qOkkgF7Jk8bp zgJpqukL#GgN;%8zh~1_k$filOh?r&7I&2g=;6bhhP|02HU`g@o5o+d-Nijw_o7wkG zfzJ1rQ;a5h{$Q9Of94eh^{K{g%=~452t5BvLp4p?{Pe=tz_m!T_zfnqwsk;^eT%sL zqWOT?Zzp;aRl6(4sM*Jn4#1r^FWv#})%uzTI`Rab;ffH6c=REuUfdh+(#c}l0j0qE z%U*SN;`QRKYKK}M(S%z*dD?bkoganB<{y~4YU6>LlC`@G@H8^^mAC%jmI6wz>V9t( zCqCRQ21@KU+rvNw6U+w3DGm#Zd~yRN*k=!Mk{KJZ@9`jd;#szn+elt#dVI&ZGB>Rh zFw5_4EqpoPQ_oJ1YpJL^TB5~EbnhjaxT(iFu8XPF|BNggdC>Te%?qNHrVQ_yWaZCz zSIt(Pw#F1+^6BbdCGDeq6ryUTXfHof%MUy5c;2EvMZ7BDNfx`BYq&XLMt!h}YeaT) zOjzogO$<9?J;Dv^7S@2J{rqT%sF50f^(Z@<@1NYbQ4_c%?!v_SZ(7$yhZwn`C#V+B za8bibs>x`X*Pz+&_zfgt$9O6x z?1~1G^33AL(XFC;oh+kecz{{ksM*D<*{&(6H-~e*G5fuqc+j8q6@92qf+rfA{kZVD z;B`T9ZB+-UswufXLQiA%x|U|sWqJOOP{`;Bk~PNMB(KEXTVoL0jW%^L2Vpfun`rT- z<8t`U<1Dx9!n>j{7(xNc)z+GFLr&qZfY~Gu(UxULRfSL`;2fnX3cUi)%p0-1-)lSn zwVt3f2u*JiP0;&#VR&W`^ykh2#|1DgjSO4n5pJB zTdW>u>rF7#i?)QMczFsxHPZWWCHhmsxa}GeXn!GsE#iMcor6zWiMc6H5!n#wlC4?p zjv2|uJx+~jurK}S%62491y1hp=#v9>_7jx<~cBd8PIr8o3+R$tyEwU;Cv=AvBU zbc#6k%n|CQ9wuK3K0dK0r8>Z54&JGM+}5qcECg~#(KElB%?_3Ihb{TvsaPl(H|i;R zE9+Z-Ku{eBpf^!-av<1c;8~?K+f_6ct;U^}Tay9|I<+v59EBIVCM3#?x<4iB_B!3P z;|;eA^&I0fe|-dYz5(rQd(CGVR%;lc#q2pp32+A5!qn_OM0Qtb!3Jv z+8PQx8alT^!DUu0AD;HG&8EvD9UW6Ff7CB8IP|-TPfow>Eogr38;{$uN0G&XzNd}P zu1C_&l6WG|?wVAe7u>-<+N!5o7o2bpbq?~J`iT?O1BHJByc|RX$si&S82U}x-lZh{ z1aXEVI86KuKN&mK5`T76x+c>UqQw9m!0+=SBs;X%&=5VV5#q?lMt=)yMysdzP2P;- zEajd}1IeRD!3|;KVb?pT2Qcd4A)gfI_>T69R$M7IxfED5eRGz2C1Ojr|oP$C)dXO6jOBa>{4oMan;ie+cRQibcU;Q;F5F4~y61Ne>aaS$PnADb>kc?;q!rJBd5R z9WCAZF>+irYuQ1`fs(+#M1&U-NjqE7Smh)D#I{_Y~OX=bV~O#n;e37B|vwBv8A z$i88I2`^*Vh39QghmFExdw+~)UPB(bJ?gQUoN0$5v` z`Ul}J@lHy<{w*uYJ;iWj0U%ZVu0U+Gp%XUYS_EgYj{cSXQ7-p*XwL8Q;p=amsHv>zz};R8?! z=;XOIwCRz9-oiR)cUyD$-~l>Eh6Vlu+Wd{c?T;DgV4oU&FaB75{C$>`s~qcb+3ANX zymp<0HnXH4IU zKjN+`{CqZ+$n)7gKmbx-F#3lFg~i-h;2%vpWLNCt?WKl!u9()XtNMobCVKtem#yGz zmSBO509FH-$X8cXQxR=&PO75A+3|!UH0wW|oYbepM#BnIs#CV8kKhN zH&h?`loCzbSXH|kbc55$G<0y0|BU9sTklG7Rcf`P7qwr*?ys3Kg{YJ&zQj!jmby7O zGbjaRCnN4-mVB7}G2>EQF9e&WWf#Dl@I(sp-FTXn$%`K&^NG6w29y^HI1G`FB2O+T zhr2B(y-7aD%u{A+*V&O@`I=76zxk^{?Se;2X}Zt#t`q~lMgqlp%Pb=z!+Uo;tS&b$ zZd0o3(P=6xe7ES^^y$6XOd)pn*^mAR{0l3vK!5VU+eRDu51xAdF2G?`WCqhczebyM z>sTr?1=16hWmh1xv|yM_W^T8~N799j3^}TReMl>*!mgzFt^2J` z$O-$Vdd-`gvn)}ehVNtT$S7^HJB3)J!qmk@+v%1Cl6TmA@MQKn%)I1G`NV+% z;vnE~z&{XrQr@228a(kNM^?7v*)~(a%dr;)btH+Q4DZZDq7`@Hfv_DJ%sckxpbT@< z{aG2E!o|{0ovB$g;L}&oDCEVUtfO{bEv%TX`HS1icm8^k_Upm73Qmu(KUX$?0ZzNh z6E7cH)qKZ0_4QV-oq37A*82Otw`osUzb=pc8LNskW{?{N%Qw3$N%Z}e%7%O-sLb8)BlQJRYUfsrQHxwQ(tr!!eq$d9kGwGwCi32C`^%>Q#R|aIjKX_fDvn3` z##fZ{=T6ZzMkgLuIFH;mT_3A-w|ftAgc5fAHFO|`=T?!4ar||-LcG4*f@Tj(ZyWFT zDy@D7iZ#WGd?HYWVq0f0R_YS|ZMmtBv_6A3lS}GUEsYhjDb#-NWO~BkII&{GCcfMq zGH3Xe-s}C$@L9+&={%Mo#nnJ5$uMY7*orhed>&716E|-2nEy4x55ClN@D{YI19*D> zh9{s5-2gPBhx$59`}vl`SY{VHq(fNQ7cjr;gNLFu#yV9a1U}-o(b?P{*3nz}2+rvk zk3S%HQ$uf;`fZnHzLDFh?0$BIkSj8n$1GX=XPeyu+s^4tCCw~LsG!D$Gj~#brq*;O z9L$+ls{6xEy8yxLgZ#As_EvswgW!#TF;LjoV5N7Ru-argFRTGWC_6+tZEkX2A}~{a zC(vO{tfQj2+L}>vF;P2si!7jQaCwJ*0zmhF_(TX<;`hV7qRwB==?#-3bTCN08NIW;O!2tJ8PuSX`>#@fU!JJ^UK;>U`-p1n-m= zN-w<1(O`BG!5GL5b@EU%MWW&@S~00qGQZwFT`Mn>l739s0nDxfs4B#BOkh%yxJqS#y6!B9yqX}hQe7$ys$T_>m#bb?>)w$(9KOChQ-0Z<_s_%5fq1c4CNX&|((=jm zlvCY|PfS=~bjqx*O*A%Y3lJl!+&hfCQHa=uL=!vku7$FJcI8_9M%v_USPn#tmN-jGT$*FLM%Q_18?mR0fbLNITnsjn-M7R-{y#c?fh2XocyVoT(o3^gQ zUZO%EhYh9;<*nxxnfzq}4FhPQ*hyTXdV>@t;g6JL-;GY&h?;IVg zaJOjTyH&?0Jt1W#M4NFU6I^a;@ta-d$Z=Fh_4ihpiu-g|RIOZIeX+1SISi0}0-cF0 zPrr@!<_GB%h?E19vjA#_uaVwUvCj)9re!8^qTAj3>X*V{EWG_~mgRR2r!+FR#Vc9i zD{TlbLLa42wH|s7)0Y{rKQCg@E)=-eF5=J!SR-wc`%pvc#Jti>UxP?>TGG>h4e1E5 zL!LPzlLwYlLD{vk{}}O8A<1p2egHb@9_y)5S*)kXqWRI?dWTB~Vgs2r_PoT^etxr0 znOsFh`5L9ag`vVL@VZi~QuKp_JGQ&`Z6i9K{aybmaAj>n|Jy0vest{s%y_$lljGG_T#g22BO}w#-mjrE>(4E-c%bm|y1uRZ z8maL9dITA+7IY2Ce0g&2((Ct*L$^S$psi=~Im37g9z?V)cu)qNAQw~1O;56^Xo|R1 zA}%k<9}kpJq)Zduhik2dv*})@vlo5?D%YPEB8A);(UOhY6j_Muht46Wvdrnrnzaveg-00Z{Z}(Brg9g z(h{Gq{40*3h{WfeteQq?ih^d1t{6jGknQLjc~lK{Ab~YC+mF%>OmAHyVNql z;{!mS@B#?Z#wk}0_e{rb-1->$6nfNPGLWJ;n&&3vHjsn4@9=5}nwm)G0;`sIWj4*d z&8^`SV?17IGC~+Q{~SJr)-xgzoP?M9Nn)R)6MEfu&ipw-#9+^TJDc~E=nQY1>JQ+I zPH2y7`D8dbx%%JVx&zAnQi!QC!GT6l1ta`D%FH*%T_vFi2zobRBydNQf-B=@iTLT0tq znC|4yqDYVJjk{W-$}+)tZq4yrVx1*TesO6hLR>07)YJNnS5MC=`O>!2Vl6 zf;oOX-B13P?(wE>0af5`<21=HUD>T00Fzc~7bZ=xv> z-O-(I0o}f%2BTw|vl_ThV4Pe{b=}UQ*>M%bFnQg6`D3AgA9q}K#ve=*yrhaQI5L~K znonRn%zR^eYsD_2FM{R+h)^bC&E=8Uu9z!;pmIOiriX%8nqtok=0cm*c%Bl=V!8nl zSNu|fyW-&Kwp={63oj;5xr_lH7imGjWvH^cKF+9>jIHAN>ufK8s4R9ud`+ENfq3MD ztJU5$rT9)L)PO4eJC|OMtcTi7JdeEm`f}~=#J$f?d29%`D=^D(^@VixKdlI2Z2W#$ zuwf)eF;;5{lj7Yy4igqcBDgpvcL?dOmOeh*Xy4h})nQkXJt-c&=_>O&;W%n|&V z59>FAu0QdD<$eLr;K3KF{T}Kre{4!S3^QKeq$=_v# zLYBtJ!PXf3nF1Y!{gn8-TODzK&Cu(!)c>^~|2<0-T2N2M{F^(Pe`At=&%c6hrqkae z^Eb4D%7&}ywGKri$_avWLaX^C<;J3y%;Y{)6RcVMZ@0~gtw$B*<#(FgZQ%;+zh$({ zPGHJ(RAVa5C@;B<{2Tc9gTpKSMuj4A%ujWr)!p6VMlZRiIo1Z*>j{-Q${}jgaT1Oyv1^Dd{ z0bs(2gW{md)36~^-#3W&YuA280R{yt=`NPdR9G5mAK1^ls{)3R9<&jN+X{On--tU^ z{#m#Puy83xm*_eSuy-srSKm$FEVy;9*btkkE$ql$w+Uqgg2p8;Fq~(l=QwKox%uS2 zGb#+8mD~Nf0Ju7D7Shl5YfOVV;zDF&>(lL#l5EC}tL?IOx7+{T0HoBwAPhaeZEud$ zYtkbimcw^~kvkYwZJB+t)tNVy;N3 zNg0>wbrEhy|1+zR2fFu8>~*~f_$o78qB#dSkCm^IkbLKVj@r*Vr%m5m`0{;Pf@29O zmsOdlBCgs2Ijk<_t{dSOO451)HlA3D1h6>vd*u>a{MLmzX|^N7ik)$5{JC2t-R@7q zc;4E>9PDBX4$V9s30^_jShGzQpZX>d`R8B+4}vhz(f{1p3L^csrx%sFCn5;7UB}8Dvy$&wg;E5O}O$=d0MVU~y}zpewufuHAU~f7Tx=7P3Ygq-)&eRN{PdoSFAX z*Re7$KRWN7KD>CunzP8t0 zdLTVeOaJNMTwJA{&dH7IgLe#NP@K0a+zs8h@0YQKC3{5Yv$FM*2j=kGA3WVS0>zN+z=lyLT zpMz2cmUBDmSr(+bK|92!S@q(t#jBL7PKVYvst#}ET3qRf2Ix{yc{Hwf+c@UkFP4SS zQ8j^d@wTD2pHnhUzRM;Tc+JS@*7H3?F#fRi1i?#KB=Ql^E5J9+O-hqiM_js9`06ak z?iPT&^5wP%B*ICj04zb2&JSd>g+?m?^RWn11(<>L*W{#=3QlQS73;J*oqSu+z6+d2pSTkRJecUoguW`<&1#n%pG5q9H#Qg< zd)pjm80&R_lpbL6x}= zxG_{lCP4GO@Y&tVjc3F5qwK?Uf32^euN^~ar^UN z4B)S<8h$CCe|rTsLf_+jz>rvF2hMBy`oNx<;-B*z?7*?wTG9G4`+UN`>Fj?;d{o7p z7wunMy{^$*k(dE;$0tkB0rXqE55lr@?QPF46#)MmZ%mWSg9>Kc-&r2HQ?ok}`K6Ic z-o$iS9`x6?OM&>W={M65Z6IP11hh5705Vg*ceD?to`tbV;H_ z)S*v^Y0v?Y_cSu7aBbW-gmhRT$jjIJ7J9Qs=}pXgoxC6HF|rxOm|HS-=a+F~1IIr$ zdIZaX*FXLqt=(C$2SsJ8d7+}Y0+=h7b2W$S?adD$MD8^NGotxXn?{3g?ijLxAXs_6 z00a@FsqUEo@LB+TPn*hoA%NecxdX%%-!V5y{7CT$SCz+n--S0!k>XCIdO%pk`2z3vRzX)&@$Ex&65XFt@ zwR}Ffcr7k%T73ku?<0Py~LUXn~`4-+GH;noG-q=x;g^pw|)`rMjOCuC@+7=qKqWrBO;RfpnmP@ zjs2ge<&D@t=i9lJNj8t3t6w&{<;@u2G6^8XUJ}d@pI~+GKz)G<=nDmANB7oN?_$P7 z0odHiK!*wBq^x;NM$vIGn9#uA-YpJ?@a@T%QsbGAF(r2xDYI=%K(7ClG;eTj^3)oX zYHE$T^{MmSgm7A@?s}4Sdpo413T4Qk9ij0@#yWtQRDtlv{6=hlz9y8WYi@ZCiya(0 zVtJUG1U!iFm0*6OLWo;EM{#XNC6K51Ph?EVBm7Lo%(Te%6GH;jX@_tI?xHv;oJS~1 z^jh2p#6v`&dnX}6^(=`isB^!V3SL}`&K|9`&=Zd;r>QsSICCW>fOeVkLe+E8=yigKb!r#eG6`5_W@0!hL#lc(r~$Q5 zuZED;6=jh~&uv@~X7e0XO_tlLKMEgGW;nP<;NP$t1}k)*I!QVQpL&V1g{6f_Hm&+& ztK@!?MWkt<&yR%`w(ncV`TK|%gg?ZxUppgmgWl@FyS9sD{28k2%^qurmjYxITB5;( zukTusccw&E2BWXj*xWM3x^n}zYw6a)R_e?x+ix782h*XE5T4y|=}gY%YxlbuA3dI1 z4siGaiPd|_YEeuo-nP;A)>Sf&?p|yDSZa9J^X8r$Z(H{(Sf-&xo~iXYE1}Y@=mnp= zOTp@N3K_yxm}lcDjzt$TmZ7@+?NZy$=NX1a34AK>?rRw}=-HUPRF~RH0j-0gPXQ&ri84Z3m+QC`b;h{u8-YZtrOr(yycCGs#@TY4*a0E2DR@_I=;+ zFQbr>3_d?GdY+;`S2y4FD19c06mRDsW8TYVq;HIMflVkk`Cf6X$)a@3YjGAOrf02Q z#Ek0uAqFEj2RlE?b-4AtE46HXB2va;Y(?(HYt204sig_i=A;HXTOr+4y))mC?-PZl zXZO*X-bL$zcQP7qWfo$ZFMC@{JyR4%VyswZ&?`g0L@z4|Ulkg!HN?wk!v!Ki;psli zz;V-}2XwocU2N3+J)ikIAK?dO{3d9>j)~8Wt3Y1nG9{I=Ghvc^v;L_hGMxW8BBojr zLE_Hpy~<3BkR+z(h-dRTo?WnJdYBR87=O9;jW=LLEN8w+T+~YaBz6B`-jFE_C1Awu z=Y7>d$*@o`K}%x zb3l%BY#!*5^WX+%o?Y@L?h;ZMK09@n%ZGIj?-i9Q%zk$k%))c-`w^ph6f`9*Foj32 z{$yr%(aYTu!d_zE}08+eW9aN9Mtg**5ApNU3W& zcRkg)D&G<+mI9-{VE6^<-?)B&2-K2v`WKwYk_^TaSbjDC)zlc4i0`_^08{xX7A8$u zE?QXXeE3d&EiXCL2J|kZD4Q=t$vxQM;98Alr($Io6{_E8?MSYk3~R-U1MPuRn+}1( z=Qw`FA7dv(W|om}OttnjO5?S2&%3Bz)u0HhY0m4m4mI-`@{Y&7hQ&9X ztNh+3)Z+sD8pB;yN8Gz*zkp$M?J4=W-gVvoA|?thFqEz3FVJHPN8+jlY7CN#a?-UH z?&lS}NKgUxn=&|AbpKV-rkqH+jHIpY5Z7uE@z`4X{S=#*@_#})Vv0>|4l=41&Ii;NNfM&j1JZ@Arhj36q%AckX0VaQhJ(NL?jW@os zYMbNBz&r=R=gYQ_zTTPmd$QB<25q0-c0+}~+ciF($xS?^zO*m<=N1ojS^w15|4ryQ zg7avTG=NRTyR1QN3tym!Gd=M4o(;JjD9q?nj!I+QXsNn6fXtxfNMu+8;q=I!?HK41 z`!d0aKKNA%#qGb2f8H~=H0bxg68IqK8^gP;_n<)QUb3+?Q{NuwETwm0ADeAV4O<&> z%+r}@nv+>7>({vb!tsB84)DS~4A5uxB+1-a72Zy&=gFwu7w&SN{jwu?-uVa(w7~vb z5NK_{hSEa#qrM97gXj#8{~vJl-<5OX&f@7WXBMsdpPwBOL{uyb1kDQ|#JB(yU2k-N zH~er#;{gWYLr0j3d29U*U;)WWkB2^|AWk5<8TJ3Uakx2x-aU%_T~x~!Ltw10a~m~ zjV`ViqP7L1YJ1nkJx}b}-&GNN0*Y5aNa}|C(Y!9pvtkEA2LLsgjd6oa{>62_eV9W0 zdaj;+MwQ<$+8|LwXEYpr)!AA`)ZpKH!jly?k%Jhfo>+)qy8vnYeOIbY`>f4GJp50_ zg~t2T_UXN?SH3~oW^1jfk}j8p(xiW5B$x|w{}qYJk|9P*`X!vnS%Fw1D~_J{c~f9X z87RuS=By6atnjk8*7ibyDCF%N7lu1;Y!d$N+Mj}?mYXQ4R#0hcdra#R@sl0=S040V z-vJ516Kb%qNZXs+*BZ(S{kN(FdU~nrSFJ3*j)MJoK8}gA-j2b@uQ}iUoSe>`JWs$A zVNBUSirf|#cmD%=oV;hxLm0v@c@}n;X%eDKfdQl!Ln_rGj!R$lFlk_LmDU)2^f6E7d8eK4SJ~Q#kw!r~f(M z<<#}D=R=p?Hwerr59M{<)c;DSb0vGY%CkUJ`~643 zRNsF$;BA$li1RvtYjcL1xVgW_4~9BxWMFqiS9lJSi02FehNz&Zc7NgC!S<3(!}r$b zy^`Oma-FZsLy<6Qdv$xwncY+EZp*GF8x*H!?Y7{}DX{tefY;}RO<k#;<2}l&DxXi< z$jEY+GjM>$v$NxBN;1)y}lC7#S8gDHa;E-#lGIXmr!(=fl`c@1V*3{peISL(~l4D{QTJP z?34{E(rvuv!?gC2!TXKykQ46(4&ZCRKgku?couz{Yo+wT12i>RCX-mv(kVs-{b&LD7i5Yk8Vj!0^$1iA0gnbauJCWwiV zPU;qD-dPjk#h3vUk&aeS9xW@|nPJ>xgz=VkVT5z_4-a?z!oz4ls3WE*iz3<)gZVI0 zrkg6YN0{cd)&*ZmI&a39^-+@F^;FMUz*uXzSMT-nfUCGkSx$hF) zbUWrWU1%kdXF58=@@<2I!h~w*^cM&Z{AEr(5G1DPkO)S12=A^guYKAF08ksQzM2nh z_OsR7$&GUYv%M^)-rRowx-hxv)X}AMQ*FV> zJ)cX#0}k6N6?p;qnksdxxWcfm<1H&^JXT>dVt~a5k>up$>--o#R` zFI9&&GF$P-lUaS4x5OT$HGR@9rj2@`+GK}BE+PEy z)6sfe_uj6G@Nj_3p-%uJ%8FNnadv_QktbHTWJ%WpzsRhPXNc6cB6{mGQdj?N$haI zLEkW-yR8AFKjXzqm&^AcDV$-opo81~;5$hEtCBO43pkD;4Fn~xdJVW`nbVX{4 z-3BnM@UZ$OT&<4s#TqTtCe{FL0C7jtiFVEcrosj-)tDkqZdn<%dqwE`H~FPQ0U$^$ z{E_r|q>b#y^8%Hmd6^1<{ol3aPiyjnB_}4PPrMq>2&C0&&zxl*pslY&i~B+AE)UA3fn{5%wj;@;qUYPxq%O8wOJ%?Rr+7Q9)>t`OXBvw|5y0A2~}4~ zqQm}m#eXkWmJVub-yM4n|E*E}xhd`3RdAWptW^IiV78Y5WrXsftiAs?3_7`gPYAdS z)fMDlrog`r41_y?+gN;c17HRJ{tmi9)@g7VPfcAx>tsOlf$wg zV4)~4eik^JS-%o^e6ZkEcW@WQU_|%)r=9F!LCOR_xnAJF>}ZW|0ThAy0JQWqRK)wW zIO{X81C{5Ya-a_=%wK|^?y~c*Cq9dv3$rko@L={IU_USdw3n=>on2S7KY!f&Yg=(o z0l$KOKIP#gxjE}wG@!xTcmu_(HfEm8<{mj1!fD{bR@nIV2~!Q|o4ih!@!Gzb1l8TD zAQdQ(8I)BW|2`;CIxF?UXI5sca?EJ%5)HR_G*Qb4RFCA2zVcjpe=es$^K#{j+a=cO z`F0`&YeOF{ivDp5!ivE9azA$L>*Nk%ByS|jA!=L+Wi)*S>6)TtK~c4!IKYj*TVPA| z2#jjW8Mu^D7XVwb0LfDGSE~cS#sfjq>^;P$(pzuWSCR>U@N-iY>23Yqt~=u%7%=z+ z5UgJ3Y9`(;?ko^-|6|Izq0TSMdR}>B7>fKUCAAB(75~ky`!NgkVrFx z2om4IT-hhzR`Q_i{y4tiao`78&Z-(X5EH|zfDqA5h)d`Rg7Ab~h^_*}XnQL_xs}GO zd5TXxDElKMgb4|k-CFVP_I(nJ@3s5;xjN1e+g(U{mXa~<2H?1>r&lnt>jZAltnxW? zoXn!vE}F>H4C^6^+kwgmkRGMT3*Q+9Bl8o_mShZj=-GAuOSeh`Y8SEHDkBR&i0qYQ z$*qW|jUfW8*164ZfKKckmN;I=m+o=h6Uf9J1Y~MoJBcN19|dL9Ccm2bCTI=Vo?AcD zk$IaCBePUJ=ySjaD&b(d?;hk#e_eN}f`aatz`!~@fyPRRxcRKr3EDwdVOFr&p{3PFI{fk^HFJ!P-Jvr7nM`i2y?&@MGRH?jT~ZT*LK{~&P)GKbbpm=d^b z+w26pY8<~0B5*6)L^Qf-Let!qVk)3MrTK3(k#q`H+b>D^9vxzX6JczQey=hHmmrN4 z2xboc!T9wY-pt=j@OA3n51J2HD3Uk7I9Onne+H49eU$(($Ti~GXO|2U@KStEe17B+ zwr|4ntM9aSzWzRyXWDvS^eWxT>>8U+LI7;`OaJ^V>*U{HMOe+WH1jE!hTbIk{1t2dHvhqInc^+7*`;za$9%PmpXt=@0)To% z+ji0qLfpZA-hNiuCDAveuJSk7DFgb)X6=`He&Ml1E%gxP1gg1z0Wzm|0bcjZ3EN;_ zbpLe-ZMeRI^tY}m(WOdUZ5QUZ3j^^2UCs(1!wWiEL!`-*X9iqa!057_jAi1jf7=Df z1=W~+sEj{|Ajbr4!rm8Kbl7YfCoa8iAK=dLtu1X;9i&Ll*nRk{?Z@91hJVSeij)-Q zaPwU@ztx86ky>z0Z=GiI(7Ogd^~ap!oSA2?x$gV^UCaHut_z->fU+O4o9u+Jk*V&iWHkDB^461eQCku8Wb2T| zR?#yU!Yq*o!+qbmXBAGf3Wg*|?a@${p-7XC#jtj!9NP6JPe7V%6l#J z(5kG_F?}~Q_n~5?(o$QAK3^I#0%|6e&J{kmFT^k`CY^@da?&lUEN~Z%hdoDNw3h-upeK2LVm=fjeAH zeVnbDdM3N)0V0@Y7aB`^bVr=|pqH1rQ!)$%bC+N(JV#{{R~-s&%jqEzLqW zM8g(NJg?I1Xv?m>zkG8Js6Nb^q5^fW`6E(5 zWkH0dD*uj@f5jpHJsyN~|0xG-@BelVgm(Yejultm^w1+k$Smv?SR~IDm;q;(H;-oD zsSI)3CGG`ZEgH`o@xQ})l&K@JpAq_(s67JtwyT@NJGCHH?z9Rjb<7~VJ)lyfgN`WJK_pFtI_582LXE4Lb-H(q% zo614=<$JIkzF@k~3y5O3MR%z!9P8b3G(48p?TF=x9*&S2l^aj))9xfF?lg zIvvN=0lW2BU(GYPeGF9?D^|Y!eGsW+2Dtg;?aTzGQHbByUZ@@M{B84rw|$L+Yu^on ztDqv;=TNE2H3z$D@3(diQ&1fnmfqs-^g!m3+&L@upS)fGqS<3=sHARoxPtK4*1IZ!JxvKomW0W zb;hv)9L??2np{R|v1%xO2P61h8R5Iph3=Ve&-fgdyfs@l*1+y6FLVBy-o-~8du6Jm zZ|L#ep}9jNE38J#RuO2m;OY7$`-A5^A?SIyo1msw?_#ZKIx?K-86$Bzaw6=Wf zMRU+4c+Tzo%dhfr;^kKl(~L&A^hb8B11?(v)w#bSLr6RO5b=6)rB>I|(6nquC~ELx z9Sdi|WDy-Jq!{%o076kmv~i#(K~>e2&ZY;PoT45Ot*+8r%JhT*^~5JbgB$?{Xh`aW zNhc`#%Y9J?Sdt$4+ytX5m#ckmmbi~LOoIDFr*cqO9DlPDdW-`4L3q7?R+^qil~v@ks9KFIcBBs$Wr+=o|U?C1Kr$uDuowe%8(vAq+OSq&ur>T;`)cB=ZsoM3&a5eQ|P-u8L&kuL@ zY$*HDh$S$Hd+%qIYjMN5D5v3#7-a6idClIPS?fE&1edNC}6;!iO$Q2NvwR{AcR#waXH27?_=JoJ!;I`7fD8}M43 zm^v?i@ACOoW%m(O<=E70L7ine1_ZWAg1YgsXk@>n3l}o5qW_hfRqp!DLq_q;;~Wmj z7`}1leSW1hYJ}+zS$3!Ma4KJ~et4`Y_^c=x;nP1=8e?Z}Q1L{9EVRh1Qh9Kxe9$#H zmoGkbk@weD$#@u&S-?{IML45A*dRv_b!~vhv*XUOT!~lCR5}-2+kGv%M7Y-T9mHDt+b3cztA4EQVy0<)Y!34IpFs#~5 z&wfTU2oXWpB49e@!-r6lfPCio71Y%k3yznXk~@m*Z~R?g_I~ZxN{NG&DztSmqo$P< zi8_$L&VY)5%NgW!wD{*fIsb+~7AmL~e7lsSGmxnf-jnn|xNUxk-sf={%yUcV4g4m2k+-MmzD(53ZG#737gqbNU zjDD72-CCLFD_#rb*s+ipZXxa>>gi`gTOb-%JMxuMB54x zn4m<}AhiGXFep4)0kKXT`YZ&Dt_F=?&gHUsI1|$uvqu+^&zASPEu3rMrGC)WERvW~tS#>b z?pB}r_etH6!LPC$+bq+LvaJF0XER2zizzH*!1+q(ssP+zg}_*i<$2&}ZfO#MapeoH!i-KmJ}?S=AgT51r$2jbMm%>@2ln zYD^vvro`}3u~lg|=XS+WCKo8sv>P8l8BAGVM(5L0WOeQcSlbW6&ox?YI8d}8d;~fb zMR1Ts)dnBH7&{F3s0)1vCX~ct1F(%neR>4C?DSB&)=y4bvX1+*(eNucTwSNEyl)qI zgp*U}!<%Jvstn`3$b>^VULI>kJni+K8xG0MT0c@G{nsXX{JO zb${(SL~=<->mgE{HKxZ-pz{e?68Yb4GE~nFc>lCBK#x@iq={acuIbReqz+|`Q!>c0 z9VZ&lrFCj89&2HqW3Y&*p66z%ebyiWYFqul9MSepvuC86d6Sz9byoB6wv|nWhox$Z zV@jtDR`uNVUgOTre{2N`T>(^C@iAa`lcCz+lyBO?f>FnbMf>;d9?3tkbjLh|Ps}+Tk^*V%w zr$4e6UTdMy=N2#}^21hu7kOYlFZKL6_ZD_K0lD|scmut7Hwh+)XD4c!WGm90Xf=uzW#>?)GS53Z~zfbZ1u>)y49>+ z08FHX*eNO~XJ`%v5Uv2mRENc|!gaX9Z>7@lSx<9PJ=O=HydxAP%7ZrykdMa z>*Lvxu`m%;%A!W)-91>mD3^lAMvfD?PkG?l6N$7Otxd#DV)MCGynAb`baRzohlVn) z93Ou&m%}4TCsVZLliKcCV0$KEJ^ft5HJDM@J(@}ZHCjXuFMz0e zU7CDjbRyyVc_rUCLTC?~4z@io_I=R+G3n~qJh=d75NLkyMc0xr%8z@Exsob zqMjzym2yG~KBJ%X;{``19DLuCE%;mZUa8<0z8Z^G$+hsXfC^|TW2VCPa_R`%rkV(L zNB-l}B?1(=Ix|oXW2C&KiaVihH1FHmhC88tK9tQ=7REjrC9T=VKh4aM%OcBC%YksK zzP|ZF*IHLz8jGLd8D#CNaK+!Vh?j9&`u4YA_4!I0UdI@$NPGVw(rr}B_6KBppHF;A zuy~FO`Kls8PHER+34xW-GGWU8CKzHx4)U?dNZSp8$zTYr6qeS+HuG$_2#j}@C56Eg zcPbmVP-H}KUwD~es+!Hu$3FQvJGRkr|rFdWREPCM!xN zp^mC~fhI>Bl?Pcw9r=&G+)C?JxrY_yY3~p7o719G7gCEkSy{Aa_Ni=Z&_8;R2YQe% zBj{4o-b1N#tKF?XGi}4eKPvS`7=2QQL61uJlz>r@_#5dr4_rPX(LJqE$~Wz~4c2To ziEmE#11 z#WmOhi8-fT2k)84;_&dT8zg5dfY9G6fT1O$>SL&4jXrZeWT_sR!E~N{G@N$a zTCl!M<6{3vgX;Hmk=g4>E+v*Ev2LOUt@I$M53+hqc=5i z`){v)hz(Z^f9gHmHjB$GwENOuy>LZg&^8H`$ahvV>gb?g>?9@|;clKrDppx|WS>e} zlCB%B>i9>xFhPLN@h=FX8TG@A2ZC5a7|zy=3UdmnOklG6NyZCuJI3hp_p3hN`CTS4 z<(-u31%;YVN;NA1!)s3Me!mP@Js;Row?CCi(6S}Qq1+ZnG3r)sG&Qp%$pwsR!fHU{ zm4PY~&V;0N!a#TBi@~q&dVha1gLZp!AH!|eFQi|aXBWzq*b6`QiqxfwWuV|y&mz3Q zy-V+JFa=($&d-(l1^ke{l^_cq;+pX5;Xk}T)bB<8@2&r7;Qy}%uwqVLwyKN~$@Lp6 zQ^`SNV7_EB$TK+)?Q7mLwVtD0$JkD9)o^XntWf@PP)(M* z{_cd3VVd*g`*bTytDudEy;+%VDyn8{{?)WBWP^0%7YfMMb-LAU^`t0yf5P?}(UFJn&S%nNvfb5K82^e8-zq7s7S?#xy1QmGUPS^S#AvY0kd%bG5c}sg!QMTzIG@V8!yFLuCtyf-0^8-dO+4-WE3p8@ zt%<(CdwU!FmM7zvKL??)UI`E+Np(v^(R;qMHC3SiP>aF1!I}s!9i;buH1QB87(Skx( zz31TZmsB@I6kTikmU_3G3Tn#Dp*)tuhsTp3`{0^iIhiEFyeQ3XL`A(Vb+OP|t;BzI z#vWAKU)b6z`eA|+MS5i*HcSDK)0 z0NRjM>Uw@NMC!GZgF=rtiP(aJ5)RU)njz^swP{a05yz@=Cg8>>h^k; zqUWUE3dmr^N28SvE7q)h%0W1*h4LYMq?tG5BEFaD z!e-Ngr#GG*S^|41>!)s{3;BxKu)reSr+6vfq2g4&7M1KkW`SvDs>4he8?xH#K}O1D zzPc4SQj`)g^j5HUxoD_cY3t_sADizuSi=*T0ragG8_zM>f!}TTx|aSb;NAvD&Sg6J zar)ligw}=Y#?)o)%M26gl<*z z22G>R@F&VS-wamKC>@@d5hvVj+!x2dKG#CetuPAGT1DxmiedcVH?ZMu2Xw>t+p9_2+% uu|Lrvqdo(2bq Date: Wed, 4 Jun 2025 13:45:09 -0400 Subject: [PATCH 04/11] added first pass of scheduler fundamentals tutorial --- .../04_scheduler_fundamentals.mdx | 238 + .../static/restaurant_queue_manager.svg | 7038 +++++++++++++++++ 2 files changed, 7276 insertions(+) create mode 100644 docs/hpc/14_tutorial_intro_hpc/04_scheduler_fundamentals.mdx create mode 100644 docs/hpc/14_tutorial_intro_hpc/static/restaurant_queue_manager.svg diff --git a/docs/hpc/14_tutorial_intro_hpc/04_scheduler_fundamentals.mdx b/docs/hpc/14_tutorial_intro_hpc/04_scheduler_fundamentals.mdx new file mode 100644 index 0000000000..868679e166 --- /dev/null +++ b/docs/hpc/14_tutorial_intro_hpc/04_scheduler_fundamentals.mdx @@ -0,0 +1,238 @@ +# Scheduler Fundamentals + +:::info[Overview] +Questions +- What is a scheduler and why does a cluster need one? +- How do I launch a program to run on a compute node in the cluster? +- How do I capture the output of a program that is run on a node in the cluster? + +Objectives +- Submit a simple script to the cluster. +- Monitor the execution of jobs using command line tools. +- Inspect the output and error files of your jobs. +- Find the right place to put large datasets on the cluster. +::: + +## Job Scheduler +An HPC system might have thousands of nodes and thousands of users. How do we decide who gets what and when? How do we ensure that a task is run with the resources it needs? This job is handled by a special piece of software called the *scheduler*. On an HPC system, the scheduler manages which jobs run where and when. + +The following illustration compares these tasks of a job scheduler to a waiter in a restaurant. If you can relate to an instance where you had to wait for a while in a queue to get in to a popular restaurant, then you may now understand why sometimes your job do not start instantly as in your laptop. + +![Job Scheduler to Waiter in Restaurant](./static/restaurant_queue_manager.svg) + +The scheduler used in this lesson is Slurm. Although Slurm is not used everywhere, running jobs is quite similar regardless of what software is being used. The exact syntax might change, but the concepts remain the same. + +## Running a Batch Job +The most basic use of the scheduler is to run a command non-interactively. Any command (or series of commands) that you want to run on the cluster is called a `job`, and the process of using a scheduler to run the job is called `batch job submission`. + +In this case, the job we want to run is a shell script – essentially a text file containing a list of UNIX commands to be executed in a sequential manner. Our shell script will have three parts: + +1. On the very first line, add `#!/bin/bash`. The `#!` (pronounced “hash-bang” or “shebang”) tells the computer what program is meant to process the contents of this file. In this case, we are telling it that the commands that follow are written for the command-line shell (what we’ve been doing everything in so far). +1. Anywhere below the first line, we’ll add an `echo` command with a friendly greeting. When run, the shell script will print whatever comes after `echo` in the terminal. + - `echo -n` will print everything that follows, *without* ending the line by printing the new-line character. +1. On the last line, we’ll invoke the `hostname` command, which will print the name of the machine the script is run on. +```bash +[NetID@log-1 ~]$ nano example-job.sh +``` +```bash +#!/bin/bash + +echo -n "This script is running on " +hostname +``` + +
+ +:::info[Creating Our Test Job] +Run the script. Does it execute on the cluster or just our login node? +::: +**[Click for Solution]** + +:::tip[Solution] +```bash +[NetID@log-1 ~]$ bash example-job.sh +This script is running on log-1 +``` +::: +
+ +This script ran on the login node, but we want to take advantage of the compute nodes: we need the scheduler to queue up `example-job.sh` to run on a compute node. + +To submit this task to the scheduler, we use the `sbatch` command. This creates a *job* which will run the *script* when *dispatched* to a compute node which the queuing system has identified as being available to perform the work. +```bash +[NetID@log-1 ~]$ sbatch example-job.sh +Submitted batch job 137860 +``` + +And that’s all we need to do to submit a job. Our work is done – now the scheduler takes over and tries to run the job for us. While the job is waiting to run, it goes into a list of jobs called the `queue`. To check on our job’s status, we check the queue using the command `squeue -u NetID`. +```bash +[NetID@log-1 ~]$ squeue -u NetID +JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) +137860 normal example- usernm R 0:02 1 c5-59 +``` +:::info[Where’s the Output?] +On the login node, this script printed output to the terminal – but now, when `squeue` shows the job has finished, nothing was printed to the terminal. + +Cluster job output is typically redirected to a file in the directory you launched it from. Use `ls` to find the file and `cat` to read it. +::: + +## Customising a Job +The job we just ran used all of the scheduler’s default options. In a real-world scenario, that’s probably not what we want. The default options represent a reasonable minimum. Chances are, we will need more cores, more memory, more time, among other special considerations. To get access to these resources we must customize our job script. + +Comments in UNIX shell scripts (denoted by `#`) are typically ignored, but there are exceptions. For instance the special `#!` comment at the beginning of scripts specifies what program should be used to run it (you’ll typically see `#!/usr/bin/env bash`). Schedulers like Slurm also have a special comment used to denote special scheduler-specific options. Though these comments differ from scheduler to scheduler, Slurm’s special comment is `#SBATCH`. Anything following the `#SBATCH` comment is interpreted as an instruction to the scheduler. + +Let’s illustrate this by example. By default, a job’s name is the name of the script, but the `-J` option can be used to change the name of a job. Add an option to the script: +```bash +[NetID@log-1 ~]$ cat example-job.sh +#!/bin/bash +#SBATCH -J hello-world + +echo -n "This script is running on " +hostname +``` +Submit the job and monitor its status: +```bash +[NetID@log-1 ~]$ sbatch example-job.sh +[NetID@log-1 ~]$ squeue -u NetID +JOBID ACCOUNT NAME ST REASON START_TIME TIME TIME_LEFT NODES CPUS +38191 yourAccount hello-wo PD Priority N/A 0:00 1:00:00 1 1 +``` +Fantastic, we’ve successfully changed the name of our job! + +## Resource Requests +What about more important changes, such as the number of cores and memory for our jobs? One thing that is absolutely critical when working on an HPC system is specifying the resources required to run a job. This allows the scheduler to find the right time and place to schedule our job. If you do not specify requirements (such as the amount of time you need), you will likely be stuck with your site’s default resources, which is probably not what you want. + +The following are several key resource requests: +- `--ntasks=` or `-n `: How many CPU cores does your job need, in total? +- `--time ` or `-t `: How much real-world time (walltime) will your job take to run? The `` part can be omitted. +- `--mem=`: How much memory on a node does your job need in megabytes? You can also specify gigabytes using by adding a little `g` afterwards (example: `--mem=5g`) +- `--nodes=` or `-N `: How many separate machines does your job need to run on? Note that if you set ntasks to a number greater than what one machine can offer, Slurm will set this value automatically. + +:::note +Just requesting these resources does not make your job run faster, nor does it necessarily mean that you will consume all of these resources. It only means that these are made available to you. Your job may end up using less memory, or less time, or fewer nodes than you have requested, and it will still run. +::: + +It’s best if your requests accurately reflect your job’s requirements. We’ll talk more about how to make sure that you’re using resources effectively in a later episode of this lesson. + +
+ +:::info[Submitting Resource Requests] +Modify our `hostname` script so that it runs for a minute, then submit a job for it on the cluster.
+**[Click for Solution]** +::: + +:::tip[Solution] +```bash +NetID@log-1 ~]$ cat example-job.sh +``` +```bash +#!/bin/bash +#SBATCH -t 00:01 # timeout in HH:MM + +echo -n "This script is running on " +sleep 20 # time in seconds +hostname +``` +```bash +[NetID@log-1 ~]$ sbatch example-job.sh +``` +Why are the Slurm runtime and sleep time not identical? +::: +
+ +Resource requests are typically binding. If you exceed them, your job will be killed. Let’s use wall time as an example. We will request 1 minute of wall time, and attempt to run a job for two minutes. +```bash +[NetID@log-1 ~]$ cat example-job.sh +``` +```bash +#!/bin/bash +#SBATCH -J long_job +#SBATCH -t 00:01 # timeout in HH:MM + +echo "This script is running on ... " +sleep 240 # time in seconds +hostname +``` +Submit the job and wait for it to finish. Once it is has finished, check the log file. +```bash +[NetID@log-1 ~]$ sbatch example-job.sh +[NetID@log-1 ~]$ squeue -u NetID +cat slurm-38193.out +This job is running on: c1-14 +slurmstepd: error: *** JOB 38193 ON gra533 CANCELLED AT 2017-07-02T16:35:48 +DUE TO TIME LIMIT *** +``` + +Our job was killed for exceeding the amount of resources it requested. Although this appears harsh, this is actually a feature. Strict adherence to resource requests allows the scheduler to find the best possible place for your jobs. Even more importantly, it ensures that another user cannot use more resources than they’ve been given. If another user messes up and accidentally attempts to use all of the cores or memory on a node, Slurm will either restrain their job to the requested resources or kill the job outright. Other jobs on the node will be unaffected. This means that one user cannot mess up the experience of others, the only jobs affected by a mistake in scheduling will be their own. + +## Cancelling a Job +Sometimes we’ll make a mistake and need to cancel a job. This can be done with the `scancel` command. Let’s submit a job and then cancel it using its job number (remember to change the walltime so that it runs long enough for you to cancel it before it is killed!). +```bash +[NetID@log-1 ~]$ sbatch example-job.sh +[NetID@log-1 ~]$ squeue -u NetID +Submitted batch job 38759 + +JOBID ACCOUNT NAME ST REASON TIME TIME_LEFT NODES CPUS +38759 yourAccount example-job.sh PD Priority 0:00 1:00 1 1 +``` +Now cancel the job with its job number (printed in your terminal). A clean return of your command prompt indicates that the request to cancel the job was successful. +```bash +[NetID@log-1 ~]$ scancel 38759 +# It might take a minute for the job to disappear from the queue... +[NetID@log-1 ~]$ squeue -u NetID +JOBID USER ACCOUNT NAME ST REASON START_TIME TIME TIME_LEFT NODES CPUS +``` + +:::tip[Cancelling multiple jobs] +We can also all of our jobs at once using the `-u` option. This will delete all jobs for a specific user (in this case us). Note that you can only delete your own jobs. + +Try submitting multiple jobs and then cancelling them all with `scancel -u NetID`. +::: + +## Other Types of Jobs +Up to this point, we’ve focused on running jobs in batch mode. Slurm also provides the ability to start an interactive session. + +There are very frequently tasks that need to be done interactively. Creating an entire job script might be overkill, but the amount of resources required is too much for a login node to handle. A good example of this might be building a genome index for alignment with a tool like [HISAT2](https://daehwankimlab.github.io/hisat2/). Fortunately, we can run these types of tasks as a one-off with `srun`. + +`srun` runs a single command on the cluster and then exits. Let’s demonstrate this by running the `hostname` command with `srun`. (We can cancel an `srun` job with `Ctrl-c`.) +```bash + srun hostname +gra752 +``` +`srun` accepts all of the same options as `sbatch`. However, instead of specifying these in a script, these options are specified on the command-line when starting a job. To submit a job that uses 2 CPUs for instance, we could use the following command: +```bash +srun -n 2 echo "This job will use 2 CPUs." +This job will use 2 CPUs. +This job will use 2 CPUs. +``` +Typically, the resulting shell environment will be the same as that for `sbatch`. + +## Interactive jobs +Sometimes, you will need a lot of resource for interactive use. Perhaps it’s our first time running an analysis or we are attempting to debug something that went wrong with a previous job. Fortunately, Slurm makes it easy to start an interactive job with `srun`: +```bash +srun --pty bash +``` +You should be presented with a bash prompt. Note that the prompt will likely change to reflect your new location, in this case the compute node we are logged on. You can also verify this with `hostname`. + +:::tip[Creating remote graphics] +To see graphical output inside your jobs, you need to use X11 forwarding. To connect with this feature enabled, use the `-Y` option when you login with `ssh` with the command `ssh -Y username@host`. + +To demonstrate what happens when you create a graphics window on the remote node, use `gnuplot` as shown in the following example after you have created your `ssh` session with X11 forwarding: +```bash +[NetID@log-1 ~]$ srun --x11 -c4 -t2:00:00 --mem=4000 --pty /bin/bash +[NetID@cm034 ~]$ module load gnuplot/gcc/5.4.1 +[NetID@cm034 ~]$ gnuplot +gnuplot> test +``` +If X11 forwarding is working you will see a test plot window open. + +Greene has the [slurm-spank-x11](https://github.com/hautreux/slurm-spank-x11) plugin installed, so you can ensure X11 forwarding within interactive jobs by using the `--x11` option for `srun` with the command `srun --x11 --pty bash`. +::: + +When you are done with the interactive job, type `exit` to quit your session. + +:::tip[Key Points] +- The scheduler handles how compute resources are shared between users. +- A job is just a shell script. +- Request `slightly` more resources than you will need. +::: diff --git a/docs/hpc/14_tutorial_intro_hpc/static/restaurant_queue_manager.svg b/docs/hpc/14_tutorial_intro_hpc/static/restaurant_queue_manager.svg new file mode 100644 index 0000000000..5cdada84e0 --- /dev/null +++ b/docs/hpc/14_tutorial_intro_hpc/static/restaurant_queue_manager.svg @@ -0,0 +1,7038 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + HPC diners + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + Login + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Queue manager + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + The queue + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + } + + + Idle compute node with 6 cores + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Idle compute core + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Reserved + + + + } + + + Reserved compute node with 4 cores + + + + + + + + + + + + + + + + + + + + + + + + + + + + Reserved compute core + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Busy compute core + + + + + + User + + } + + + Busy compute node with 5 cores + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + } + + Compute node with 4 cores + + + + + + Allocated + + + + + + Idle + + + + + + + + + + + + + + + + + + + + + + + + + + Busy + + + + + + Busy + + + CC-BY-2.0, sabryr + + From d92b771285a2bc3c4573df5d3803bd6e3d207c53 Mon Sep 17 00:00:00 2001 From: Robert Young Date: Thu, 5 Jun 2025 11:11:06 -0400 Subject: [PATCH 05/11] first pass on environment variables tutorial --- .../05_environment_variables.mdx | 161 ++++++++++++++++++ 1 file changed, 161 insertions(+) create mode 100644 docs/hpc/14_tutorial_intro_hpc/05_environment_variables.mdx diff --git a/docs/hpc/14_tutorial_intro_hpc/05_environment_variables.mdx b/docs/hpc/14_tutorial_intro_hpc/05_environment_variables.mdx new file mode 100644 index 0000000000..82373d82c2 --- /dev/null +++ b/docs/hpc/14_tutorial_intro_hpc/05_environment_variables.mdx @@ -0,0 +1,161 @@ +# Environment Variables +:::info[Overview] +Questions +- How are variables set and accessed in the Unix shell? +- How can I use variables to change how a program runs? + +Objectives +- Understand how variables are implemented in the shell +- Read the value of an existing variable +- Create new variables and change their values +- Change the behaviour of a program using an environment variable +- Explain how the shell uses the PATH variable to search for executables +::: + +:::info[Provenance] +This episode has been remixed from the [Shell Extras episode on Shell Variables](https://github.com/carpentries-incubator/shell-extras/blob/gh-pages/_episodes/08-environment-variables.md) and the [HPC Shell episode on scripts](https://github.com/hpc-carpentry/hpc-shell/blob/gh-pages/_episodes/05-scripts.md) +::: + +The shell is just a program, and like other programs, it has variables. Those variables control its execution, so by changing their values you can change how the shell behaves (and with a little more effort how other programs behave). + +Variables are a great way of saving information under a name you can access later. In programming languages like Python and R, variables can store pretty much anything you can think of. In the shell, they usually just store text. The best way to understand how they work is to see them in action. + +Let’s start by running the command `set` and looking at some of the variables in a typical shell session: +```bash +$ set +SHELL=/bin/bash +WINDOWID=87 +SINGULARITY_CACHEDIR=/state/partition1/NYUNetID-singularity-cache +COLORTERM=truecolor +HISTCONTROL=ignoredups +HISTSIZE=1000 +HOSTNAME=log-3 +FPATH=/usr/share/zsh/5.5.1/functions:/usr/share/zsh/5.5.1/functions:/share/apps/lmod/8.4.9/lmod/lmod/init/ksh_funcs +SSH_AUTH_SOCK=/tmp/ssh-XXXXb32gan/agent.3025643 +__LMOD_REF_COUNT_MODULEPATH=/share/apps/modulefiles:1 +VAST=/vast/NYUNetID +LMOD_DIR=/share/apps/lmod/8.4.9/lmod/lmod/libexec +... +``` +As you can see, there are quite a few — in fact, four or five times more than what’s shown here. And yes, using `set` to *show* things might seem a little strange, even for Unix, but if you don’t give it any arguments, it might as well show you things you *could* set. + +Every variable has a name. All shell variables’ values are strings, even those (like `UID`) that look like numbers. It’s up to programs to convert these strings to other types when necessary. For example, if a program wanted to find out how many processors the computer had, it would convert the value of the `NUMBER_OF_PROCESSORS` variable from a string to an integer. + +## Showing the Value of a Variable +Let’s show the value of the variable HOME: +```bash +$ echo HOME +HOME +``` +That just prints “HOME”, which isn’t what we wanted (though it is what we actually asked for). Let’s try this instead: +```bash +$ echo $HOME +/home/NetID +``` +The dollar sign tells the shell that we want the `value` of the variable rather than its name. This works just like wildcards: the shell does the replacement *before* running the program we’ve asked for. Thanks to this expansion, what we actually run is `echo /home/NetID`, which displays the right thing. + +## Creating and Changing Variables +Creating a variable is easy — we just assign a value to a name using “=” (we just have to remember that the syntax requires that there are *no* spaces around the `=`!): +```bash +$ SECRET_IDENTITY=Dracula +$ echo $SECRET_IDENTITY +Dracula +``` +To change the value, just assign a new one: +```bash +$ SECRET_IDENTITY=Camilla +$ echo $SECRET_IDENTITY +Camilla +``` + +## Environment variables +When we ran the `printenv` command we saw there were a lot of variables whose names were in upper case. That’s because, by convention, variables that are also available to use by *other* programs are given upper-case names. Such variables are called *environment variables* as they are shell variables that are defined for the current shell and are inherited by any child shells or processes. + +To create an environment variable you need to `export` a shell variable. For example, to make our `SECRET_IDENTITY` available to other programs that we call from our shell we can do: +```bash +$ SECRET_IDENTITY=Camilla +$ export SECRET_IDENTITY +``` +You can also create and export the variable in a single step: +```bash +$ export SECRET_IDENTITY=Camilla +``` + +
+ +:::info[Using environment variables to change program behaviour] +Set a shell variable `TIME_STYLE` to have a value of `iso` and check this value using the `echo` command. + +Now, run the command `ls` with the option `-l` (which gives a long format). + +`export` the variable and rerun the `ls -l` command. Do you notice any difference?
+**[Click for Solution]** +::: + +:::tip[Solution] +The `TIME_STYLE` variable is not *seen* by `ls` until is exported, at which point it is used by `ls` to decide what date format to use when presenting the timestamp of files. +::: +
+ +You can see the complete set of environment variables in your current shell session with the command `env` (which returns a subset of what the command `set` gave us). The complete set of environment variables is called your *runtime environment* and can affect the behaviour of the programs you run. + +
+ +:::info[Job environment variables] +When Slurm runs a job, it sets a number of environment variables for the job. One of these will let us check what directory our job script was submitted from. The `SLURM_SUBMIT_DIR` variable is set to the directory from which our job was submitted. Using the `SLURM_SUBMIT_DIR` variable, modify your job so that it prints out the location from which the job was submitted.
+**[Click for Solution]** +::: + +:::tip[Solution] +```bash +[NetID@log-1 ~]$ nano example-job.sh +[NetID@log-1 ~]$ cat example-job.sh +#!/bin/bash +#SBATCH -t 00:00:30 + +echo -n "This script is running on " +hostname + +echo "This job was launched in the following directory:" +echo ${SLURM_SUBMIT_DIR} +::: +
+ +To remove a variable or environment variable you can use the `unset` command, for example: +```bash +$ unset SECRET_IDENTITY +``` + +## The `PATH` Environment Variable +Similarly, some environment variables (like `PATH`) store lists of values. In this case, the convention is to use a colon ‘:’ as a separator. If a program wants the individual elements of such a list, it’s the program’s responsibility to split the variable’s string value into pieces. + +Let’s have a closer look at that `PATH` variable. Its value defines the shell’s search path for executables, i.e., the list of directories that the shell looks in for runnable programs when you type in a program name without specifying what directory it is in. + +For example, when we type a command like `analyze`, the shell needs to decide whether to run `./analyze` or `/bin/analyze`. The rule it uses is simple: the shell checks each directory in the `PATH` variable in turn, looking for a program with the requested name in that directory. As soon as it finds a match, it stops searching and runs the program. + +To show how this works, here are the components of `PATH` listed one per line: +```bash +/share/apps/singularity/bin +/share/apps/local/bin +/home/NetID/.local/bin +/home/NetID/bin +/share/apps/singularity/bin +/share/apps/local/bin +/usr/local/bin +/usr/bin +/usr/local/sbin +/usr/sbin +/usr/lpp/mmfs/bin +/opt/slurm/bin +``` +On our computer, there are actually three programs called `analyze` in three different directories: `/bin/analyze`, `/usr/local/bin/analyze`, and `/users/NetID/analyze`. Since the shell searches the directories in the order they’re listed in `PATH`, it finds `/bin/analyze` first and runs that. Notice that it will never find the program `/users/NetID/analyze` unless we type in the full path to the program, since the directory `/users/NetID` isn’t in `PATH`. + +This means that I can have executables in lots of different places as long as I remember that I need to update my `PATH` so that my shell can find them. + +What if I want to run two different versions of the same program? Since they share the same name, if I add them both to my PATH the first one found will always win. In the next episode we’ll learn how to use helper tools to help us manage our runtime environment to make that possible without us needing to do a lot of bookkeeping on what the value of `PATH` (and other important environment variables) is or should be. + +:::tip[Key Points] +- Shell variables are by default treated as strings +- Variables are assigned using `=` and recalled using the variable’s name prefixed by `$` +- Use `export` to make an variable available to other programs +- The `PATH` variable defines the shell’s search path From c4dc9bf31a4869068d6bd83c903c1ea651656ba1 Mon Sep 17 00:00:00 2001 From: Robert Young Date: Thu, 5 Jun 2025 12:14:01 -0400 Subject: [PATCH 06/11] first pass on modules tutorial --- docs/hpc/14_tutorial_intro_hpc/06_modules.mdx | 217 ++++++++++++++++++ 1 file changed, 217 insertions(+) create mode 100644 docs/hpc/14_tutorial_intro_hpc/06_modules.mdx diff --git a/docs/hpc/14_tutorial_intro_hpc/06_modules.mdx b/docs/hpc/14_tutorial_intro_hpc/06_modules.mdx new file mode 100644 index 0000000000..39f950559f --- /dev/null +++ b/docs/hpc/14_tutorial_intro_hpc/06_modules.mdx @@ -0,0 +1,217 @@ +# Accessing software via Modules +:::info[Overview] +Questions +- How do we load and unload software packages? + +Objectives +- Load and use a software package. +- Explain how the shell environment changes when the module mechanism loads or unloads packages. +::: + +On a high-performance computing system, it is seldom the case that the software we want to use is available when we log in. It is installed, but we will need to “load” it before it can run. + +Before we start using individual software packages, however, we should understand the reasoning behind this approach. The three biggest factors are: +- software incompatibilities +- versioning +- dependencies + +Software incompatibility is a major headache for programmers. Sometimes the presence (or absence) of a software package will break others that depend on it. Two well known examples are Python and C compiler versions. Python 3 famously provides a `python` command that conflicts with that provided by Python 2. Software compiled against a newer version of the C libraries and then run on a machine that has older C libraries installed will result in a nasty `'GLIBCXX_3.4.20' not found` error. + +Software versioning is another common issue. A team might depend on a certain package version for their research project - if the software version was to change (for instance, if a package was updated), it might affect their results. Having access to multiple software versions allows a set of researchers to prevent software versioning issues from affecting their results. + +Dependencies are where a particular software package (or even a particular version) depends on having access to another software package (or even a particular version of another software package). For example, the VASP materials science software may depend on having a particular version of the FFTW (Fastest Fourier Transform in the West) software library available for it to work. + +## Environment Modules +Environment modules are the solution to these problems. A *module* is a self-contained description of a software package – it contains the settings required to run a software package and, usually, encodes required dependencies on other software packages. + +There are a number of different environment module implementations commonly used on HPC systems: the two most common are *TCL modules* and *Lmod*. Both of these use similar syntax and the concepts are the same so learning to use one will allow you to use whichever is installed on the system you are using. In both implementations the `module` command is used to interact with environment modules. An additional subcommand is usually added to the command to specify what you want to do. For a list of subcommands you can use `module -h` or `module help`. As for all commands, you can access the full help on the *man* pages with `man module`. + +On login you may start out with a default set of modules loaded or you may start out with an empty environment; this depends on the setup of the system you are using. + +# Listing Available Modules +To see available software modules, use `module avail`: +```bash +[NetID@log-1 ~]$ module avail + +----------------------------------------------- /share/apps/modulefiles ----------------------------------------------- + abyss/intel/2.3.0 liftoff/1.6.1 + admixtools/intel/7.0.2 liftoff/1.6.3 + admixture/1.3.0 llvm/11.0.0 + advanpix-mct/4.9.3.15018 lofreq/2.1.5 + amber/openmpi/24.00 lp_solve/intel/5.5.2.9 + amber/openmpi/intel/20.06 lua/5.3.6 + amber/openmpi/intel/20.11 macs2/2.1.1.20160309 + amber/openmpi/intel/22.00 macs2/intel/2.2.7.1 + amber/openmpi/intel/22.03 macs3/intel/3.0.0a5 + ampl/20240308 mafft/intel/7.475 + amplgurobilink/11.0.1 mambaforge/23.1.0 + +[removed most of the output here for clarity] +``` + +## Listing Currently Loaded Modules +You can use the `module list` command to see which modules you currently have loaded in your environment. If you have no modules loaded, you will see a message telling you so +```bash +[NetID@log-1 ~]$ module list +No modules loaded +``` + +## Loading and Unloading Software +To load a software module, use `module load`. In this example we will use R. + +Initially, R is not loaded. We can test this by using the `which` command. `which` looks for programs the same way that Bash does, so we can use it to tell us where a particular piece of software is stored. +```bash +[NetID@log-1 ~]$ which R +/usr/bin/which: no R in +(share/apps/singularity/bin: +/share/apps/local/bin: +/home/yourUsername/.local/bin: +/home/yourUsername/bin: +/share/apps/singularity/bin: +/share/apps/local/bin: +/usr/local/bin: +/usr/bin: +/usr/local/sbin: +/usr/sbin: +/usr/lpp/mmfs/bin: +/opt/slurm/bin:) +``` +We can load the `R` command with `module load`: +```bash +[NetID@log-1 ~]$ module load r/gcc/4.4.0 +[NetID@log-1 ~]$ which R +/share/apps/r/4.4.0/gcc/bin/R +``` + +So, what just happened? + +To understand the output, first we need to understand the nature of the `$PATH` environment variable. `$PATH` is a special environment variable that controls where a UNIX system looks for software. Specifically `$PATH` is a list of directories (separated by :) that the OS searches through for a command before giving up and telling us it can’t find it. As with all environment variables we can print it out using `echo`. +```bash +[NetID@log-1 ~]$ echo $PATH +/share/apps/r/4.4.0/gcc/bin: +/share/apps/singularity/bin: +/share/apps/local/bin: +/home/yourUsername/.local/bin: +/home/yourUsername/bin: +/usr/local/bin: +/usr/bin: +/usr/local/sbin: +/usr/sbin: +/usr/lpp/mmfs/bin: +/opt/slurm/bin: +``` +You’ll notice a similarity to the output of the `which` command. In this case, there’s only one difference: the different directory at the beginning. When we ran the `module load` command, it added a directory to the beginning of our `$PATH`. Let’s examine what’s there: + +```bash +[NetID@log-1 ~]$ ls /share/apps/r/4.4.0/gcc/bin +R Rscript __run_base.bash +``` + +Taking this to its conclusion, `module load` will add software to your `$PATH`. It “loads” software. A special note on this - depending on which version of the `module` program that is installed at your site, `module load` will also load required software dependencies. + +Let's see an example of this when loading the module `lammps/openmpi/intel/20231214`: + +To start, let’s use module list. module list shows all loaded software modules. +```bash +[NetID@log-1 ~]$ module list +No modules loaded +[NetID@log-1 ~]$ module load lammps/openmpi/intel/20231214 +[NetID@log-1 ~]$ module list + +Currently Loaded Modules: + 1) szip/intel/2.1.1 5) gsl/intel/2.6 9) python/intel/3.8.6 + 2) hdf5/intel/1.12.0 6) openmpi/intel/4.0.5 10) boost/intel/1.74.0 + 3) netcdf-c/intel/4.7.4 7) fftw/openmpi/intel/3.3.9 11) plumed/openmpi/intel/2.8.3 + 4) pnetcdf/openmpi/intel/1.12.1 8) intel/19.1.2 12) lammps/openmpi/intel/20231214 +``` + +So in this case, loading the `lammps` module (a molecular dynamics software package), also loaded 11 other modules as well. Let’s try unloading the `lammps` package. +```bash +[NetID@log-1 ~]$ module unload lammps/openmpi/intel/20231214 +[NetID@log-1 ~]$ module list +No modules loaded +``` +So using `module unload` “un-loads” a module along with its dependencies. If we wanted to unload everything at once, we could run `module purge` (unloads everything). +```bash +[NetID@log-1 ~]$ module purge +No modules loaded +``` + +:::note +This module loading process happens principally through the manipulation of environment variables like `$PATH`. There is usually little or no data transfer involved. +::: + +The module loading process manipulates other special environment variables as well, including variables that influence where the system looks for software libraries, and sometimes variables which tell commercial software packages where to find license servers. + +The module command also restores these shell environment variables to their previous state when a module is unloaded. + +## Software Versioning +So far, we’ve learned how to load and unload software packages. This is very useful. However, we have not yet addressed the issue of software versioning. At some point or other, you will run into issues where only one particular version of some software will be suitable. Perhaps a key bugfix only happened in a certain version, or version X broke compatibility with a file format you use. In either of these example cases, it helps to be very specific about what software is loaded. + +Let’s examine the output of `module avail` more closely. +```bash +[NetID@log-1 ~]$ module avail + +----------------------------------------------- /share/apps/modulefiles ----------------------------------------------- + abyss/intel/2.3.0 liftoff/1.6.1 + admixtools/intel/7.0.2 liftoff/1.6.3 + admixture/1.3.0 llvm/11.0.0 + advanpix-mct/4.9.3.15018 lofreq/2.1.5 + amber/openmpi/24.00 lp_solve/intel/5.5.2.9 + amber/openmpi/intel/20.06 lua/5.3.6 + amber/openmpi/intel/20.11 macs2/2.1.1.20160309 + amber/openmpi/intel/22.00 macs2/intel/2.2.7.1 + amber/openmpi/intel/22.03 macs3/intel/3.0.0a5 + ampl/20240308 mafft/intel/7.475 + amplgurobilink/11.0.1 mambaforge/23.1.0 + +[removed most of the output here for clarity] +``` +Let’s take a closer look at the `nextflow` module. Nextflow is a scientific workflow system predominantly used for bioinformatic data analysis In this case, we have: +```bash + nextflow/20.07.1 nextflow/20.11.0-edge nextflow/21.10.5 nextflow/23.04.1 + nextflow/20.10.0 nextflow/21.04.3 nextflow/21.10.6 nextflow/24.04.3 +``` +How do we load each copy and which copy is the default? +```bash +[NetID@log-1 ~]$ module load Nextflow +Lmod has detected the following error: The following module(s) are unknown: "Nextflow" + +Please check the spelling or version number. Also try "module spider ..." +It is also possible your cache file is out-of-date; it may help to try: + $ module --ignore-cache load "Nextflow" + +Also make sure that all modulefiles written in TCL start with the string #%Module +``` + +To load a software module we must specify the full module name: +```bash +[NYUNetID@log-1 ~]$ module load nextflow/24.04.3 +[NYUNetID@log-1 ~]$ nextflow -version +``` + +
+ +:::info[Using Software Modules in Scripts] +Create a job that is able to run `R --version`. Remember, no software is loaded by default! Running a job is just like logging on to the system (you should not assume a module loaded on the login node is loaded on a compute node).
+**[Click for Solution]** +::: + +:::tip[Solution] +```bash +[NetID@log-1 ~]$ nano r-module.sh +[NetID@log-1 ~]$ cat r-module.sh +#!/bin/bash +#SBATCH #SBATCH -t 00:00:30 + +module load r/gcc/4.4.0 + +R --version +[NetID@log-1 ~]$ sbatch r-module.sh +::: +
+ +:::tip[Key Points] +- Load software with `module load softwareName`. +- Unload software with `module unload` +- The module system handles software versioning and package conflicts for you automatically. From 4b48b40073a32ddf6247878fb66aed6dbd472659 Mon Sep 17 00:00:00 2001 From: Robert Young Date: Fri, 6 Jun 2025 10:39:46 -0400 Subject: [PATCH 07/11] first pass on file transfer tutorial --- .../07_transferring_files_remote.mdx | 251 ++++++++++++++++++ 1 file changed, 251 insertions(+) create mode 100644 docs/hpc/14_tutorial_intro_hpc/07_transferring_files_remote.mdx diff --git a/docs/hpc/14_tutorial_intro_hpc/07_transferring_files_remote.mdx b/docs/hpc/14_tutorial_intro_hpc/07_transferring_files_remote.mdx new file mode 100644 index 0000000000..a39a319bfc --- /dev/null +++ b/docs/hpc/14_tutorial_intro_hpc/07_transferring_files_remote.mdx @@ -0,0 +1,251 @@ +# Transferring files with remote computers +:::info[Overview] +Questions +- How do I transfer files to (and from) the cluster? + +Objectives +- Transfer files to and from a computing cluster. +::: + +Performing work on a remote computer is not very useful if we cannot get files to or from the cluster. There are several options for transferring data between computing resources using CLI and GUI utilities, a few of which we will cover. + +# Download Lesson Files From the Internet +One of the most straightforward ways to download files is to use either `curl` or `wget`. One of these is usually installed in most Linux shells, on Mac OS terminal and in GitBash. Any file that can be downloaded in your web browser through a direct link can be downloaded using curl or wget. This is a quick way to download datasets or source code. The syntax for these commands is: + +- `wget [-O new_name] https://some/link/to/a/file` +- `curl [-o new_name] https://some/link/to/a/file` + +Try it out by downloading some material we’ll use later on, from a terminal on your local machine, using the URL of the current codebase: + +[https://github.com/hpc-carpentry/amdahl/tarball/main](https://github.com/hpc-carpentry/amdahl/tarball/main) + +
+ +:::info[Download the “Tarball”] +The word “tarball” in the above URL refers to a compressed archive format commonly used on Linux, which is the operating system the majority of HPC cluster machines run. A tarball is a lot like a `.zip` file. The actual file extension is `.tar.gz`, which reflects the two-stage process used to create the file: the files or folders are merged into a single file using `tar`, which is then compressed using `gzip`, so the file extension is “tar-dot-g-z.” That’s a mouthful, so people often say “the xyz tarball” instead. + +You may also see the extension `.tgz`, which is just an abbreviation of `.tar.gz`. + +By default, `curl` and `wget` download files to the same name as the URL: in this case, `main`. Use one of the above commands to save the tarball as `amdahl.tar.gz`.
+**[Click for Solution]** +::: + +:::tip[wget and curl Commands] +```bash +[user@laptop ~]$ wget -O amdahl.tar.gz https://github.com/hpc-carpentry/amdahl/tarball/main +# or +[user@laptop ~]$ curl -o amdahl.tar.gz -L https://github.com/hpc-carpentry/amdahl/tarball/main +``` +The `-L` option to `curl` tells it to follow URL redirects (which `wget` does by default). +::: +
+ +After downloading the file, use `ls` to see it in your working directory: +```bash +[user@laptop ~]$ ls +``` + +## Archiving Files +One of the biggest challenges we often face when transferring data between remote HPC systems is that of large numbers of files. There is an overhead to transferring each individual file and when we are transferring large numbers of files these overheads combine to slow down our transfers to a large degree. + +The solution to this problem is to *archive* multiple files into smaller numbers of larger files before we transfer the data to improve our transfer efficiency. Sometimes we will combine archiving with *compression* to reduce the amount of data we have to transfer and so speed up the transfer. The most common archiving command you will use on a (Linux) HPC cluster is `tar`. + +`tar` can be used to combine files and folders into a single archive file and, optionally, compress the result. Let’s look at the file we downloaded from the lesson site, `amdahl.tar.gz`. + +The `.gz` part stands for `gzip`, which is a compression library. It’s common (but not necessary!) that this kind of file can be interpreted by reading its name: it appears somebody took files and folders relating to something called “amdahl,” wrapped them all up into a single file with `tar`, then compressed that archive with `gzip` to save space. + +Let’s see if that is the case, *without* unpacking the file. `tar` prints the “table of contents” with the `-t` flag, for the file specified with the `-f` flag followed by the filename. Note that you can concatenate the two flags: writing `-t -f` is interchangeable with writing `-tf` together. However, the argument following `-f` must be a filename, so writing `-ft` will *not* work. +```bash +[user@laptop ~]$ tar -tf amdahl.tar.gz +hpc-carpentry-amdahl-46c9b4b/ +hpc-carpentry-amdahl-46c9b4b/.github/ +hpc-carpentry-amdahl-46c9b4b/.github/workflows/ +hpc-carpentry-amdahl-46c9b4b/.github/workflows/python-publish.yml +hpc-carpentry-amdahl-46c9b4b/.gitignore +hpc-carpentry-amdahl-46c9b4b/LICENSE +hpc-carpentry-amdahl-46c9b4b/README.md +hpc-carpentry-amdahl-46c9b4b/amdahl/ +hpc-carpentry-amdahl-46c9b4b/amdahl/__init__.py +hpc-carpentry-amdahl-46c9b4b/amdahl/__main__.py +hpc-carpentry-amdahl-46c9b4b/amdahl/amdahl.py +hpc-carpentry-amdahl-46c9b4b/requirements.txt +hpc-carpentry-amdahl-46c9b4b/setup.py +``` + +This example output shows a folder which contains a few files, where `46c9b4b` is an 8-character [git](https://swcarpentry.github.io/git-novice/) commit hash that will change when the source material is updated. + +Now let’s unpack the archive. We’ll run `tar` with a few common flags: + +- `-x` to extract the archive +- `-v` for verbose output +- `-z` for gzip compression +- `-f «tarball»` for the file to be unpacked + +:::tip[Extract the Archive] +Using the flags above, unpack the source code tarball into a new directory named “amdahl” using `tar`. +```bash +[user@laptop ~]$ tar -xvzf amdahl.tar.gz +hpc-carpentry-amdahl-46c9b4b/ +hpc-carpentry-amdahl-46c9b4b/.github/ +hpc-carpentry-amdahl-46c9b4b/.github/workflows/ +hpc-carpentry-amdahl-46c9b4b/.github/workflows/python-publish.yml +hpc-carpentry-amdahl-46c9b4b/.gitignore +hpc-carpentry-amdahl-46c9b4b/LICENSE +hpc-carpentry-amdahl-46c9b4b/README.md +hpc-carpentry-amdahl-46c9b4b/amdahl/ +hpc-carpentry-amdahl-46c9b4b/amdahl/__init__.py +hpc-carpentry-amdahl-46c9b4b/amdahl/__main__.py +hpc-carpentry-amdahl-46c9b4b/amdahl/amdahl.py +hpc-carpentry-amdahl-46c9b4b/requirements.txt +hpc-carpentry-amdahl-46c9b4b/setup.py +``` +Note that we did not need to type out `-x -v -z -f`, thanks to flag concatenation, though the command works identically either way – so long as the concatenated list ends with `f`, because the next string must specify the name of the file to extract. +::: + +The folder has an unfortunate name, so let’s change that to something more convenient. +```bash +[user@laptop ~]$ mv hpc-carpentry-amdahl-46c9b4b amdahl +``` +Check the size of the extracted directory and compare to the compressed file size, using du for “disk usage”. +```bash +[user@laptop ~]$ du -sh amdahl.tar.gz +8.0K amdahl.tar.gz +[user@laptop ~]$ du -sh amdahl +48K amdahl +``` +Text files (including Python source code) compress nicely: the “tarball” is one-sixth the total size of the raw data! + +If you want to reverse the process – compressing raw data instead of extracting it – set a `c` flag instead of `x`, set the archive filename, then provide a directory to compress: +```bash +[user@laptop ~]$ tar -cvzf compressed_code.tar.gz amdahl +amdahl/ +amdahl/.github/ +amdahl/.github/workflows/ +amdahl/.github/workflows/python-publish.yml +amdahl/.gitignore +amdahl/LICENSE +amdahl/README.md +amdahl/amdahl/ +amdahl/amdahl/__init__.py +amdahl/amdahl/__main__.py +amdahl/amdahl/amdahl.py +amdahl/requirements.txt +amdahl/setup.py +``` +If you give `amdahl.tar.gz` as the filename in the above command, `tar` will update the existing tarball with any changes you made to the files. That would mean adding the new `amdahl` folder to the *existing* folder (`hpc-carpentry-amdahl-46c9b4b`) inside the tarball, doubling the size of the archive! + +:::tip[Working with Windows] +When you transfer text files from a Windows system to a Unix system (Mac, Linux, BSD, Solaris, etc.) this can cause problems. Windows encodes its files slightly different than Unix, and adds an extra character to every line. + +On a Unix system, every line in a file ends with a `\n` (newline). On Windows, every line in a file ends with a `\r\n` (carriage return + newline). This causes problems sometimes. + +Though most modern programming languages and software handles this correctly, in some rare instances, you may run into an issue. The solution is to convert a file from Windows to Unix encoding with the `dos2unix` command. + +You can identify if a file has Windows line endings with `cat -A filename`. A file with Windows line endings will have `^M$` at the end of every line. A file with Unix line endings will have `$` at the end of a line. + +To convert the file, just run `dos2unix filename`. (Conversely, to convert back to Windows format, you can run `unix2dos filename`.) +::: + +## Transferring Single Files and Folders With `scp` +To copy a single file to or from the cluster, we can use `scp` (“secure copy”). The syntax can be a little complex for new users, but we’ll break it down. The `scp` command is a relative of the `ssh` command we used to access the system, and can use the same public-key authentication mechanism. + +To *upload* to another computer, the template command is +```bash +[user@laptop ~]$ scp local_file NetID@greene.hpc.nyu.edu:remote_destination +``` +in which `@` and `:` are field separators and `remote_destination` is a path relative to your remote home directory, or a new filename if you wish to change it, or both a relative path *and* a new filename. If you don’t have a specific folder in mind you can omit the `remote_destination` and the file will be copied to your home directory on the remote computer (with its original name). If you include a `remote_destination`, note that `scp` interprets this the same way `cp` does when making local copies: if it exists and is a folder, the file is copied inside the folder; if it exists and is a file, the file is overwritten with the contents of `local_file`; if it does not exist, it is assumed to be a destination filename for `local_file`. + +Upload the lesson material to your remote home directory like so: +```bash +[user@laptop ~]$ scp amdahl.tar.gz NYUNetID@greene.hpc.nyu.edu: +``` + +## Transferring a Directory +To transfer an entire directory, we add the `-r` flag for “recursive”: copy the item specified, and every item below it, and every item below those… until it reaches the bottom of the directory tree rooted at the folder name you provided. +```bash +[user@laptop ~]$ scp -r amdahl NYUNetID@greene.hpc.nyu.edu: +``` +:::warning[Caution] +For a large directory – either in size or number of files – copying with `-r` can take a long time to complete. +::: + +When using `scp`, you may have noticed that a `:` always follows the remote computer name. A string *after* the `:` specifies the remote directory you wish to transfer the file or folder to, including a new name if you wish to rename the remote material. If you leave this field blank, `scp` defaults to your home directory and the name of the local material to be transferred. + +On Linux computers, `/` is the separator in file or directory paths. A path starting with a `/` is called *absolute*, since there can be nothing above the root `/`. A path that does not start with `/` is called *relative*, since it is not anchored to the root. + +If you want to upload a file to a location inside your home directory – which is often the case – then you don’t need a *leading* `/`. After the `:`, you can type the destination path relative to your home directory. If your home directory *is* the destination, you can leave the destination field blank, or type `~` – the shorthand for your home directory – for completeness. + +With `scp`, a trailing slash on the target directory is optional, and has no effect. A trailing slash on a source directory is important for other commands, like `rsync`. + +:::tip[A Note on rsync] +As you gain experience with transferring files, you may find the `scp` command limiting. The [rsync](https://rsync.samba.org/) utility provides advanced features for file transfer and is typically faster compared to both `scp` and `sftp` (see below). It is especially useful for transferring large and/or many files and for synchronizing folder contents between computers. + +The syntax is similar to `scp`. To transfer to another computer with commonly used options: +```bash +[user@laptop ~]$ rsync -avP amdahl.tar.gz NYUNetID@greene.hpc.nyu.edu: +``` +The options are: +- `-a` (archive) to preserve file timestamps, permissions, and folders, among other things; implies recursion +- `-v` (verbose) to get verbose output to help monitor the transfer +- `-P` (partial/progress) to preserve partially transferred files in case of an interruption and also displays the progress of the transfer. + +To recursively copy a directory, we can use the same options: +```bash +[user@laptop ~]$ rsync -avP amdahl NYUNetID@greene.hpc.nyu.edu:~/ +``` +As written, this will place the local directory and its contents under your home directory on the remote system. If a trailing slash is added to the source, a new directory corresponding to the transferred directory will not be created, and the contents of the source directory will be copied directly into the destination directory. + +To download a file, we simply change the source and destination: +```bash +[user@laptop ~]$ rsync -avP NYUNetID@greene.hpc.nyu.edu:amdahl ./ +``` +::: + +File transfers using both `scp` and `rsync` use SSH to encrypt data sent through the network. So, if you can connect via SSH, you will be able to transfer files. By default, SSH uses network port 22. If a custom SSH port is in use, you will have to specify it using the appropriate flag, often `-p`, `-P`, or `--port`. Check `--help` or the `man` page if you’re unsure. + +
+ +:::info[Change the Rsync Port] +Say we have to connect `rsync` through port 768 instead of 22. How would we modify this command? +```bash +[user@laptop ~]$ rsync amdahl.tar.gz NYUNetID@greene.hpc.nyu.edu: +``` +Hint: check the `man` page or “help” for `rsync`.
+**[Click for Solution]** +::: + +::::tip[Solution] +```bash +[user@laptop ~]$ man rsync +[user@laptop ~]$ rsync --help | grep port + --port=PORT specify double-colon alternate port number +See http://rsync.samba.org/ for updates, bug reports, and answers +[user@laptop ~]$ rsync --port=768 amdahl.tar.gz NYUNetID@greene.hpc.nyu.edu: +``` +:::note +This command will fail, as the correct port in this case is the default: 22. +::: +:::: +
+ +## Transferring Files Interactively with FileZilla +FileZilla is a cross-platform client for downloading and uploading files to and from a remote computer. It is absolutely fool-proof and always works quite well. It uses the `sftp` protocol. You can read more about using the sftp protocol in the command line in the [lesson discussion](https://nyuhpc.github.io/hpc-intro/discuss/index.html). + +Download and install the FileZilla client from [https://filezilla-project.org](https://filezilla-project.org). After installing and opening the program, you should end up with a window with a file browser of your local system on the left hand side of the screen. When you connect to the cluster, your cluster files will appear on the right hand side. + +To connect to the cluster, we’ll just need to enter our credentials at the top of the screen: + +- Host: `sftp://greene.hpc.nyu.edu` +- User: Your NetID +- Password: Your NetID password +- Port: (leave blank to use the default port) + +Hit “Quickconnect” to connect. You should see your remote files appear on the right hand side of the screen. You can drag-and-drop files between the left (local) and right (remote) sides of the screen to transfer files. + +Finally, if you need to move large files (typically larger than a gigabyte) from one remote computer to another remote computer, SSH in to the computer hosting the files and use `scp` or `rsync` to transfer over to the other. This will be more efficient than using FileZilla (or related applications) that would copy from the source to your local machine, then to the destination machine. + +:::tip[Key Points] +- `wget` and `curl -O` download a file from the internet. +- `scp` and `rsync` transfer files to and from your computer. +- You can use an SFTP client like FileZilla to transfer files through a GUI. +::: From 0f31bc12281235383a719e11303354710c7672a3 Mon Sep 17 00:00:00 2001 From: Robert Young Date: Fri, 6 Jun 2025 12:44:09 -0400 Subject: [PATCH 08/11] first pass on running parallel job tutorial --- .../07_transferring_files_remote.mdx | 14 +- .../08_running_parallel_job.mdx | 304 ++++++++++++++++++ 2 files changed, 311 insertions(+), 7 deletions(-) create mode 100644 docs/hpc/14_tutorial_intro_hpc/08_running_parallel_job.mdx diff --git a/docs/hpc/14_tutorial_intro_hpc/07_transferring_files_remote.mdx b/docs/hpc/14_tutorial_intro_hpc/07_transferring_files_remote.mdx index a39a319bfc..1c14f8d590 100644 --- a/docs/hpc/14_tutorial_intro_hpc/07_transferring_files_remote.mdx +++ b/docs/hpc/14_tutorial_intro_hpc/07_transferring_files_remote.mdx @@ -157,13 +157,13 @@ in which `@` and `:` are field separators and `remote_destination` is a path rel Upload the lesson material to your remote home directory like so: ```bash -[user@laptop ~]$ scp amdahl.tar.gz NYUNetID@greene.hpc.nyu.edu: +[user@laptop ~]$ scp amdahl.tar.gz NetID@greene.hpc.nyu.edu: ``` ## Transferring a Directory To transfer an entire directory, we add the `-r` flag for “recursive”: copy the item specified, and every item below it, and every item below those… until it reaches the bottom of the directory tree rooted at the folder name you provided. ```bash -[user@laptop ~]$ scp -r amdahl NYUNetID@greene.hpc.nyu.edu: +[user@laptop ~]$ scp -r amdahl NetID@greene.hpc.nyu.edu: ``` :::warning[Caution] For a large directory – either in size or number of files – copying with `-r` can take a long time to complete. @@ -182,7 +182,7 @@ As you gain experience with transferring files, you may find the `scp` command l The syntax is similar to `scp`. To transfer to another computer with commonly used options: ```bash -[user@laptop ~]$ rsync -avP amdahl.tar.gz NYUNetID@greene.hpc.nyu.edu: +[user@laptop ~]$ rsync -avP amdahl.tar.gz NetID@greene.hpc.nyu.edu: ``` The options are: - `-a` (archive) to preserve file timestamps, permissions, and folders, among other things; implies recursion @@ -191,13 +191,13 @@ The options are: To recursively copy a directory, we can use the same options: ```bash -[user@laptop ~]$ rsync -avP amdahl NYUNetID@greene.hpc.nyu.edu:~/ +[user@laptop ~]$ rsync -avP amdahl NetID@greene.hpc.nyu.edu:~/ ``` As written, this will place the local directory and its contents under your home directory on the remote system. If a trailing slash is added to the source, a new directory corresponding to the transferred directory will not be created, and the contents of the source directory will be copied directly into the destination directory. To download a file, we simply change the source and destination: ```bash -[user@laptop ~]$ rsync -avP NYUNetID@greene.hpc.nyu.edu:amdahl ./ +[user@laptop ~]$ rsync -avP NetID@greene.hpc.nyu.edu:amdahl ./ ``` ::: @@ -208,7 +208,7 @@ File transfers using both `scp` and `rsync` use SSH to encrypt data sent through :::info[Change the Rsync Port] Say we have to connect `rsync` through port 768 instead of 22. How would we modify this command? ```bash -[user@laptop ~]$ rsync amdahl.tar.gz NYUNetID@greene.hpc.nyu.edu: +[user@laptop ~]$ rsync amdahl.tar.gz NetID@greene.hpc.nyu.edu: ``` Hint: check the `man` page or “help” for `rsync`.
**[Click for Solution]** @@ -220,7 +220,7 @@ Hint: check the `man` page or “help” for `rsync`.
[user@laptop ~]$ rsync --help | grep port --port=PORT specify double-colon alternate port number See http://rsync.samba.org/ for updates, bug reports, and answers -[user@laptop ~]$ rsync --port=768 amdahl.tar.gz NYUNetID@greene.hpc.nyu.edu: +[user@laptop ~]$ rsync --port=768 amdahl.tar.gz NetID@greene.hpc.nyu.edu: ``` :::note This command will fail, as the correct port in this case is the default: 22. diff --git a/docs/hpc/14_tutorial_intro_hpc/08_running_parallel_job.mdx b/docs/hpc/14_tutorial_intro_hpc/08_running_parallel_job.mdx new file mode 100644 index 0000000000..1e47e305de --- /dev/null +++ b/docs/hpc/14_tutorial_intro_hpc/08_running_parallel_job.mdx @@ -0,0 +1,304 @@ +# Running a parallel job +:::info[Overview] +Questions +- How do we execute a task in parallel? +- What benefits arise from parallel execution? +- What are the limits of gains from execution in parallel? + +Objectives +- Install a Python package using `pip` +- Prepare a job submission script for the parallel executable. +- Launch jobs with parallel execution. +- Record and summarize the timing and accuracy of jobs. +- Describe the relationship between job parallelism and performance. +::: + +We now have the tools we need to run a multi-processor job. This is a very important aspect of HPC systems, as parallelism is one of the primary tools we have to improve the performance of computational tasks. + +If you disconnected, log back in to the cluster. +```bash +[user@laptop ~]$ ssh NetID@greene.hpc.nyu.edu +``` + +## Install the Amdahl Program +With the Amdahl source code on the cluster, we can install it, which will provide access to the amdahl executable. + +The Amdahl code has one dependency: `mpi4py`. Package Installer for Python (pip) will collect `mpi4py` from the Internet and install it for you, but it needs an active `mpi` module. Here we will load `openmpi/gcc/4.1.6` to build `mpi4py`. + +Move into the extracted directory, the use pip, to install it in your (“user”) home directory: +```bash +[NetID@log-1 ~]$ cd amdahl +[NetID@log-1 ~]$ module load python/intel/3.8.6 +[NetID@log-1 ~]$ python -m venv ./test_venv +[NetID@log-1 ~]$ source ./test_venv/bin/activate +[NetID@log-1 ~]$ module load openmpi/gcc/4.1.6 +[NetID@log-1 ~]$ python -m pip install . +Processing /home/NetID/packages/temp/hpc-carpentry-amdahl-46c9b4b +Collecting mpi4py + Using cached mpi4py-4.0.0.tar.gz (464 kB) + Installing build dependencies ... done + Getting requirements to build wheel ... done + Installing backend dependencies ... done + Preparing wheel metadata ... done +Building wheels for collected packages: amdahl, mpi4py + Building wheel for amdahl (setup.py) ... done + Created wheel for amdahl: filename=amdahl-0.3.1-py3-none-any.whl size=6996 sha256=13a95c3e6fbc53fde1c90a4a9bbb3fd3179d5e3afa3e19b4131a05d9ac798981 + Stored in directory: /home/NetID/.cache/pip/wheels/2c/53/fc/19c3053b3a1d3625ac26158b28f263783f66ec258df97aefcf + Building wheel for mpi4py (PEP 517) ... done + Created wheel for mpi4py: filename=mpi4py-4.0.0-cp38-cp38-linux_x86_64.whl size=5169079 sha256=9afceb56e22608a7de33442a60bbde3cbd4aa06947d48de5f6dc63932d34bc9f + Stored in directory: /home/NetID/.cache/pip/wheels/31/3b/6f/dc579e9ff3e2273078596b0cbc1e8d6cbf5a3a05cfad4a380a +Successfully built amdahl mpi4py +Installing collected packages: mpi4py, amdahl +Successfully installed amdahl-0.3.1 mpi4py-4.0.0 +WARNING: You are using pip version 20.2.3; however, version 24.2 is available. +You should consider upgrading via the '/home/NetID/packages/temp/hpc-carpentry-amdahl-46c9b4b/test_venv/bin/python3 -m pip install --upgrade pip' command. +``` + +:::note[Amdahl is Python Code] +The Amdahl program is written in Python, and installing or using it requires locating the python executable on the login node. +::: + +## Help! +Many command-line programs include a “help” message. Try it with `amdahl`: +```bash +[NetID@log-1 ~]$ amdahl --help +usage: amdahl [-h] [-p [PARALLEL_PROPORTION]] [-w [WORK_SECONDS]] [-t] [-e] [-j [JITTER_PROPORTION]] + +optional arguments: + -h, --help show this help message and exit + -p [PARALLEL_PROPORTION], --parallel-proportion [PARALLEL_PROPORTION] + Parallel proportion: a float between 0 and 1 + -w [WORK_SECONDS], --work-seconds [WORK_SECONDS] + Total seconds of workload: an integer greater than 0 + -t, --terse Format output as a machine-readable object for easier analysis + -e, --exact Exactly match requested timing by disabling random jitter + -j [JITTER_PROPORTION], --jitter-proportion [JITTER_PROPORTION] + Random jitter: a float between -1 and +1 +``` +This message doesn’t tell us much about what the program does, but it does tell us the important flags we might want to use when launching it. + +## Running the Job on a Compute Node +Create a submission file, requesting one task on a single node, then launch it. +```bash +[NetID@log-1 ~]$ nano serial-job.sh +[NetID@log-1 ~]$ cat serial-job.sh +``` +```bash +#!/bin/bash +#SBATCH -J solo-job +#SBATCH --nodes=1 +#SBATCH --ntasks-per-node=1 +#SBATCH --mem=3G + +# Load the computing environment we need +module load python/intel/3.8.6 +module load openmpi/gcc/4.1.6 +source /home/yourUsername/amdahl/test_venv/bin/activate + +# Execute the task +srun amdahl +``` +```bash +[NetID@log-1 ~]$ sbatch serial-job.sh +``` +As before, use the Slurm status commands to check whether your job is running and when it ends: +```bash +[NetID@log-1 ~]$ squeue -u NetID +``` +Use `ls` to locate the output file. The `-t` flag sorts in reverse-chronological order: newest first. What was the output? + +
+ +:::info[Read the Job Output] +**[Click for Output]** +::: + +:::tip[Output] +The cluster output should be written to a file in the folder you launched the job from. For example, +```bash +[NetID@log-1 ~]$ ls -t +slurm-347087.out serial-job.sh amdahl README.md LICENSE.txt +[NetID@log-1 ~]$ cat slurm-347087.out +Doing 30.000 seconds of 'work' on 1 processor, +which should take 30.000 seconds with 0.850 parallel proportion of the workload. + + Hello, World! I am process 0 of 1 on cs. I will do all the serial 'work' for 4.500 seconds. + Hello, World! I am process 0 of 1 on cs. I will do parallel 'work' for 25.500 seconds. + +Total execution time (according to rank 0): 30.033 seconds +::: +
+ +As we saw before, two of the `amdahl` program flags set the amount of work and the proportion of that work that is parallel in nature. Based on the output, we can see that the code uses a default of 30 seconds of work that is 85% parallel. The program ran for just over 30 seconds in total, and if we run the numbers, it is true that 15% of it was marked ‘serial’ and 85% was ‘parallel’. + +Since we only gave the job one CPU, this job wasn’t really parallel: the same processor performed the ‘serial’ work for 4.5 seconds, then the ‘parallel’ part for 25.5 seconds, and no time was saved. The cluster can do better, if we ask. + +## Running the Parallel Job +The `amdahl` program uses the Message Passing Interface (MPI) for parallelism – this is a common tool on HPC systems. + +:::tip[What is MPI?] +The Message Passing Interface is a set of tools which allow multiple tasks running simultaneously to communicate with each other. Typically, a single executable is run multiple times, possibly on different machines, and the MPI tools are used to inform each instance of the executable about its sibling processes, and which instance it is. MPI also provides tools to allow communication between instances to coordinate work, exchange information about elements of the task, or to transfer data. An MPI instance typically has its own copy of all the local variables. +::: + +While MPI-aware executables can generally be run as stand-alone programs, in order for them to run in parallel they must use an MPI *run-time environment*, which is a specific implementation of the MPI *standard*. To activate the MPI environment, the program should be started via a command such as `mpiexec` (or `mpirun`, or `srun`, etc. depending on the MPI run-time you need to use), which will ensure that the appropriate run-time support for parallelism is included. + +:::tip[MPI Runtime Arguments] +On their own, commands such as `mpiexec` can take many arguments specifying how many machines will participate in the execution, and you might need these if you would like to run an MPI program on your own (for example, on your laptop). In the context of a queuing system, however, it is frequently the case that MPI run-time will obtain the necessary parameters from the queuing system, by examining the environment variables set when the job is launched. +::: + +Let’s modify the job script to request more cores and use the MPI run-time. +```bash +[NetID@log-1 ~]$ cp serial-job.sh parallel-job.sh +[NetID@log-1 ~]$ nano parallel-job.sh +[NetID@log-1 ~]$ cat parallel-job.sh +#!/bin/bash +#SBATCH -J parallel-pi +#SBATCH --nodes=4 +#SBATCH --ntasks-per-node=1 +#SBATCH --mem=3G + +# Load the computing environment we need +module load python/intel/3.8.6 +module load openmpi/gcc/4.1.6 +source /home/yourUsername/amdahl/test_venv/bin/activate + +# Execute the task +srun amdahl +``` + +Then submit your job. Note that the submission command has not really changed from how we submitted the serial job: all the parallel settings are in the batch file rather than the command line. +```bash +[NetID@log-1 ~]$ sbatch parallel-job.sh +``` + +As before, use the status commands to check when your job runs. +```bash +[NetID@log-1 ~]$ ls -t +slurm-347178.out parallel-job.sh slurm-347087.out serial-job.sh amdahl README.md LICENSE.txt +[NetID@log-1 ~]$ cat slurm-347178.out +Doing 30.000 seconds of 'work' on 4 processors, +which should take 10.875 seconds with 0.850 parallel proportion of the workload. + + Hello, World! I am process 0 of 4 on cs. I will do all the serial 'work' for 4.500 seconds. + Hello, World! I am process 2 of 4 on cs. I will do parallel 'work' for 6.375 seconds. + Hello, World! I am process 1 of 4 on cs. I will do parallel 'work' for 6.375 seconds. + Hello, World! I am process 3 of 4 on cs. I will do parallel 'work' for 6.375 seconds. + Hello, World! I am process 0 of 4 on cs. I will do parallel 'work' for 6.375 seconds. + +Total execution time (according to rank 0): 10.888 seconds +``` + +
+ +:::info[Is it 4× faster?] +The parallel job received 4× more processors than the serial job: does that mean it finished in ¼ the time?
+**[Click for Solution]** +::: + +:::tip[Solution] +The parallel job did take less time: 11 seconds is better than 30! But it is only a 2.7× improvement, not 4×. + +Look at the job output: +- While “process 0” did serial work, processes 1 through 3 did their parallel work. +- While process 0 caught up on its parallel work, the rest did nothing at all. + +Process 0 always has to finish its serial task before it can start on the parallel work. This sets a lower limit on the amount of time this job will take, no matter how many cores you throw at it. + +This is the basic principle behind [Amdahl’s Law](https://en.wikipedia.org/wiki/Amdahl's_law), which is one way of predicting improvements in execution time for a fixed workload that can be subdivided and run in parallel to some extent. +::: +
+ +## How Much Does Parallel Execution Improve Performance? +In theory, dividing up a perfectly parallel calculation among *n* MPI processes should produce a decrease in total run time by a factor of *n*. As we have just seen, real programs need some time for the MPI processes to communicate and coordinate, and some types of calculations can’t be subdivided: they only run effectively on a single CPU. + +Additionally, if the MPI processes operate on different physical CPUs in the computer, or across multiple compute nodes, even more time is required for communication than it takes when all processes operate on a single CPU. + +In practice, it’s common to evaluate the parallelism of an MPI program by: +- running the program across a range of CPU counts +- recording the execution time on each run +- comparing each execution time to the time when using a single CPU + +Since “more is better” – improvement is easier to interpret from increases in some quantity than decreases – comparisons are made using the speedup factor *S*, which is calculated as the single-CPU execution time divided by the multi-CPU execution time. For a perfectly parallel program, a plot of the speedup S versus the number of CPUs *n* would give a straight line, S = n. + +Let’s run one more job, so we can see how close to a straight line our amdahl code gets. +```bash +[NetID@log-1 ~]$ nano parallel-job.sh +[NetID@log-1 ~]$ cat parallel-job.sh +#!/bin/bash +#SBATCH -J parallel-pi +#SBATCH --nodes=8 +#SBATCH --ntasks-per-node=1 +#SBATCH --mem=3G + +# Load the computing environment we need +module load python/intel/3.8.6 +module load openmpi/gcc/4.1.6 +source /home/yourUsername/amdahl/test_venv/bin/activate + +# Execute the task +srun amdahl +``` + +Then submit your job. Note that the submission command has not really changed from how we submitted the serial job: all the parallel settings are in the batch file rather than the command line. +```bash +[NetID@log-1 ~]$ sbatch parallel-job.sh +``` +As before, use the status commands to check when your job runs. +```bash +[NetID@log-1 ~]$ ls -t +slurm-347271.out parallel-job.sh slurm-347178.out slurm-347087.out serial-job.sh amdahl README.md LICENSE.txt +[NetID@log-1 ~]$ cat slurm-347178.out +which should take 7.688 seconds with 0.850 parallel proportion of the workload. + + Hello, World! I am process 4 of 8 on cs. I will do parallel 'work' for 3.188 seconds. + Hello, World! I am process 0 of 8 on cs. I will do all the serial 'work' for 4.500 seconds. + Hello, World! I am process 2 of 8 on cs. I will do parallel 'work' for 3.188 seconds. + Hello, World! I am process 1 of 8 on cs. I will do parallel 'work' for 3.188 seconds. + Hello, World! I am process 3 of 8 on cs. I will do parallel 'work' for 3.188 seconds. + Hello, World! I am process 5 of 8 on cs. I will do parallel 'work' for 3.188 seconds. + Hello, World! I am process 6 of 8 on cs. I will do parallel 'work' for 3.188 seconds. + Hello, World! I am process 7 of 8 on cs. I will do parallel 'work' for 3.188 seconds. + Hello, World! I am process 0 of 8 on cs. I will do parallel 'work' for 3.188 seconds. + +Total execution time (according to rank 0): 7.697 seconds +``` + +:::info[Non-Linear Output] +When we ran the job with 4 parallel workers, the serial job wrote its output first, then the parallel processes wrote their output, with process 0 coming in first and last. + +With 8 workers, this is not the case: since the parallel workers take less time than the serial work, it is hard to say which process will write its output first, except that it will not be process 0! +::: + +Now, let’s summarize the amount of time it took each job to run: + +| Number of CPUs | Runtime (sec) | +| -------------- | ------------- | +| 1 | 30.033 | +| 4 | 10.888 | +| 8 | 7.697 | + +Then, use the first row to compute speedups S, using Python as a command-line calculator: +```bash +[NetID@log-1 ~]$ for n in 30.033 10.888 7.697; do python3 -c "print(30.033 / $n)"; done +``` +| Number of CPUs | Speedup | Ideal | +| -------------- | ------- | ----- | +| 1 | 1.0 | 1 | +| 4 | 2.75 | 4 | +| 8 | 3.90 | 8 | + +The job output files have been telling us that this program is performing 85% of its work in parallel, leaving 15% to run in serial. This seems reasonably high, but our quick study of speedup shows that in order to get a 4× speedup, we have to use 8 or 9 processors in parallel. In real programs, the speedup factor is influenced by + +- CPU design +- communication network between compute nodes +- MPI library implementations +- details of the MPI program itself + +Using Amdahl’s Law, you can prove that with this program, it is *impossible* to reach 8× speedup, no matter how many processors you have on hand. + +In an HPC environment, we try to reduce the execution time for all types of jobs, and MPI is an extremely common way to combine dozens, hundreds, or thousands of CPUs into solving a single problem. To learn more about parallelization, see the [parallel novice lesson](https://www.hpc-carpentry.org/hpc-parallel-novice/). + +Key Points +- Parallel programming allows applications to take advantage of parallel hardware. +- The queuing system facilitates executing parallel tasks. +- Performance improvements from parallel execution do not scale linearly. From d59635cefb2fbc0e1ecd82ce5ae9ec104442b6e9 Mon Sep 17 00:00:00 2001 From: Robert Young Date: Fri, 6 Jun 2025 13:50:34 -0400 Subject: [PATCH 09/11] added first pass of using resources effectively tutorial --- .../08_running_parallel_job.mdx | 2 +- .../09_using_resources_effectively.mdx | 69 +++++++++++++++++++ 2 files changed, 70 insertions(+), 1 deletion(-) create mode 100644 docs/hpc/14_tutorial_intro_hpc/09_using_resources_effectively.mdx diff --git a/docs/hpc/14_tutorial_intro_hpc/08_running_parallel_job.mdx b/docs/hpc/14_tutorial_intro_hpc/08_running_parallel_job.mdx index 1e47e305de..c88eb00cb8 100644 --- a/docs/hpc/14_tutorial_intro_hpc/08_running_parallel_job.mdx +++ b/docs/hpc/14_tutorial_intro_hpc/08_running_parallel_job.mdx @@ -294,7 +294,7 @@ The job output files have been telling us that this program is performing 85% of - MPI library implementations - details of the MPI program itself -Using Amdahl’s Law, you can prove that with this program, it is *impossible* to reach 8× speedup, no matter how many processors you have on hand. +Using Amdahl’s Law, you can prove that with this program, it is *impossible* to reach 8× speedup, no matter how many processors you have on hand. We'll discuss estimating job resources more in the next tutorial [Using Resources Effectively](./09_using_resources_effectively.mdx). In an HPC environment, we try to reduce the execution time for all types of jobs, and MPI is an extremely common way to combine dozens, hundreds, or thousands of CPUs into solving a single problem. To learn more about parallelization, see the [parallel novice lesson](https://www.hpc-carpentry.org/hpc-parallel-novice/). diff --git a/docs/hpc/14_tutorial_intro_hpc/09_using_resources_effectively.mdx b/docs/hpc/14_tutorial_intro_hpc/09_using_resources_effectively.mdx new file mode 100644 index 0000000000..f2515a4618 --- /dev/null +++ b/docs/hpc/14_tutorial_intro_hpc/09_using_resources_effectively.mdx @@ -0,0 +1,69 @@ +# Using resources effectively +:::info[Overview] +Questions +- How can I review past jobs? +- How can I use this knowledge to create a more accurate submission script? + +Objectives +- Look up job statistics. +- Make more accurate resource requests in job scripts based on data describing past performance. +::: + +We’ve touched on all the skills you need to interact with an HPC cluster: logging in over SSH, loading software modules, submitting parallel jobs, and finding the output. Let’s learn about estimating resource usage and why it might matter. + +## Estimating Required Resources Using the Scheduler +Although we covered requesting resources from the scheduler earlier with the Amdahl Program, how do we know what type of resources the software will need in the first place, and its demand for each? In general, unless the software documentation or user testimonials provide some idea, we won’t know how much memory or compute time a program will need. + +:::tip[Read the Documentation] +Most HPC facilities maintain documentation as a wiki, a website, or a document sent along when you register for an account. Take a look at these resources, and search for the software you plan to use: somebody might have written up guidance for getting the most out of it. +::: + +A convenient way of figuring out the resources required for a job to run successfully is to submit a test job, and then ask the scheduler about its impact using `sacct -u $USER`. You can use this knowledge to set up the next job with a closer estimate of its load on the system. A good general rule is to ask the scheduler for 20% to 30% more time and memory than you expect the job to need. This ensures that minor fluctuations in run time or memory use will not result in your job being cancelled by the scheduler. Keep in mind that if you ask for too much, your job may not run even though enough resources are available, because the scheduler will be waiting for other people’s jobs to finish and free up the resources needed to match what you asked for. + +## Stats +Since we already submitted amdahl to run on the cluster, we can query the scheduler to see how long our job took and what resources were used. We will use `sacct -u $USER` to get statistics about `parallel-job.sh`. +```bash +[NYUNetID@log-1 ~]$ sacct -u $USER + JobID JobName Partition Account AllocCPUS State ExitCode +------------ ---------- ---------- ---------- ---------- ---------- -------- +991167 Sxxxx normal nn9299k 128 COMPLETED 0:0 +``` +This shows all the jobs we ran today (note that there are multiple entries per job). To get info about a specific job (for example, 347087), we change command slightly. +```bash +[NYUNetID@log-1 ~]$ sacct -u $USER -l -j 347087 +``` +It will show a lot of info; in fact, every single piece of info collected on your job by the scheduler will show up here. It may be useful to redirect this information to `less` to make it easier to view (use the left and right arrow keys to scroll through fields). + +```bash +[NYUNetID@log-1 ~]$ sacct -u $USER -l -j 347087 | less -S +``` + +:::tip[Discussion] +This view can help compare the amount of time requested and actually used, duration of residence in the queue before launching, and memory footprint on the compute node(s). + +How accurate were our estimates? +::: + +## Improving Resource Requests +From the job history, we see that `amdahl` jobs finished executing in at most a few minutes, once dispatched. The time estimate we provided in the job script was far too long! This makes it harder for the queuing system to accurately estimate when resources will become free for other jobs. Practically, this means that the queuing system waits to dispatch our amdahl job until the full requested time slot opens, instead of “sneaking it in” a much shorter window where the job could actually finish. Specifying the expected runtime in the submission script more accurately will help alleviate cluster congestion and may get your job dispatched earlier. + +
+ +:::info[Narrow the Time Estimate] +Edit `parallel_job.sh` to set a better time estimate. How close can you get? + +Hint: use `-t`.
+**[Click for Solution]** +::: + +:::tip[Solution] +The following line tells Slurm that our job should finish within 2 minutes: +```bash +#SBATCH -t 00:02:00 +``` +::: +
+ +:::tip[Key Points] +- Accurate job scripts help the queuing system efficiently allocate shared resources. +::: From ea3404924cc69650c60e29a5e302bcf191b38499 Mon Sep 17 00:00:00 2001 From: Robert Young Date: Fri, 6 Jun 2025 15:44:21 -0400 Subject: [PATCH 10/11] added first pass of using resources responsibly tutorial --- .../10_using_resources_responsibly.mdx | 153 +++ .../static/schematic_network_bandwidth.svg | 895 ++++++++++++++++++ 2 files changed, 1048 insertions(+) create mode 100644 docs/hpc/14_tutorial_intro_hpc/10_using_resources_responsibly.mdx create mode 100644 docs/hpc/14_tutorial_intro_hpc/static/schematic_network_bandwidth.svg diff --git a/docs/hpc/14_tutorial_intro_hpc/10_using_resources_responsibly.mdx b/docs/hpc/14_tutorial_intro_hpc/10_using_resources_responsibly.mdx new file mode 100644 index 0000000000..5fadba6783 --- /dev/null +++ b/docs/hpc/14_tutorial_intro_hpc/10_using_resources_responsibly.mdx @@ -0,0 +1,153 @@ +# Using shared resources responsibly +:::info[Overview] +Questions +- How can I be a responsible user? +- How can I protect my data? +- How can I best get large amounts of data off an HPC system? + +Objectives +- Describe how the actions of a single user can affect the experience of others on a shared system. +- Discuss the behaviour of a considerate shared system citizen. +- Explain the importance of backing up critical data. +- Describe the challenges with transferring large amounts of data off HPC systems. +- Convert many files to a single archive file using tar. +::: + +One of the major differences between using remote HPC resources and your own system (e.g. your laptop) is that remote resources are shared. How many users the resource is shared between at any one time varies from system to system, but it is unlikely you will ever be the only user logged into or using such a system. + +The widespread usage of scheduling systems where users submit jobs on HPC resources is a natural outcome of the shared nature of these resources. There are other things you, as an upstanding member of the community, need to consider. + +## Be Kind to the Login Nodes +The login node is often busy managing all of the logged in users, creating and editing files and compiling software. If the machine runs out of memory or processing capacity, it will become very slow and unusable for everyone. While the machine is meant to be used, be sure to do so responsibly – in ways that will not adversely impact other users’ experience. + +Login nodes are always the right place to launch jobs. Cluster policies vary, but they may also be used for proving out workflows, and in some cases, may host advanced cluster-specific debugging or development tools. The cluster may have modules that need to be loaded, possibly in a certain order, and paths or library versions that differ from your laptop, and doing an interactive test run on the head node is a quick and reliable way to discover and fix these issues. + +:::warning[Login Nodes Are a Shared Resource] +Remember, the login node is shared with all other users and your actions could cause issues for other people. Think carefully about the potential implications of issuing commands that may use large amounts of resource. + +Unsure? Ask your friendly systems administrator (“sysadmin”) if the thing you’re contemplating is suitable for the login node, or if there’s another mechanism to get it done safely. Please email [hpc@nyu.edu](mailto:hpc@nyu.edu) with questions. +::: + +You can always use the commands `top` and `ps ux` to list the processes that are running on the login node along with the amount of CPU and memory they are using. If this check reveals that the login node is somewhat idle, you can safely use it for your non-routine processing task. If something goes wrong – the process takes too long, or doesn’t respond – you can use the `kill` command along with the *PID* to terminate the process. + +
+ +:::info[Login Node Etiquette] +Which of these commands would be a routine task to run on the login node? +- `python physics_sim.py` +- `make` +- `create_directories.sh` +- `molecular_dynamics_2` +- `tar -xzf R-3.3.0.tar.gz` + +**[Click for Solution]** +::: + +::::tip[Solution] +Building software, creating directories, and unpacking software are common and acceptable tasks for the login node: options #2 (`make`), #3 (`mkdir`), and #5 (`tar`) are probably OK. +:::note +Script names do not always reflect their contents though, so before launching #3, please `less create_directories.sh` and make sure it’s not a Trojan horse. +::: +Running resource-intensive applications is frowned upon. Unless you are sure it will not affect other users, do not run jobs like #1 (`python`) or #4 (`custom MD code`).
+**If you’re unsure, ask your friendly sysadmin for advice by emailing [hpc@nyu.edu](mailto:hpc@nyu.edu).** +:::: +
+ +If you experience performance issues with a login node you should report it to the system staff by sending email to [hpc@nyu.edu](mailto:hpc@nyu.edu) for them to investigate. + +## Test Before Scaling +Remember that you are generally charged for usage on shared systems. A simple mistake in a job script can end up costing a large amount of resource budget. Imagine a job script with a mistake that makes it sit doing nothing for 24 hours on 1000 cores or one where you have requested 2000 cores by mistake and only use 100 of them! This problem can be compounded when people write scripts that automate job submission (for example, when running the same calculation or analysis over lots of different parameters or files). When this happens it hurts both you (as you waste lots of charged resource) and other users (who are blocked from accessing the idle compute nodes). On very busy resources you may wait many days in a queue for your job to fail within 10 seconds of starting due to a trivial typo in the job script. **This is extremely frustrating!** + +Most systems provide dedicated resources for testing that have short wait times to help you avoid this issue. + +:::tip[Test Job Submission Scripts That Use Large Amounts of Resources] +Before submitting a large run of jobs, submit one as a test first to make sure everything works as expected. + +Before submitting a very large or very long job submit a short truncated test to ensure that the job starts as expected. +::: + +## Have a Backup Plan +Although many HPC systems keep backups, it does not always cover all the file systems available and may only be for disaster recovery purposes (i.e. for restoring the whole file system if lost rather than an individual file or directory you have deleted by mistake). Please see the [available storage options on Greene](../03_storage/01_intro_and_data_management.mdx) for specific information about Greene, but remember that rotecting critical data from corruption or deletion is primarily your responsibility: keep your own backup copies. + +Version control systems (such as Git) often have free, cloud-based offerings (e.g., GitHub and GitLab) that are generally used for storing source code. Even if you are not writing your own programs, these can be very useful for storing job scripts, analysis scripts and small input files. + +If you are building software, you may have a large amount of source code that you compile to build your executable. Since this data can generally be recovered by re-downloading the code, or re-running the checkout operation from the source code repository, this data is also less critical to protect. + +For larger amounts of data, especially important results from your runs, which may be irreplaceable, you should make sure you have a robust system in place for taking copies of data off the HPC system wherever possible to backed-up storage. Tools such as rsync can be very useful for this. + +Your access to the shared HPC system will generally be time-limited so you should ensure you have a plan for transferring your data off the system before your access finishes. The time required to transfer large amounts of data should not be underestimated and you should ensure you have planned for this early enough (ideally, before you even start using the system for your research). + +In all these cases, please contact [hpc@nyu.edu](mailto:hpc@nyu.edu) if you have questions about data transfer and storage for the volumes of data you will be using. + +:::warning[Your Data Is Your Responsibility] +Make sure you understand what the backup policy is on the file systems on the system you are using and what implications this has for your work if you lose your data on the system. Plan your backups of critical data and how you will transfer data off the system throughout the project. +::: + +## Transferring Data +As mentioned above, many users run into the challenge of transferring large amounts of data off HPC systems at some point (this is more often in transferring data off than onto systems but the advice below applies in either case). Data transfer speed may be limited by many different factors so the best data transfer mechanism to use depends on the type of data being transferred and where the data is going. + +The components between your data’s source and destination have varying levels of performance, and in particular, may have different capabilities with respect to bandwidth and latency. + +**Bandwidth** is generally the raw amount of data per unit time a device is capable of transmitting or receiving. It’s a common and generally well-understood metric. + +**Latency** is a bit more subtle. For data transfers, it may be thought of as the amount of time it takes to get data out of storage and into a transmittable form. Latency issues are the reason it’s advisable to execute data transfers by moving a small number of large files, rather than the converse. + +Some of the key components and their associated issues are: + +- **Disk speed**: File systems on HPC systems are often highly parallel, consisting of a very large number of high performance disk drives. This allows them to support a very high data bandwidth. Unless the remote system has a similar parallel file system you may find your transfer speed limited by disk performance at that end. +- **Meta-data performance**: Meta-data operations such as opening and closing files or listing the owner or size of a file are much less parallel than read/write operations. If your data consists of a very large number of small files you may find your transfer speed is limited by meta-data operations. Meta-data operations performed by other users of the system can also interact strongly with those you perform so reducing the number of such operations you use (by combining multiple files into a single file) may reduce variability in your transfer rates and increase transfer speeds. +- **Network speed**: Data transfer performance can be limited by network speed. More importantly it is limited by the slowest section of the network between source and destination. If you are transferring to your laptop/workstation, this is likely to be its connection (either via LAN or WiFi). +- **Firewall speed**: Most modern networks are protected by some form of firewall that filters out malicious traffic. This filtering has some overhead and can result in a reduction in data transfer performance. The needs of a general purpose network that hosts email/web-servers and desktop machines are quite different from a research network that needs to support high volume data transfers. If you are trying to transfer data to or from a host on a general purpose network you may find the firewall for that network will limit the transfer rate you can achieve. + +As mentioned above, if you have related data that consists of a large number of small files it is strongly recommended to pack the files into a larger *archive* file for long term storage and transfer. A single large file makes more efficient use of the file system and is easier to move, copy and transfer because significantly fewer metadata operations are required. Archive files can be created using tools like `tar` and `zip`. We have already met `tar` when we talked about data transfer earlier. + +
+![Schematic of Network Bandwidth](./static/schematic_network_bandwidth.svg) + +*Schematic diagram of bandwidth and latency for disk and network I/O. Each of the components on the figure is connected by a blue line of width proportional to the interface bandwidth.
The small mazes at the link points illustrate the latency of the link, with more tortuous mazes indicating higher latency.* +
+ +
+ +:::info[Consider the Best Way to Transfer Data] +If you are transferring large amounts of data you will need to think about what may affect your transfer performance. It is always useful to run some tests that you can use to extrapolate how long it will take to transfer your data. + +Say you have a “data” folder containing 10,000 or so files, a healthy mix of small and large ASCII and binary data. Which of the following would be the best way to transfer them to Greene? +```bash +[user@laptop ~]$ scp -r data NYUNetID@greene.hpc.nyu.edu:~/ +``` +```bash +[user@laptop ~]$ rsync -ra data NYUNetID@greene.hpc.nyu.edu:~/ +``` +```bash +[user@laptop ~]$ rsync -raz data NYUNetID@greene.hpc.nyu.edu:~/ +``` +```bash +[user@laptop ~]$ tar -cvf data.tar data +``` +```bash +[user@laptop ~]$ rsync -raz data.tar NYUNetID@greene.hpc.nyu.edu:~/ +``` +```bash +[user@laptop ~]$ tar -cvzf data.tar.gz data +``` +```bash +[user@laptop ~]$ rsync -ra data.tar.gz NYUNetID@greene.hpc.nyu.edu:~/ +``` +**[Click for Solution]** +::: + +:::tip[Solution] +1. `scp` will recursively copy the directory. This works, but without compression. +1. `rsync -ra` works like `scp -r`, but preserves file information like creation times. This is marginally better. +1. `rsync -raz` adds compression, which will save some bandwidth. If you have a strong CPU at both ends of the line, and you’re on a slow network, this is a good choice. +1. This command first uses `tar` to merge everything into a single file, then `rsync -z` to transfer it with compression. If you have a large number of files, metadata overhead can hamper your transfer, so this is a good idea. +1. This command uses `tar -z` to compress the archive, then `rsync` to transfer it. This may perform similarly to the command directly above, but in most cases (for large datasets), it’s the best combination of high throughput and low latency (making the most of your time and network connection). +::: +
+ +:::tip[Key Points] +- Be careful how you use the login node. +- Your data on the system is your responsibility. +- Plan and test large data transfers. +- It is often best to convert many files to a single archive file before transferring. diff --git a/docs/hpc/14_tutorial_intro_hpc/static/schematic_network_bandwidth.svg b/docs/hpc/14_tutorial_intro_hpc/static/schematic_network_bandwidth.svg new file mode 100644 index 0000000000..69ff64555e --- /dev/null +++ b/docs/hpc/14_tutorial_intro_hpc/static/schematic_network_bandwidth.svg @@ -0,0 +1,895 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + From c058b285247cf01910d8ce0d40b01ea4b363dc15 Mon Sep 17 00:00:00 2001 From: Robert Young Date: Mon, 9 Jun 2025 10:11:49 -0400 Subject: [PATCH 11/11] typo fix --- docs/hpc/14_tutorial_intro_hpc/01_intro_hpc.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hpc/14_tutorial_intro_hpc/01_intro_hpc.mdx b/docs/hpc/14_tutorial_intro_hpc/01_intro_hpc.mdx index c6552e1af5..cf7a78e1d1 100644 --- a/docs/hpc/14_tutorial_intro_hpc/01_intro_hpc.mdx +++ b/docs/hpc/14_tutorial_intro_hpc/01_intro_hpc.mdx @@ -1,6 +1,6 @@ # Introduction to High-Performance Computing -This tutorial is an introduction to using the Greene high-performance computing systems at NYU effectively. It is not intended to be an exhaustive course on parallel programming. The goal is to give new users of Greene an introduciton and overview of the tools available and how to use them effectively. +This tutorial is an introduction to using the Greene high-performance computing systems at NYU effectively. It is not intended to be an exhaustive course on parallel programming. The goal is to give new users of Greene an introduction and overview of the tools available and how to use them effectively. :::warning[Prerequisites] Command line experience is necessary for this lesson. We recommend the participants to go through our [Introduction to Using the Shell on Greene](../13_tutorial_intro_shell_hpc/01_intro.mdx), if new to the command line (also known as terminal or shell).