Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import TabItem from '@theme/TabItem';

# How to request an HPC account

First, check if you already have an account. You can check this by attempting to log in to the cluster, according to the instructions at [Connecting to the HPC Cluster](../02_connecting_to_hpc/01_connecting_to_hpc.md)
First, check if you already have an account. You can check this by attempting to log in to the cluster, according to the instructions at [Connecting to the HPC Cluster](../02_connecting_to_hpc/01_connecting_to_hpc.mdx)

:::info VPN Needed
You need to be on the NYU VPN to perform this task!
Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
# How to approve an HPC Account Request

When someone nominates you as their HPC sponsor, you should be notified by email.

:::info VPN Needed
You need to be on the NYU VPN to perform this task!
:::

When someone nominates you as their HPC sponsor, you should be notified by email. You can also [log into IIQ at any time](https://iiq.nyu.edu/identityiq), and if you have a request awaiting your approval, it will appear in your "Actions Items" box, as per the following screenshot:
You can also [log into IIQ at any time](https://iiq.nyu.edu/identityiq), and if you have a request awaiting your approval, it will appear in your "Actions Items" box, as per the following screenshot:
!["Actions Items" box](./static/work_item.png)

Another way to get to pending approvals is to click on the line item in the “Latest Approvals” section which will lead directly to the approval page. For new HPC Account Requests, the page will look like this:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# Renewing your HPC Account

Login to the URL given below, using your NetID/password, to create or manage HPC Account Requests:

:::info VPN Needed
You need to be on the NYU VPN to perform this task!
:::

Login to the URL given below, using your NetID/password, to create or manage HPC Account Requests:

https://identity.it.nyu.edu/

Upon logging in, an end user’s landing page will look like this
Expand Down
Original file line number Diff line number Diff line change
@@ -1,22 +1,32 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

# Connecting to the HPC Cluster

There are several ways to interact with the Greene HPC cluster. Similar to other Linux clusters, the most common method of connection is via a Command Line Interface (CLI). A CLI is a program that allows you to create and delete files, run programs, and navigate through directories and files via a typed prompt. On Mac, the built-in CLI application is called Terminal. While Windows 11 machines support a Linux Subsystem, which allows for similar functionality, a popular tool used to connect to a Linux server is a free application called [PuTTY][putty link].

:::tip
If you'd prefer to access the HPC cluster via a web gateway instead, please refer to [the section describing OpenOnDemand](./03_ood.md).
:::

The following sections will outline basic ways to connect to the Greene cluster. Access to the clusters is primarily handled via the Secure Shell Protocol, or ssh. Below we outline ways to connect via ssh on Mac, Linux, and Windows machines.

## Remote Connections with the NYU VPN & HPC Gateway Server
:::info Connecting to Torch from outside NYU Network

If you are connecting from a remote location that is not on the NYU network (your home for example), you have two options:

1. **VPN Option:** [Set up your computer to use the NYU VPN][nyu vpn link]. Once you've created a VPN connection, you can proceed as if you were connected to the NYU net

2. **Gateway Option:** Go through our gateway servers (example below). Gateways are designed to support only a very minimal set of commands and their only purpose is to let users connect HPC systems without needing to first connect to the VPN
<Tabs>
<TabItem value="VPN" label="VPN (Preferred)">
[Set up your computer to use the NYU VPN][nyu vpn link]. Once you've created a VPN connection, you can proceed as if you were connected to the NYU net
</TabItem>
<TabItem value="Gateway" label="Gateway">
Go through our gateway servers (example below). Gateways are designed to support only a very minimal set of commands and their only purpose is to let users connect HPC systems without needing to first connect to the VPN
</TabItem>
</Tabs>
:::

You do not need to use the NYU VPN or gateways if you are connected to the NYU network (wired connection in your office or WiFi) or if you have VPN connection initiated. In this case you can ssh directly to the clusters.

## Command Line Interface (Use Terminal)
## Command Line Interface (with a Terminal)

### Mac & Linux Access

Expand Down Expand Up @@ -48,7 +58,7 @@ Instructions on WSL installation can be found here: [https://docs.microsoft.com/
- If you are using WSL 2 (Windows subsystem for Linux), you may not be able to access internet when Cisco AnyConnect VPN, installed from exe file, is activated. A potential solution: uninstall Cisco AnyConnect and install AnyConnect using Microsoft Store, and then setup new VPN connection using settings described on [IT webpage][install vpn on windows link].
:::

### Setting up SSH Keys
## Setting up SSH Keys

Instead of typing your password every time you need to log in, you can also specify an ssh key.

Expand Down
2 changes: 1 addition & 1 deletion docs/hpc/03_storage/06_best_practices.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,5 +48,5 @@ Your home directory has a relatively small number of inodes.
If you create a conda or python environment in you home directory, this can eat up all the inodes.
:::

Please review the [Package Management section](../06_tools_and_software/05_software_on_greene.md#package-management-for-r-python--julia-and-conda-in-general) of the [Greene Software Page](../06_tools_and_software/05_software_on_greene.md).
Please review the [Package Management section](../06_tools_and_software/01_software_on_greene.md#package-management-for-r-python--julia-and-conda-in-general) of the [Greene Software Page](../06_tools_and_software/01_software_on_greene.md).

2 changes: 1 addition & 1 deletion docs/hpc/03_storage/07_large_number_of_small_files.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ This option may require a lot of RAM - thus you may have to wait a long time for
Special containers, which allow to either load data fast fully or access chosen elements without loading the whole dataset into RAM.

### SQLite
If you have structured data, a good option would be to use SQLite. Please see [SQLite: Handling Large Structured Data](../06_tools_and_software/07_sqlite_handling_large_structured_data.md) for more information.
If you have structured data, a good option would be to use SQLite. Please see [SQLite: Handling Large Structured Data](../06_tools_and_software/05_sqlite_handling_large_structured_data.md) for more information.

### HDF5
One can think about HDF5 file as a "container file" (database of a sort), which holds a lot of objects inside.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ There are specific instruction on the [rclone web site](https://rclone.org/drive

### Step 1: Login to Greene:

Follow [instructions](../02_connecting_to_hpc/01_connecting_to_hpc.md) to log into the Greene HPC cluster.
Follow [instructions](../02_connecting_to_hpc/01_connecting_to_hpc.mdx) to log into the Greene HPC cluster.

### Step 2: Load the rclone module
```sh
Expand Down
5 changes: 2 additions & 3 deletions docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
- HPC workloads are usually better suited to *batch processing* than *interactive* working.
- A batch job is sent to the system when submitted with an **sbatch** command.
- The working pattern we are all familiar with is *interactive* - where we type ( or click ) something interactively, and the computer performs the associated action. Then we type ( or click ) the next thing.
- Comments at the start of the script, which match a special pattern ( #SBATCH ) are read as Slurm options.
- Comments at the start of the script, which match a special pattern ( `#SBATCH` ) are read as Slurm options.

### The trouble with interactive environments

Expand All @@ -15,8 +15,7 @@ There is a reason why GUIs are less common in HPC environments: **point-and-clic
:::tip
The job might not start immediately, and might take hours or days, so we prefer a *batch* approach:

- Plan the sequence of commands which will perform the actions we need
- Write them into a script.
- Plan the sequence of commands which will perform the actions we need and write the commands into a script.

You can now run the script interactively, which is a great way to save effort if i frequently use the same workflow, or ...
- Submit the script to a batch system, to run on dedicated resources when they become available.
Expand Down
1 change: 0 additions & 1 deletion docs/hpc/06_tools_and_software/01_intro.md

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ There are different types of software packages available
- This includes the licensed software listed below
- Singularity Containers
- You can find those already built and ready to use, at location `/scratch/work/public/singularity/`
- For more information on running software with Singularity, [click here](../06_tools_and_software/06_singularity_run_custom_applications_with_containers.md).
- For more information on running software with Singularity, [click here](../07_containers/01_intro.md).
- Python/R/Julia packages can be installed by a user

If you need another linux program installed, please contact us at [hpc@nyu.edu](mailto:hpc@nyu.edu)
Expand Down
2 changes: 1 addition & 1 deletion docs/hpc/06_tools_and_software/02_conda_environments.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Conda provides a great way to install packages that are already compiled, so you

:::note
Reproducibility:
One of the ways to ensure the reproducibility of your results is to have an independent conda environment in the directory of each project (one of the options shown below). This will also keep conda environment files away from your /home/$USER directory.
One of the ways to ensure the reproducibility of your results is to have an independent conda environment in the directory of each project (one of the options shown below). This will also keep conda environment files away from your `/home/$USER` directory.
:::

## Advantages/disadvantages of using Conda
Expand Down

This file was deleted.

Loading