Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 48 additions & 46 deletions docs/hpc/02_connecting_to_hpc/01_connecting_to_hpc.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,28 +12,33 @@ If you'd prefer to access the HPC cluster via a web gateway instead, please refe

The following sections will outline basic ways to connect to the Greene cluster. Access to the clusters is primarily handled via the Secure Shell Protocol, or ssh. Below we outline ways to connect via ssh on Mac, Linux, and Windows machines.

## Quick Start HPC Connection Guide
### Configuring Your SSH Client
To connect to HPC systems, it's important to configure your machine's SSH client. For Linux and Mac machines, the configuration file is the ~/.ssh/config file on your computer. These are the basic lines that should be added to your ~/.ssh/config file:
## Configuring Your SSH Client

To connect to HPC systems, you have to configure the SSH client on your machine. We recommend the following configuration:
```sh
Host greene.hpc.nyu.edu dtn.hpc.nyu.edu
User <NetID>
StrictHostKeyChecking no
ServerAliveInterval 60
ForwardAgent yes
UserKnownHostsFile /dev/null
LogLevel ERROR

Host greene
HostName localhost
Port 8027
ForwardX11 yes

Host torch login.torch.hpc.nyu.edu
Hostname login.torch.hpc.nyu.edu
User <NetID>
StrictHostKeyChecking no
ServerAliveInterval 60
ForwardAgent yes
UserKnownHostsFile /dev/null
LogLevel ERROR
User <Your NetID>
```
### Setting Up an SSH Config File on Windows
For Windows, you can now follow a similar configuration setup. Using Powershell you can use the following to create and open a Windows SSH config file:
### MacOS/Linux

MacOS/Linux machines have SSH clients pre-installed. Using the editor of your choice, open the `~/.ssh/config` file on your machine and add the contents listed above.

### Windows
Using Powershell you can use the following to create and open a Windows SSH config file:
```sh
# Create the config file with Powershell
New-Item -Path $HOME\.ssh\config -ItemType File
Expand All @@ -42,12 +47,11 @@ New-Item -Path $HOME\.ssh\config -ItemType File
C:\WINDOWS\System32\notepad.exe $HOME\.ssh\config
```
Configuring the file above will allow you to use the cmd terminal to connect to Greene.
### Avoiding Man in the Middle Warning.

If you see this warning:

:::warning
### Common issues

:::warning Man in the Middle warning
If you see this warning:
```sh
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
Expand All @@ -56,29 +60,14 @@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
```

Do not be alarmed - this is an issue that occurs because the Greene and Torch clusters have multiple login nodes. For instance, `greene.hpc.nyu.edu` resolves to one of three possible login nodes (`log-1`, `log-2`, and `log-3`). The configuration directive `StrictHostKeyChecking no` referenced in the recommended config above will prevent this error. Ensure that you have not missed adding it.
:::

Do not be alarmed - this is an issue that occurs because the cluster has multiple login nodes (`log-1`, `log-2`, and `log-3`) that `greene.hpc.nyu.edu` resolves to.

To avoid this warning, you can add these lines to your SSH configuration file. Open `~/.ssh/config` and place the following lines in it:

:::tip
This segment, referenced in the recommended config above will nullify the error:

```sh
Host greene.hpc.nyu.edu dtn.hpc.nyu.edu
StrictHostKeyChecking no
ServerAliveInterval 60
ForwardAgent yes
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
LogLevel ERROR
```
The above will also fix SSH timeout errors by extending the `ServerAliveInterval` argument.
:::info SSH Timeout errors
SSH timeout errors can be fixed by extending the `ServerAliveInterval` argument.
:::

### Connecting to the NYU Network
## Connecting to the NYU Network
To access the HPC systems, you must either be on the NYU secure network (such as the NYU WiFi or a wired connection from an NYU office) or use the NYU VPN.

:::info Connecting to the HPC clusters from outside NYU Network
Expand Down Expand Up @@ -110,26 +99,39 @@ With [Windowns Terminal](https://apps.microsoft.com/detail/9n0dx20hk701?hl=en-US
- If you are using WSL 2 (Windows subsystem for Linux 2), you may not be able to access internet when Cisco AnyConnect VPN, installed from exe file, is activated. A potential solution: uninstall Cisco AnyConnect and install AnyConnect using Microsoft Store, and then setup new VPN connection using settings described on [IT webpage][install vpn on windows link].
:::

## Additional Tips
Below are some additional options and tips for connecting to the HPC resources.
### Setting up SSH Keys
## Two-factor authentication for Torch
There is an extra layer of security when accessing Torch which is described below:

- Login as usual via the `ssh torch` command:
![torch-ssh-2fa-1](static/torch-ssh-2fa-1.png)
- Open [https://microsoft.com/devicelogin](https://microsoft.com/devicelogin)
![torch-ssh-2fa-2](static/torch-ssh-2fa-2.png)
- Copy and enter pin from terminal
![torch-ssh-2fa-3](static/torch-ssh-2fa-3.png)
- Enter or select your `<NetID>@nyu.edu` account
![torch-ssh-2fa-4](static/torch-ssh-2fa-4.png)
- Enter your password
![torch-ssh-2fa-5](static/torch-ssh-2fa-5.png)
- Complete MFA (click on Approve with MFA (Duo)
![torch-ssh-2fa-6](static/torch-ssh-2fa-6.png)
![torch-ssh-2fa-7](static/torch-ssh-2fa-7.png)
![torch-ssh-2fa-8](static/torch-ssh-2fa-8.png)
- Return to terminal and hit Enter
![torch-ssh-2fa-9](static/torch-ssh-2fa-9.png)

## SSH Keys for Greene
:::warning
SSH keys are not supported on Torch!
:::

Instead of typing your password every time you need to log in, you can also specify an ssh key.

- Only do that on the computer you trust

- Generate ssh key pair (terminal in Linux/Mac or cmd/WSL in Windows):
[https://www.ssh.com/ssh/keygen/][ssh instructions keygen link]

- Generate ssh key pair (terminal in Linux/Mac or cmd/WSL in Windows): [https://www.ssh.com/ssh/keygen/][ssh instructions keygen link]
- Note the path to ssh key files. Don't share key files with anybody - anybody with this key file can login to your account

- Log into cluster using regular login/password and then add the content of generated public key file (the one with .pub) to `$HOME/.ssh/authorized_keys` on cluster

- Next time you will log into cluster no password will be required

For additional recommendations on how to configure your SSH sessions, see the [ssh configuring and x11 forwarding page](./02_ssh_tunneling_and_x11_forwarding.md).



[nyu vpn link]: https://www.nyu.edu/life/information-technology/infrastructure/network-services/vpn.html

Expand Down

This file was deleted.

14 changes: 14 additions & 0 deletions docs/hpc/02_connecting_to_hpc/02_x11_forwarding.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# X11 Forwarding

[xquartz]: https://www.xquartz.org/

In rare cases when you need to interact with GUI applications on HPC clusters, you need to enable X11 forwarding for your SSH connection. Mac and Linux users will need to run the ssh commands described above with an additional flag:

```sh
ssh -Y <NYU_NetID>@greene.hpc.nyu.edu
```

However, Mac users need to install [XQuartz][xquartz], since X-server is no longer shipped with the macOS.

Windows users will also need to install X server software. We recommend using MobaXTerm. Further instructions are provided in the [introductory HPC tutorial](../12_tutorial_intro_shell_hpc/02_connecting_to_hpc.mdx).

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 changes: 0 additions & 3 deletions docs/hpc/03_storage/03_data_transfers.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,9 +45,6 @@ where username would be your user name, project1 a directory to be copied to the
#### File Transfer Clients
Windows 10 machines may have the Linux Subsystem installed, which will allow for the use of Linux tools, as listed above, but generally it is recommended to use a client such as [WinSCP](https://winscp.net/eng/docs/tunneling) or [FileZilla](https://filezilla-project.org/) to transfer data. Additionally, Windows users may also take advantage of [Globus](./04_globus.md) to transfer files.

### Tunneling
[Read the detailed instructions for setting up tunnels.](../02_connecting_to_hpc/02_ssh_tunneling_and_x11_forwarding.md)

### Globus
Globus is the recommended tool to use for large-volume data transfers. It features automatic performance tuning and automatic retries in cases of file-transfer failures. Data-transfer tasks can be submitted via a web portal. The Globus service will take care of the rest, to make sure files are copied efficiently, reliably, and securely. Globus is also a tool for you to share data with collaborators, for whom you only need to provide the email addresses.

Expand Down
2 changes: 1 addition & 1 deletion docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -564,7 +564,7 @@ To use any GUI-based program within the interactive batch session you will need
[NetID@log-1 ~]$ gnuplot
gnuplot> test
```
If a window opens on your display with a gnuplot test window, you know that Xforwarding is working. Please see [SSH Tunneling and X11 Forwarding](../02_connecting_to_hpc/02_ssh_tunneling_and_x11_forwarding.md) for details.
If a window opens on your display with a gnuplot test window, you know that Xforwarding is working. Please see the [X11 Forwarding](../02_connecting_to_hpc/02_x11_forwarding.md) section for details.

### Request Resources

Expand Down
2 changes: 1 addition & 1 deletion pixi.toml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[project]
[workspace]
channels = ["conda-forge"]
description = "Add a short description here"
name = "rts-docs-dev"
Expand Down
Loading