diff --git a/docs/hpc/02_connecting_to_hpc/01_connecting_to_hpc.md b/docs/hpc/02_connecting_to_hpc/01_connecting_to_hpc.md index 60a154f1ef..1ed4046e6e 100644 --- a/docs/hpc/02_connecting_to_hpc/01_connecting_to_hpc.md +++ b/docs/hpc/02_connecting_to_hpc/01_connecting_to_hpc.md @@ -12,28 +12,33 @@ If you'd prefer to access the HPC cluster via a web gateway instead, please refe The following sections will outline basic ways to connect to the Greene cluster. Access to the clusters is primarily handled via the Secure Shell Protocol, or ssh. Below we outline ways to connect via ssh on Mac, Linux, and Windows machines. -## Quick Start HPC Connection Guide -### Configuring Your SSH Client -To connect to HPC systems, it's important to configure your machine's SSH client. For Linux and Mac machines, the configuration file is the ~/.ssh/config file on your computer. These are the basic lines that should be added to your ~/.ssh/config file: +## Configuring Your SSH Client + +To connect to HPC systems, you have to configure the SSH client on your machine. We recommend the following configuration: ```sh Host greene.hpc.nyu.edu dtn.hpc.nyu.edu + User StrictHostKeyChecking no ServerAliveInterval 60 ForwardAgent yes UserKnownHostsFile /dev/null LogLevel ERROR - -Host greene - HostName localhost - Port 8027 - ForwardX11 yes + +Host torch login.torch.hpc.nyu.edu + Hostname login.torch.hpc.nyu.edu + User StrictHostKeyChecking no + ServerAliveInterval 60 + ForwardAgent yes UserKnownHostsFile /dev/null LogLevel ERROR - User ``` -### Setting Up an SSH Config File on Windows -For Windows, you can now follow a similar configuration setup. Using Powershell you can use the following to create and open a Windows SSH config file: +### MacOS/Linux + +MacOS/Linux machines have SSH clients pre-installed. Using the editor of your choice, open the `~/.ssh/config` file on your machine and add the contents listed above. + +### Windows +Using Powershell you can use the following to create and open a Windows SSH config file: ```sh # Create the config file with Powershell New-Item -Path $HOME\.ssh\config -ItemType File @@ -42,12 +47,11 @@ New-Item -Path $HOME\.ssh\config -ItemType File C:\WINDOWS\System32\notepad.exe $HOME\.ssh\config ``` Configuring the file above will allow you to use the cmd terminal to connect to Greene. -### Avoiding Man in the Middle Warning. -If you see this warning: - -:::warning +### Common issues +:::warning Man in the Middle warning +If you see this warning: ```sh @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@ -56,29 +60,14 @@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. ``` - +Do not be alarmed - this is an issue that occurs because the Greene and Torch clusters have multiple login nodes. For instance, `greene.hpc.nyu.edu` resolves to one of three possible login nodes (`log-1`, `log-2`, and `log-3`). The configuration directive `StrictHostKeyChecking no` referenced in the recommended config above will prevent this error. Ensure that you have not missed adding it. ::: -Do not be alarmed - this is an issue that occurs because the cluster has multiple login nodes (`log-1`, `log-2`, and `log-3`) that `greene.hpc.nyu.edu` resolves to. - -To avoid this warning, you can add these lines to your SSH configuration file. Open `~/.ssh/config` and place the following lines in it: - -:::tip -This segment, referenced in the recommended config above will nullify the error: - -```sh -Host greene.hpc.nyu.edu dtn.hpc.nyu.edu - StrictHostKeyChecking no - ServerAliveInterval 60 - ForwardAgent yes - StrictHostKeyChecking no - UserKnownHostsFile /dev/null - LogLevel ERROR -``` -The above will also fix SSH timeout errors by extending the `ServerAliveInterval` argument. +:::info SSH Timeout errors +SSH timeout errors can be fixed by extending the `ServerAliveInterval` argument. ::: -### Connecting to the NYU Network +## Connecting to the NYU Network To access the HPC systems, you must either be on the NYU secure network (such as the NYU WiFi or a wired connection from an NYU office) or use the NYU VPN. :::info Connecting to the HPC clusters from outside NYU Network @@ -110,26 +99,39 @@ With [Windowns Terminal](https://apps.microsoft.com/detail/9n0dx20hk701?hl=en-US - If you are using WSL 2 (Windows subsystem for Linux 2), you may not be able to access internet when Cisco AnyConnect VPN, installed from exe file, is activated. A potential solution: uninstall Cisco AnyConnect and install AnyConnect using Microsoft Store, and then setup new VPN connection using settings described on [IT webpage][install vpn on windows link]. ::: -## Additional Tips -Below are some additional options and tips for connecting to the HPC resources. -### Setting up SSH Keys +## Two-factor authentication for Torch +There is an extra layer of security when accessing Torch which is described below: + +- Login as usual via the `ssh torch` command: +![torch-ssh-2fa-1](static/torch-ssh-2fa-1.png) +- Open [https://microsoft.com/devicelogin](https://microsoft.com/devicelogin) +![torch-ssh-2fa-2](static/torch-ssh-2fa-2.png) +- Copy and enter pin from terminal +![torch-ssh-2fa-3](static/torch-ssh-2fa-3.png) +- Enter or select your `@nyu.edu` account +![torch-ssh-2fa-4](static/torch-ssh-2fa-4.png) +- Enter your password +![torch-ssh-2fa-5](static/torch-ssh-2fa-5.png) +- Complete MFA (click on Approve with MFA (Duo) +![torch-ssh-2fa-6](static/torch-ssh-2fa-6.png) +![torch-ssh-2fa-7](static/torch-ssh-2fa-7.png) +![torch-ssh-2fa-8](static/torch-ssh-2fa-8.png) +- Return to terminal and hit Enter +![torch-ssh-2fa-9](static/torch-ssh-2fa-9.png) + +## SSH Keys for Greene +:::warning +SSH keys are not supported on Torch! +::: Instead of typing your password every time you need to log in, you can also specify an ssh key. - Only do that on the computer you trust - -- Generate ssh key pair (terminal in Linux/Mac or cmd/WSL in Windows): - [https://www.ssh.com/ssh/keygen/][ssh instructions keygen link] - +- Generate ssh key pair (terminal in Linux/Mac or cmd/WSL in Windows): [https://www.ssh.com/ssh/keygen/][ssh instructions keygen link] - Note the path to ssh key files. Don't share key files with anybody - anybody with this key file can login to your account - - Log into cluster using regular login/password and then add the content of generated public key file (the one with .pub) to `$HOME/.ssh/authorized_keys` on cluster - - Next time you will log into cluster no password will be required -For additional recommendations on how to configure your SSH sessions, see the [ssh configuring and x11 forwarding page](./02_ssh_tunneling_and_x11_forwarding.md). - - [nyu vpn link]: https://www.nyu.edu/life/information-technology/infrastructure/network-services/vpn.html diff --git a/docs/hpc/02_connecting_to_hpc/02_ssh_tunneling_and_x11_forwarding.md b/docs/hpc/02_connecting_to_hpc/02_ssh_tunneling_and_x11_forwarding.md deleted file mode 100644 index 8f64a760f8..0000000000 --- a/docs/hpc/02_connecting_to_hpc/02_ssh_tunneling_and_x11_forwarding.md +++ /dev/null @@ -1,52 +0,0 @@ -# SSH Tunneling and X11 Forwarding - -[xquartz]: https://www.xquartz.org/ - -## SSH Tunneling (Mac, Linux) - -Setting up your workstation for SSH tunneling will make logging in and transferring files significantly easier, and installing and running an X server will allow you to use graphical software on the HPC clusters. X server is a software package that draws on your local screen windows created on a remote computer such as on the remote HPC. - -Linux users have X set up already. Mac users can download and install [XQuartz][xquartz]. - -### Set up a reusable tunnel - -To avoid repeatedly setting up a tunnel, you can write the details of the tunnel into your SSH configuration file. Using your favorite editor, open the file `~/.ssh/config` and place the following lines in it: - -```sh -Host greene - HostName localhost - Port 8027 - ForwardX11 yes - StrictHostKeyChecking no - UserKnownHostsFile /dev/null - LogLevel ERROR - User -``` - -Create this file/directory In case you don't have it. Make sure that ".ssh" directory has correct permissions (it should be "700" or "drwx------"). If needed, set permissions with: - -```sh -chmod 700 ~/.ssh -``` - -You may also need to setup permissions on your local computer: - -```sh -chmod 700 $HOME -chmod 700 $HOME/.ssh -## to be safe, all files inside ~/.ssh should be set 600 -chmod 600 ~/.ssh/* -``` - -## X11 Forwarding - -In rare cases when you need to interact with GUI applications on HPC clusters, you need to enable X11 forwarding for your SSH connection. Mac and Linux users will need to run the ssh commands described above with an additional flag: - -```sh -ssh -Y @greene.hpc.nyu.edu -``` - -However, Mac users need to install [XQuartz][xquartz], since X-server is no longer shipped with the macOS. - -Windows users will also need to install X server software. We recommend using MobaXTerm. Further instructions are provided in the [introductory HPC tutorial](../12_tutorial_intro_shell_hpc/02_connecting_to_hpc.mdx). - diff --git a/docs/hpc/02_connecting_to_hpc/02_x11_forwarding.md b/docs/hpc/02_connecting_to_hpc/02_x11_forwarding.md new file mode 100644 index 0000000000..5eb4feb4e1 --- /dev/null +++ b/docs/hpc/02_connecting_to_hpc/02_x11_forwarding.md @@ -0,0 +1,14 @@ +# X11 Forwarding + +[xquartz]: https://www.xquartz.org/ + +In rare cases when you need to interact with GUI applications on HPC clusters, you need to enable X11 forwarding for your SSH connection. Mac and Linux users will need to run the ssh commands described above with an additional flag: + +```sh +ssh -Y @greene.hpc.nyu.edu +``` + +However, Mac users need to install [XQuartz][xquartz], since X-server is no longer shipped with the macOS. + +Windows users will also need to install X server software. We recommend using MobaXTerm. Further instructions are provided in the [introductory HPC tutorial](../12_tutorial_intro_shell_hpc/02_connecting_to_hpc.mdx). + diff --git a/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-1.png b/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-1.png new file mode 100644 index 0000000000..94861fbcc1 Binary files /dev/null and b/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-1.png differ diff --git a/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-2.png b/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-2.png new file mode 100644 index 0000000000..c3dbeea989 Binary files /dev/null and b/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-2.png differ diff --git a/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-3.png b/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-3.png new file mode 100644 index 0000000000..79e1f9792f Binary files /dev/null and b/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-3.png differ diff --git a/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-4.png b/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-4.png new file mode 100644 index 0000000000..3967663535 Binary files /dev/null and b/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-4.png differ diff --git a/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-5.png b/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-5.png new file mode 100644 index 0000000000..2ffe2933d8 Binary files /dev/null and b/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-5.png differ diff --git a/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-6.png b/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-6.png new file mode 100644 index 0000000000..a78a46e73b Binary files /dev/null and b/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-6.png differ diff --git a/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-7.png b/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-7.png new file mode 100644 index 0000000000..212aa12655 Binary files /dev/null and b/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-7.png differ diff --git a/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-8.png b/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-8.png new file mode 100644 index 0000000000..cb619f28eb Binary files /dev/null and b/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-8.png differ diff --git a/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-9.png b/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-9.png new file mode 100644 index 0000000000..9267c28979 Binary files /dev/null and b/docs/hpc/02_connecting_to_hpc/static/torch-ssh-2fa-9.png differ diff --git a/docs/hpc/03_storage/03_data_transfers.md b/docs/hpc/03_storage/03_data_transfers.md index 622e794c6a..fc580b049a 100644 --- a/docs/hpc/03_storage/03_data_transfers.md +++ b/docs/hpc/03_storage/03_data_transfers.md @@ -45,9 +45,6 @@ where username would be your user name, project1 a directory to be copied to the #### File Transfer Clients Windows 10 machines may have the Linux Subsystem installed, which will allow for the use of Linux tools, as listed above, but generally it is recommended to use a client such as [WinSCP](https://winscp.net/eng/docs/tunneling) or [FileZilla](https://filezilla-project.org/) to transfer data. Additionally, Windows users may also take advantage of [Globus](./04_globus.md) to transfer files. -### Tunneling -[Read the detailed instructions for setting up tunnels.](../02_connecting_to_hpc/02_ssh_tunneling_and_x11_forwarding.md) - ### Globus Globus is the recommended tool to use for large-volume data transfers. It features automatic performance tuning and automatic retries in cases of file-transfer failures. Data-transfer tasks can be submitted via a web portal. The Globus service will take care of the rest, to make sure files are copied efficiently, reliably, and securely. Globus is also a tool for you to share data with collaborators, for whom you only need to provide the email addresses. diff --git a/docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md b/docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md index 7a2f32f97d..c782d504be 100644 --- a/docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md +++ b/docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md @@ -564,7 +564,7 @@ To use any GUI-based program within the interactive batch session you will need [NetID@log-1 ~]$ gnuplot gnuplot> test ``` -If a window opens on your display with a gnuplot test window, you know that Xforwarding is working. Please see [SSH Tunneling and X11 Forwarding](../02_connecting_to_hpc/02_ssh_tunneling_and_x11_forwarding.md) for details. +If a window opens on your display with a gnuplot test window, you know that Xforwarding is working. Please see the [X11 Forwarding](../02_connecting_to_hpc/02_x11_forwarding.md) section for details. ### Request Resources diff --git a/pixi.toml b/pixi.toml index 7c8dc73579..3a2195ee78 100644 --- a/pixi.toml +++ b/pixi.toml @@ -1,4 +1,4 @@ -[project] +[workspace] channels = ["conda-forge"] description = "Add a short description here" name = "rts-docs-dev"