Merge pull request #199 from NYU-RTS/gateways_what_are_those

gateways, what are those?

Author: s-sajid-ali

docs/hpc/01_getting_started/03_walkthrough_request_hpc_account.mdx

@@ -4,7 +4,7 @@ import TabItem from '@theme/TabItem';
 # How to request an HPC account
 
 :::tip
-Make sure you don't already have an HPC account. You can check this by attempting to log in to the cluster, according to the instructions at [Connecting to the HPC Cluster](../02_connecting_to_hpc/01_connecting_to_hpc.mdx).
+Make sure you don't already have an HPC account. You can check this by attempting to log in to the cluster, according to the instructions at [Connecting to the HPC Cluster](../02_connecting_to_hpc/01_connecting_to_hpc.md).
 :::
 
 Login to the URL given below, using your NetID/password, to create or manage HPC Account Requests:

docs/hpc/01_getting_started/06_hpc_accounts_external_collaborators.md

@@ -13,7 +13,7 @@ External (non-NYU) collaborators can access, with proper sponsorship, the NYU HP
 :::note
 Once the sponsoring faculty approves the account request, the HPC account is created within one hour.
 :::
--   Once the HPC account is created, the external collaborator can access HPC resources as described in our documentation page [Connecting to the HPC Cluster](../02_connecting_to_hpc/01_connecting_to_hpc.mdx).
+-   Once the HPC account is created, the external collaborator can access HPC resources as described in our documentation page [Connecting to the HPC Cluster](../02_connecting_to_hpc/01_connecting_to_hpc.md).
 
 :::note
 As with all sponsored accounts, HPC accounts for external collaborators are valid for a period of 12 months, at which point a renewal process is required to continue access to the NYU HPC environment.  You can find information about renewing your account on our documentation page [Getting and Renewing an HPC Account](./02_getting_and_renewing_an_account.mdx).

…nnecting_to_hpc/01_connecting_to_hpc.mdx …onnecting_to_hpc/01_connecting_to_hpc.mddocs/hpc/02_connecting_to_hpc/01_connecting_to_hpc.mdx renamed to docs/hpc/02_connecting_to_hpc/01_connecting_to_hpc.md docs/hpc/02_connecting_to_hpc/01_connecting_to_hpc.mdx renamed to docs/hpc/02_connecting_to_hpc/01_connecting_to_hpc.md

@@ -1,9 +1,10 @@
-import Tabs from '@theme/Tabs';
-import TabItem from '@theme/TabItem';
-
 # Connecting to the HPC Cluster
 
-There are several ways to interact with the Greene HPC cluster. Similar to other Linux clusters, the most common method of connection is via a Command Line Interface (CLI). A CLI is a program that allows you to create and delete files, run programs, and navigate through directories and files via a typed prompt. On Mac, the built-in CLI application is called Terminal. While Windows 11 machines support a Linux Subsystem, which allows for similar functionality, a popular tool used to connect to a Linux server is a free application called [PuTTY][putty link]. 
+:::tip
+This page gives an overview of connecting to the HPC cluster, for a tutorial on this topic, head to the [this section](../12_tutorial_intro_shell_hpc/02_connecting_to_hpc.mdx)! 
+:::
+
+There are several ways to interact with the Greene HPC cluster. Similar to other Linux clusters, the most common method of connection is via a Command Line Interface (CLI). A CLI is a program that allows you to create and delete files, run programs, and navigate through directories and files via a typed prompt. On Mac, the built-in CLI application is called Terminal. While Windows 11 machines support a Linux Subsystem, which allows for similar functionality, a popular tool used to connect to a Linux server is a free application called [MoabXterm](https://mobaxterm.mobatek.net/documentation.html) 
 
 :::tip
 If you'd prefer to access the HPC cluster via a web gateway instead, please refer to [the section describing Open OnDemand (OOD)](../09_ood/01_ood_intro.md).
@@ -15,21 +16,13 @@ The following sections will outline basic ways to connect to the Greene cluster.
 ### Configuring Your SSH Client
 To connect to HPC systems, it's important to configure your machine's SSH client. For Linux and Mac machines, the configuration file is the ~/.ssh/config file on your computer. These are the basic lines that should be added to your ~/.ssh/config file:
 ```sh
-Host greene.hpc.nyu.edu dtn.hpc.nyu.edu gw.hpc.nyu.edu
+Host greene.hpc.nyu.edu dtn.hpc.nyu.edu
   StrictHostKeyChecking no
   ServerAliveInterval 60
   ForwardAgent yes
   UserKnownHostsFile /dev/null
   LogLevel ERROR
 
-Host hpcgwtunnel
-  HostName gw.hpc.nyu.edu
-  ForwardX11 no
-  StrictHostKeyChecking no
-  LocalForward 8027 greene.hpc.nyu.edu:22
-  UserKnownHostsFile /dev/null
-  User <Your NetID>
-
 Host greene
   HostName localhost
   Port 8027
@@ -74,7 +67,7 @@ To avoid this warning, you can add these lines to your SSH configuration file. O
 This segment, referenced in the recommended config above will nullify the error:
 
 ```sh
-Host greene.hpc.nyu.edu dtn.hpc.nyu.edu gw.hpc.nyu.edu
+Host greene.hpc.nyu.edu dtn.hpc.nyu.edu
   StrictHostKeyChecking no
   ServerAliveInterval 60
   ForwardAgent yes
@@ -86,63 +79,35 @@ The above will also fix SSH timeout errors by extending the `ServerAliveInterval
 :::
 
 ### Connecting to the NYU Network
-To access the HPC systems, you must either be on the NYU secure network (such as the NYU WiFi or a wired connection from an NYU office) or use a method of accessing it, such as the NYU VPN or the HPC Gateway servers.
+To access the HPC systems, you must either be on the NYU secure network (such as the NYU WiFi or a wired connection from an NYU office) or use the NYU VPN.
 
 :::info Connecting to the HPC clusters from outside NYU Network
-<Tabs>
-  <TabItem value="VPN" label="VPN (Preferred)">
-    [Set up your computer to use the NYU VPN][nyu vpn link]. Once you've created a VPN connection, you can proceed as if you were connected to the NYU net
-  </TabItem>
-  <TabItem value="Gateway" label="Gateway">
-    Go through our gateway servers (example below). Gateways are designed to support only a very minimal set of commands and their only purpose is to let users connect HPC systems without needing to first connect to the VPN
-  </TabItem>
-</Tabs>
+[Set up your computer to use the NYU VPN][nyu vpn link]. Once you've created a VPN connection, you can proceed as if you were connected to the NYU net
 :::
 
 ## Command Line Interface (with a Terminal)
 
 ### Mac & Linux Access
 
-#### VPN Connection
-If you're on the VPN or the main NYU network on campus you can SSH directly to greene like so:
+Once you're on the VPN or the main NYU network on campus you can SSH directly to greene like so:
 
 ```sh
 ssh <Your NetID>@greene.hpc.nyu.edu
 ```
 
-#### Gateway / SSH Tunneling Option
-
-To connect to the gateway servers, after you've configured your ~/.ssh/config file, simply open a [terminal application][apple terminal link] and follow these steps:
-
-1. Open a terminal
-2. Connect to the gateway via the ssh command:
-```sh
-ssh hpcgwtunnel
-```
-3. Authenticate with your password and Duo MFA
-
-4. Open a second terminal
-5. Connect to the Greene login nodes via the ssh command:
-```sh
-ssh greene
-```
-6. Authenticate
-
-After typing in your password you will be logged in to the cluster.
+### Windows
 
-### Windows CMD
+#### Powershell
+You can access the ssh client via Powershell, either by installing [Windows Terminal](https://apps.microsoft.com/detail/9n0dx20hk701?hl=en-US&gl=US) or [Powershell](https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-windows) directly. With powershell, you would log into the HPC systems the same way as with a Linux terminal.
 
-Windows 11 users have several options. First, the CMD program should contain an ssh client, allowing you to log into the HPC systems the same way as with a Linux terminal.
+#### Windows WSL2
 
-### Windows WSL2
-
-If you run Windows 10, you can install WSL, and then install Ubuntu or other Linux distribution (for example, from Microsoft Store). You will have a fully functional Ubuntu with terminal and can connect to cluster using instructions provided above for Linux/Mac users.
-
-Instructions on WSL installation can be found here: [https://docs.microsoft.com/en-us/windows/wsl/install-win10][wsl installation link]
+Alternatively, you can install WSL2, and then install Ubuntu or other Linux distribution (for example, from Microsoft Store). You will have a fully functional Ubuntu with terminal and can connect to cluster using instructions provided above for Linux/Mac users. Instructions on WSL installation can be found here: [https://docs.microsoft.com/en-us/windows/wsl/install-win10][wsl installation link]
 
+With [Windowns Terminal](https://apps.microsoft.com/detail/9n0dx20hk701?hl=en-US&gl=US), you can access both the Linux WSL2 shell and the Windows Powershell from the same application.
+ 
 :::tip
--   One of many options to get terminal that support tabs, etc. is to install 'Windows Terminal' from Microsoft Store.
--   If you are using WSL 2 (Windows subsystem for Linux), you may not be able to access internet when Cisco AnyConnect VPN, installed from exe file, is activated. A potential solution: uninstall Cisco AnyConnect and install AnyConnect using Microsoft Store, and then setup new VPN connection using settings described on [IT webpage][install vpn on windows link].
+-   If you are using WSL 2 (Windows subsystem for Linux 2), you may not be able to access internet when Cisco AnyConnect VPN, installed from exe file, is activated. A potential solution: uninstall Cisco AnyConnect and install AnyConnect using Microsoft Store, and then setup new VPN connection using settings described on [IT webpage][install vpn on windows link].
 :::
 
 ## Additional Tips
@@ -164,30 +129,7 @@ Instead of typing your password every time you need to log in, you can also spec
 
 For additional recommendations on how to configure your SSH sessions, see the [ssh configuring and x11 forwarding page](./02_ssh_tunneling_and_x11_forwarding.md).
 
-### PuTTY (Only for Windows)
-
-There are many SSH clients for Windows OS, but we recommend using [PuTTY SSH][putty link] if you have not already. Once it is installed, launch PuTTY and configure new session "Session" category as in the screenshot below:
-
-![putty configuration image](./static/putty_config.png)
-
-Here we are instructing PuTTY to connect to host gw.hpc.nyu.edu on port 22 using SSH protocol (note, that this interface allows you to save this connection configuration for future). Just like for Linux and Mac users, if you are connecting from the outside of NYU network, you need to go through the gateway servers.
-
-Once you click "Open", a terminal window with prompt for password will pop up. Enter your NetID password and you should be authorized on the gateway server. Gateways are designed to support only a very minimal set of commands and their only purpose it to let users access HPC systems. Once you are there type in an ssh command that will let you connect to Greene cluster :
-
-```sh
-# Gateway Login
-ssh gw.hpc.nyu.edu
-```
-
-A new command line interface window will open up that prompts you for your password on the gateway server, from there you can connect to Greene by entering the following:
-
-```sh
-ssh greene.hpc.nyu.edu
-```
-
-
 
-[putty link]: https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html
 
 [nyu vpn link]: https://www.nyu.edu/life/information-technology/infrastructure/network-services/vpn.html
 

docs/hpc/02_connecting_to_hpc/02_ssh_tunneling_and_x11_forwarding.md

@@ -13,20 +13,6 @@ Linux users have X set up already. Mac users can download and install [XQuartz][
 To avoid repeatedly setting up a tunnel, you can write the details of the tunnel into your SSH configuration file. Using your favorite editor, open the file `~/.ssh/config` and place the following lines in it: 
 
 ```sh
-# first we create the tunnel, with instructions to pass incoming
-# packets on ports 8027 and 8028 through it and to specific locations
-
-Host hpcgwtunnel
-  HostName gw.hpc.nyu.edu
-  ForwardX11 no
-  StrictHostKeyChecking no
-  LocalForward 8027 greene.hpc.nyu.edu:22
-  UserKnownHostsFile /dev/null 
-  User <Your NetID>
-
-# next we create an alias for incoming packets on the port
-# The alias corresponds to where the tunnel forwards these packets
-
 Host greene
   HostName localhost
   Port 8027
@@ -52,80 +38,6 @@ chmod 700 $HOME/.ssh
 chmod 600 ~/.ssh/*
 ```
 
-### Start the tunnel
-
-To create the tunnel, ssh to it with the following command:
-```sh
-ssh hpcgwtunnel
-```
-
-:::tip
-
-You must leave this window open for the tunnel to remain open. It is best to start a new terminal window for subsequent logins. 
-
-:::
-
-### Log in via the tunnel
-
-Open a new terminal window and use ssh to log in to the cluster, as shown below.
-
-```sh
-ssh greene
-```
-
-Note that you must use the short name defined above in your .ssh/config file, not the fully qualified domain name:
-
-**Creating a once-off tunnel.**
-
-Alternatively, you can set up a once-off tunnel without editing .ssh/config by running the following command:
-
-```sh
-ssh -L 8027:greene:22 NetID@gw.hpc.nyu.edu # to set up a tunnel
-ssh -Y -p 8027 NetID@localhost
-```
-
-This is the equivalent to running "ssh hpcgwtunnel" in the reusable tunnel instructions, but the port forwarding is specified on the command line.
-
-
-## Tunneling (Windows)
-
-### Creating the tunnel
-
-1.  First open Putty and prepare to log in to gw.hpc.nyu.edu. If you saved your session during that process, you can load it by selecting from the "Saved Sessions" box and hitting "Load". Don't hit "Open" yet!
-
-2.  Under "Connection" -> "SSH", just below "X11", select "Tunnels
-
-3.  Write "8026" (the port number) in the "Source port" box, and "greene.hpc.nyu.edu:22" (the machine you wish to tunnel to - 22 is the port that ssh listens on) in the 	"Destination" box
-
-4.  Click "Add". You can repeat step 3 with a different port number and a different destination. If you do this you will create multiple tunnels, one to each destination
-
-5.  Before hitting "Open", go back to the "Sessions" page, give the session a name ("hpcgw_tunnel") and hit "Save". Then next time you need not do all this again, just load the saved session
-
-6.  Hit "Open" to login in to gw.hpc.nyu.edu and create the tunnel. A terminal window will appear, asking for your login name (NYU NetID) and password (NYU password). 	Windows may also ask you to allow certain connections through its firewall - this is so you can ssh to port 8026 on your workstation - the entrance to the tunnel
-
-
-:::note
-You can add other NYU hosts to the tunnel by adding a new source port and destination and clicking "Add". For example, you could add "Source port = 8025" and "Destination = EXAMPLE.hpc.nyu.edu:22", then press "Add". You would then perform Step 2 (below) twice - once for greene on port 8026 and once for an example server on port 8025.
-:::
-
-Using your SSH tunnel: To log in via the tunnel, first the tunnel must be open. If you've just completed Step 1, it will be open and you can jump down to "Step 2: Logging in via your SSH tunnel". If you completed Step 1 yesterday, and now want to re-use the tunnel you created, first start the tunnel:
-
-Starting the tunnel: During a session, you need only do this once - as long as the tunnel is open, new connections will go over it.
-
-1.  Start Putty.exe (again, if necessary), and load the session you saved in settings during procedure above
-
-2.  Hit "Open", and log in to the bastion host with your NYU NetID and password. This will create the tunnel. 
-
-### Logging in via your SSH tunnel
-
-1.  Start the second Putty.exe. In the "Host Name" box, write "localhost" and in the "Port" box, write "8026" (or whichever port number you specified when you set up the tunnel in the procedure above). We use "localhost" because the entrance of the tunnel is actually on this workstation, at port 8026
-
-2.  Go to "Connections" -> "SSH" -> "X11" and check "Enable X11 forwarding"
-
-3.  Optionally, give this session a name (in "Saved Sessions") and hit "Save" to save it. Then next time instead of steps 1 and 2 you can simply load this saved session
-
-4.  Hit "Open". You will again get a terminal window asking for your login (NYU NetID) and password (NYU password). You are now logged in to the HPC cluster!
-
 ## X11 Forwarding
 
 In rare cases when you need to interact with GUI applications on HPC clusters, you need to enable X11 forwarding for your SSH connection. Mac and Linux users will need to run the ssh commands described above with an additional flag:
@@ -136,7 +48,5 @@ ssh -Y <NYU_NetID>@greene.hpc.nyu.edu
 
 However, Mac users need to install [XQuartz][xquartz], since X-server is no longer shipped with the macOS.
 
-Windows users will also need to install X server software. We recommend two options out there. We recommend installing Xming. Start Xming application and configure PuTTY to support X11 forwarding:
-
-
+Windows users will also need to install X server software. We recommend using MobaXTerm. Further instructions are provided in the [introductory HPC tutorial](../12_tutorial_intro_shell_hpc/02_connecting_to_hpc.mdx).
 

docs/hpc/02_connecting_to_hpc/static/putty_config.png

@@ -29,7 +29,7 @@ There are specific instruction on the [rclone web site](https://rclone.org/drive
 
 ### Step 1: Login to Greene:
 
-Follow [instructions](../02_connecting_to_hpc/01_connecting_to_hpc.mdx) to log into the Greene HPC cluster.
+Follow [instructions](../02_connecting_to_hpc/01_connecting_to_hpc.md) to log into the Greene HPC cluster.
 
 ### Step 2: Load the rclone module
 ```sh

docs/hpc/03_storage/08_transferring_cloud_storage_data_with_rclone.md

@@ -5,7 +5,7 @@ Here we provide an example of how one can run a Hugging Face Large-language mode
 ## Prepare environment
 ### Create project directory
 
-After [logging on to a Greene login node](../02_connecting_to_hpc/01_connecting_to_hpc.mdx), make a directory for this project:
+After [logging on to a Greene login node](../02_connecting_to_hpc/01_connecting_to_hpc.md), make a directory for this project:
 ```bash
 [NetID@log-1 ~]$ mkdir -p /scratch/NetID/llm_example
 [NetID@log-1 ~]$ cd /scratch/NetID/llm_example

docs/hpc/08_ml_ai_hpc/03_run_hf_model.md

@@ -63,4 +63,4 @@ Just click on the `Session ID` link and a tab will open with the contents of the
 
 #### From terminal
 
-If your session is no longer visible from within OOD you may still be able to find your logs via the terminal.  Simply [log into Greene](../02_connecting_to_hpc/01_connecting_to_hpc.mdx) and `cd` to `/home/$USER/ondemand/data/sys/dashboard/batch_connect/sys/` and then `cd` into the directory for the app that you're interested in.  You should find the file `output.log` there.
+If your session is no longer visible from within OOD you may still be able to find your logs via the terminal.  Simply [log into Greene](../02_connecting_to_hpc/01_connecting_to_hpc.md) and `cd` to `/home/$USER/ondemand/data/sys/dashboard/batch_connect/sys/` and then `cd` into the directory for the app that you're interested in.  You should find the file `output.log` there.

docs/hpc/09_ood/01_ood_intro.md

@@ -4,7 +4,7 @@
 This page describes how to use your Singularity with conda environment in OOD GUI at Greene. 
 
 ### Log Into Greene via the Terminal
-The following commands must be run from the terminal. Information on accessing via the terminal can be found at [Connecting to the HPC](../02_connecting_to_hpc/01_connecting_to_hpc.mdx) or you can log into OOD and select the `Clusters` tab at the top of the page and select `Greene Shell Access`.
+The following commands must be run from the terminal. Information on accessing via the terminal can be found at [Connecting to the HPC](../02_connecting_to_hpc/01_connecting_to_hpc.md) or you can log into OOD and select the `Clusters` tab at the top of the page and select `Greene Shell Access`.
 
 ### Preinstallation Warning
 :::warning