Skip to content

minor cleanup #127

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
s-sajid-ali merged 1 commit into from
Jun 13, 2025
Merged

minor cleanup #127

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 9 additions & 10 deletions docs/hpc/03_storage/03_data_transfers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,17 @@
# Data Transfers

:::tip Globus
Globus is the recommended tool to use for large-volume data transfers due to the efficiency, reliability, security and ease of use. Use other tools only if you really need to. Detailed instructions available at [Globus](./04_globus.md)
:::

## Data-Transfer nodes
Attached to the NYU HPC cluster Greene, the Greene Data Transfer Node (gDTN) are nodes optimized for transferring data between cluster file systems (e.g. scratch) and other endpoints outside the NYU HPC clusters, including user laptops and desktops. The gDTNs have 100-Gb/s Ethernet connections to the High Speed Research Network (HSRN) and are connected to the HDR Infiniband fabric of the HPC clusters. More information on the hardware characteristics is available at [Greene spec sheet](../10_spec_sheet.mdx).

### Data Transfer Node Access
The HPC cluster filesystems include `/home`, `/scratch`, `/archive` and the [HPC Research Project Space](./05_research_project_space.mdx) are available on the gDTN.

The Data-Transfer Node (DTN) can be accessed in a variety of ways
The HPC cluster filesystems include `/home`, `/scratch`, `/archive` and the [HPC Research Project Space](./05_research_project_space.mdx) are available on the gDTN. The Data-Transfer Node (DTN) can be accessed in a variety of ways
- From NYU-net and the High Speed Research Network: use SSH to the DTN hostname `gdtn.hpc.nyu.edu`
- From the Greene cluster (e.g., the login nodes): the hostname can be shortened to `gdtn`
:::tip
:::info
For example, to log in to a DTN from the Greene cluster, to carry out some copy operation, and to log back out, you can use a command sequence like:
```sh
ssh gdtn
Expand All @@ -19,6 +21,7 @@ logout
:::
- Via specific tools like [Globus](#globus)


## Tools for Data Transfer

### Linux & Mac Tools
Expand Down Expand Up @@ -48,14 +51,10 @@ Windows 10 machines may have the Linux Subsystem installed, which will allow for
### Globus
Globus is the recommended tool to use for large-volume data transfers. It features automatic performance tuning and automatic retries in cases of file-transfer failures. Data-transfer tasks can be submitted via a web portal. The Globus service will take care of the rest, to make sure files are copied efficiently, reliably, and securely. Globus is also a tool for you to share data with collaborators, for whom you only need to provide the email addresses.

The Globus endpoint for Greene is available at `nyu#greene`. The endpoint `nyu#prince` has been retired.

Detailed instructions available at [Globus](./04_globus.md)
The Globus endpoint for Greene is available at `nyu#greene`. The endpoint `nyu#prince` has been retired. Detailed instructions available at [Globus](./04_globus.md)

### rclone
rclone - rsync for cloud storage, is a command line program to sync files and directories to and from cloud storage systems such as Google Drive, Amazon Drive, S3, B2 etc. rclone is available on DTNs.

[Please see the documentation for how to use it.](https://rclone.org/)
rclone - rsync for cloud storage, is a command line program to sync files and directories to and from cloud storage systems such as Google Drive, Amazon Drive, S3, B2 etc. rclone is available on DTNs. [Please see the documentation for how to use it.](https://rclone.org/)

### Open OnDemand (OOD)
One can use [Open OnDemand (OOD)](../09_ood/01_open_on_demand.md) interface to upload data.
Expand Down
2 changes: 1 addition & 1 deletion docs/hpc/03_storage/09_sharing_data_on_hpc.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Sharing Data on HPC

## Introduction
To share files on the cluster with other users, we recommend using file access control lists (FACL) for a user to share access to their data with others. FACL mechanism allows a fine-grained control access to any files by any users or groups of users. We discourage users from setting '777' permissions with `chmod`, because this can lead to data loss (by a malicious user or unintentionally, by accident). Linux commands `getfacl` and `setfacl` are used to view and set access.
To share files on the cluster with other users, we recommend using file access control lists (FACL) for a user to share access to their data with others. FACL mechanism allows a fine-grained control access to any files by any users or groups of users. We discourage users from setting `777` permissions with `chmod`, because this can lead to data loss (by a malicious user or unintentionally, by accident). Linux commands `getfacl` and `setfacl` are used to view and set access.

ACL mechanism, just like regular Linux POSIX, allows three different levels of access control:

Expand Down
6 changes: 3 additions & 3 deletions docs/hpc/07_containers/01_intro.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Singularity: Run Custom Applications with Containers
# Custom Applications with Containers

## What is Singularity
Singularity is a container based Linux kernel workspace that works just like docker. You can run pre-built programs in containers without having to worry about the pre-install environment.
Expand All @@ -7,14 +7,14 @@ For users who are familiar with Docker containers, Singularity works very simila

For a detailed introduction on Singularity, visit their official site [here](https://apptainer.org/documentation/)

## why do we use Singularity
## Why do we use Singularity
There are multiple reasons to use Singularity on the HPC clusters:
1. Security: Singularity provides a layer of security as it does not require any root access on our clusters. This makes it safer against malware and bad scripts that might jeopardize the outer system. Thus we only support Singularity on our clusters(there are not other options such as Kubernetes or Docker on our clusters right now)
2. Containerization: Singularity will run all your images(packaged and pre-built programs) inside of its containers, each container works like a small vm. They contain all the required environment and files of a single Linux kernel and you don't have to worry about any pre-installation nonsense
3. Inter-connectivity: Containers are able to talk to each other, as well as the home system, so while each container has its own small space, they are still a part of a big interconnected structure. Thus enabling you to connect your programs.
4. Accessibility: Probably the most important feature of all, Singularity allows you to run your program in 2 to 3 simple steps, as shown in the topic how to run a singularity container.

## how to run a singularity container
## How to run a singularity container
There are 3 steps to run a Singularity container on our clusters:

pulling a image from Singularity hub or Docker hub
Expand Down