Skip to content

Commit 79fa339

Browse files
authored
Update 07_transferring_cloud_storage_data_with_rclone.md
1 parent 204c2d7 commit 79fa339

1 file changed

Lines changed: 5 additions & 5 deletions

File tree

docs/hpc/03_storage/07_transferring_cloud_storage_data_with_rclone.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
1-
# Transferring Cloud Storage Data with rclone
1+
# Transferring Cloud Storage Data with Rclone
22

33
:::tip Globus
44
Globus is the recommended tool to use for large-volume data transfers due to the efficiency, reliability, security and ease of use. Use other tools only if you really need to. Detailed instructions available at [Globus](./03_globus.md)
55
:::
66

7-
## Transferring files to and from Google Drive with RCLONE
7+
## Transferring Files to and from Google Drive with Rclone
88
Having access to Google Drive from the HPC environment provides an option to archive data and even share data with collaborators who have no access to the NYU HPC environment. Other options to archiving data include the HPC Archive file system and using [Globus](./03_globus.md) to share data with collaborators.
99

1010
Access to Google Drive is provided by [rclone](https://rclone.org/drive/) - rsync for cloud storage - a command line program to sync files and directories to and from cloud storage systems such as Google Drive, Amazon Drive, S3, B2 etc. [rclone](https://rclone.org/drive/) is available on Torch cluster as a module, the module versions currently available (March 2025) are:
@@ -26,7 +26,7 @@ rclone --transfers=32 --checkers=16 --drive-chunk-size=16384k --drive-upload-cut
2626

2727
This option works great for file sizes 1Gb+ to 250GB. Keep in mind that there is a rate limiting of 2 files/sec for upload into Google Drive. Small file transfers don’t work that well. If you have many small jobs, please tar the parent directory of such folders and split the tar file into 100GB chunks and then uploads then into Google Drive.
2828

29-
## rclone Configuration
29+
## Rclone Configuration
3030
You need to configure rclone before you will be able to move files between the HPC Environment and Google Drive
3131

3232
There are specific instruction on the [rclone web site](https://rclone.org/drive/) and here is an example of the process for configuring rclone for use on Torch:
@@ -35,12 +35,12 @@ There are specific instruction on the [rclone web site](https://rclone.org/drive
3535

3636
Follow [instructions](../02_connecting_to_hpc/01_connecting_to_hpc.mdx) to log into the Torch HPC cluster.
3737

38-
### Step 2: Load the rclone module
38+
### Step 2: Load the Rclone Module
3939
```sh
4040
$ module load rclone/1.68.2
4141
```
4242

43-
### Step 3: Configure rclone
43+
### Step 3: Configure Rclone
4444

4545
Configuring rclone and setting up remote access to your Google Drive, using the command:
4646
```sh

0 commit comments

Comments
 (0)