You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/hpc/03_storage/07_transferring_cloud_storage_data_with_rclone.md
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,10 +1,10 @@
1
-
# Transferring Cloud Storage Data with rclone
1
+
# Transferring Cloud Storage Data with Rclone
2
2
3
3
:::tip Globus
4
4
Globus is the recommended tool to use for large-volume data transfers due to the efficiency, reliability, security and ease of use. Use other tools only if you really need to. Detailed instructions available at [Globus](./03_globus.md)
5
5
:::
6
6
7
-
## Transferring files to and from Google Drive with RCLONE
7
+
## Transferring Files to and from Google Drive with Rclone
8
8
Having access to Google Drive from the HPC environment provides an option to archive data and even share data with collaborators who have no access to the NYU HPC environment. Other options to archiving data include the HPC Archive file system and using [Globus](./03_globus.md) to share data with collaborators.
9
9
10
10
Access to Google Drive is provided by [rclone](https://rclone.org/drive/) - rsync for cloud storage - a command line program to sync files and directories to and from cloud storage systems such as Google Drive, Amazon Drive, S3, B2 etc. [rclone](https://rclone.org/drive/) is available on Torch cluster as a module, the module versions currently available (March 2025) are:
This option works great for file sizes 1Gb+ to 250GB. Keep in mind that there is a rate limiting of 2 files/sec for upload into Google Drive. Small file transfers don’t work that well. If you have many small jobs, please tar the parent directory of such folders and split the tar file into 100GB chunks and then uploads then into Google Drive.
28
28
29
-
## rclone Configuration
29
+
## Rclone Configuration
30
30
You need to configure rclone before you will be able to move files between the HPC Environment and Google Drive
31
31
32
32
There are specific instruction on the [rclone web site](https://rclone.org/drive/) and here is an example of the process for configuring rclone for use on Torch:
@@ -35,12 +35,12 @@ There are specific instruction on the [rclone web site](https://rclone.org/drive
35
35
36
36
Follow [instructions](../02_connecting_to_hpc/01_connecting_to_hpc.mdx) to log into the Torch HPC cluster.
37
37
38
-
### Step 2: Load the rclone module
38
+
### Step 2: Load the Rclone Module
39
39
```sh
40
40
$ module load rclone/1.68.2
41
41
```
42
42
43
-
### Step 3: Configure rclone
43
+
### Step 3: Configure Rclone
44
44
45
45
Configuring rclone and setting up remote access to your Google Drive, using the command:
0 commit comments