You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We facilitate access to Google Cloud Platform (GCP) for your research projects. NYU is a member of the [Internet2 Net+ GCP program][internet2-gcp] that allows for community-negotiated GCP terms that provides NYU researchers with benefits that include, amongst other things, price discounts, [waivers for data egress fees][waive-egress], and [NIH STRIDES][nih-strides] initiative benefits.
11
12
@@ -24,12 +25,9 @@ NYU researchers who work with the Research Cloud team to deploy projects on GCP
24
25
There are a number of ways you can get started with using GCP in research projects:
25
26
- To start using GCP resources in a research project, request a consultation with the Research Cloud team (via email research-cloud-support@nyu.edu). The Research Cloud team can advise on ways to set up your research project, available discounts, etc.
26
27
- If a project involves Data Science and Machine Learning, consult with the DS3 team before starting your project on GCP.
27
-
Please note:
28
-
- Creating a GCP project using your NYU account (NetID@nyu.edu) will place the project under the nyu.edu organization, an environment managed by the NYU Research Cloud team. However, your project doesn't automatically qualify for the price discounts and benefits that NYU has negotiated with GCP.
29
-
- For non-NYU work on GCP, please use your personal, non-NYU, Google email when creating GCP projects.
30
-
31
28
:::tip
32
-
For non-NYU work on GCP, please use your personal, non-NYU, Google email when creating GCP projects.
29
+
- Creating a GCP project using your NYU account (NetID@nyu.edu) will place the project under the nyu.edu organization, an environment managed by the NYU Research Cloud team. However, your project doesn't automatically qualify for the price discounts and benefits that NYU has negotiated with GCP.
30
+
- For non-NYU work on GCP, please use your personal, non-NYU, Google email when creating GCP projects.
33
31
:::
34
32
35
33
The NYU Research Cloud team does not currently offer training on how to deploy and utilize resources on GCP in research projects or teaching. If you are new to GCP and you want to learn the GCP fundamentals or if you want to learn how to perform specific tasks on GCP (obtain skill badges), please consider the following resources:
@@ -43,7 +41,7 @@ Apply for credits using your NYU account (https://cloud.google.com/free/). After
Copy file name to clipboardExpand all lines: docs/cloud/04_dataproc/02_data_management.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ HDFS stands for Hadoop Distributed File System. HDFS is a highly fault-tolerant
4
4
5
5
### File Permissions and Access Control Lists
6
6
7
-
You can share files with others using access control lists (ACLs). An ACL gives you per-file, per-directory and per-user control over who has permission to access files. You can see the ACL for a file or directory with the getfacl command:
7
+
You can share files with others using [access control lists (ACLs)](../../hpc/03_storage/09_sharing_data_on_hpc.md). An ACL gives you per-file, per-directory and per-user control over who has permission to access files. You can see the ACL for a file or directory with the getfacl command:
8
8
```sh
9
9
hdfs dfs -getfacl /user/<net_id>_nyu_edu/testdir
10
10
```
@@ -67,7 +67,7 @@ Copy and paste the command that you are given into a terminal application on you
67
67
68
68
Copy the URL that you are given in the terminal window, and paste it into your Torch session on the line where gcloud asks for it.
69
69
70
-
Type gcloud auth list to verify that you are logged in:
70
+
Type `gcloud auth list` to verify that you are logged in:
The following command will run an example Word Count job (described in more detail here) with the Sherlock Holmes book as its input.
20
+
The following command will run an example Word Count job (described in more detail [here](https://cwiki.apache.org/confluence/display/HADOOP2/WordCount)) with the Sherlock Holmes book as its input.
21
21
```sh
22
22
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount -D mapreduce.job.maps=6 -D mapreduce.job.reduces=2 /user/<netid>_nyu_edu/input.txt /user/<netid>_nyu_edu/output
23
23
```
@@ -32,25 +32,27 @@ Spark provides an interactive shell that you can use to learn the Spark API and
Note: NYU Dataproc deploys Spark applications in cluster mode by default. The following error indicates that you are trying to deploy an interactive shell, which must use client mode:
35
+
:::tip
36
+
NYU Dataproc deploys Spark applications in cluster mode by default. The following error indicates that you are trying to deploy an interactive shell, which must use client mode:
37
+
:::
36
38
```sh
37
39
Exception in thread "main" org.apache.spark.SparkException: Cluster deploy mode is not applicable to Spark shells.
38
40
```
39
-
To resolve this error, either use the command line flag indicated above (--deploy-mode client) or set the spark.submit.deployMode property in your Spark configuration to client. More details about the difference between cluster and client mode can be found [here](https://spark.apache.org/docs/latest/running-on-yarn.html#launching-spark-on-yarn).
41
+
To resolve this error, either use the command line flag indicated above (`--deploy-mode client`) or set the `spark.submit.deployMode` property in your Spark configuration to client. More details about the difference between cluster and client mode can be found [here](https://spark.apache.org/docs/latest/running-on-yarn.html#launching-spark-on-yarn).
40
42
41
43
## YARN Scheduler
42
44
43
45
YARN is the resource manager and job scheduler in the Hadoop cluster. YARN allows you to use various data processing engines for batch, interactive, and real-time stream processing of data stored in HDFS.
44
46
45
-
Application status and logs: Please find the list of current running apps using 'Yarn' script. Running the yarn script without any arguments prints the description for all commands
47
+
-Application status and logs: Please find the list of current running apps using 'Yarn' script. Running the yarn script without any arguments prints the description for all commands
46
48
```sh
47
49
yarn application -list
48
50
```
49
-
To kill a currently running app because the submitted app started malfunctioning or in worst case scenario, it's stuck in an infinite loop. Get the app ID and then kill it as given below
51
+
-To kill a currently running app because the submitted app started malfunctioning or in worst case scenario, it's stuck in an infinite loop. Get the app ID and then kill it as given below
50
52
```sh
51
53
yarn application -kill <application_ID>
52
54
```
53
-
To download application logs for examination on the command line
55
+
-To download application logs for examination on the command line
54
56
```sh
55
57
yarn logs -applicationId <application_ID>
56
58
```
@@ -79,8 +81,9 @@ Access to Hive databases on NYU Dataproc is derived from HDFS permissions becaus
Outside of NYU, other Hadoop installations may use a different mechanism to share databases with other colleagues– it is common for Hadoop installations to use a SQL style grant/revoke mechanism for sharing databases ([SQL Standards Based Authorization](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Authorization#LanguageManualAuthorization-2SQLStandardsBasedAuthorizationinHiveServer2)). This mechanism is not used at NYU and it is important to bear in mind that external documentation referring to grant/revoke statements is not applicable to NYU Dataproc.
0 commit comments