Skip to content

Commit 59e54f9

Browse files
committed
incorporate reviewer feedback
1 parent a936f01 commit 59e54f9

4 files changed

Lines changed: 19 additions & 16 deletions

File tree

docs/cloud/02_gcp_self_managed/01_intro.md

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
[internet2-cc]: https://internet2.edu/services/cloud-connect/
77
[google-for-edu]: https://cloud.google.com/edu?hl=en
88
[gcp-get-started]: https://cloud.google.com/docs/get-started/
9+
[gcp-credits]: https://cloud.google.com/edu/researchers?hl=en
910

1011
We facilitate access to Google Cloud Platform (GCP) for your research projects. NYU is a member of the [Internet2 Net+ GCP program][internet2-gcp] that allows for community-negotiated GCP terms that provides NYU researchers with benefits that include, amongst other things, price discounts, [waivers for data egress fees][waive-egress], and [NIH STRIDES][nih-strides] initiative benefits.
1112

@@ -24,12 +25,9 @@ NYU researchers who work with the Research Cloud team to deploy projects on GCP
2425
There are a number of ways you can get started with using GCP in research projects:
2526
- To start using GCP resources in a research project, request a consultation with the Research Cloud team (via email research-cloud-support@nyu.edu). The Research Cloud team can advise on ways to set up your research project, available discounts, etc.
2627
- If a project involves Data Science and Machine Learning, consult with the DS3 team before starting your project on GCP.
27-
Please note:
28-
- Creating a GCP project using your NYU account (NetID@nyu.edu) will place the project under the nyu.edu organization, an environment managed by the NYU Research Cloud team. However, your project doesn't automatically qualify for the price discounts and benefits that NYU has negotiated with GCP.
29-
- For non-NYU work on GCP, please use your personal, non-NYU, Google email when creating GCP projects.
30-
3128
:::tip
32-
For non-NYU work on GCP, please use your personal, non-NYU, Google email when creating GCP projects.
29+
- Creating a GCP project using your NYU account (NetID@nyu.edu) will place the project under the nyu.edu organization, an environment managed by the NYU Research Cloud team. However, your project doesn't automatically qualify for the price discounts and benefits that NYU has negotiated with GCP.
30+
- For non-NYU work on GCP, please use your personal, non-NYU, Google email when creating GCP projects.
3331
:::
3432

3533
The NYU Research Cloud team does not currently offer training on how to deploy and utilize resources on GCP in research projects or teaching. If you are new to GCP and you want to learn the GCP fundamentals or if you want to learn how to perform specific tasks on GCP (obtain skill badges), please consider the following resources:
@@ -43,7 +41,7 @@ Apply for credits using your NYU account (https://cloud.google.com/free/). After
4341
- https://edu.google.com/programs/?modal_active=none
4442
### Sources of funding for GCP project
4543
Please consider options below and explore other options which may exist for your specific field.
46-
- Google Cloud research credits
44+
- [Google Cloud research credits][gcp-credits]
4745
- NIH STRIDES
4846
- NSF CloudBank
4947
- GCP seed grant through NYU HPC

docs/cloud/03_hpc_bursting_to_cloud/01_intro.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,9 @@ To request access to the HPC Bursting capabilities, [please complete this form][
1212
:::
1313

1414
## Running a Bursting Job
15-
Note: this is not public, only per request of eligible classes or researchers
15+
:::note
16+
This is not public, only per request of eligible classes or researchers
17+
:::
1618
```sh
1719
ssh <NetID>@greene.hpc.nyu.edu
1820
```

docs/cloud/04_dataproc/02_data_management.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ HDFS stands for Hadoop Distributed File System. HDFS is a highly fault-tolerant
44

55
### File Permissions and Access Control Lists
66

7-
You can share files with others using access control lists (ACLs). An ACL gives you per-file, per-directory and per-user control over who has permission to access files. You can see the ACL for a file or directory with the getfacl command:
7+
You can share files with others using [access control lists (ACLs)](../../hpc/03_storage/09_sharing_data_on_hpc.md). An ACL gives you per-file, per-directory and per-user control over who has permission to access files. You can see the ACL for a file or directory with the getfacl command:
88
```sh
99
hdfs dfs -getfacl /user/<net_id>_nyu_edu/testdir
1010
```
@@ -67,7 +67,7 @@ Copy and paste the command that you are given into a terminal application on you
6767

6868
Copy the URL that you are given in the terminal window, and paste it into your Torch session on the line where gcloud asks for it.
6969

70-
Type gcloud auth list to verify that you are logged in:
70+
Type `gcloud auth list` to verify that you are logged in:
7171
```sh
7272
[NetID@hlog-1 ~]$ gcloud auth list
7373
Credentialed Accounts

docs/cloud/04_dataproc/03_computation.md

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ Ingest a text file into HDFS. In the below example, the hdfs -put command is co
1717
curl -o input.txt https://www.gutenberg.org/files/1661/1661-0.txt
1818
hdfs dfs -put input.txt
1919
```
20-
The following command will run an example Word Count job (described in more detail here) with the Sherlock Holmes book as its input.
20+
The following command will run an example Word Count job (described in more detail [here](https://cwiki.apache.org/confluence/display/HADOOP2/WordCount)) with the Sherlock Holmes book as its input.
2121
```sh
2222
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount -D mapreduce.job.maps=6 -D mapreduce.job.reduces=2 /user/<netid>_nyu_edu/input.txt /user/<netid>_nyu_edu/output
2323
```
@@ -32,25 +32,27 @@ Spark provides an interactive shell that you can use to learn the Spark API and
3232
```sh
3333
spark-shell --deploy-mode client --num-executors=1 --driver-memory=1G --executor-memory=1G
3434
```
35-
Note: NYU Dataproc deploys Spark applications in cluster mode by default. The following error indicates that you are trying to deploy an interactive shell, which must use client mode:
35+
:::tip
36+
NYU Dataproc deploys Spark applications in cluster mode by default. The following error indicates that you are trying to deploy an interactive shell, which must use client mode:
37+
:::
3638
```sh
3739
Exception in thread "main" org.apache.spark.SparkException: Cluster deploy mode is not applicable to Spark shells.
3840
```
39-
To resolve this error, either use the command line flag indicated above (--deploy-mode client) or set the spark.submit.deployMode property in your Spark configuration to client. More details about the difference between cluster and client mode can be found [here](https://spark.apache.org/docs/latest/running-on-yarn.html#launching-spark-on-yarn).
41+
To resolve this error, either use the command line flag indicated above (`--deploy-mode client`) or set the `spark.submit.deployMode` property in your Spark configuration to client. More details about the difference between cluster and client mode can be found [here](https://spark.apache.org/docs/latest/running-on-yarn.html#launching-spark-on-yarn).
4042

4143
## YARN Scheduler
4244

4345
YARN is the resource manager and job scheduler in the Hadoop cluster. YARN allows you to use various data processing engines for batch, interactive, and real-time stream processing of data stored in HDFS.
4446

45-
Application status and logs: Please find the list of current running apps using 'Yarn' script. Running the yarn script without any arguments prints the description for all commands
47+
- Application status and logs: Please find the list of current running apps using 'Yarn' script. Running the yarn script without any arguments prints the description for all commands
4648
```sh
4749
yarn application -list
4850
```
49-
To kill a currently running app because the submitted app started malfunctioning or in worst case scenario, it's stuck in an infinite loop. Get the app ID and then kill it as given below
51+
- To kill a currently running app because the submitted app started malfunctioning or in worst case scenario, it's stuck in an infinite loop. Get the app ID and then kill it as given below
5052
```sh
5153
yarn application -kill <application_ID>
5254
```
53-
To download application logs for examination on the command line
55+
- To download application logs for examination on the command line
5456
```sh
5557
yarn logs -applicationId <application_ID>
5658
```
@@ -79,8 +81,9 @@ Access to Hive databases on NYU Dataproc is derived from HDFS permissions becaus
7981
```
8082
hdfs dfs -setfacl -R -m user:<OTHER_PERSON_NETID>_nyu_edu:r-x /user/hive/warehouse/NetID_nyu_edu.db
8183
```
84+
:::warning
8285
Outside of NYU, other Hadoop installations may use a different mechanism to share databases with other colleagues– it is common for Hadoop installations to use a SQL style grant/revoke mechanism for sharing databases ([SQL Standards Based Authorization](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Authorization#LanguageManualAuthorization-2SQLStandardsBasedAuthorizationinHiveServer2)). This mechanism is not used at NYU and it is important to bear in mind that external documentation referring to grant/revoke statements is not applicable to NYU Dataproc.
83-
86+
:::
8487

8588
## Using Trino
8689

0 commit comments

Comments
 (0)