diff --git a/docs/hpc/09_ood/01_ood_intro.md b/docs/hpc/09_ood/01_ood_intro.md index f80f0fb28d..97d2b0726c 100644 --- a/docs/hpc/09_ood/01_ood_intro.md +++ b/docs/hpc/09_ood/01_ood_intro.md @@ -18,7 +18,7 @@ Under the **clusters** menu you can select the **Greene Shell Access** option to ![img](./static/open_ondemand_gif.gif) -Please see our documentation on [Submitting Jobs](http://localhost:3000/rts-docs-dev/docs/hpc/submitting_jobs/slurm_submitting_jobs/) if you'd like detailed instructions. +Please see our documentation on [Submitting Jobs](../05_submitting_jobs/01_slurm_submitting_jobs.md) if you'd like detailed instructions. **Interactive Applications** @@ -63,4 +63,4 @@ Just click on the `Session ID` link and a tab will open with the contents of the #### From terminal -If your session is no longer visible from within OOD you may still be able to find your logs via the terminal. Simply [log into Greene](https://sites.google.com/nyu.edu/nyu-hpc/accessing-hpc) and `cd` to `/home/$USER/ondemand/data/sys/dashboard/batch_connect/sys/` and then `cd` into the directory for the app that you're interested in. You should find the file `output.log` there. +If your session is no longer visible from within OOD you may still be able to find your logs via the terminal. Simply [log into Greene](../02_connecting_to_hpc/01_connecting_to_hpc.mdx) and `cd` to `/home/$USER/ondemand/data/sys/dashboard/batch_connect/sys/` and then `cd` into the directory for the app that you're interested in. You should find the file `output.log` there. diff --git a/docs/hpc/09_ood/02_CellACDC.mdx b/docs/hpc/09_ood/02_CellACDC.mdx new file mode 100644 index 0000000000..b4a781ba89 --- /dev/null +++ b/docs/hpc/09_ood/02_CellACDC.mdx @@ -0,0 +1,32 @@ +# Cell-ACDC in OOD + +[Cell-ACDC](https://cell-acdc.readthedocs.io) is a GUI-based Python framework for segmentation, tracking, cell cycle annotations and quantification of microscopy data. + +## Getting Started +You can run Cell-ACDC in OOD by going to the URL [ood.hpc.nyu.edu](http://ood.hpc.nyu.edu) in your browser and selecting `Cell-ACDC` from the `Interactive Apps` pull-down menu at the top of the page. Once you've used it and other interactive apps they'll show up on your home screen under the `Recently Used Apps` header. + +:::note +Be aware that when you start from `Recently Used Apps` it will start with the same configuration that you used previously. If you'd like to configure your Cell-ACDC session differently, you'll need to select it from the menu. +::: + +## Configuration + +You can select the number of cores, amount of memory, and number of hours. + +![OOD Cell-ACDC Configuration](./static/ood_cellacdc_config.png) + +## Cell-ACDC running in OOD + +After you hit the `Launch` button you'll have to wait for the scheduler to find node(s) for you to run on: +![OOD Cell-ACDC in queue](./static/ood_cellacdc_in_queue.png) + +Then you'll have a short wait for the Cell-ACDC itself to start up.
+Once that happens you'll get one last page that will give you links to: +- open a terminal window on the compute node your Cell-ACDC session is running on +- go to the directory associated with your Session ID that stores output, config and other related files for your session +- make changes to compression and image qualtiy +- get a link that you can share that will allow others to view your Cell-ACDC session + +![Pre-launch Cell-ACDC OOD](./static/ood_cellacdc_prelaunch.png) + +Please click the `Launch Cell-ACDC` button and a Cell-ACDC window will open. \ No newline at end of file diff --git a/docs/hpc/09_ood/03_Dask.mdx b/docs/hpc/09_ood/03_Dask.mdx new file mode 100644 index 0000000000..3964291d4a --- /dev/null +++ b/docs/hpc/09_ood/03_Dask.mdx @@ -0,0 +1,113 @@ +# Dask in Jupyter Notebook in OOD + +[Dask](https://docs.dask.org/en/stable/) is a Python library for parallel and distributed computing. + +## Getting Started +You can run Dask in a Jupyter Notebook in OOD by going to the URL [ood.hpc.nyu.edu](http://ood.hpc.nyu.edu) in your browser and selecting `DS-GA.1004 - Jupyter Dask` from the `Interactive Apps` pull-down menu at the top of the page. Once you've used it and other interactive apps they'll show up on your home screen under the `Recently Used Apps` header. + +:::note +Be aware that when you start from `Recently Used Apps` it will start with the same configuration that you used previously. If you'd like to configure your Dask session differently, you'll need to select it from the menu. +::: + +## Configuration + +You can select the Dask version, number of cores, amount of memory, root directory, number of hours, and optional Slurm options. + +![OOD Dask Configuration](./static/ood_dask_config.png) + +:::warning +If you select to use `/home` as your root directory be careful not to go over your quota. You can find your current usage with the `myquota` command. Please see our [Storage documentation](../03_storage/01_intro_and_data_management.mdx) for details about your storage options. +::: + +## Dask with Jupyter Notebook running in OOD + +After you hit the `Launch` button you'll have to wait for the scheduler to find node(s) for you to run on: +![OOD Dask in queue](./static/ood_dask_in_queue.png) + +Then you'll have a short wait for Dask itself to start up.
+Once that happens you'll get one last page that will give you links to: +- open a terminal window on the compute node your Dask session is running on +- go to the directory associated with your Session ID that stores output, config and other related files for your session + +![Pre-launch Dask OOD](./static/ood_dask_prelaunch.png) + +Please click the `Connect to Jupyter` button and a Jupyter window will open. + +## Dask Example + +Start a new Jupyter notebook with 4 cores, 16GB memory, and set your root directory to `/scratch`. Enter the following code in the first cell and execute it by pressing the `Shift` and `Enter` keys at the same time. +```python +import os +import pandas as pd +import numpy as np +import time + +# Create a directory for the large files +output_dir = "tmp/large_data_files" +os.makedirs(output_dir, exist_ok=True) + +num_files = 5 # Number of files to create +rows_per_file = 10_000_000 # 10 million rows per file +for i in range(num_files): + data = { + 'col1': np.random.randint(0, 100, size=rows_per_file), + 'value': np.random.rand(rows_per_file) * 100 + } + df = pd.DataFrame(data) + df.to_csv(os.path.join(output_dir, f'data_{i}.csv'), index=False) +print(f"{num_files} large CSV files created in '{output_dir}'.") + +import dask.dataframe as dd +from dask.distributed import Client +import time +import os + +# Start a Dask client for distributed processing (optional but recommended) +# This allows you to monitor the computation with the Dask dashboard +client = Client(n_workers=4, threads_per_worker=2, memory_limit='16GB') # Adjust these as per your system resources +print(client) + +# Load multiple CSV files into a Dask DataFrame +# Dask will automatically partition and parallelize the reading of these files +output_dir = '/scratch/rjy1/tmp/large_data_files' +dask_df = dd.read_csv(os.path.join(output_dir, 'data_*.csv')) + +# Perform a calculation (e.g., calculate the mean of the 'value' column) +# This operation will be parallelized across the available workers +result_dask = dask_df['value'].mean() + +# Trigger the computation and measure the time +start_time = time.time() +computed_result_dask = result_dask.compute() +end_time = time.time() + +print(f"Dask took {end_time - start_time} seconds to compute the mean across {num_files} files.") +print(f"Result (Dask): {computed_result_dask}") + +import pandas as pd +import time +import os + +# Perform the same calculation sequentially with Pandas +start_time_pandas = time.time() +total_mean = 0 +total_count = 0 +for i in range(num_files): + df = pd.read_csv(os.path.join(output_dir, f'data_{i}.csv')) + total_mean += df['value'].sum() + total_count += len(df) +computed_result_pandas = total_mean / total_count +end_time_pandas = time.time() + +print(f"Pandas took {end_time_pandas - start_time_pandas} seconds to compute the mean across {num_files} files.") +print(f"Result (Pandas): {computed_result_pandas}") +``` +You should get output like: +``` +5 large CSV files created in 'tmp/large_data_files'. + +Dask took 3.448112726211548 seconds to compute the mean across 5 files. +Result (Dask): 50.010815178612596 +Pandas took 9.641847610473633 seconds to compute the mean across 5 files. +Result (Pandas): 50.01081517861258 +``` \ No newline at end of file diff --git a/docs/hpc/09_ood/04_Desktop.mdx b/docs/hpc/09_ood/04_Desktop.mdx new file mode 100644 index 0000000000..4318b5d15c --- /dev/null +++ b/docs/hpc/09_ood/04_Desktop.mdx @@ -0,0 +1,32 @@ +# Desktop in OOD + +You can get a basic desktop interface to HPC resources. + +## Getting Started +You can get a desktop in OOD by going to the URL [ood.hpc.nyu.edu](http://ood.hpc.nyu.edu) in your browser and selecting `Desktop` from the `Interactive Apps` pull-down menu at the top of the page. Once you've used it and other interactive apps they'll show up on your home screen under the `Recently Used Apps` header. + +:::note +Be aware that when you start from `Recently Used Apps` it will start with the same configuration that you used previously. If you'd like to configure your Desktop session differently, you'll need to select it from the menu. +::: + +## Configuration + +You can select the number of cores, amount of memory, GPU type (if any), number of hours, and optional Slurm options. + +![OOD Desktop Configuration](./static/ood_desktop_config.png) + +## Desktop running in OOD + +After you hit the `Launch` button you'll have to wait for the scheduler to find node(s) for you to run on: +![OOD Desktop in queue](./static/ood_desktop_in_queue.png) + +Then you'll have a short wait for the Desktop itself to start up.
+Once that happens you'll get one last page that will give you links to: +- open a terminal window on the compute node your Desktop session is running on +- go to the directory associated with your Session ID that stores output, config and other related files for your session +- make changes to compression and image qualtiy +- get a link that you can share that will allow others to view your Desktop session + +![Pre-launch Desktop OOD](./static/ood_desktop_prelaunch.png) + +Please click the `Launch Desktop` button and a Desktop window will open. \ No newline at end of file diff --git a/docs/hpc/09_ood/05_FileZilla.mdx b/docs/hpc/09_ood/05_FileZilla.mdx deleted file mode 100644 index 8d2db5a628..0000000000 --- a/docs/hpc/09_ood/05_FileZilla.mdx +++ /dev/null @@ -1,24 +0,0 @@ -# FileZilla - -As mentioned in [File Transfer Clients](../03_storage/03_data_transfers.md#file-transfer-clients), FileZilla is a client for transferring files. It has been installed on OOD as a convenience, but be aware that you can use many of the other methods described in the [Data Transfers](../03_storage/03_data_transfers.md) section of our documentation for transferring your files on OOD, since they all use the same underlying filesystems. - -## FileZilla running in OOD - -First, you'll be able to select the number of cores, amount of memory, job duration and optional slurm options, you'd like. -![OOD FileZilla initial options](./static/ood_filezilla_init_options.png) - -After you hit the `Launch` button you'll have to wait for the scheduler to find you node(s) to run on, and then you'll have a short wait for FileZilla itself to start up.
- -After all that, you'll get the chance to set some options for your environment: -- Compression -- Image Quality - -:::tip -You can also get a 'view only' link that you can share with others and a link to your session files for debugging -::: - -![OOD FileZilla in queue](./static/ood_filezilla_startup.png) - -After you hit the `Launch FileZilla` button you'll have the familiar FileZilla Desktop to use. - -![OOD FileZilla Running](./static/ood_filezilla_running.png) \ No newline at end of file diff --git a/docs/hpc/09_ood/07_igv.mdx b/docs/hpc/09_ood/05_igv.mdx similarity index 83% rename from docs/hpc/09_ood/07_igv.mdx rename to docs/hpc/09_ood/05_igv.mdx index 40005bb226..c8149ccf85 100644 --- a/docs/hpc/09_ood/07_igv.mdx +++ b/docs/hpc/09_ood/05_igv.mdx @@ -1,4 +1,4 @@ -# Integrative Genomics Viewer (IGV) +# Integrative Genomics Viewer (IGV) in OOD The IGV is a high-performance, easy-to-use, interactive tool for the visual exploration of genomic data. @@ -7,14 +7,12 @@ Please see the following links for details: - [Tutorial Videos](https://www.youtube.com/channel/UCb5W5WqauDOwubZHb-IA_rA) ## Getting Started -You can run IGV in OOD by going to the URL [ood.hpc.nyu.edu](http://ood.hpc.nyu.edu) in your browser and selecting `IGV` from the `Interactive Apps` pull-down menu at the top of the page. As you can see below, once you've used it and other interactive apps they'll show up on your home screen under the `Recently Used Apps` header. +You can run IGV in OOD by going to the URL [ood.hpc.nyu.edu](http://ood.hpc.nyu.edu) in your browser and selecting `IGV` from the `Interactive Apps` pull-down menu at the top of the page. Once you've used it and other interactive apps they'll show up on your home screen under the `Recently Used Apps` header. :::note Be aware that when you start from `Recently Used Apps` it will start with the same configuration that you used previously. If you'd like to configure your IGV session differently, you'll need to select it from the menu. ::: -![OOD Interactive Apps menu IGV](./static/ood_interactive_apps_igv.png) - ## Configuration You can select the number or cores, amount of memory, amount of time, and optional Slurm options. diff --git a/docs/hpc/09_ood/06_Alphafold2.mdx b/docs/hpc/09_ood/06_Alphafold2.mdx deleted file mode 100644 index 2e0377988e..0000000000 --- a/docs/hpc/09_ood/06_Alphafold2.mdx +++ /dev/null @@ -1,61 +0,0 @@ -# AlphaFold2 - -The AlphaFold2 software predicts protein structure from an amino acid sequence. Please see the paper [Highly accurate protein structure prediction with AlphaFold](https://www.nature.com/articles/s41586-021-03819-2) for details. - -## Getting Started -You can run AlphaFold2 in OOD by going to the URL [ood.hpc.nyu.edu](http://ood.hpc.nyu.edu) in your browser and selecting `AlphaFold2` from the `Interactive Apps` pull-down menu at the top of the page. As you can see below, once you've used it and other interactive apps they'll show up on your home screen under the `Recently Used Apps` header. - -:::note -Be aware that when you start from `Recently Used Apps` it will start with the same configuration that you used previously. If you'd like to configure your AlphaFold2 session differently, you'll need to select it from the menu. -::: - -![OOD Interactive Apps menu](./static/ood_interactive_apps_alphafold2.png) - -## Configuration - -You can select the version of AlphaFold2 to use, the number or cores, amount of memory, GPU type (if any), amount of time, and optional Slurm options. You'll also be able to select either `/home` or `/scratch` to use as your root directory. - -![OOD AlphaFold2 Configuration](./static/ood_alphafold2_config.png) - -:::warning -If you select to use `/home` as your root directory be careful not to go over your quota. You can find your current usage with the `myquota` command. Please see our [Storage documentation](../03_storage/01_intro_and_data_management.mdx) for details about your storage options. -::: - -## AlphaFold2 running in OOD - -After you hit the `Launch` button you'll have to wait for the scheduler to find you node(s) to run on: -![OOD AlphaFold2 in queue](./static/ood_alphafold2_in_queue.png) - -Once that's done, just hit the `Connect to Jupyter` button to see your Jupyter environment. - -## Running the example notebook - -Please follow these steps to run the example notebook: -1. Get a shell prompt (This can be done in either of the following two ways): - - In OOD: - 1. Select the `Clusters` pull-down menu at the top of the page - 1. Select `Greene Shell Access` - - Use your terminal to log into Greene: - 1. Follow the instructions on [Connecting to HPC](../02_connecting_to_hpc/01_connecting_to_hpc.mdx) -1. Create a directory to save the example notebook: - ```bash - [NetID@log-1 ~]$ mkdir /scratch/$USER/AlphaFold2_example - ``` -1. `cd` into this new directory: - ```bash - [NetID@log-1 ~]$ cd /scratch/$USER/AlphaFold2_example - ``` -1. Copy the example notebook into your directory: - ```bash - [NetID@log-1 AlphaFold2_example]$ cp /scratch/work/public/examples/alphafold/alphafold-test.ipynb . - ``` -1. You can now go back to the main OOD site in your browser at [ood.hpc.nyu.edu](http://ood.hpc.nyu.edu) and start an AlphaFold2 interactive Session. - :::tip - - You'll need a GPU for the example notebook so select either version `2.0.0` or `2.1.1 amber-gpu` - - One core is enough for the example notebook - - You can select more if you'd like, but your job may sit in the queue longer before starting. - - 4GB of memory is enough for the example notebook - - Again, you can select more memory if you'd like, but it may take longer for you job to get picked up from the queue. - - Select `any` type of GPU for the least amount of time spent in the queue - ::: -1. When your Jupyter session starts you should be able to navigate to the new directory you created above and work through the example notebook. \ No newline at end of file diff --git a/docs/hpc/09_ood/09_jbrowse_genome_browser.mdx b/docs/hpc/09_ood/06_jbrowse_genome_browser.mdx similarity index 99% rename from docs/hpc/09_ood/09_jbrowse_genome_browser.mdx rename to docs/hpc/09_ood/06_jbrowse_genome_browser.mdx index 3cf74d07e8..ca99479056 100644 --- a/docs/hpc/09_ood/09_jbrowse_genome_browser.mdx +++ b/docs/hpc/09_ood/06_jbrowse_genome_browser.mdx @@ -1,4 +1,4 @@ -# JBrowse Genome Browser +# JBrowse Genome Browser in OOD [JBrowse](https://jbrowse.org/jbrowse1.html) is a web-based genome browser for visualizing genomic features in common file formats, such as variants (VCF), genes (GFF3, BigBed) and gene expression (BigWig), and sequence alignments (BAM, CRAM, and GFF3). diff --git a/docs/hpc/09_ood/02_jupyter_with_conda_singularity.mdx b/docs/hpc/09_ood/07_jupyter_with_conda_singularity.mdx similarity index 99% rename from docs/hpc/09_ood/02_jupyter_with_conda_singularity.mdx rename to docs/hpc/09_ood/07_jupyter_with_conda_singularity.mdx index 3085945120..86563ca22c 100644 --- a/docs/hpc/09_ood/02_jupyter_with_conda_singularity.mdx +++ b/docs/hpc/09_ood/07_jupyter_with_conda_singularity.mdx @@ -1,4 +1,4 @@ -# Jupyter with Conda/Singularity in OOD +# Jupyter Notebook with Conda/Singularity in OOD ## OOD + Singularity + conda This page describes how to use your Singularity with conda environment in OOD GUI at Greene. diff --git a/docs/hpc/09_ood/03_matlab.mdx b/docs/hpc/09_ood/08_matlab_proxy.mdx similarity index 71% rename from docs/hpc/09_ood/03_matlab.mdx rename to docs/hpc/09_ood/08_matlab_proxy.mdx index c0a82fb169..8be215675d 100644 --- a/docs/hpc/09_ood/03_matlab.mdx +++ b/docs/hpc/09_ood/08_matlab_proxy.mdx @@ -1,17 +1,15 @@ -# Matlab in OOD +# Matlab-Proxy in OOD ## Getting Started -You can run Matlab in OOD by going to the URL [ood.hpc.nyu.edu](http://ood.hpc.nyu.edu) in your browser and selecting `MATLAB` from the `Interactive Apps` pull-down menu at the top of the page. As you can see below, once you've used it and other interactive apps they'll show up on your home screen under the `Recently Used Apps` header. +You can run Matlab-Proxy in OOD by going to the URL [ood.hpc.nyu.edu](http://ood.hpc.nyu.edu) in your browser and selecting `MATLAB` from the `Interactive Apps` pull-down menu at the top of the page. Once you've used it and other interactive apps they'll show up on your home screen under the `Recently Used Apps` header. :::note Be aware that when you start from `Recently Used Apps` it will start with the same configuration that you used previously. If you'd like to configure your Matlab session differently, you'll need to select it from the menu. ::: -![OOD Interactive Apps menu](./static/ood_interactive_apps_matlab.png) - ## Configuration -You can select the version of Matlab to use, the number or cores, amount of memory, GPU type (if any), amount of time, and optional Slurm options. +You can select the version of Matlab to use, the number or cores, amount of memory, GPU type (if any), amount of time, account, and optional Slurm options. ![OOD Matlab Configuration](./static/ood_matlab_config.png) @@ -47,13 +45,10 @@ After you hit the `Launch` button you'll have to wait for the scheduler to find ![OOD Matlab in queue](./static/ood_matlab_in_queue.png) Then you'll have a short wait for Matlab itself to start up.
-Once that happens you'll get one last form that will allow you to: -- make changes to compression and image qualtiy +Once that happens you'll get a form that will allow you to: - open a terminal window on the compute node your Matlab session is running on -- get a link that you can share that will allow others to view your Matlab session +- go to the directory associated with your Session ID that stores output, config and other related files for your session ![Pre-launch matlab OOD](./static/ood_matlab_prelaunch.png) -Then after you hit the `Launch Matlab` button you'll have the familiar Matlab Desktop to use. - -![OOD Matlab Running](./static/ood_matlab_running.png) +Then after you hit the `Connect to Matlab` button you'll have the familiar Matlab Desktop to use. \ No newline at end of file diff --git a/docs/hpc/09_ood/04_RStudio.mdx b/docs/hpc/09_ood/09_RStudio.mdx similarity index 87% rename from docs/hpc/09_ood/04_RStudio.mdx rename to docs/hpc/09_ood/09_RStudio.mdx index 0d465bf799..ab2fb7f5fd 100644 --- a/docs/hpc/09_ood/04_RStudio.mdx +++ b/docs/hpc/09_ood/09_RStudio.mdx @@ -2,14 +2,12 @@ ## Getting Started -You can run RStudio in OOD by going to the URL [ood.hpc.nyu.edu](http://ood.hpc.nyu.edu) in your browser and selecting `RStudio Server` from the `Interactive Apps` pull-down menu at the top of the page. As you can see below, once you've used it and other interactive apps they'll show up on your home screen under the `Recently Used Apps` header. +You can run RStudio in OOD by going to the URL [ood.hpc.nyu.edu](http://ood.hpc.nyu.edu) in your browser and selecting `RStudio Server` from the `Interactive Apps` pull-down menu at the top of the page. Once you've used it and other interactive apps they'll show up on your home screen under the `Recently Used Apps` header. :::note Be aware that when you start from `Recently Used Apps` it will start with the same configuration that you used previously. If you'd like to configure your RStudio session differently, you'll need to select it from the menu. ::: -![OOD Interactive Apps menu](./static/ood_interactive_apps_rstudio.png) - ## Configuration You can select the version of R to use, the R package location, the number or cores, amount of memory, GPU type (if any), amount of time, and optional Slurm options. diff --git a/docs/hpc/09_ood/10_Spark.mdx b/docs/hpc/09_ood/10_Spark.mdx new file mode 100644 index 0000000000..78a25e15bf --- /dev/null +++ b/docs/hpc/09_ood/10_Spark.mdx @@ -0,0 +1,114 @@ +# Spark Standalone Cluster with Jupyter Notebook in OOD + +## Getting Started +You can run a Spark Standalone Cluster with Jupyter Notebook in OOD by going to the URL [ood.hpc.nyu.edu](http://ood.hpc.nyu.edu) in your browser and selecting `Spark Standalone Cluster` from the `Interactive Apps` pull-down menu at the top of the page. Once you've used it and other interactive apps they'll show up on your home screen under the `Recently Used Apps` header. + +:::note +Be aware that when you start from `Recently Used Apps` it will start with the same configuration that you used previously. If you'd like to configure your Spark Standalone Cluster with Jupyter Notebook session differently, you'll need to select it from the menu. +::: + +## Configuration + +You can select the Spark version, amount of time, number of nodes and cores, amount of memory, gpu type (if any), Jupyter notebook root directory, path to custom pyspark overlay (if any), and optional Slurm options. + +![OOD Spark Configuration](./static/ood_spark_config.png) + + +:::warning +If you select to use `/home` as your root directory be careful not to go over your quota. You can find your current usage with the `myquota` command. Please see our [Storage documentation](../03_storage/01_intro_and_data_management.mdx) for details about your storage options. +::: + +## Spark Standalone Cluster with Jupyter Notebook running in OOD + +After you hit the `Launch` button you'll have to wait for the scheduler to find node(s) for you to run on: +![OOD Spark in queue](./static/ood_spark_in_queue.png) + +Then you'll have a short wait for Spark itself to start up.
+Once that happens you'll get one last page that will give you links to: +- open a terminal window on the compute node your Spark session is running on +- go to the directory associated with your Session ID that stores output, config and other related files for your session + +![Pre-launch Spark OOD](./static/ood_spark_prelaunch.png) + +Please click the `Connect to the Jupyter Notebook Environment` button and a Jupyter window will open. Please select to create a new notebook and you're ready to go. + +### Spark Standalone Cluster Jupyter Notebook Example + +Please enter the following commands into the first cell of your Jupyter notebook and execute them by typing `Shift` and `Enter` at the same time. +```python +from pyspark import SparkContext +import requests + +# Create a SparkContext +sc = SparkContext("local", "WordCountExample") + +# Get text of Moby Dick from Project Gutenberg +file_url = 'https://www.gutenberg.org/ebooks/2701.txt.utf-8' +try: + response = requests.get(file_url) + response.raise_for_status() +except requests.exceptions.RequestException as e: + print(f"Error during request: {e}") +# Save text to temp file +with open('moby_dick_temp_spark_example.txt', "w") as file: + file.write(response.text) + +# Create an RDD from a text file +lines = sc.textFile("/scratch/rjy1/moby_dick_temp_spark_example.txt") + +# FlatMap to split lines into words and convert to lowercase +words = lines.flatMap(lambda line: line.lower().split(" ")) + +# Map each word to a (word, 1) tuple +word_pairs = words.map(lambda word: (word, 1)) + +# ReduceByKey to sum the counts for each word +word_counts = word_pairs.reduceByKey(lambda a, b: a + b) + +# Collect the results to the driver program +results = word_counts.collect() + +# Print the word counts +for word, count in results: + print(f"{word}: {count}") +``` + +You should get output like: +``` +the: 14512 +project: 87 +gutenberg: 25 +ebook: 8 +of: 6682 +moby: 81 +dick;: 10 +or,: 17 +whale: 533 +: 4318 +this: 1277 +is: 1601 +for: 1555 +use: 39 +anyone: 5 +anywhere: 11 +in: 4126 +united: 24 +states: 13 +and: 6321 +most: 284 +other: 360 +parts: 32 +world: 79 +at: 1310 +no: 488 +cost: 3 +with: 1750 +almost: 189 +restrictions: 2 +whatsoever.: 5 +you: 843 +may: 227 +copy: 15 +... +``` +etc. diff --git a/docs/hpc/09_ood/11_Stata.mdx b/docs/hpc/09_ood/11_Stata.mdx new file mode 100644 index 0000000000..05572a0eb3 --- /dev/null +++ b/docs/hpc/09_ood/11_Stata.mdx @@ -0,0 +1,43 @@ +# Stata in OOD + +## Getting Started +You can run Stata in OOD by going to the URL [ood.hpc.nyu.edu](http://ood.hpc.nyu.edu) in your browser and selecting `STATA` from the `Interactive Apps` pull-down menu at the top of the page. Once you've used it and other interactive apps they'll show up on your home screen under the `Recently Used Apps` header. + +:::note +Be aware that when you start from `Recently Used Apps` it will start with the same configuration that you used previously. If you'd like to configure your NGLView Jupyter Notebook session differently, you'll need to select it from the menu. +::: + +## Configuration + +You can select the Stata version, number or cores, amount of memory, amount of time, and optional Slurm options. + +![OOD Stata Configuration](./static/ood_stata_config.png) + +## Stata running in OOD + +After you hit the `Launch` button you'll have to wait for the scheduler to find node(s) for you to run on: +![OOD Stata in queue](./static/ood_stata_in_queue.png) + +Then you'll have a short wait for Stata itself to start up.
+Once that happens you'll get one last form that will allow you to: +- open a terminal window on the compute node your Stata session is running on +- go to the directory associated with your Session ID that stores output, config and other related files for your session +- set compression and image quality for your app +- get a share-able, view only link to your app + +![Pre-launch Stata OOD](./static/ood_stata_prelaunch.png) + +Then after you hit the `Launch STATA` button, your Stata window will be displayed. + +### Stata example + +Please enter the following commands into the Stata command window and execute them by hitting `enter`: +``` +. sysuse auto +. describe +. summarize +. twoway (scatter mpg weight) +``` + +You should get output like this: +![OOD Stata example](./static/ood_stata_example.png) diff --git a/docs/hpc/09_ood/static/ood_cellacdc_config.png b/docs/hpc/09_ood/static/ood_cellacdc_config.png new file mode 100644 index 0000000000..dd6076bd08 Binary files /dev/null and b/docs/hpc/09_ood/static/ood_cellacdc_config.png differ diff --git a/docs/hpc/09_ood/static/ood_cellacdc_in_queue.png b/docs/hpc/09_ood/static/ood_cellacdc_in_queue.png new file mode 100644 index 0000000000..8a2cfaea90 Binary files /dev/null and b/docs/hpc/09_ood/static/ood_cellacdc_in_queue.png differ diff --git a/docs/hpc/09_ood/static/ood_cellacdc_prelaunch.png b/docs/hpc/09_ood/static/ood_cellacdc_prelaunch.png new file mode 100644 index 0000000000..0fb1a0f576 Binary files /dev/null and b/docs/hpc/09_ood/static/ood_cellacdc_prelaunch.png differ diff --git a/docs/hpc/09_ood/static/ood_dask_config.png b/docs/hpc/09_ood/static/ood_dask_config.png new file mode 100644 index 0000000000..3a9e06c420 Binary files /dev/null and b/docs/hpc/09_ood/static/ood_dask_config.png differ diff --git a/docs/hpc/09_ood/static/ood_dask_in_queue.png b/docs/hpc/09_ood/static/ood_dask_in_queue.png new file mode 100644 index 0000000000..8f49c6d860 Binary files /dev/null and b/docs/hpc/09_ood/static/ood_dask_in_queue.png differ diff --git a/docs/hpc/09_ood/static/ood_dask_prelaunch.png b/docs/hpc/09_ood/static/ood_dask_prelaunch.png new file mode 100644 index 0000000000..f755181996 Binary files /dev/null and b/docs/hpc/09_ood/static/ood_dask_prelaunch.png differ diff --git a/docs/hpc/09_ood/static/ood_desktop_config.png b/docs/hpc/09_ood/static/ood_desktop_config.png new file mode 100644 index 0000000000..352880c84d Binary files /dev/null and b/docs/hpc/09_ood/static/ood_desktop_config.png differ diff --git a/docs/hpc/09_ood/static/ood_desktop_in_queue.png b/docs/hpc/09_ood/static/ood_desktop_in_queue.png new file mode 100644 index 0000000000..4c2906f877 Binary files /dev/null and b/docs/hpc/09_ood/static/ood_desktop_in_queue.png differ diff --git a/docs/hpc/09_ood/static/ood_desktop_prelaunch.png b/docs/hpc/09_ood/static/ood_desktop_prelaunch.png new file mode 100644 index 0000000000..d552d48b6d Binary files /dev/null and b/docs/hpc/09_ood/static/ood_desktop_prelaunch.png differ diff --git a/docs/hpc/09_ood/static/ood_filezilla_init_options.png b/docs/hpc/09_ood/static/ood_filezilla_init_options.png deleted file mode 100644 index 2693d02ee7..0000000000 Binary files a/docs/hpc/09_ood/static/ood_filezilla_init_options.png and /dev/null differ diff --git a/docs/hpc/09_ood/static/ood_filezilla_running.png b/docs/hpc/09_ood/static/ood_filezilla_running.png deleted file mode 100644 index 11e99f35a3..0000000000 Binary files a/docs/hpc/09_ood/static/ood_filezilla_running.png and /dev/null differ diff --git a/docs/hpc/09_ood/static/ood_filezilla_startup.png b/docs/hpc/09_ood/static/ood_filezilla_startup.png deleted file mode 100644 index 943d53c0ff..0000000000 Binary files a/docs/hpc/09_ood/static/ood_filezilla_startup.png and /dev/null differ diff --git a/docs/hpc/09_ood/static/ood_interactive_apps_alphafold2.png b/docs/hpc/09_ood/static/ood_interactive_apps_alphafold2.png deleted file mode 100644 index cb3bf226eb..0000000000 Binary files a/docs/hpc/09_ood/static/ood_interactive_apps_alphafold2.png and /dev/null differ diff --git a/docs/hpc/09_ood/static/ood_interactive_apps_igv.png b/docs/hpc/09_ood/static/ood_interactive_apps_igv.png deleted file mode 100644 index 0c29f86856..0000000000 Binary files a/docs/hpc/09_ood/static/ood_interactive_apps_igv.png and /dev/null differ diff --git a/docs/hpc/09_ood/static/ood_interactive_apps_matlab.png b/docs/hpc/09_ood/static/ood_interactive_apps_matlab.png deleted file mode 100644 index 8e19cea25c..0000000000 Binary files a/docs/hpc/09_ood/static/ood_interactive_apps_matlab.png and /dev/null differ diff --git a/docs/hpc/09_ood/static/ood_interactive_apps_rstudio.png b/docs/hpc/09_ood/static/ood_interactive_apps_rstudio.png deleted file mode 100644 index db40f44339..0000000000 Binary files a/docs/hpc/09_ood/static/ood_interactive_apps_rstudio.png and /dev/null differ diff --git a/docs/hpc/09_ood/static/ood_matlab_config.png b/docs/hpc/09_ood/static/ood_matlab_config.png index 7bef8c6853..3a0950138d 100644 Binary files a/docs/hpc/09_ood/static/ood_matlab_config.png and b/docs/hpc/09_ood/static/ood_matlab_config.png differ diff --git a/docs/hpc/09_ood/static/ood_matlab_in_queue.png b/docs/hpc/09_ood/static/ood_matlab_in_queue.png index 0b53745f0a..16cd6050ba 100644 Binary files a/docs/hpc/09_ood/static/ood_matlab_in_queue.png and b/docs/hpc/09_ood/static/ood_matlab_in_queue.png differ diff --git a/docs/hpc/09_ood/static/ood_matlab_prelaunch.png b/docs/hpc/09_ood/static/ood_matlab_prelaunch.png index 6a0b0f620a..8512161466 100644 Binary files a/docs/hpc/09_ood/static/ood_matlab_prelaunch.png and b/docs/hpc/09_ood/static/ood_matlab_prelaunch.png differ diff --git a/docs/hpc/09_ood/static/ood_matlab_running.png b/docs/hpc/09_ood/static/ood_matlab_running.png deleted file mode 100644 index 8aa2cf5705..0000000000 Binary files a/docs/hpc/09_ood/static/ood_matlab_running.png and /dev/null differ diff --git a/docs/hpc/09_ood/static/ood_spark_config.png b/docs/hpc/09_ood/static/ood_spark_config.png new file mode 100644 index 0000000000..e0d00cb1fe Binary files /dev/null and b/docs/hpc/09_ood/static/ood_spark_config.png differ diff --git a/docs/hpc/09_ood/static/ood_spark_in_queue.png b/docs/hpc/09_ood/static/ood_spark_in_queue.png new file mode 100644 index 0000000000..d0cb0e3ccd Binary files /dev/null and b/docs/hpc/09_ood/static/ood_spark_in_queue.png differ diff --git a/docs/hpc/09_ood/static/ood_spark_prelaunch.png b/docs/hpc/09_ood/static/ood_spark_prelaunch.png new file mode 100644 index 0000000000..61ebd79b7d Binary files /dev/null and b/docs/hpc/09_ood/static/ood_spark_prelaunch.png differ diff --git a/docs/hpc/09_ood/static/ood_stata_config.png b/docs/hpc/09_ood/static/ood_stata_config.png new file mode 100644 index 0000000000..523d31cd68 Binary files /dev/null and b/docs/hpc/09_ood/static/ood_stata_config.png differ diff --git a/docs/hpc/09_ood/static/ood_stata_example.png b/docs/hpc/09_ood/static/ood_stata_example.png new file mode 100644 index 0000000000..f98fc5bc7d Binary files /dev/null and b/docs/hpc/09_ood/static/ood_stata_example.png differ diff --git a/docs/hpc/09_ood/static/ood_stata_in_queue.png b/docs/hpc/09_ood/static/ood_stata_in_queue.png new file mode 100644 index 0000000000..f47e338366 Binary files /dev/null and b/docs/hpc/09_ood/static/ood_stata_in_queue.png differ diff --git a/docs/hpc/09_ood/static/ood_stata_prelaunch.png b/docs/hpc/09_ood/static/ood_stata_prelaunch.png new file mode 100644 index 0000000000..81f743ed46 Binary files /dev/null and b/docs/hpc/09_ood/static/ood_stata_prelaunch.png differ