OOD Tutorials

docs/hpc/09_ood/01_ood_intro.md

@@ -18,7 +18,7 @@ Under the **clusters** menu you can select the **Greene Shell Access** option to
 
 ![img](./static/open_ondemand_gif.gif)
 
-Please see our documentation on [Submitting Jobs](http://localhost:3000/rts-docs-dev/docs/hpc/submitting_jobs/slurm_submitting_jobs/) if you'd like detailed instructions.
+Please see our documentation on [Submitting Jobs](../05_submitting_jobs/01_slurm_submitting_jobs.md) if you'd like detailed instructions.
 
 **Interactive Applications**
 
@@ -63,4 +63,4 @@ Just click on the `Session ID` link and a tab will open with the contents of the
 
 #### From terminal
 
-If your session is no longer visible from within OOD you may still be able to find your logs via the terminal.  Simply [log into Greene](https://sites.google.com/nyu.edu/nyu-hpc/accessing-hpc) and `cd` to `/home/$USER/ondemand/data/sys/dashboard/batch_connect/sys/` and then `cd` into the directory for the app that you're interested in.  You should find the file `output.log` there.
+If your session is no longer visible from within OOD you may still be able to find your logs via the terminal.  Simply [log into Greene](../02_connecting_to_hpc/01_connecting_to_hpc.mdx) and `cd` to `/home/$USER/ondemand/data/sys/dashboard/batch_connect/sys/` and then `cd` into the directory for the app that you're interested in.  You should find the file `output.log` there.

docs/hpc/09_ood/02_CellACDC.mdx

@@ -0,0 +1,32 @@
+# Cell-ACDC in OOD
+
+[Cell-ACDC](https://cell-acdc.readthedocs.io) is a GUI-based Python framework for segmentation, tracking, cell cycle annotations and quantification of microscopy data.
+
+## Getting Started
+You can run Cell-ACDC in OOD by going to the URL [ood.hpc.nyu.edu](http://ood.hpc.nyu.edu) in your browser and selecting `Cell-ACDC` from the `Interactive Apps` pull-down menu at the top of the page.  Once you've used it and other interactive apps they'll show up on your home screen under the `Recently Used Apps` header.
+
+:::note
+Be aware that when you start from `Recently Used Apps` it will start with the same configuration that you used previously.  If you'd like to configure your Cell-ACDC session differently, you'll need to select it from the menu.
+:::
+
+## Configuration
+
+You can select the number of cores, amount of memory, and number of hours.
+
+![OOD Cell-ACDC Configuration](./static/ood_cellacdc_config.png)
+
+## Cell-ACDC running in OOD
+
+After you hit the `Launch` button you'll have to wait for the scheduler to find node(s) for you to run on:
+![OOD Cell-ACDC in queue](./static/ood_cellacdc_in_queue.png)
+
+Then you'll have a short wait for the Cell-ACDC itself to start up.<br />
+Once that happens you'll get one last page that will give you links to:
+-   open a terminal window on the compute node your Cell-ACDC session is running on
+-   go to the directory associated with your Session ID that stores output, config and other related files for your session
+-   make changes to compression and image qualtiy
+-   get a link that you can share that will allow others to view your Cell-ACDC session
+
+![Pre-launch Cell-ACDC OOD](./static/ood_cellacdc_prelaunch.png)
+
+Please click the `Launch Cell-ACDC` button and a Cell-ACDC window will open.

docs/hpc/09_ood/03_Dask.mdx

@@ -0,0 +1,113 @@
+# Dask in Jupyter Notebook in OOD
+
+[Dask](https://docs.dask.org/en/stable/) is a Python library for parallel and distributed computing.
+
+## Getting Started
+You can run Dask in a Jupyter Notebook in OOD by going to the URL [ood.hpc.nyu.edu](http://ood.hpc.nyu.edu) in your browser and selecting `DS-GA.1004 - Jupyter Dask` from the `Interactive Apps` pull-down menu at the top of the page.  Once you've used it and other interactive apps they'll show up on your home screen under the `Recently Used Apps` header.
+
+:::note
+Be aware that when you start from `Recently Used Apps` it will start with the same configuration that you used previously.  If you'd like to configure your Dask session differently, you'll need to select it from the menu.
+:::
+
+## Configuration
+
+You can select the Dask version, number of cores, amount of memory, root directory, number of hours, and optional Slurm options.
+
+![OOD Dask Configuration](./static/ood_dask_config.png)
+
+:::warning
+If you select to use `/home` as your root directory be careful not to go over your quota.  You can find your current usage with the `myquota` command.  Please see our [Storage documentation](../03_storage/01_intro_and_data_management.mdx) for details about your storage options.
+:::
+
+## Dask with Jupyter Notebook running in OOD
+
+After you hit the `Launch` button you'll have to wait for the scheduler to find node(s) for you to run on:
+![OOD Dask in queue](./static/ood_dask_in_queue.png)
+
+Then you'll have a short wait for Dask itself to start up.<br />
+Once that happens you'll get one last page that will give you links to:
+-   open a terminal window on the compute node your Dask session is running on
+-   go to the directory associated with your Session ID that stores output, config and other related files for your session
+
+![Pre-launch Dask OOD](./static/ood_dask_prelaunch.png)
+
+Please click the `Connect to Jupyter` button and a Jupyter window will open.
+
+## Dask Example
+
+Start a new Jupyter notebook with 4 cores, 16GB memory, and set your root directory to `/scratch`.  Enter the following code in the first cell and execute it by pressing the `Shift` and `Enter` keys at the same time.
+```python
+import os
+import pandas as pd
+import numpy as np
+import time
+
+# Create a directory for the large files
+output_dir = "tmp/large_data_files"
+os.makedirs(output_dir, exist_ok=True)
+
+num_files = 5  # Number of files to create
+rows_per_file = 10_000_000  # 10 million rows per file
+for i in range(num_files):
+    data = {
+        'col1': np.random.randint(0, 100, size=rows_per_file),
+        'value': np.random.rand(rows_per_file) * 100
+    }
+    df = pd.DataFrame(data)
+    df.to_csv(os.path.join(output_dir, f'data_{i}.csv'), index=False)
+print(f"{num_files} large CSV files created in '{output_dir}'.")
+
+import dask.dataframe as dd
+from dask.distributed import Client
+import time
+import os
+
+# Start a Dask client for distributed processing (optional but recommended)
+# This allows you to monitor the computation with the Dask dashboard
+client = Client(n_workers=4, threads_per_worker=2, memory_limit='16GB') # Adjust these as per your system resources
+print(client)
+
+# Load multiple CSV files into a Dask DataFrame
+# Dask will automatically partition and parallelize the reading of these files
+output_dir = '/scratch/rjy1/tmp/large_data_files'
+dask_df = dd.read_csv(os.path.join(output_dir, 'data_*.csv'))
+
+# Perform a calculation (e.g., calculate the mean of the 'value' column)
+# This operation will be parallelized across the available workers
+result_dask = dask_df['value'].mean()
+
+# Trigger the computation and measure the time
+start_time = time.time()
+computed_result_dask = result_dask.compute()
+end_time = time.time()
+
+print(f"Dask took {end_time - start_time} seconds to compute the mean across {num_files} files.")
+print(f"Result (Dask): {computed_result_dask}")
+
+import pandas as pd
+import time
+import os
+
+# Perform the same calculation sequentially with Pandas
+start_time_pandas = time.time()
+total_mean = 0
+total_count = 0
+for i in range(num_files):
+    df = pd.read_csv(os.path.join(output_dir, f'data_{i}.csv'))
+    total_mean += df['value'].sum()
+    total_count += len(df)
+computed_result_pandas = total_mean / total_count
+end_time_pandas = time.time()
+
+print(f"Pandas took {end_time_pandas - start_time_pandas} seconds to compute the mean across {num_files} files.")
+print(f"Result (Pandas): {computed_result_pandas}")
+```
+You should get output like:
+```
+5 large CSV files created in 'tmp/large_data_files'.
+<Client: 'tcp://127.0.0.1:45511' processes=4 threads=8, memory=59.60 GiB>
+Dask took 3.448112726211548 seconds to compute the mean across 5 files.
+Result (Dask): 50.010815178612596
+Pandas took 9.641847610473633 seconds to compute the mean across 5 files.
+Result (Pandas): 50.01081517861258
+```

docs/hpc/09_ood/04_Desktop.mdx

@@ -0,0 +1,32 @@
+# Desktop in OOD
+
+You can get a basic desktop interface to HPC resources.
+
+## Getting Started
+You can get a desktop in OOD by going to the URL [ood.hpc.nyu.edu](http://ood.hpc.nyu.edu) in your browser and selecting `Desktop` from the `Interactive Apps` pull-down menu at the top of the page.  Once you've used it and other interactive apps they'll show up on your home screen under the `Recently Used Apps` header.
+
+:::note
+Be aware that when you start from `Recently Used Apps` it will start with the same configuration that you used previously.  If you'd like to configure your Desktop session differently, you'll need to select it from the menu.
+:::
+
+## Configuration
+
+You can select the number of cores, amount of memory, GPU type (if any), number of hours, and optional Slurm options.
+
+![OOD Desktop Configuration](./static/ood_desktop_config.png)
+
+## Desktop running in OOD
+
+After you hit the `Launch` button you'll have to wait for the scheduler to find node(s) for you to run on:
+![OOD Desktop in queue](./static/ood_desktop_in_queue.png)
+
+Then you'll have a short wait for the Desktop itself to start up.<br />
+Once that happens you'll get one last page that will give you links to:
+-   open a terminal window on the compute node your Desktop session is running on
+-   go to the directory associated with your Session ID that stores output, config and other related files for your session
+-   make changes to compression and image qualtiy
+-   get a link that you can share that will allow others to view your Desktop session
+
+![Pre-launch Desktop OOD](./static/ood_desktop_prelaunch.png)
+
+Please click the `Launch Desktop` button and a Desktop window will open.

docs/hpc/09_ood/07_igv.mdx → docs/hpc/09_ood/05_igv.mdx

@@ -1,4 +1,4 @@
-# Integrative Genomics Viewer (IGV) 
+# Integrative Genomics Viewer (IGV) in OOD
 
 The IGV is a high-performance, easy-to-use, interactive tool for the visual exploration of genomic data.
 
@@ -7,14 +7,12 @@ Please see the following links for details:
 -   [Tutorial Videos](https://www.youtube.com/channel/UCb5W5WqauDOwubZHb-IA_rA)
 
 ## Getting Started
-You can run IGV in OOD by going to the URL [ood.hpc.nyu.edu](http://ood.hpc.nyu.edu) in your browser and selecting `IGV` from the `Interactive Apps` pull-down menu at the top of the page.  As you can see below, once you've used it and other interactive apps they'll show up on your home screen under the `Recently Used Apps` header.
+You can run IGV in OOD by going to the URL [ood.hpc.nyu.edu](http://ood.hpc.nyu.edu) in your browser and selecting `IGV` from the `Interactive Apps` pull-down menu at the top of the page.  Once you've used it and other interactive apps they'll show up on your home screen under the `Recently Used Apps` header.
 
 :::note
 Be aware that when you start from `Recently Used Apps` it will start with the same configuration that you used previously.  If you'd like to configure your IGV session differently, you'll need to select it from the menu.
 :::
 
-![OOD Interactive Apps menu IGV](./static/ood_interactive_apps_igv.png)
-
 ## Configuration
 
 You can select the number or cores, amount of memory, amount of time, and optional Slurm options.

docs/hpc/09_ood/09_jbrowse_genome_browser.mdx → docs/hpc/09_ood/06_jbrowse_genome_browser.mdx

@@ -1,4 +1,4 @@
-# JBrowse Genome Browser
+# JBrowse Genome Browser in OOD
 
 [JBrowse](https://jbrowse.org/jbrowse1.html) is a web-based genome browser for visualizing genomic features in common file formats, such as variants (VCF), genes (GFF3, BigBed) and gene expression (BigWig), and sequence alignments (BAM, CRAM, and GFF3).
 

docs/hpc/09_ood/02_jupyter_with_conda_singularity.mdx → docs/hpc/09_ood/07_jupyter_with_conda_singularity.mdx

@@ -1,4 +1,4 @@
-# Jupyter with Conda/Singularity in OOD
+# Jupyter Notebook with Conda/Singularity in OOD
 
 ## OOD + Singularity + conda
 This page describes how to use your Singularity with conda environment in OOD GUI at Greene. 

docs/hpc/09_ood/03_matlab.mdx → docs/hpc/09_ood/08_matlab_proxy.mdx

@@ -1,17 +1,15 @@
-# Matlab in OOD
+# Matlab-Proxy in OOD
 
 ## Getting Started
-You can run Matlab in OOD by going to the URL [ood.hpc.nyu.edu](http://ood.hpc.nyu.edu) in your browser and selecting `MATLAB` from the `Interactive Apps` pull-down menu at the top of the page.  As you can see below, once you've used it and other interactive apps they'll show up on your home screen under the `Recently Used Apps` header.
+You can run Matlab-Proxy in OOD by going to the URL [ood.hpc.nyu.edu](http://ood.hpc.nyu.edu) in your browser and selecting `MATLAB` from the `Interactive Apps` pull-down menu at the top of the page.  Once you've used it and other interactive apps they'll show up on your home screen under the `Recently Used Apps` header.
 
 :::note
 Be aware that when you start from `Recently Used Apps` it will start with the same configuration that you used previously.  If you'd like to configure your Matlab session differently, you'll need to select it from the menu.
 :::
 
-![OOD Interactive Apps menu](./static/ood_interactive_apps_matlab.png)
-
 ## Configuration
 
-You can select the version of Matlab to use, the number or cores, amount of memory, GPU type (if any), amount of time, and optional Slurm options.
+You can select the version of Matlab to use, the number or cores, amount of memory, GPU type (if any), amount of time, account, and optional Slurm options.
 
 ![OOD Matlab Configuration](./static/ood_matlab_config.png)
 
@@ -47,13 +45,10 @@ After you hit the `Launch` button you'll have to wait for the scheduler to find
 ![OOD Matlab in queue](./static/ood_matlab_in_queue.png)
 
 Then you'll have a short wait for Matlab itself to start up.<br />
-Once that happens you'll get one last form that will allow you to:
--   make changes to compression and image qualtiy
+Once that happens you'll get a form that will allow you to:
 -   open a terminal window on the compute node your Matlab session is running on
--   get a link that you can share that will allow others to view your Matlab session
+-   go to the directory associated with your Session ID that stores output, config and other related files for your session
 
 ![Pre-launch matlab OOD](./static/ood_matlab_prelaunch.png)
 
-Then after you hit the `Launch Matlab` button you'll have the familiar Matlab Desktop to use.
-
-![OOD Matlab Running](./static/ood_matlab_running.png)
+Then after you hit the `Connect to Matlab` button you'll have the familiar Matlab Desktop to use.