New tutorials and polish (#23)

- Add / update new tutorials formatted as Juypter notbooks
- Add "scripts to rule them all" and used the `em-core` scripts as a template
- Updated and define S3 client
- Added persistent system environment variable instructions
- README Updates
- Replaced static links to original repo with dynamic links for this repo
- Improved organization
- Various markdown linting

Co-authored-by: Ryan Earley <66645596+rearley516@users.noreply.github.com>
Co-authored-by: Nadia Dimitrova <54898993+dnadia@users.noreply.github.com>
Co-authored-by: imgbot[bot] <31301654+imgbot[bot]@users.noreply.github.com>

Author: 4 people

README.md

@@ -1,45 +1,67 @@
-# Tutorials Guide
+# LADI Tutorials
+
+Tutorials for the Low Altitude Disaster Imagery (LADI) dataset. This tutorial was originally forked from a [Penn State Learning Factory](https://www.lf.psu.edu/) capstone project.
+
+- [LADI Tutorials](#ladi-tutorials)
+  - [Point of Contact](#point-of-contact)
+  - [Initial Setup](#initial-setup)
+    - [Persistent System Environment Variables](#persistent-system-environment-variables)
+    - [Scripts](#scripts)
+  - [Tutorials - Accessing the Dataset](#tutorials---accessing-the-dataset)
+  - [Tutorials - Metadata Analysis](#tutorials---metadata-analysis)
+    - [Tutorials - Machine Learning](#tutorials---machine-learning)
+  - [Distribution Statement](#distribution-statement)
 
-Tutorial for the Low Altitude Disaster Imagery (LADI) dataset. This tutorial was originally forked from a [Penn State Learning Factory](https://www.lf.psu.edu/) capstone project
+## Point of Contact
 
-- [Tutorials Guide](#tutorials-guide)
-  - [Getting Started](#getting-started)
-  - [Clean and Validate LADI Dataset](#clean-and-validate-ladi-dataset)
-  - [PyTorch Data Loading](#pytorch-data-loading)
-  - [Train and Test A Classifier](#train-and-test-a-classifier)
-  - [Fine Tuning Torchvision Models](#fine-tuning-torchvision-models)
-  - [Distribution Statement](#distribution-statement)
+We encourage the use of the [GitHub Issues](https://guides.github.com/features/issues/) but when email is required, please contact the administrators at [ladi-dataset-admin@mit.edu](mailto:ladi-dataset-admin@mit.edu). As the public safety and computer vision communities adopt the dataset, a separate mailing list for development may be created.
 
-## Getting Started
+## Initial Setup
 
-[Getting Started](./Tutorials/Get_Started.md)
+This section specifies the run order and requirements for the initial setup the repository. Other repositories in this organization may be reliant upon this setup being completed.
 
-This documentation is about installing AWS tools and configuring AWS environment to download LADI dataset and load dataset in Python locally and remotely.
+### Persistent System Environment Variables
 
-## Clean and Validate LADI Dataset
+Immediately after cloning this repository, [create a persistent system environment](https://superuser.com/q/284342/44051) variable titled `LADI_DIR_TUTORIAL` with a value of the full path to this repository root directory.
 
-[Clean and Validate LADI Dataset](./Tutorials/Clean_Validate.md)
+On unix there are many ways to do this, here is an example using [`/etc/profile.d`](https://unix.stackexchange.com/a/117473). Create a new file `ladi-env.sh` using `sudo vi /etc/profile.d/ladi-env.sh` and add the command to set the variable:
 
-This documentation is about clean the LADI dataset. For this project, we have only extracted 2000 images for training.
+```bash
+export LADI_DIR_TUTORIAL=PATH TO /ladi-tutorial
+```
 
-## PyTorch Data Loading
+You can confirm `LADI_DIR_TUTORIAL` was set in unix by inspecting the output of `env`.
 
-[PyTorch Data Loading](./Tutorials/Pytorch_Data_Load.md)
+### Scripts
 
-This documentation is about loading LADI dataset in PyTorch framework including examples of writing custom `Dataset`, `Transforms` and `Dataloader`.
+This is a set of boilerplate scripts describing the [normalized script pattern that GitHub uses in its projects](https://github.blog/2015-06-30-scripts-to-rule-them-all/). The [GitHub Scripts To Rule Them All](https://github.com/github/scripts-to-rule-them-all) was used as a template. Refer to the [script directory README](./script/README.md) for more details.
+
+You will need to run these scripts in this order to download package dependencies and download all of the necessary data to get you started.
+
+## Tutorials - Accessing the Dataset
 
-## Train and Test A Classifier
+A set of tutorials focused on installing AWS tools and configuring AWS environment to download LADI dataset and load dataset in Python locally and remotely. There is also a short tutorial on how to clean and validate data.
 
-[Train and Test A Classifier](./Tutorials/Train_Test_Classifier.md)
+- [Getting Started](./tutorials/Get_Started.md)
+- [Clean and Validate LADI Dataset](./tutorials/Clean_Validate.md)
 
-This documentation is about training and testing a classifier model using Convolutional Neural Network (CNN) from scratch.
+## Tutorials - Metadata Analysis
 
-## Fine Tuning Torchvision Models
+A set of tutorials are Jupyter Python 3.X notebooks that demonstrate on how to perform geospatial analysis by enhancing the LADI metadata with third party GIS information. One tutorial identifies the number of images taken within an administrative boundary (e.g. USA states) and assigns each state a color based on the number of images taken. The other tutorial filters images based on an specific annotation and performs various geospatial measurements on this subset.
 
-[Fine Tuning Torchvision Models](./Tutorials/Fine_Tune_Torchvision_Models.md)
+- [ISO-3166-2 Administrative Boundaries](./tutorials/Geospatial-Hurricane-Analysis.ipynb)
+- [Geospatial Hurricane Analysis](./tutorials/Geospatial-Hurricane-Analysis.ipynb)
 
-This documentation is about training and testing a classifier model using pre-trained ResNet and AlexNet.
+### Tutorials - Machine Learning
+
+These tutorials focus on how to training and testing a classifier model using Convolutional Neural Network (CNN) from scratch or using pre-trained ResNet and AlexNet.
+
+- [PyTorch Data Loading](./tutorials/Pytorch_Data_Load.md)
+- [Train and Test A Classifier](./tutorials/Train_Test_Classifier.md)
+- [Fine Tuning Torchvision Models](./tutorials/Fine_Tune_Torchvision_Models.md)
+
+This documentation is about loading LADI dataset in PyTorch framework including examples of writing custom `Dataset`, `Transforms` and `Dataloader`.
 
 ## Distribution Statement
 
-[BSD -Clause License](https://github.com/LADI-Dataset/ladi-tutorial/blob/master/LICENSE)
+[BSD 3-Clause License](LICENSE)

Tutorials/Model Script/README.md

@@ -0,0 +1,19 @@
+# Census-AHS
+The Census' American Housing Survey data provides information about the physical costs and conditions of homes, characteristics of the people living in these houses, and characteristics for disaster response of more than 60,000 Americans
+
+## Download Instructions
+
+### Script (Recommended)
+
+[`script/setup.sh`](../../script/setup.sh) is used to set up the project in an initial state. It will download and extract the data.
+
+### Manual
+
+Although not recommended, the data can be downloaded manually:
+
+1. Go to the US-Census National public use file page: [Census](https://www.census.gov/programs-surveys/ahs/data/2017/ahs-2017-public-use-file--puf-/ahs-2017-national-public-use-file--puf-.html)
+2. download the AHS 2017 National PUF v3.0 CSV (or the latest version)
+
+## Distribution Statement
+
+[BSD -Clause License](https://github.com/LADI-Dataset/ladi-tutorial/blob/master/LICENSE)

data/Census-AHS/README.md

@@ -0,0 +1,20 @@
+# Census-CBSA
+
+Census-CBSA defines US Metropolitan and Micropolitan statistical areas. This Census data is provided as a geospatial-enabled file format.
+
+## Download Instructions
+
+### Script (Recommended)
+
+[`script/setup.sh`](../../script/setup.sh) is used to set up the project in an initial state. It will download and extract the data.
+
+### Manual
+
+Although not recommended, the data can be downloaded manually:
+
+1. Go to the US-Census Bureau open data website: [US-CBSA](https://catalog.data.gov/dataset/tiger-line-shapefile-2019-nation-u-s-current-metropolitan-statistical-area-micropolitan-statist)
+2. download the latest version of the Shapefile Zip File
+
+## Distribution Statement
+
+[BSD -Clause License](https://github.com/LADI-Dataset/ladi-tutorial/blob/master/LICENSE)

data/Census-CBSA/README.md

@@ -0,0 +1,21 @@
+# Census-State
+
+Census-State defines statistical areas for all US-States. This Census data is provided as a geospatial-enabled file format.
+
+
+## Download Instructions
+
+### Script (Recommended)
+
+[`script/setup.sh`](../../script/setup.sh) is used to set up the project in an initial state. It will download and extract the data.
+
+### Manual
+
+Although not recommended, the data can be downloaded manually:
+
+1. Go to the US-Census Bureau open data website: [US-State](https://catalog.data.gov/dataset/tiger-line-shapefile-2017-nation-u-s-current-state-and-equivalent-national)
+2. download the latest version of the Shapefile Zip File
+
+## Distribution Statement
+
+[BSD -Clause License](https://github.com/LADI-Dataset/ladi-tutorial/blob/master/LICENSE)

data/Census-State/README.md

@@ -0,0 +1,16 @@
+# FAA Airports
+
+Airport defines area on land or water intended to be used either wholly or in part for the arrival; departure and surface movement of aircraft/helicopters. This airport data is provided as a vector geospatial-enabled file format.
+
+## Download Instructions
+
+### Script (Recommended)
+
+[`script/setup.sh`](../../script/setup.sh) is used to set up the project in an initial state. It will download and extract the data.
+
+### Manual
+
+Although not recommended, the data can be downloaded manually:
+
+1. Go to the FAA open data website: [Airports](https://ais-faa.opendata.arcgis.com/datasets/e747ab91a11045e8b3f8a3efd093d3b5_0)
+2. Select shapefile from the Download drop down

data/FAA-Airports/README.md

@@ -0,0 +1,17 @@
+# Natural-Earth
+The Natural Earth file is a comprehensive map of the world and it's administrative boundaries (states, fips, countries). This map is provided as a geospatial-enabled file format.
+
+### Script (Recommended)
+
+[`script/setup.sh`](../../script/setup.sh) is used to set up the project in an initial state. It will download and extract the data.
+
+### Manual
+
+Although not recommended, the data can be downloaded manually:
+
+1. Go to the Natural Earth download page: [Natural-Earth](https://www.naturalearthdata.com/downloads/10m-cultural-vectors/10m-admin-1-states-provinces/)
+2. Download states and provinces version 4.1.0 (or the latest version)
+
+## Distribution Statement
+
+[BSD -Clause License](https://github.com/LADI-Dataset/ladi-tutorial/blob/master/LICENSE)

data/Natural-Earth/README.md

@@ -1,6 +1,6 @@
-# Tutorials
+# Data
 
-Default directory for tutorials markdowns.
+Default directory for sample data used by tutorials.
 
 ## Distribution Statement