revise lab_explore_data and solutions

Author: fabridamicelli

book/_quarto.yml

@@ -50,7 +50,7 @@ book:
     - 081_pandas.ipynb
     - 09_plotting.ipynb
     - 10_testing.ipynb
-    - lab_explore_data_exercises.ipynb
+    - lab_explore_data.ipynb
     - 11_cli.ipynb
     - 13_tools.ipynb
     - 051_oop.ipynb

book/lab_explore_data.ipynb

@@ -0,0 +1,168 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "0076254c-d77a-4333-9c71-807d9c680054",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "title: \"Lab: Exploring a Dataset\"\n",
+    "toc: true\n",
+    "output-file: lab_explore_dataset.html\n",
+    "---"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "86550260-7939-4970-a420-5eb24e8158c2",
+   "metadata": {},
+   "source": [
+    "We want to take a look at this real-world dataset: [https://github.com/OpenNeuroDatasets/ds005420](https://github.com/OpenNeuroDatasets/ds005420)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "931fe244-be1a-4935-9e18-081f3ac4f08c",
+   "metadata": {},
+   "source": [
+    "## Download data\n",
+    "\n",
+    "### Clone the repository \n",
+    "Inside the `/pycourse/data` folder:\n",
+    "```bash\n",
+    "git clone https://github.com/OpenNeuroDatasets/ds005420\n",
+    "```\n",
+    "\n",
+    "### Install `git-annex`\n",
+    "The files with the actual data are not there, but we have the references to them so that we can pull them down.\n",
+    "We will need the tool [git-annex tool](https://git-annex.branchable.com/install/).\n",
+    "\n",
+    "\n",
+    "### Pull the data\n",
+    "Open a terminal inside `ds005420` and run:\n",
+    "```bash\n",
+    "git-annex get .\n",
+    "```\n",
+    "You should see a progress dialog showing `...from s3-PUBLIC...`  \n",
+    "After that, you're ready to go with the exercises."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2f3d879b-f266-4b1f-96eb-cec0e3c4d2b2",
+   "metadata": {},
+   "source": [
+    ":::{ .callout-tip }\n",
+    "Jupyter Notebooks are ideal for this kind of exploratory tasks.\n",
+    "Make a directory called `/python/notebooks` and open there a jupyter lab instance.\n",
+    "Having the notebooks there will help us keeping things tidy for later reproducibility of our workflow.\n",
+    ":::"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8570f8a3-9719-4978-a648-7b5971b8062d",
+   "metadata": {},
+   "source": [
+    "## Explore files\n",
+    "1) List only the sub-directories in path.  \n",
+    "2) List only the sub-directories with subject data.\n",
+    "3) Write a function that lists sub-directories with subject data."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3ebcd69a-5222-4514-9531-7cdf54f782dd",
+   "metadata": {},
+   "source": [
+    "## Validating the data\n",
+    "We will start by making sure our data/metadata contains the information we expect at a high level.\n",
+    "\n",
+    "1) Write a unit test (inside `/pycourse/tests/test_data.py`) to make sure the number of subject sub-directories corresponds to actual the number of subjects. **Hint:** Look at the metadata.\n",
+    "2) Verify that all subject directories have a eeg sub-directory.  \n",
+    "3) Verify that all data in a subject directories matches with the subject number.  \n",
+    "4) Assert that EEG data for all subjects was taken using 20 channels and sampling frequency 500.  \n",
+    "5) (Optional) Write a file (`discarded_subjects.txt`) with the subject numbers that do not match that criterion.  "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3af93b8f-e20d-42f8-8c17-0830ba14c8b6",
+   "metadata": {},
+   "source": [
+    "## Exploratory data analysis \n",
+    "Now we want to look at the data.\n",
+    "We find that the data is in a particular format `.edf` that we cannot directly read in python.  \n",
+    "**Hint:**\n",
+    "We need to install a third-party library `mne` to read `.edf` files.  \n",
+    "You can check out the [library documentation here](https://mne.tools/dev/)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "de6b3872-f101-4fe1-ac5c-079d7a13887d",
+   "metadata": {},
+   "source": [
+    ":::{ .callout-tip }\n",
+    "It's a *very* good idea to first take a look at the documentation of a tool before installing it.\n",
+    "Executing someone else's code is a potential risk so you should try to find out if you can actually trust the source.\n",
+    ":::"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bb817e5c-3096-4d05-9120-ba7b05844403",
+   "metadata": {},
+   "source": [
+    "1) Plot one time series.  \n",
+    "2) Plot all time series with labels according to channel name.  \n",
+    "3) Plot the channels that start with \"T\" and \"O\".  \n",
+    "4) Plot a correlation plot of the \"T\" and \"O\" channels as a heatmap.\n",
+    "5) Plot a histogram of `RecordingDuration` across all subjects.  "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "196f319e-51a8-48bd-91f0-931978a5ec04",
+   "metadata": {},
+   "source": [
+    "## Process data\n",
+    "After having taken this quick look at the data, we want to start processing the data.\n",
+    "\n",
+    "1) Clean the column names removing \"EEG\", eg \"EEG C4-A1A2\" -> \"C4-A1A2\"\n",
+    "2) Substract the mean from each channel  \n",
+    "3) Plot correlation matrix of all-vs-all channels. **Hint:** Look at seaborn documentation on heatmaps.\n",
+    "4) Save the correlation plot as vector graphics."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9b7edbe2-0c6f-4dd1-bf92-74f080b2e00f",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}