Skip to content

refactor(notebooks/part0_python_intro/03_useful-std-library-modules.i… #167

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mnfienen merged 1 commit into from
Sep 19, 2025
Merged

refactor(notebooks/part0_python_intro/03_useful-std-library-modules.i… #167

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 63 additions & 46 deletions notebooks/part0_python_intro/03_useful-std-library-modules.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@
"# 03: Useful standard library modules\n",
"(pathlib, shutil, sys, os, subprocess, zipfile, etc.)\n",
"\n",
"These packages are part of the standard python library and provide very useful functionality for working with your operating system and files. This notebook will provide explore these packages and demonstrate some of their functionality. Online documentation is at https://docs.python.org/3/library/.\n",
"These packages are part of the standard python library and provide very useful functionality for working with your operating system and files. This notebook will provide these packages and demonstrate some of their functionality. Online documentation is at https://docs.python.org/3/library/.\n",
"\n",
"\n",
"#### Topics covered:\n",
"## Topics covered:\n",
"* **pathlib**:\n",
" * listing files\n",
" * creating, moving and deleting files\n",
Expand Down Expand Up @@ -62,7 +62,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Make a ``Path()`` object for the current folder"
"### Listing files\n",
"\n",
"#### Start by making a ``Path()`` object for the current folder"
]
},
{
Expand All @@ -75,13 +77,6 @@
"cwd"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Listing files"
]
},
{
"cell_type": "code",
"execution_count": null,
Expand Down Expand Up @@ -185,7 +180,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Create a new path for the data subfolder"
"### Creating files and folders\n",
"\n",
"#### make a ``Path`` object for a new subdirectory"
]
},
{
Expand All @@ -194,8 +191,8 @@
"metadata": {},
"outputs": [],
"source": [
"data_path = cwd / 'data'\n",
"data_path"
"new_folder = cwd / 'more_files'\n",
"new_folder"
]
},
{
Expand Down Expand Up @@ -235,28 +232,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### Creating files and folders\n",
"\n",
"#### make a new subdirectory"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"new_folder = cwd / 'more_files'\n",
"new_folder"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"new_folder.exists()"
"#### make the actual folder"
]
},
{
Expand Down Expand Up @@ -793,7 +769,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"**Note**: Generally, importing code using ``sys.path`` is considered bad practice, because \n",
"**Note**: Generally, importing code using ``sys.path`` is often considered bad practice, because \n",
"\n",
"* it can hide dependencies. \n",
"\n",
Expand All @@ -802,7 +778,9 @@
"\n",
"* importing code using ``sys.path`` is also sensitive to the location of the script relative to the path. If the script is moved or used on someone else's computer with a different file structure, it'll break.\n",
"\n",
"In general, [installing reusable code in a package is the best way to go](https://nsls-ii.github.io/scientific-python-cookiecutter/). Packages provide a framework for organizing, documenting, testing and sharing code in a way that is easily understood by others.\n",
"* this all said, sometimes using ``sys.path`` is expedient in reproducible workflows in that it can allow code to be consolidated and re-used across multiple scripts in various locations\n",
"\n",
"For code that is useful across multiple projects, [installing reusable code in a package can be the best way to go](https://learn.scientific-python.org/development/tutorials/). Packages provide a framework for organizing, documenting, testing and sharing code in a way that is easily understood by others.\n",
"\n",
"Whatever you do, avoid importing with an `*` (i.e. ``from mycode import *``) at all costs. This imports everything from the namespace of a module, which can lead to unintended consequences."
]
Expand All @@ -816,7 +794,7 @@
"\n",
"### Changing the current working directory\n",
"``pathlib`` doesn't do this. \n",
"Note: this can obviously lead to trouble in scripts, so should usually be avoided, but sometimes it is necessary."
"Note: this can obviously lead to trouble in scripts, so should usually be avoided, but sometimes it is necessary. In groundwater modeling workflows, for example, this can help keep flow and transport model files organized in separate folders."
]
},
{
Expand Down Expand Up @@ -912,6 +890,33 @@
"os.environ['CONDA_PREFIX']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Running system commands\n",
"`os.system` provides a limited way to run system commands. For more flexibility, use `subprocess` (below)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"os.system('ls -l')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# on Windows\n",
"os.system('dir')"
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand Down Expand Up @@ -943,7 +948,7 @@
"outputs": [],
"source": [
"# if on mac/unix\n",
"print(subprocess.run(['ls', '-l'], shell=True))"
"print(subprocess.run(['ls', '-l']))"
]
},
{
Expand All @@ -959,7 +964,7 @@
"metadata": {},
"outputs": [],
"source": [
"print(subprocess.run(['ls', '-l'], shell=True, cwd='..'))"
"print(subprocess.run(['ls', '-l'], cwd='..'))"
]
},
{
Expand All @@ -978,7 +983,7 @@
"source": [
"## ``zipfile`` — Work with ZIP archives\n",
"\n",
"#### zip up one of the files in data/"
"### zip up one of the files in data/"
]
},
{
Expand All @@ -995,7 +1000,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"#### now extract it"
"### now extract it"
]
},
{
Expand All @@ -1014,7 +1019,7 @@
"source": [
"## Testing Your Skills with a truly awful example:\n",
"\n",
"#### the problem:\n",
"### the problem:\n",
"Pretend that the file `data/fileio/netcdf_data.zip` contains some climate data (in the NetCDF format with the ``*.nc`` extension) that we downloaded. If you open `data/fileio/netcdf_data.zip`, you'll see that within a subfolder `zipped` are a bunch of additional subfolders, each for a different year. Within each subfolder is another zipfile. Within each of these zipfiles is yet another subfolder, inside of which is the actual data file we want (`prcp.nc`). "
]
},
Expand All @@ -1033,7 +1038,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"#### the goal:\n",
"### the goal:\n",
"To extract all of these `prcp.nc` files into a single folder, after renaming them with their respective years (obtained from their enclosing folders or zip files). e.g. \n",
"```\n",
"prcp_1980.nc\n",
Expand All @@ -1042,7 +1047,7 @@
"```\n",
"This will allow us to open them together as a dataset in `xarray` (more on that later). Does this sound awful? I'm not making this up. This is the kind of structure you get when downloading tiles of climate data with the [Daymet Tile Selection Tool](https://daymet.ornl.gov/gridded/)\n",
"\n",
"#### hint:\n",
"### hint:\n",
"you might find these functions helpful:\n",
"```\n",
"ZipFile.extractall\n",
Expand All @@ -1061,7 +1066,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"#### hint: start by using ``ZipFile.extractall()`` to extract all of the individual zip files from the main zip archive\n",
"### hint: start by using ``ZipFile.extractall()`` to extract all of the individual zip files from the main zip archive\n",
"This extracts the entire contents of the zip file to a designated folder"
]
},
Expand Down Expand Up @@ -1252,6 +1257,18 @@
"display_name": "pyclass",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.11"
}
},
"nbformat": 4,
Expand Down
Loading