Skip to content

Fix ch5 build hang: add network timeouts and reduce memory #51

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tyfolino merged 4 commits into from
Mar 16, 2026
Merged

Fix ch5 build hang: add network timeouts and reduce memory #51

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
4c0f7bf
Fix ch5 build hang: add network timeouts and reduce memory
tyfolino Mar 13, 2026
6ffc5e5
Re-raise data fetch errors instead of swallowing silently
tyfolino Mar 13, 2026
1635c12
Fix s3fs timeout config: use config_kwargs not client_kwargs
tyfolino Mar 13, 2026
4260b58
Restore 2-hour freshness check with proper error handling
tyfolino Mar 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 25 additions & 85 deletions notebooks/ch5_realtimeData.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@
"By the end of this chapter, you will produce an **interactive visualization** of MRMS imagery for your chosen region and product. If you wish to continue working with near real-time MRMS data beyond this notebook, there are three bonus challenges at the end of the notebook that encourage the user to further apply their skills. \n",
"\n",
"### Estimated Time\n",
"- **15 minutes** — Run the notebook and review the code. \n",
"- **30 minutes** — Build enough familiarity to reproduce the workflow independently and begin to tackle the bonus challenges.\n",
"- **15 minutes** \u2014 Run the notebook and review the code. \n",
"- **30 minutes** \u2014 Build enough familiarity to reproduce the workflow independently and begin to tackle the bonus challenges.\n",
"- **2 hours** - Complete all bonus steps and begin to integrate these concepts into your own workflow. \n"
]
},
Expand All @@ -44,7 +44,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## 📦 Imports"
"## \ud83d\udce6 Imports"
]
},
{
Expand Down Expand Up @@ -88,7 +88,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## 🌧️ About MRMS"
"## \ud83c\udf27\ufe0f About MRMS"
]
},
{
Expand All @@ -112,7 +112,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## ☁️ About AWS and NOAA's Open Data Dissemination Program"
"## \u2601\ufe0f About AWS and NOAA's Open Data Dissemination Program"
]
},
{
Expand All @@ -129,10 +129,7 @@
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Initialize the S3 filesystem as anonymous\n",
"aws = s3fs.S3FileSystem(anon=True)"
]
"source": "# Initialize the S3 filesystem as anonymous\naws = s3fs.S3FileSystem(anon=True, config_kwargs={'connect_timeout': 30, 'read_timeout': 60})"
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -164,7 +161,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## 🎯 Data selection\n",
"## \ud83c\udfaf Data selection\n",
"\n",
"For ease of use, I've integrated widgets (drop-down menus!) that allow you to make selections from AWS, and refined a selection of data variables as a demonstration. You can choose between the QC'd Merged Reflectivity Composite[^MRQC], a 12-hour multisensor QPE from Pass 2[^MSQPE], and the Probability of Severe Hail[^POSH]. \n",
"\n",
Expand All @@ -173,7 +170,7 @@
"[^MRQC]: **Merged Reflectivity Composite** \n",
" ![MRMS Visualization](images/QCReflectivity.png) \n",
" **Description:** The maximum reflectivity in a vertical column, from the merged product. \n",
" **Spatial Resolution:** 0.01º Latitude (~1.11 km) x 0.01º Longitude (~1.01 km at 25ºN and 0.73 km at 49ºN) \n",
" **Spatial Resolution:** 0.01\u00ba Latitude (~1.11 km) x 0.01\u00ba Longitude (~1.01 km at 25\u00baN and 0.73 km at 49\u00baN) \n",
" **Temporal Resolution:** 2 minutes \n",
" **AWS Variable:** \"MergedReflectivityQCComposite_00.50\" \n",
"\n",
Expand All @@ -187,7 +184,7 @@
"[^POSH]: **Probability of Severe Hail**\n",
" ![MRMS Visualization](images/POSH.png) \n",
" **Description:** The probability of 0.75-inch diameter hail. \n",
" **Spatial Resolution:** 0.01º Latitude (~1.11 km) x 0.01º Longitude (~1.01 km at 25ºN and 0.73 km at 49ºN) \n",
" **Spatial Resolution:** 0.01\u00ba Latitude (~1.11 km) x 0.01\u00ba Longitude (~1.01 km at 25\u00baN and 0.73 km at 49\u00baN) \n",
" **Temporal Resolution:** 2 minutes \n",
" **AWS Variable:** \"POSH_00.50\""
]
Expand Down Expand Up @@ -224,7 +221,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"🎉 Congratulations, you've made your data selection!"
"\ud83c\udf89 Congratulations, you've made your data selection!"
]
},
{
Expand Down Expand Up @@ -262,7 +259,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## 📡 Data request\n",
"## \ud83d\udce1 Data request\n",
"Now that you've made your variable selection, it's time to read in the data from AWS. First, we retrieve the current UTC datetime so that we can request files from today's S3 bucket."
]
},
Expand All @@ -289,14 +286,7 @@
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Query the S3 bucket for the available files that meet the criteria\n",
"try:\n",
" data_files = aws.ls(f'noaa-mrms-pds/{region}/{product}/{datestring}/', refresh=True) \n",
"except Exception as e:\n",
" print(f\"Error accessing S3 bucket: {e}\")\n",
" data_files = []"
]
"source": "# Query the S3 bucket for the available files that meet the criteria\ntry:\n data_files = aws.ls(f'noaa-mrms-pds/{region}/{product}/{datestring}/', refresh=True) \nexcept Exception as e:\n raise RuntimeError(f\"Error accessing S3 bucket: {e}\") from e"
},
{
"cell_type": "markdown",
Expand All @@ -310,27 +300,7 @@
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"if data_files:\n",
" # Choose the last file from S3 for the most recent data\n",
" most_recent_file = data_files[-1]\n",
"\n",
" # Check that the most recent file is within 2 hours of current time\n",
" timestamp_str = most_recent_file.split('_')[-1].replace('.grib2.gz', '')\n",
" dt = datetime.datetime.strptime(timestamp_str, \"%Y%m%d-%H%M%S\").replace(tzinfo=timezone.utc)\n",
" if abs((now - dt).total_seconds()) <= 120 * 60:\n",
" # Download file to memory, decompress from .gz, and read into xarray\n",
" try:\n",
" response = urllib.request.urlopen(f\"https://noaa-mrms-pds.s3.amazonaws.com/{most_recent_file[14:]}\")\n",
" compressed_file = response.read()\n",
"\n",
" with tempfile.NamedTemporaryFile(suffix=\".grib2\") as f:\n",
" f.write(gzip.decompress(compressed_file))\n",
" f.flush()\n",
" data = xr.load_dataarray(f.name, engine=\"cfgrib\", decode_timedelta=True)\n",
" except Exception as e:\n",
" print(f\"Failed to process {product}: {e}\")"
]
"source": "if data_files:\n # Choose the last file from S3 for the most recent data\n most_recent_file = data_files[-1]\n\n # Check that the most recent file is within 2 hours of current time\n timestamp_str = most_recent_file.split('_')[-1].replace('.grib2.gz', '')\n dt = datetime.datetime.strptime(timestamp_str, \"%Y%m%d-%H%M%S\").replace(tzinfo=timezone.utc)\n if abs((now - dt).total_seconds()) > 120 * 60:\n raise RuntimeError(f\"Most recent MRMS file is more than 2 hours old ({dt}). Data may not be current \u2014 check the MRMS S3 bucket for availability.\")\n\n # Download file to memory, decompress from .gz, and read into xarray\n try:\n response = urllib.request.urlopen(f\"https://noaa-mrms-pds.s3.amazonaws.com/{most_recent_file[14:]}\", timeout=60)\n compressed_file = response.read()\n\n with tempfile.NamedTemporaryFile(suffix=\".grib2\") as f:\n f.write(gzip.decompress(compressed_file))\n f.flush()\n data = xr.load_dataarray(f.name, engine=\"cfgrib\", decode_timedelta=True)\n # Decimate to reduce memory usage (~17x reduction, safe for display)\n data = data[::4, ::4]\n except Exception as e:\n raise RuntimeError(f\"Failed to load MRMS data for {product}: {e}\") from e"
},
{
"cell_type": "markdown",
Expand All @@ -350,7 +320,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## 🗺️ Visualization\n",
"## \ud83d\uddfa\ufe0f Visualization\n",
"Now that we have the data read into memory using xarray, it is quite simple to plot. Here, we use hvplot to make an interactive visualization that allows the user to zoom in to a region of interest and mouse over values to better understand the product's functionality over a specific region."
]
},
Expand All @@ -359,37 +329,7 @@
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Mask data for neater visualization\n",
"data = data.where(data > 0, np.nan)\n",
"\n",
"# Get the NWS Reflectivity colormap and normalize range\n",
"ref_norm, ref_cmap = ctables.registry.get_with_steps('NWSReflectivity', 5, 5)\n",
"\n",
"# Convert to hex colors for Bokeh\n",
"norm = Normalize(vmin=ref_norm.vmin, vmax=ref_norm.vmax)\n",
"hex_cmap = [ref_cmap(norm(val)) for val in range(ref_norm.vmin, ref_norm.vmax + 5, 5)]\n",
"hex_cmap = [mcls.to_hex(c) for c in hex_cmap]\n",
"\n",
"# Plot using hvplot\n",
"reflectivity_plot = data.hvplot.image(\n",
" x=\"longitude\", y=\"latitude\",\n",
" cmap=hex_cmap,\n",
" colorbar=True,\n",
" geo=True, \n",
" tiles=True, \n",
" alpha=0.7,\n",
" clim=(ref_norm.vmin, ref_norm.vmax),\n",
" title=f\"{product} - {pd.to_datetime(data.time.values).strftime('%b %d, %Y at %H:%M:%S')} UTC\",\n",
" frame_width=700,\n",
" frame_height=500,\n",
" xlabel='Longitude',\n",
" ylabel='Latitude',\n",
" tools=['hover']\n",
")\n",
"\n",
"reflectivity_plot"
]
"source": "# Mask data for neater visualization\ndata = data.where(data > 0, np.nan)\n\n# Get the NWS Reflectivity colormap and normalize range\nref_norm, ref_cmap = ctables.registry.get_with_steps('NWSReflectivity', 5, 5)\n\n# Convert to hex colors for Bokeh\nnorm = Normalize(vmin=ref_norm.vmin, vmax=ref_norm.vmax)\nhex_cmap = [ref_cmap(norm(val)) for val in range(ref_norm.vmin, ref_norm.vmax + 5, 5)]\nhex_cmap = [mcls.to_hex(c) for c in hex_cmap]\n\n# Plot using hvplot\nreflectivity_plot = data.hvplot.image(\n x=\"longitude\", y=\"latitude\",\n cmap=hex_cmap,\n colorbar=True,\n geo=True, \n tiles=True, \n alpha=0.7,\n clim=(ref_norm.vmin, ref_norm.vmax),\n title=f\"{product} - {pd.to_datetime(data.time.values).strftime('%b %d, %Y at %H:%M:%S')} UTC\",\n frame_width=700,\n frame_height=500,\n xlabel='Longitude',\n ylabel='Latitude',\n tools=['hover']\n)\n\nreflectivity_plot"
},
{
"cell_type": "markdown",
Expand All @@ -402,7 +342,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## 🏆 Bonus Challenges\n",
"## \ud83c\udfc6 Bonus Challenges\n",
"Congratulations on the completion of this notebook! You have successfully selected a region and product, queried the AWS S3 bucket, and visualized MRMS data in near real-time.\n",
"\n",
"If you'd like to continue this analysis, I've provided a couple of bonus challenges. Click on the drop-down menu to view the bonus challenge according to your desired level of difficulty. "
Expand All @@ -412,25 +352,25 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"```{dropdown} 🟢 Challenge (easy) -- make a new data selection\n",
"```{dropdown} \ud83d\udfe2 Challenge (easy) -- make a new data selection\n",
":class: tip\n",
"\n",
"Use the drop-down widgets in this notebook to plot a different product and region from your initial run!\n",
"\n",
"```{dropdown} 💡 Hint\n",
"```{dropdown} \ud83d\udca1 Hint\n",
"- Scroll up to the drop-down menus, make new region/product selections, and run all cells *below* the drop-down menus to see your new visualization!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```{dropdown} 🟡 Challenge (medium) -- plot a new variable from AWS\n",
"```{dropdown} \ud83d\udfe1 Challenge (medium) -- plot a new variable from AWS\n",
":class: warning\n",
"\n",
"Browse the [AWS S3 bucket](https://noaa-mrms-pds.s3.amazonaws.com/index.html) and the [NSSL Variable Table](https://www.nssl.noaa.gov/projects/mrms/operational/tables.php) and find an MRMS product that was not covered in this notebook. Alter the provided code to read in and plot your new variable!\n",
"\n",
"```{dropdown} 💡 Hints\n",
"```{dropdown} \ud83d\udca1 Hints\n",
"**Step-by-step:**\n",
"1. Delete the widget-generating cell in the \"Data selection\" section.\n",
"2. Hard-code the \"region\" and \"product\" variables with the *exact* strings that correspond to your data product on AWS. For example: \n",
Expand All @@ -448,12 +388,12 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"```{dropdown} 🔴 Challenge (difficult) -- create a cron job to update your MRMS plot hourly\n",
"```{dropdown} \ud83d\udd34 Challenge (difficult) -- create a cron job to update your MRMS plot hourly\n",
":class: danger\n",
"\n",
"Turn this notebook into a Python script, then use cron to create an updated plot from MRMS data every hour. Incorporate this plot into a web page, send it to your friend, or try it just for fun! \n",
"\n",
"```{dropdown} 💡 Hints\n",
"```{dropdown} \ud83d\udca1 Hints\n",
"**Step-by-step:**\n",
"1. Delete the widget-generating cell in the \"Data selection\" section.\n",
"2. Hard-code the \"region\" and \"product\" variables with the *exact* strings that correspond to your data product on AWS. For example: \n",
Expand Down Expand Up @@ -489,7 +429,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## 📚 Resources and references"
"## \ud83d\udcda Resources and references"
]
},
{
Expand All @@ -509,7 +449,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## 🛠️ Appendix\n",
"## \ud83d\udee0\ufe0f Appendix\n",
"If you'd prefer to plot these data as a static plot, below is some sample code to kickstart your plotting journey. "
]
},
Expand Down Expand Up @@ -653,4 +593,4 @@
},
"nbformat": 4,
"nbformat_minor": 4
}
}
Loading