diff --git a/notebooks/ch5_realtimeData.ipynb b/notebooks/ch5_realtimeData.ipynb index 4f7fa784b..05268dad8 100644 --- a/notebooks/ch5_realtimeData.ipynb +++ b/notebooks/ch5_realtimeData.ipynb @@ -28,8 +28,8 @@ "By the end of this chapter, you will produce an **interactive visualization** of MRMS imagery for your chosen region and product. If you wish to continue working with near real-time MRMS data beyond this notebook, there are three bonus challenges at the end of the notebook that encourage the user to further apply their skills. \n", "\n", "### Estimated Time\n", - "- **15 minutes** โ€” Run the notebook and review the code. \n", - "- **30 minutes** โ€” Build enough familiarity to reproduce the workflow independently and begin to tackle the bonus challenges.\n", + "- **15 minutes** \u2014 Run the notebook and review the code. \n", + "- **30 minutes** \u2014 Build enough familiarity to reproduce the workflow independently and begin to tackle the bonus challenges.\n", "- **2 hours** - Complete all bonus steps and begin to integrate these concepts into your own workflow. \n" ] }, @@ -44,7 +44,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## ๐Ÿ“ฆ Imports" + "## \ud83d\udce6 Imports" ] }, { @@ -88,7 +88,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## ๐ŸŒง๏ธ About MRMS" + "## \ud83c\udf27\ufe0f About MRMS" ] }, { @@ -112,7 +112,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## โ˜๏ธ About AWS and NOAA's Open Data Dissemination Program" + "## \u2601\ufe0f About AWS and NOAA's Open Data Dissemination Program" ] }, { @@ -129,10 +129,7 @@ "execution_count": null, "metadata": {}, "outputs": [], - "source": [ - "# Initialize the S3 filesystem as anonymous\n", - "aws = s3fs.S3FileSystem(anon=True)" - ] + "source": "# Initialize the S3 filesystem as anonymous\naws = s3fs.S3FileSystem(anon=True, config_kwargs={'connect_timeout': 30, 'read_timeout': 60})" }, { "cell_type": "markdown", @@ -164,7 +161,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## ๐ŸŽฏ Data selection\n", + "## \ud83c\udfaf Data selection\n", "\n", "For ease of use, I've integrated widgets (drop-down menus!) that allow you to make selections from AWS, and refined a selection of data variables as a demonstration. You can choose between the QC'd Merged Reflectivity Composite[^MRQC], a 12-hour multisensor QPE from Pass 2[^MSQPE], and the Probability of Severe Hail[^POSH]. \n", "\n", @@ -173,7 +170,7 @@ "[^MRQC]: **Merged Reflectivity Composite** \n", " ![MRMS Visualization](images/QCReflectivity.png) \n", " **Description:** The maximum reflectivity in a vertical column, from the merged product. \n", - " **Spatial Resolution:** 0.01ยบ Latitude (~1.11 km) x 0.01ยบ Longitude (~1.01 km at 25ยบN and 0.73 km at 49ยบN) \n", + " **Spatial Resolution:** 0.01\u00ba Latitude (~1.11 km) x 0.01\u00ba Longitude (~1.01 km at 25\u00baN and 0.73 km at 49\u00baN) \n", " **Temporal Resolution:** 2 minutes \n", " **AWS Variable:** \"MergedReflectivityQCComposite_00.50\" \n", "\n", @@ -187,7 +184,7 @@ "[^POSH]: **Probability of Severe Hail**\n", " ![MRMS Visualization](images/POSH.png) \n", " **Description:** The probability of 0.75-inch diameter hail. \n", - " **Spatial Resolution:** 0.01ยบ Latitude (~1.11 km) x 0.01ยบ Longitude (~1.01 km at 25ยบN and 0.73 km at 49ยบN) \n", + " **Spatial Resolution:** 0.01\u00ba Latitude (~1.11 km) x 0.01\u00ba Longitude (~1.01 km at 25\u00baN and 0.73 km at 49\u00baN) \n", " **Temporal Resolution:** 2 minutes \n", " **AWS Variable:** \"POSH_00.50\"" ] @@ -224,7 +221,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "๐ŸŽ‰ Congratulations, you've made your data selection!" + "\ud83c\udf89 Congratulations, you've made your data selection!" ] }, { @@ -262,7 +259,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## ๐Ÿ“ก Data request\n", + "## \ud83d\udce1 Data request\n", "Now that you've made your variable selection, it's time to read in the data from AWS. First, we retrieve the current UTC datetime so that we can request files from today's S3 bucket." ] }, @@ -289,14 +286,7 @@ "execution_count": null, "metadata": {}, "outputs": [], - "source": [ - "# Query the S3 bucket for the available files that meet the criteria\n", - "try:\n", - " data_files = aws.ls(f'noaa-mrms-pds/{region}/{product}/{datestring}/', refresh=True) \n", - "except Exception as e:\n", - " print(f\"Error accessing S3 bucket: {e}\")\n", - " data_files = []" - ] + "source": "# Query the S3 bucket for the available files that meet the criteria\ntry:\n data_files = aws.ls(f'noaa-mrms-pds/{region}/{product}/{datestring}/', refresh=True) \nexcept Exception as e:\n raise RuntimeError(f\"Error accessing S3 bucket: {e}\") from e" }, { "cell_type": "markdown", @@ -310,27 +300,7 @@ "execution_count": null, "metadata": {}, "outputs": [], - "source": [ - "if data_files:\n", - " # Choose the last file from S3 for the most recent data\n", - " most_recent_file = data_files[-1]\n", - "\n", - " # Check that the most recent file is within 2 hours of current time\n", - " timestamp_str = most_recent_file.split('_')[-1].replace('.grib2.gz', '')\n", - " dt = datetime.datetime.strptime(timestamp_str, \"%Y%m%d-%H%M%S\").replace(tzinfo=timezone.utc)\n", - " if abs((now - dt).total_seconds()) <= 120 * 60:\n", - " # Download file to memory, decompress from .gz, and read into xarray\n", - " try:\n", - " response = urllib.request.urlopen(f\"https://noaa-mrms-pds.s3.amazonaws.com/{most_recent_file[14:]}\")\n", - " compressed_file = response.read()\n", - "\n", - " with tempfile.NamedTemporaryFile(suffix=\".grib2\") as f:\n", - " f.write(gzip.decompress(compressed_file))\n", - " f.flush()\n", - " data = xr.load_dataarray(f.name, engine=\"cfgrib\", decode_timedelta=True)\n", - " except Exception as e:\n", - " print(f\"Failed to process {product}: {e}\")" - ] + "source": "if data_files:\n # Choose the last file from S3 for the most recent data\n most_recent_file = data_files[-1]\n\n # Check that the most recent file is within 2 hours of current time\n timestamp_str = most_recent_file.split('_')[-1].replace('.grib2.gz', '')\n dt = datetime.datetime.strptime(timestamp_str, \"%Y%m%d-%H%M%S\").replace(tzinfo=timezone.utc)\n if abs((now - dt).total_seconds()) > 120 * 60:\n raise RuntimeError(f\"Most recent MRMS file is more than 2 hours old ({dt}). Data may not be current \u2014 check the MRMS S3 bucket for availability.\")\n\n # Download file to memory, decompress from .gz, and read into xarray\n try:\n response = urllib.request.urlopen(f\"https://noaa-mrms-pds.s3.amazonaws.com/{most_recent_file[14:]}\", timeout=60)\n compressed_file = response.read()\n\n with tempfile.NamedTemporaryFile(suffix=\".grib2\") as f:\n f.write(gzip.decompress(compressed_file))\n f.flush()\n data = xr.load_dataarray(f.name, engine=\"cfgrib\", decode_timedelta=True)\n # Decimate to reduce memory usage (~17x reduction, safe for display)\n data = data[::4, ::4]\n except Exception as e:\n raise RuntimeError(f\"Failed to load MRMS data for {product}: {e}\") from e" }, { "cell_type": "markdown", @@ -350,7 +320,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## ๐Ÿ—บ๏ธ Visualization\n", + "## \ud83d\uddfa\ufe0f Visualization\n", "Now that we have the data read into memory using xarray, it is quite simple to plot. Here, we use hvplot to make an interactive visualization that allows the user to zoom in to a region of interest and mouse over values to better understand the product's functionality over a specific region." ] }, @@ -359,37 +329,7 @@ "execution_count": null, "metadata": {}, "outputs": [], - "source": [ - "# Mask data for neater visualization\n", - "data = data.where(data > 0, np.nan)\n", - "\n", - "# Get the NWS Reflectivity colormap and normalize range\n", - "ref_norm, ref_cmap = ctables.registry.get_with_steps('NWSReflectivity', 5, 5)\n", - "\n", - "# Convert to hex colors for Bokeh\n", - "norm = Normalize(vmin=ref_norm.vmin, vmax=ref_norm.vmax)\n", - "hex_cmap = [ref_cmap(norm(val)) for val in range(ref_norm.vmin, ref_norm.vmax + 5, 5)]\n", - "hex_cmap = [mcls.to_hex(c) for c in hex_cmap]\n", - "\n", - "# Plot using hvplot\n", - "reflectivity_plot = data.hvplot.image(\n", - " x=\"longitude\", y=\"latitude\",\n", - " cmap=hex_cmap,\n", - " colorbar=True,\n", - " geo=True, \n", - " tiles=True, \n", - " alpha=0.7,\n", - " clim=(ref_norm.vmin, ref_norm.vmax),\n", - " title=f\"{product} - {pd.to_datetime(data.time.values).strftime('%b %d, %Y at %H:%M:%S')} UTC\",\n", - " frame_width=700,\n", - " frame_height=500,\n", - " xlabel='Longitude',\n", - " ylabel='Latitude',\n", - " tools=['hover']\n", - ")\n", - "\n", - "reflectivity_plot" - ] + "source": "# Mask data for neater visualization\ndata = data.where(data > 0, np.nan)\n\n# Get the NWS Reflectivity colormap and normalize range\nref_norm, ref_cmap = ctables.registry.get_with_steps('NWSReflectivity', 5, 5)\n\n# Convert to hex colors for Bokeh\nnorm = Normalize(vmin=ref_norm.vmin, vmax=ref_norm.vmax)\nhex_cmap = [ref_cmap(norm(val)) for val in range(ref_norm.vmin, ref_norm.vmax + 5, 5)]\nhex_cmap = [mcls.to_hex(c) for c in hex_cmap]\n\n# Plot using hvplot\nreflectivity_plot = data.hvplot.image(\n x=\"longitude\", y=\"latitude\",\n cmap=hex_cmap,\n colorbar=True,\n geo=True, \n tiles=True, \n alpha=0.7,\n clim=(ref_norm.vmin, ref_norm.vmax),\n title=f\"{product} - {pd.to_datetime(data.time.values).strftime('%b %d, %Y at %H:%M:%S')} UTC\",\n frame_width=700,\n frame_height=500,\n xlabel='Longitude',\n ylabel='Latitude',\n tools=['hover']\n)\n\nreflectivity_plot" }, { "cell_type": "markdown", @@ -402,7 +342,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## ๐Ÿ† Bonus Challenges\n", + "## \ud83c\udfc6 Bonus Challenges\n", "Congratulations on the completion of this notebook! You have successfully selected a region and product, queried the AWS S3 bucket, and visualized MRMS data in near real-time.\n", "\n", "If you'd like to continue this analysis, I've provided a couple of bonus challenges. Click on the drop-down menu to view the bonus challenge according to your desired level of difficulty. " @@ -412,12 +352,12 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "```{dropdown} ๐ŸŸข Challenge (easy) -- make a new data selection\n", + "```{dropdown} \ud83d\udfe2 Challenge (easy) -- make a new data selection\n", ":class: tip\n", "\n", "Use the drop-down widgets in this notebook to plot a different product and region from your initial run!\n", "\n", - "```{dropdown} ๐Ÿ’ก Hint\n", + "```{dropdown} \ud83d\udca1 Hint\n", "- Scroll up to the drop-down menus, make new region/product selections, and run all cells *below* the drop-down menus to see your new visualization!" ] }, @@ -425,12 +365,12 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "```{dropdown} ๐ŸŸก Challenge (medium) -- plot a new variable from AWS\n", + "```{dropdown} \ud83d\udfe1 Challenge (medium) -- plot a new variable from AWS\n", ":class: warning\n", "\n", "Browse the [AWS S3 bucket](https://noaa-mrms-pds.s3.amazonaws.com/index.html) and the [NSSL Variable Table](https://www.nssl.noaa.gov/projects/mrms/operational/tables.php) and find an MRMS product that was not covered in this notebook. Alter the provided code to read in and plot your new variable!\n", "\n", - "```{dropdown} ๐Ÿ’ก Hints\n", + "```{dropdown} \ud83d\udca1 Hints\n", "**Step-by-step:**\n", "1. Delete the widget-generating cell in the \"Data selection\" section.\n", "2. Hard-code the \"region\" and \"product\" variables with the *exact* strings that correspond to your data product on AWS. For example: \n", @@ -448,12 +388,12 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "```{dropdown} ๐Ÿ”ด Challenge (difficult) -- create a cron job to update your MRMS plot hourly\n", + "```{dropdown} \ud83d\udd34 Challenge (difficult) -- create a cron job to update your MRMS plot hourly\n", ":class: danger\n", "\n", "Turn this notebook into a Python script, then use cron to create an updated plot from MRMS data every hour. Incorporate this plot into a web page, send it to your friend, or try it just for fun! \n", "\n", - "```{dropdown} ๐Ÿ’ก Hints\n", + "```{dropdown} \ud83d\udca1 Hints\n", "**Step-by-step:**\n", "1. Delete the widget-generating cell in the \"Data selection\" section.\n", "2. Hard-code the \"region\" and \"product\" variables with the *exact* strings that correspond to your data product on AWS. For example: \n", @@ -489,7 +429,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## ๐Ÿ“š Resources and references" + "## \ud83d\udcda Resources and references" ] }, { @@ -509,7 +449,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## ๐Ÿ› ๏ธ Appendix\n", + "## \ud83d\udee0\ufe0f Appendix\n", "If you'd prefer to plot these data as a static plot, below is some sample code to kickstart your plotting journey. " ] }, @@ -653,4 +593,4 @@ }, "nbformat": 4, "nbformat_minor": 4 -} +} \ No newline at end of file