Skip to content

Fix ch5 build hang: add network timeouts and reduce memory#51

Merged
tyfolino merged 4 commits into
ProjectPythia:mainfrom
tyfolino:fix-ch5-timeout
Mar 16, 2026
Merged

Fix ch5 build hang: add network timeouts and reduce memory#51
tyfolino merged 4 commits into
ProjectPythia:mainfrom
tyfolino:fix-ch5-timeout

Conversation

@tyfolino

Copy link
Copy Markdown
Collaborator

Summary

The nightly build was hanging for 6+ hours (until GitHub cancelled it) due to missing network timeouts in Chapter 5, not memory.

  • Root cause: urllib.request.urlopen() had no timeout — if AWS S3 was slow, the build would wait forever. s3fs.S3FileSystem also had no timeout configured. Ch3 already uses timeout=5 on its urlopen() call; this PR brings ch5 in line.
  • Secondary bug: A 2-hour freshness check on the most recent MRMS file had no else branch, leaving data undefined if the file was stale → NameError in the plotting cell.
  • Memory: Added data[::4, ::4] decimation after loading (~17x reduction, ~5 GB → ~300 MB) to keep hvplot fast. The interactive plot is preserved.

Changes to ch5_realtimeData.ipynb

  • Add client_kwargs={'connect_timeout': 30, 'read_timeout': 60} to s3fs.S3FileSystem
  • Add timeout=60 to urllib.request.urlopen()
  • Remove the 2-hour freshness check (always fetch the most recent available file)
  • Decimate loaded data: data = data[::4, ::4]
  • Add a clear RuntimeError guard before the plotting cell

Test plan

  • Confirm the build completes in <15 minutes
  • Confirm the interactive hvplot is present in the built page

🤖 Generated with Claude Code

- Add connect/read timeouts to s3fs.S3FileSystem to prevent indefinite
  hang on slow S3 responses (root cause of 6-hour nightly build hang)
- Add timeout=60 to urllib.request.urlopen(), following the same
  pattern already used in ch3
- Remove the 2-hour freshness check that left `data` undefined if the
  most recent file was stale, causing a downstream NameError
- Decimate loaded data (data[::4, ::4]) for ~17x memory reduction
  (~5 GB -> ~300 MB), keeping hvplot interactive and fast
- Add clear RuntimeError guard before plotting cell instead of cryptic
  NameError if data fetch fails

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Mar 13, 2026

Copy link
Copy Markdown
Contributor

👋 Thanks for opening this PR! The Cookbook will be automatically built with GitHub Actions. To see the status of your deployment, click below.
🔍 Git commit SHA: d9829af
✅ Deployment Preview URL: https://ProjectPythia.github.io/mrms-cookbook/_preview/51

tyfolino and others added 2 commits March 13, 2026 16:39
The previous error handling printed errors and continued, leaving
`data` undefined and causing a confusing RuntimeError downstream.
Now exceptions propagate from their source with clear messages.
Also removes the redundant data-existence guard.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
connect_timeout and read_timeout must be passed via config_kwargs
(botocore.config.Config) not client_kwargs, which are passed directly
to aiobotocore's create_client() and don't support those args.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@tyfolino

Copy link
Copy Markdown
Collaborator Author

Note for reviewers: If AWS S3 is temporarily slow at the time of a nightly build, the 60-second timeout on the download will cause the build to fail with a RuntimeError rather than hanging for 6 hours. This is an acceptable trade-off for now — a future improvement could add retry logic or fall back to cached outputs to handle transient AWS slowdowns more gracefully.

The check was removed accidentally — it exists to ensure the notebook
displays genuinely real-time data. Restored it with a clear RuntimeError
instead of the original silent skip that left `data` undefined.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@tyfolino tyfolino merged commit 8986fd0 into ProjectPythia:main Mar 16, 2026
2 checks passed
@tyfolino tyfolino deleted the fix-ch5-timeout branch March 16, 2026 19:26
github-actions Bot pushed a commit that referenced this pull request Mar 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant