debug_utils: async buffer sanity inspector + --inspect-buffers-on-shutdown

[DLWoodruff]

Summary

Adds a new package mpisppy.debug_utils containing a passive content-check utility for the MPI-RMA send and receive buffers used by the hub-and-spoke system, plus an opt-in CLI flag that runs the check at cylinder shutdown.

Motivation. An xhat spoke was observed receiving a SHUTDOWN signal that Hub.send_terminate had not produced — suggesting that the SHUTDOWN buffer region was being written to from somewhere unexpected. mpi-sppy had no runtime sanity check on buffer contents; _validate_recv_field only checks layout at registration time. This PR adds the missing layer.

What's in it

• mpisppy/debug_utils/buffer_inspect.py — the inspector. Single entry point inspect_buffer(buf, fld, ctx, *, send, verbose). InspectContext is duck-typed (accepts SPBase via getattr); explicit fields take precedence.
• Generic checks (every field): trailing write_id slot is finite / non-negative / integer-valued; send-side id matches buf.id(); recv-side id does not regress below buf.id() or optional ctx.last_write_id; no inf in data; no NaN once write_id >= 1; padding region (between logical_len and padded_len) stays NaN.
• Per-Field checks: SHUTDOWN, NONANT, NONANT_LOWER_BOUNDS, NONANT_UPPER_BOUNDS, OBJECTIVE_INNER_BOUND, OBJECTIVE_OUTER_BOUND, BEST_XHAT. Adding a new checker is one function and one dict entry.
• --inspect-buffers-on-shutdown CLI flag in popular_args, propagated through cfg_vanilla.shared_options. _BoundSpoke.got_kill_signal reads it and runs the inspector only when shutdown is detected (the diagnostic moment). Off by default; no overhead when unset.
• Producers are NOT touched. The inspector is purely a consumer concern.

Tests (34 new)

mpisppy/tests/test_buffer_inspect.py covers: every generic check, every per-field check, SPBase fallback for nonant length, verbose dump, severity ladder, and the config-flag wiring (default false, propagation through shared_options). Wired into the existing unit-tests CI job so codecov picks it up.

Documentation

• doc/src/debug_utils.rst — user-facing page under For Developers and Contributors.
• doc/designs/async_buffer_sanity_design.md — internal design doc with invariants, check tables, and explicit non-goals (cross-cylinder consensus, module-level history state).

Draft because

Opened as a draft so the design can be reviewed before any further hot-path integrations (e.g., wiring the inspector into update_nonants or sync_bounds) are added. The current scope is intentionally narrow: prove out the SHUTDOWN check, then expand.

Test plan

• python -m pytest mpisppy/tests/test_config.py mpisppy/tests/test_buffer_inspect.py — 112 passed
• ruff check clean on all touched files
• Manual smoke: stomped recv buffer (data=1.0, write_id=0) produces the expected SHUTDOWN-specific finding
• Manual smoke: padding-region overrun produces a padding finding
• Manual smoke: --inspect-buffers-on-shutdown propagates through shared_options for hub and spoke; unset default propagates as False
• Real reproduction of the motivating spurious-SHUTDOWN incident with the flag enabled

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 95.13514% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.26%. Comparing base (b58f561) to head (2d36a25).

Files with missing lines	Patch %	Lines
mpisppy/debug_utils/buffer_inspect.py	96.57%	6 Missing ⚠️
mpisppy/cylinders/spoke.py	57.14%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #683      +/-   ##
==========================================
+ Coverage   71.05%   71.26%   +0.21%     
==========================================
  Files         154      156       +2     
  Lines       19252    19436     +184     
==========================================
+ Hits        13679    13852     +173     
- Misses       5573     5584      +11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.