Inital verification of monitor shm message correctness

Author: stephenlienharrell

.cursor/hooks/hook_task_router.py

@@ -320,6 +320,30 @@ class RouterEntry(TypedDict):
             "monitor-debug-shm.mdc",
         ],
     },
+    {
+        "id": "monitor_shm_message_correctness",
+        "patterns": [
+            "HPCPerfStats/monitor/scripts/emit_build_capabilities.py",
+            "HPCPerfStats/monitor/scripts/build_message_expectations.py",
+            "HPCPerfStats/monitor/scripts/validate_shm_messages.py",
+            "HPCPerfStats/monitor/scripts/lib/*",
+            "HPCPerfStats/monitor/tests/test_shm_message_correctness.sh",
+            "HPCPerfStats/monitor/tests/expected/shm_*",
+            "HPCPerfStats/monitor/tests/expected/capabilities_*",
+            "HPCPerfStats/monitor/tests/expected/expectations_*",
+            "monitor/scripts/emit_build_capabilities.py",
+            "monitor/scripts/build_message_expectations.py",
+            "monitor/scripts/validate_shm_messages.py",
+            "monitor/scripts/lib/*",
+            "monitor/tests/test_shm_message_correctness.sh",
+            "monitor/tests/expected/shm_*",
+            "monitor/tests/expected/capabilities_*",
+            "monitor/tests/expected/expectations_*",
+        ],
+        "rules": [
+            "monitor-shm-message-correctness.mdc",
+        ],
+    },
     {
         "id": "monitor_emit_contract",
         "patterns": [

.github/workflows/monitor-verify.yaml

@@ -36,6 +36,15 @@ jobs:
         working-directory: monitor/.build-static
         run: make check
 
+      - name: Emit build capabilities
+        working-directory: monitor/.build-static
+        run: make capabilities
+
+      - name: Shm message correctness (synthetic fixture; skip 77 if no slug goldens)
+        working-directory: monitor
+        run: |
+          ./tests/test_shm_message_correctness.sh .build-static || test $? -eq 77
+
       - name: distclean static build tree
         working-directory: monitor/.build-static
         run: make distclean

monitor/Makefile.am

@@ -1,5 +1,11 @@
 SUBDIRS = src tests
 
+.PHONY: capabilities
+capabilities:
+	@cd "$(top_builddir)" && python3 "$(abs_top_srcdir)/scripts/emit_build_capabilities.py" \
+	  --build-dir "$(top_builddir)" \
+	  --tier "$${CAPABILITIES_TIER:-slowtier1}"
+
 # Monitor-specific trees (not created by Autoconf); removed by distclean.
 distclean-local:
 	rm -rf .build-static

monitor/README.md

@@ -121,14 +121,44 @@ See `HPCPerfStats/docs/monitor_variable_rename_map.yaml` for rename-map entries.
 
 ### DEBUG `/dev/shm` mirror (daemon only)
 
-When built with **`--enable-debug`**, the RabbitMQ daemon overwrites the latest
-`@fast` and `@full` sample payloads under **`/dev/shm/hpcperfstatsd-debug/`**
-(files `fast` and `full`). Override the base directory with
-**`HPCPERFSTATS_DEBUG_SHM_DIR`**. Schema/`$` rotation payloads (`write_hdr=1`)
-do not update shm.
-
-Payloads contain job id, hostname, and workload metrics — treat as sensitive on
-shared nodes. Files are created mode `0600` under a `0700` directory.
+When built with **`--enable-debug`**, the RabbitMQ daemon mirrors the latest **full
+outbound payloads** under **`/dev/shm/hpcperfstatsd-debug/`**:
+
+| File | Content |
+|------|---------|
+| `schema` | Complete `$` rotation payload (`write_hdr=1`) |
+| `fast` | Latest `@fast` sample |
+| `full` | Latest `@full` (or legacy full-width) sample |
+
+Override the base directory with **`HPCPERFSTATS_DEBUG_SHM_DIR`**. Payloads
+contain job id, hostname, and workload metrics — treat as sensitive on shared
+nodes. Files are created mode `0600` under a `0700` directory (atomic `*.tmp` +
+`rename`).
+
+### `/dev/shm` message correctness testing
+
+On a data host with a DEBUG build:
+
+```bash
+cd HPCPerfStats/monitor
+./scripts/build_static_bundle.sh --enable-debug   # or add to configure args
+make -C .build-static capabilities
+# start hpcperfstatsd; wait for schema + fast + full updates under /dev/shm/...
+python3 scripts/build_message_expectations.py \
+  --capabilities .build-static/monitor-build-capabilities.json
+python3 scripts/validate_shm_messages.py \
+  --capabilities .build-static/monitor-build-capabilities.json \
+  --manifest .build-static/expectations_<capability_slug>.json \
+  --shm-dir /dev/shm/hpcperfstatsd-debug
+```
+
+**Capability slug** — `monitor-build-capabilities.json` includes
+`capability_slug` (compile flags + `slowtier0`/`slowtier1`). Golden fixtures
+and expectations must use the same slug in filenames
+(`tests/expected/shm_*_<slug>.txt`, `expectations_<slug>.json`). CI runs
+`tests/test_shm_message_correctness.sh` (synthetic fixture always; live slug
+goldens when present; **exit 77 skip** otherwise). Local run logs belong under
+**`<workspace-root>/test_runs/`** (see **test-runs-output-directory**).
 
 ## Power telemetry (DCGM, RAPL, and interpretation)
 

monitor/cursor-rules/agent-discipline-core.mdc

@@ -48,6 +48,7 @@ Use the **Read** tool on the listed `HPCPerfStats/monitor/cursor-rules/<file>.md
 | `HPCPerfStats/monitor/src/*ib*.c`, IB sysfs parsers | `monitor-ib-sysfs-parsing.mdc` |
 | `HPCPerfStats/monitor/src/*dcgm*`, `*gpu*` | `monitor-dcgm-integration.mdc` |
 | DEBUG shm mirror paths | `monitor-debug-shm.mdc` |
+| Shm message correctness scripts, slug goldens | `monitor-shm-message-correctness.mdc` |
 
 ### Emit contract, RabbitMQ, schema
 

monitor/cursor-rules/implementation-workflow-discipline.mdc

@@ -31,7 +31,8 @@ Use these during **plan**, **implement**, **self-review**, and **verification**
 | [monitor-schema-keys-headers.mdc](monitor-schema-keys-headers.mdc) | Shared KEYS headers |
 | [monitor-consumer-schema-migration.mdc](monitor-consumer-schema-migration.mdc) | Breaking output migrations |
 | [monitor-consumer-side-plan.mdc](monitor-consumer-side-plan.mdc) | Consumer handoff plans |
-| [monitor-debug-shm.mdc](monitor-debug-shm.mdc) | DEBUG `/dev/shm` mirror |
+| [monitor-debug-shm.mdc](monitor-debug-shm.mdc) | DEBUG `/dev/shm` mirror (`schema`/`fast`/`full`) |
+| [monitor-shm-message-correctness.mdc](monitor-shm-message-correctness.mdc) | Capability slug, validators, shm goldens |
 | [monitor-ib-sysfs-parsing.mdc](monitor-ib-sysfs-parsing.mdc) | IB port state parsers |
 | [monitor-ci-workflow.mdc](monitor-ci-workflow.mdc) | GitHub `monitor-verify.yaml` |
 | [monitor-cursor-rules-sync.mdc](monitor-cursor-rules-sync.mdc) | Edit both `.cursor/rules/` and `monitor/cursor-rules/` |

monitor/cursor-rules/monitor-debug-shm.mdc

@@ -1,5 +1,5 @@
 ---
-description: DEBUG /dev/shm sample mirror — security, gating, and tests
+description: DEBUG /dev/shm full-message mirror — security, gating, and tests
 globs:
   - HPCPerfStats/monitor/src/stats_buffer_debug_shm.*
   - HPCPerfStats/monitor/src/monitor_daemon.c
@@ -12,19 +12,26 @@ alwaysApply: false
 ## Scope
 
 - **DEBUG builds only** (`--enable-debug`); release uses inline no-ops in `stats_buffer_debug_shm.h`.
-- Updates **`fast`** and **`full`** under `HPCPERFSTATS_DEBUG_SHM_DIR` (default `/dev/shm/hpcperfstatsd-debug`).
+- Mirrors latest full payloads under `HPCPERFSTATS_DEBUG_SHM_DIR` (default `/dev/shm/hpcperfstatsd-debug`):
+  - **`schema`** — complete `$` rotation payload (`write_hdr=1`)
+  - **`fast`** — latest `@fast` sample
+  - **`full`** — latest `@full` (or legacy full-width) sample
 
 ## Implementation
 
 - **Permissions**: directory `0700`, files `0600` (payloads contain jobid, host, metrics).
 - **Atomic write**: tmp + `rename`; log failures via `monitor_log_debug` (do not `(void)` discard).
 - **Env**: copy `HPCPERFSTATS_DEBUG_SHM_DIR` into owned storage — do not retain `getenv()` pointer.
-- **Gating**: use `stats_buffer_debug_shm_sample_wanted(write_hdr, payload_ok)` — only routine samples (`!write_hdr`), not schema/`$` paths.
+- **Gating**:
+  - `stats_buffer_debug_shm_schema_wanted(write_hdr, payload_ok)` → schema file when `write_hdr && payload_ok`
+  - `stats_buffer_debug_shm_sample_wanted(write_hdr, payload_ok)` → sample files when `!write_hdr && payload_ok`
+- **Write API**: `stats_buffer_debug_shm_write_payload(sf, kind)` with `STATS_BUFFER_DEBUG_SHM_PAYLOAD_{SCHEMA,FAST,FULL}`.
 
 ## Tests
 
-- Unit + golden tests under `tests/test_stats_buffer_debug_shm.c`, `test_debug_shm_emit_*`.
-- Assert **`stats_buffer_debug_shm_sample_wanted`** matches daemon behavior.
+- Unit + golden: `tests/test_stats_buffer_debug_shm.c`, `test_debug_shm_emit_*`, `test_debug_shm_schema_mirror.c`.
+- Assert gating helpers match `monitor_daemon_collect_to_ring` behavior.
+- Slug-named goldens: `tests/expected/shm_{schema,fast,full}_<capability_slug>.txt` (**monitor-shm-message-correctness**).
 - Document DEBUG requirement in **`monitor/README.md`** (**monitor-readme-maintenance**).
 
-Cross-links: **monitor-debug-vs-symbols**, **monitor-readme-maintenance**.
+Cross-links: **monitor-debug-vs-symbols**, **monitor-readme-maintenance**, **monitor-shm-message-correctness**.

monitor/cursor-rules/monitor-shm-message-correctness.mdc

@@ -0,0 +1,52 @@
+---
+description: /dev/shm full-message correctness harness — capability slug, validators, goldens
+globs:
+  - HPCPerfStats/monitor/scripts/emit_build_capabilities.py
+  - HPCPerfStats/monitor/scripts/build_message_expectations.py
+  - HPCPerfStats/monitor/scripts/validate_shm_messages.py
+  - HPCPerfStats/monitor/scripts/lib/*
+  - HPCPerfStats/monitor/tests/test_shm_message_correctness.sh
+  - HPCPerfStats/monitor/tests/expected/shm_*
+  - HPCPerfStats/monitor/tests/expected/capabilities_*
+  - HPCPerfStats/monitor/tests/expected/expectations_*
+alwaysApply: false
+---
+
+# Monitor: /dev/shm message correctness
+
+## Scope
+
+- Validates **full outbound monitor payloads** mirrored under `HPCPERFSTATS_DEBUG_SHM_DIR` (`schema`, `fast`, `full`).
+- **Out of scope:** listend, archive files, RabbitMQ consumer path.
+- Requires **DEBUG build** (`--enable-debug`) for live shm capture (**monitor-debug-shm**).
+
+## Capability slug
+
+- Emitted by `scripts/emit_build_capabilities.py` → `<builddir>/monitor-build-capabilities.json`.
+- `make capabilities` in configured build tree; `build_static_bundle.sh` invokes unless `SKIP_CAPABILITIES=1`.
+- Slug tokens (omit absent): `arch`, `ver`, `debug`, `hw`, `ib`, `nvgpu`, `amdgpu`, `lustre`, `opa`, `dcgm|likwid|none`, `slowtier0|slowtier1`.
+- **All fixtures** use slug in filename: `expectations_<slug>.json`, `tests/expected/shm_{schema,fast,full}_<slug>.txt`.
+- Validator **layer 0**: manifest slug must match capabilities JSON; exit 2 on mismatch.
+- CI/integration: **exit 77 skip** when no goldens for live slug — never fall back to generic golden.
+
+## Artifacts
+
+| Local (gitignored) | Committed |
+|--------------------|-----------|
+| `test_runs/monitor/validate_<slug>_*.txt` | `tests/expected/shm_*_<slug>.txt` |
+| `test_runs/monitor/shm_snapshot_<host>_<slug>/` | `scripts/lib/*.py` |
+
+## Data-host runbook
+
+1. `build_static_bundle.sh` with `--enable-debug` → `make -C .build-static capabilities`.
+2. Run daemon; read `schema`/`fast`/`full` from shm.
+3. `build_message_expectations.py --capabilities …` → `expectations_<slug>.json`.
+4. `validate_shm_messages.py` (optional `--spot-check-values`).
+
+## Tests
+
+- C: `test_stats_buffer_debug_shm.c`, `test_debug_shm_schema_mirror.c` (**monitor-debug-shm**).
+- Shell: `test_shm_message_correctness.sh` (slug-aware).
+- Python via repo `.venv` (**python-venv-enforcement**).
+
+Cross-links: **monitor-collect-tier-gating**, **monitor-schema-keys-headers**, **monitor-c-testing-standards**, **test-runs-output-directory**.