Skip to content

Commit 7ec9b6a

Browse files
kmbandyclaude
andcommitted
DSWS flow: throttle deadman message-bus read off the idle-coast hot path
Root cause of the W=16 chunked-dispatch brick + wall instability (measured 2026-07-05): deadman_check does an s_sendmsg_rtn RTC read (an SQ-front-end op) at every loop-head. With 24 WGs x 16 waves = 384 waves and only 2 tiles/chunk, ~382 idle waves spin the coast loop and hammer that message bus ~41k times per chunk. That front-end spam (a) destabilized the coast wall -- identical STAGINSTR work measured 0.324s vs 1.996s across two chunks -- and (b) starved the compositor's SQC(inst) instruction fetch -> GCVM_L2 page fault -> MODE1. Fix: throttle the RTC read to once every DEADMAN_EVERY (=64) iterations via a per-wave counter in s71 (the high-RTC reg, unused at TRACE=0; deadman_check only reads s70). Force-retire slack = DEADMAN_EVERY iters (negligible vs 0.5s deadline). Result (12-chunk W=16, same config as the bricking run): all chunks complete, wall flat at 0.325s (6x variance gone), oracle CLEAN bad=0 max_rel=0, no brick. grow-fail=0 throughout also disproves the VGPR-starvation-livelock hypothesis. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0132aDSBLwusCJ4KzHQTnvdu
1 parent e06a599 commit 7ec9b6a

1 file changed

Lines changed: 14 additions & 1 deletion

File tree

ggml/src/ggml-cuda/aiter-integration/rdna4_fp8_gemm/spike/dvgpr_occ/occ_kernel_dsws_flow.s

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -517,20 +517,33 @@
517517
.Lcasadv_skip\@:
518518
s_mov_b32 exec_lo, s49
519519
.endm
520-
// ---- DEADMAN watchdog: s[70:71] = this wave's start RTC; deadman_check force-retires past the deadline ----
520+
// ---- DEADMAN watchdog: s70 = this wave's start RTC (low 32b); s71 = throttle counter (repurposed high-RTC
521+
// reg, which is unused at TRACE=0 -- deadman_check only reads s70). The message-bus RTC read (s_sendmsg_rtn)
522+
// is an SQ-front-end op; hundreds of idle COAST waves hitting it EVERY loop iteration spam the front-end,
523+
// starving the compositor's SQC(inst) fetch (2026-07-05 MODE1 brick) AND destabilizing the coast wall
524+
// (identical STAGINSTR work measured 0.32s vs 2.0s). THROTTLE: only read the RTC every DEADMAN_EVERY iters. ----
525+
.ifndef DEADMAN_EVERY
526+
.set DEADMAN_EVERY, 64 // message-bus RTC-read cadence (in loop iters); force-retire slack = DEADMAN_EVERY iters
527+
.endif
521528
.macro deadman_stamp // stamp start RTC (low 32b in s70) once at entry
522529
.if DEADMAN
523530
s_sendmsg_rtn_b64 s[70:71], sendmsg(MSG_RTN_GET_REALTIME)
524531
s_wait_kmcnt 0x0
532+
s_mov_b32 s71, 0 // repurpose the (TRACE=0-unused) high-RTC reg as the throttle counter
525533
.endif
526534
.endm
527535
.macro deadman_check // if alive > DEADMAN_TICKS -> clean force-retire (no wedge)
528536
.if DEADMAN
537+
s_add_u32 s71, s71, 1 // THROTTLE: touch the SQ-front-end message bus only every
538+
s_cmp_ge_u32 s71, DEADMAN_EVERY // DEADMAN_EVERY iters (idle coast waves would else spam it -> brick)
539+
s_cbranch_scc0 .Ldm_skip\@
540+
s_mov_b32 s71, 0
529541
s_sendmsg_rtn_b64 s[62:63], sendmsg(MSG_RTN_GET_REALTIME)
530542
s_wait_kmcnt 0x0
531543
s_sub_u32 s62, s62, s70 // elapsed = now_lo - start_lo (u32 wrap-safe; deadline << 42s)
532544
s_cmp_ge_u32 s62, DEADMAN_TICKS
533545
s_cbranch_scc1 .Lflow_retire
546+
.Ldm_skip\@:
534547
.endif
535548
.endm
536549
// lds_put_r (RUNTIME-addr write) is also defined inside the .if DSWS2_CONV||DSWS2_ENVELOPE block below;

0 commit comments

Comments
 (0)