|
| 1 | +# DSWS Phase B — Universal Dynamic Dispatch + Grow-into-Budget Pool Economy |
| 2 | + |
| 3 | +**Status:** design (brainstormed 2026-07-01, kmbandy). Extends and builds on the |
| 4 | +committed Phase-B substrate (Tasks 1–5, branch `feat/dsws-phaseb-conversion`, |
| 5 | +HEAD `6f3e36f45`). |
| 6 | + |
| 7 | +**Goal:** Make a wave that commits a role conversion actually *execute the new |
| 8 | +role's code*, and generalize the fixed 8-wave partition into an adaptive |
| 9 | +wave-role economy on a lean-start pool — realizing the DSWS vision where the |
| 10 | +kernel re-balances its {compute, A-feed, B-feed} mix into the VGPR budget at |
| 11 | +runtime, per shape, per moment. |
| 12 | + |
| 13 | +**Non-goal:** parking/dormant waves (explicitly dropped as YAGNI — see §7). |
| 14 | + |
| 15 | +--- |
| 16 | + |
| 17 | +## 0. Context: what already exists (Tasks 1–5) |
| 18 | + |
| 19 | +The committed substrate provides, gated under `.if DSWS2_CONV` (byte-identical |
| 20 | +to Phase A when off): |
| 21 | + |
| 22 | +- **Sensing** (`occ_sample`) — per-role occupancy proxy in `s55/s56`. |
| 23 | +- **Ticket** (`try_gate \dir,\swin`) — single-winner `(dir,epoch)` LDS-CAS. |
| 24 | +- **Reservation** (`reserve_try \delta,\won`) — signed-delta VGPR envelope vs |
| 25 | + `BUDGET`; shrink (−80) always wins, grow (+80) aborts over budget. |
| 26 | +- **Bail-time commit** (`conv_apply`) — floor-guarded role-slot swap + VGPR |
| 27 | + resize (`s_alloc_vgpr`) + `s59` role-register flip, ordered strictly *before* |
| 28 | + the `QUIESCE_CNT` bump. |
| 29 | +- **Snapshot/quiesce** — claimer snapshots the live role mix into the next |
| 30 | + epoch's parity half of `SNAP_BASE` at broadcast; per-role claim-count |
| 31 | + sentinels are sized from that snapshot; a `QUIESCE_CNT ≥ WAVES−1` cross-check |
| 32 | + is the safety net; a DIAG `occ[29]` flag asserts the two agree. |
| 33 | + |
| 34 | +**The gap this spec closes:** `conv_apply` flips `s59` but nothing reads it — a |
| 35 | +converted wave changes its role-slot count and VGPR footprint yet keeps running |
| 36 | +its *original* role's code path. Conversion is currently accounting-only. |
| 37 | + |
| 38 | +**Key constants (from `occ_kernel_dsws.s`):** `NFV=112` (fat compute VGPR), |
| 39 | +`VLEAN=32` (lean feed VGPR), conversion delta `NFV−VLEAN=80`, role slot ids |
| 40 | +`NCOMP_SLOT=24 / NAFEED_SLOT=28 / NBFEED_SLOT=32`, `SNAP_BASE=72` (u32[6], |
| 41 | +parity-doubled {nC,nA,nB}), `QUIESCE_CNT_OFF=96`. |
| 42 | + |
| 43 | +--- |
| 44 | + |
| 45 | +## 1. Architecture |
| 46 | + |
| 47 | +Under `DSWS2_CONV=1` every wave launches lean at `VLEAN=32` (already Phase-A |
| 48 | +behavior — compute grows to `NFV` per-rowblk on demand and shrinks back; nothing |
| 49 | +launches fat). The launched wave count *is* the pool size (`WAVES` = the mix |
| 50 | +sum). wid-0 is the claimer. Each non-claimer wave is **seeded** with its launch |
| 51 | +role by writing `s59` in the existing wid partition — **no launch-time grow**. |
| 52 | +Role lives entirely in `s59`, and a single dispatcher routes every wave — at |
| 53 | +entry and after every terminal bail — to the code for the role `s59` names. |
| 54 | +Waves rebalance by growing into / shedding VGPR budget (bounded by |
| 55 | +`reserve_try`/`BUDGET`), **never** by forking, merging, or parking. Scaling the |
| 56 | +launched wave count above the current mix sum (the 12–16-wave "bigger pool" for a |
| 57 | +larger feed multiplier) is a follow-on requiring an `occ_dispatch` dims change; |
| 58 | +the mechanism is proven first as role rebalancing *within* the launched mix. |
| 59 | + |
| 60 | +**Wave-count invariant (the hardware truth):** the launched wave count is fixed |
| 61 | +for the kernel's life; `s_alloc_vgpr` resizes *one existing wave*, it does not |
| 62 | +spawn or retire waves. The {compute, A, B} multiplier from the vision |
| 63 | +(shed 1 fat compute → several lean feed light up) is realized in the **VGPR |
| 64 | +budget**, not the wave count: shrinking a compute wave frees `80` VGPR of budget |
| 65 | +that other waves can grow into or that lets a feed-heavy mix stay within budget. |
| 66 | +Because feed waves are cheap (`32` VGPR), the pool is launched large enough that |
| 67 | +a feed-heavy mix has many active feed waves. |
| 68 | + |
| 69 | +**Byte-identity invariant:** all Model-B launch/dispatch lives under |
| 70 | +`.if DSWS2_CONV`. With `DSWS2_CONV=0` the kernel assembles byte-identical to the |
| 71 | +proven Phase-A fixed-partition kernel. Task 6's re-baseline runs `DSWS2_CONV=0` |
| 72 | +as the untouched golden reference. |
| 73 | + |
| 74 | +--- |
| 75 | + |
| 76 | +## 2. The dispatcher (universal dynamic dispatch) |
| 77 | + |
| 78 | +**Today:** role is a wid-partition branch taken once at entry (`occ_kernel_dsws.s` |
| 79 | +~L604–629); each role's `_follow` loop ends with an unconditional |
| 80 | +`s_branch .L<samerole>_follow` (L907/L951/compute equivalent). |
| 81 | + |
| 82 | +**New:** replace those three tail branches with one shared, scalar-only, |
| 83 | +wave-uniform trampoline: |
| 84 | + |
| 85 | +```asm |
| 86 | +.Ldispatch: // s59 = current role slot id (24/28/32) |
| 87 | + s_cmp_eq_u32 s59, NCOMP_SLOT |
| 88 | + s_cbranch_scc1 .Lcompute_follow |
| 89 | + s_cmp_eq_u32 s59, NAFEED_SLOT |
| 90 | + s_cbranch_scc1 .Lafeed_follow |
| 91 | + s_branch .Lbfeed_follow |
| 92 | +``` |
| 93 | + |
| 94 | +> **CORRECTION 2026-07-02 (Pool-T7 brick root cause).** `.Ldispatch` is a |
| 95 | +> **RE-DISPATCH-ONLY** trampoline — it is reached from a per-super-tile `_quiesce` |
| 96 | +> bail, **never from first-time entry**. Entry and re-dispatch are **NOT** the same |
| 97 | +> mechanism: first entry must run the full role `_alloc`/`_init` (see §3), and only |
| 98 | +> re-dispatch may land on `_follow`. The original text below said entry could reuse |
| 99 | +> `.Ldispatch`; that routing skipped `_alloc`/`_init` on first entry (no |
| 100 | +> `s_alloc_vgpr 32` handshake, no INITFLAG rendezvous, garbage `s35`) and hung the |
| 101 | +> first CONV=1 dispatch → desktop brick. Points 1–2 hold **for re-dispatch**; point 3 |
| 102 | +> is corrected below. |
| 103 | +
|
| 104 | +Three properties make this safe and cheap: |
| 105 | + |
| 106 | +1. **On RE-DISPATCH, lands on `_follow`, skipping `_alloc`/`_init`.** `conv_apply` |
| 107 | + already set the wave's VGPR footprint to the target role's size, and the wave |
| 108 | + already ran `_alloc`/`_init` once at first entry. Re-running `_alloc` would |
| 109 | + wrongly resize the wave; re-running `_init` would reset `s35 = 0`, breaking the |
| 110 | + "wait for the *next* epoch" contract (INITFLAG is written once, `0xACED`, and is |
| 111 | + never cleared — so it is the `s35` reset, not INITFLAG, that `_follow` preserves). |
| 112 | + **First entry must NOT take this path** — it has run neither `_alloc` nor `_init`. |
| 113 | +2. **Preserves the last-seen-epoch register** (`s35`). A re-dispatched wave |
| 114 | + therefore waits for the *next* epoch in its new role rather than |
| 115 | + re-processing the epoch it just converted in. |
| 116 | +3. **Entry and re-dispatch are DISTINCT.** First entry runs the full role entry |
| 117 | + (`.Lcompute`/`.Lafeed`/`.Lbfeed` → `_alloc` → `_init` → `_follow`), with the |
| 118 | + wid-branch *seeding* `s59` first so the role register is set before any later |
| 119 | + re-dispatch reads it. Only after a bail does a wave use `.Ldispatch`. A |
| 120 | + non-converting wave, once past first entry, reads back its own role each epoch |
| 121 | + via `.Ldispatch` and loops exactly as today. |
| 122 | + |
| 123 | +The claimer (wid-0) is **excluded**: it branches to `.Lclaimer` at entry (before |
| 124 | +the seed/`.Ldispatch` path), runs its own `.Lclaim_loop`, never converts, and is |
| 125 | +never counted among the `N_POOL−1` bailers. The dispatcher governs non-claimer |
| 126 | +waves only. |
| 127 | + |
| 128 | +Cost: initialize `s59` at each seed role (part of §3), the trampoline (~6 |
| 129 | +lines), replace 3 tail branches. Entirely additive; the proven `_follow` bodies |
| 130 | +are untouched. `s59` read + `s_branch` is scalar-only → **zero** new OOR |
| 131 | +exposure (§6). |
| 132 | + |
| 133 | +Rejected alternatives: per-bail inline role check (less DRY, no benefit); |
| 134 | +unified per-epoch dispatch loop that collapses the three `_follow` bodies |
| 135 | +(rewrites GPU-proven control flow for no functional gain). |
| 136 | + |
| 137 | +--- |
| 138 | + |
| 139 | +## 3. Launch & init (seed the lean partition) |
| 140 | + |
| 141 | +Every wave already `s_alloc_vgpr 32` at entry (feeds, claimer, **and** compute — |
| 142 | +compute grows to `NFV` per-rowblk inside its loop and shrinks back). wid-0 → |
| 143 | +claimer. So launch/init is unchanged from Phase A except one addition: |
| 144 | + |
| 145 | +**Seed `s59` by wid** (under `DSWS2_CONV`), in the *existing* partition arms — |
| 146 | +`[0,NBFEED)` → B-feed, `[NBFEED,NBFEED+NAFEED)` → A-feed, rest → compute — each |
| 147 | +arm gains a single `s_mov_b32 s59, <slot>` and then **falls into the full role |
| 148 | +entry** (`.Lbfeed`/`.Lafeed`/`.Lcompute` → `_alloc` → `_init` → `_follow`). |
| 149 | +**It must NOT branch to `.Ldispatch`** (that skips `_alloc`/`_init` on a wave that |
| 150 | +has run neither → the Pool-T7 brick; see §2 correction). `.Ldispatch` is entered |
| 151 | +only later, from a `_quiesce` bail. **No launch-time `s_alloc_vgpr` is *added*** — |
| 152 | +the seed reuses the role entry's existing `_alloc` (compute's `NFV` footprint is |
| 153 | +still handled per-rowblk). There are no `N_POOL`/`SEED_*` defsyms: the launched mix |
| 154 | +`NCOMP/NAFEED/NBFEED` *is* the seed and its sum *is* the pool size. |
| 155 | + |
| 156 | +**Defsyms:** |
| 157 | + |
| 158 | +- `BUDGET` — retuned to the real per-SIMD VGPR ceiling (headroom for growth), |
| 159 | + **not** the zero-headroom launch-footprint default Task 5 inherited. This is |
| 160 | + the knob that makes feed→compute grows able to succeed. Validated by the RGA |
| 161 | + live-VGPR gate and a real dispatch — never a guessed constant. Compile-time |
| 162 | + no-parking invariant: `WAVES × VLEAN ≤ BUDGET` (every wave always fits lean). |
| 163 | +- `VRESV` seed = `NCOMP*NFV + (NAFEED+NBFEED)*VLEAN` (unchanged; the live |
| 164 | + reservation the economy adjusts from). |
| 165 | + |
| 166 | +--- |
| 167 | + |
| 168 | +## 4. Conversion policy (sense → decide → commit → re-dispatch) |
| 169 | + |
| 170 | +At each terminal bail (`conv_apply` site), a wave runs Task 5's decision, plus |
| 171 | +two additions: |
| 172 | + |
| 173 | +- **Cooldown `K`** — a per-wave scalar counts down one per epoch; while `> 0` |
| 174 | + the wave skips the watermark check entirely (no ticket race, no conversion). |
| 175 | + On a committed conversion, reset to `K`. Default `K = 0` (spec-faithful; the |
| 176 | + storm gate sets 0 for maximal thrash). `K > 0` is a churn damper available if |
| 177 | + `s_alloc_vgpr` grow/shrink frequency starts bricking runs — its value is |
| 178 | + reducing the *number* of OOR windows per run. The cooldown counter joins the |
| 179 | + persistent scalar set (outside `s60–s65`, alongside `s57/s58/s59`). |
| 180 | +- **Force-convert bring-up hook** — a `DSWS2_FORCE` defsym path where a |
| 181 | + designated wid converts a chosen dir at a chosen epoch, watermarks bypassed; |
| 182 | + everything else static. The deterministic first-proof lever (§7). Gated, off |
| 183 | + by default, emits zero bytes when off. |
| 184 | + |
| 185 | +Direction semantics unchanged from Task 5: `occ_X < CTRL_LOW` → compute starved |
| 186 | +for X → shrink one compute → feed-X (delta −80, always wins); `occ_X > |
| 187 | +CTRL_HIGH_X` → feed-X over-serving → grow one feed-X → compute (delta +80, may |
| 188 | +abort over `BUDGET` → stay). Floor guard keeps every role ≥ 1 wave. |
| 189 | + |
| 190 | +After commit (or no-op), the wave bumps `QUIESCE_CNT` (Task 5 ordering) and |
| 191 | +falls into `.Ldispatch`. |
| 192 | + |
| 193 | +--- |
| 194 | + |
| 195 | +## 5. Quiesce reconciliation (the correctness crux) |
| 196 | + |
| 197 | +**Invariant that must hold every epoch:** *the claimer's per-epoch snapshot of |
| 198 | +the role mix equals the role mix the waves actually execute that epoch.* |
| 199 | + |
| 200 | +Re-dispatch closes Task 5's loop: |
| 201 | + |
| 202 | +1. A converting wave commits its slot swap in epoch *E* at bail time, **before** |
| 203 | + bumping `QUIESCE_CNT` (Task 5 ordering). |
| 204 | +2. The claimer waits for quiesce (all `N_POOL−1` non-claimer waves bailed **and** |
| 205 | + the snapshot sentinels), then snapshots the **live** (now-updated) slot counts |
| 206 | + into the *E+1* parity half of `SNAP_BASE`, resets `QUIESCE_CNT`, and bumps the |
| 207 | + epoch last. |
| 208 | +3. The converting wave, having flipped `s59`, enters its **new** role's `_follow` |
| 209 | + via `.Ldispatch` and wakes for *E+1*. |
| 210 | +4. In *E+1* the wave contributes to the claim counters **as its new role** — |
| 211 | + exactly the mix the claimer snapshotted for *E+1*. Sentinels match observed |
| 212 | + contributions. |
| 213 | + |
| 214 | +The `QUIESCE_CNT ≥ N_POOL−1` cross-check holds unchanged: **every wave bails |
| 215 | +exactly once per epoch regardless of role** (no parking), so the count is |
| 216 | +role-agnostic and invariant under conversion. The DIAG `occ[29]` flag asserts |
| 217 | +snapshot-sentinels ⟺ quiesce-count agree; with re-dispatch wired it reads |
| 218 | +*agree* even when conversions fire (whereas Task 5 alone would diverge — the |
| 219 | +divergence the flag was built to catch). Task 7's dynamic gate verifies this on |
| 220 | +silicon. |
| 221 | + |
| 222 | +--- |
| 223 | + |
| 224 | +## 6. OOR-poison window (SPEC §4, #1 brick risk) |
| 225 | + |
| 226 | +- **The dispatcher adds zero OOR exposure:** `.Ldispatch` is `s59` read + |
| 227 | + `s_branch`, scalar-only. A re-dispatched wave enters its new `_follow` with its |
| 228 | + footprint already correctly sized (`conv_apply` closed its grow window before |
| 229 | + the bump) and no pending grow. |
| 230 | +- **`conv_apply`'s `s_alloc_vgpr` grow** is unchanged from Task 5's audit: waves |
| 231 | + are lean-32 at every bail, every pre-grow LDS/atomic temp ≤ v15. **This is the |
| 232 | + only GROW in the design** — seeding adds none (compute's per-rowblk grow is the |
| 233 | + existing, already-audited Phase-A path). |
| 234 | + |
| 235 | +--- |
| 236 | + |
| 237 | +## 7. Failure modes, floors, and no-parking |
| 238 | + |
| 239 | +- **Role floors:** `conv_dec_floor` keeps compute ≥ 1 and each feed ≥ 1 — a role |
| 240 | + can never empty. |
| 241 | +- **Budget safety:** `N_POOL × VLEAN ≤ BUDGET` guarantees the lean floor always |
| 242 | + fits; `reserve_try` guarantees grows only commit within the real ceiling; the |
| 243 | + atomic-add-then-undo-on-abort path leaves `vgpr_reserved` exactly restored (no |
| 244 | + leaked reservation, Task 5-verified). |
| 245 | +- **No parking (YAGNI):** because the pool is sized so all waves stay |
| 246 | + active-lean, a wave never needs to go dormant. Surplus capacity does useful |
| 247 | + lean-feed work instead of sleeping. This is what preserves the simple |
| 248 | + `QUIESCE_CNT ≥ N_POOL−1` accounting and avoids a race-prone wake/sleep |
| 249 | + handshake on the brick-risk path. |
| 250 | + |
| 251 | +--- |
| 252 | + |
| 253 | +## 8. Validation ladder |
| 254 | + |
| 255 | +Offline (no GPU) → then supervised GPU gates, each needing an individual |
| 256 | +greenlight; any brick/hang/DMESG-fault/DIAG-mismatch = full STOP + bisect, never |
| 257 | +auto-advance. |
| 258 | + |
| 259 | +1. **Offline** — assemble all mixes `DSWS2_CONV=1 DIAG=1` → `ASSEMBLE_OK`; RGA |
| 260 | + `SGPR_SPILLS=0 VGPR_SPILLS=0` and live-VGPR within `BUDGET`; `DSWS2_CONV=0` |
| 261 | + sha256 byte-identical to pre-change; CPU control model `ALL PASS`; dry-print |
| 262 | + sanity. |
| 263 | +2. **Force-convert gate (SUPERVISED)** — `DSWS2_FORCE` one designated wave, one |
| 264 | + dir, one known epoch, watermarks bypassed. Expect oracle-CLEAN, `bad=0`, |
| 265 | + `occ[0]=0`, DIAG `occ[29]` agree, dmesg silent. The deterministic debut of a |
| 266 | + wave executing a converted footprint's role code. |
| 267 | +3. **Dynamic gate (SUPERVISED)** — watermark-driven conversions fire; oracle |
| 268 | + stays CLEAN as roles move; `occ[29]` agree across all mixes × tiers. |
| 269 | +4. **Storm gate (SUPERVISED)** — `K=0`, `EPOCH_SHIFT=0`, ×10 repeats — lock-free |
| 270 | + race-hunt under maximal conversion frequency. |
| 271 | + |
| 272 | +--- |
| 273 | + |
| 274 | +## 9. Open parameters (resolved during implementation, not guessed) |
| 275 | + |
| 276 | +- `BUDGET`, `K` — empirically tuned defsyms with the principled defaults in |
| 277 | + §3–§4; final values set by RGA + real-dispatch measurement on the target ml8 |
| 278 | + shapes, not chosen a priori. (No `N_POOL`/`SEED_*`: pool size == launched mix |
| 279 | + sum, seed == launch partition — see §3 correction.) |
| 280 | +- Persistent-scalar assignment for the cooldown counter (must sit outside |
| 281 | + `s60–s65` and every macro's clobber set; candidates alongside `s57/s58/s59`) — |
| 282 | + fixed at implementation time against the live register map. |
0 commit comments