|
| 1 | +# DSWS Phase B — Universal Dynamic Dispatch + Grow-into-Budget Pool Economy |
| 2 | + |
| 3 | +**Status:** design (brainstormed 2026-07-01, kmbandy). Extends and builds on the |
| 4 | +committed Phase-B substrate (Tasks 1–5, branch `feat/dsws-phaseb-conversion`, |
| 5 | +HEAD `6f3e36f45`). |
| 6 | + |
| 7 | +**Goal:** Make a wave that commits a role conversion actually *execute the new |
| 8 | +role's code*, and generalize the fixed 8-wave partition into an adaptive |
| 9 | +wave-role economy on a lean-start pool — realizing the DSWS vision where the |
| 10 | +kernel re-balances its {compute, A-feed, B-feed} mix into the VGPR budget at |
| 11 | +runtime, per shape, per moment. |
| 12 | + |
| 13 | +**Non-goal:** parking/dormant waves (explicitly dropped as YAGNI — see §7). |
| 14 | + |
| 15 | +--- |
| 16 | + |
| 17 | +## 0. Context: what already exists (Tasks 1–5) |
| 18 | + |
| 19 | +The committed substrate provides, gated under `.if DSWS2_CONV` (byte-identical |
| 20 | +to Phase A when off): |
| 21 | + |
| 22 | +- **Sensing** (`occ_sample`) — per-role occupancy proxy in `s55/s56`. |
| 23 | +- **Ticket** (`try_gate \dir,\swin`) — single-winner `(dir,epoch)` LDS-CAS. |
| 24 | +- **Reservation** (`reserve_try \delta,\won`) — signed-delta VGPR envelope vs |
| 25 | + `BUDGET`; shrink (−80) always wins, grow (+80) aborts over budget. |
| 26 | +- **Bail-time commit** (`conv_apply`) — floor-guarded role-slot swap + VGPR |
| 27 | + resize (`s_alloc_vgpr`) + `s59` role-register flip, ordered strictly *before* |
| 28 | + the `QUIESCE_CNT` bump. |
| 29 | +- **Snapshot/quiesce** — claimer snapshots the live role mix into the next |
| 30 | + epoch's parity half of `SNAP_BASE` at broadcast; per-role claim-count |
| 31 | + sentinels are sized from that snapshot; a `QUIESCE_CNT ≥ WAVES−1` cross-check |
| 32 | + is the safety net; a DIAG `occ[29]` flag asserts the two agree. |
| 33 | + |
| 34 | +**The gap this spec closes:** `conv_apply` flips `s59` but nothing reads it — a |
| 35 | +converted wave changes its role-slot count and VGPR footprint yet keeps running |
| 36 | +its *original* role's code path. Conversion is currently accounting-only. |
| 37 | + |
| 38 | +**Key constants (from `occ_kernel_dsws.s`):** `NFV=112` (fat compute VGPR), |
| 39 | +`VLEAN=32` (lean feed VGPR), conversion delta `NFV−VLEAN=80`, role slot ids |
| 40 | +`NCOMP_SLOT=24 / NAFEED_SLOT=28 / NBFEED_SLOT=32`, `SNAP_BASE=72` (u32[6], |
| 41 | +parity-doubled {nC,nA,nB}), `QUIESCE_CNT_OFF=96`. |
| 42 | + |
| 43 | +--- |
| 44 | + |
| 45 | +## 1. Architecture |
| 46 | + |
| 47 | +Under `DSWS2_CONV=1` the workgroup becomes a **lean-start pool of `N_POOL` |
| 48 | +waves** (empirically-tuned defsym, 12–16 range), all launched at `VLEAN=32`. |
| 49 | +wid-0 is the claimer. A seed subset grows to fat compute at init; the remainder |
| 50 | +are lean A-feed / B-feed. Role lives entirely in the scalar `s59`, and a single |
| 51 | +dispatcher routes every wave — at entry and after every terminal bail — to the |
| 52 | +code for the role `s59` names. Waves rebalance by growing into / shedding VGPR |
| 53 | +budget (bounded by `reserve_try`/`BUDGET`), **never** by forking, merging, or |
| 54 | +parking. |
| 55 | + |
| 56 | +**Wave-count invariant (the hardware truth):** the launched wave count is fixed |
| 57 | +for the kernel's life; `s_alloc_vgpr` resizes *one existing wave*, it does not |
| 58 | +spawn or retire waves. The {compute, A, B} multiplier from the vision |
| 59 | +(shed 1 fat compute → several lean feed light up) is realized in the **VGPR |
| 60 | +budget**, not the wave count: shrinking a compute wave frees `80` VGPR of budget |
| 61 | +that other waves can grow into or that lets a feed-heavy mix stay within budget. |
| 62 | +Because feed waves are cheap (`32` VGPR), the pool is launched large enough that |
| 63 | +a feed-heavy mix has many active feed waves. |
| 64 | + |
| 65 | +**Byte-identity invariant:** all Model-B launch/dispatch lives under |
| 66 | +`.if DSWS2_CONV`. With `DSWS2_CONV=0` the kernel assembles byte-identical to the |
| 67 | +proven Phase-A fixed-partition kernel. Task 6's re-baseline runs `DSWS2_CONV=0` |
| 68 | +as the untouched golden reference. |
| 69 | + |
| 70 | +--- |
| 71 | + |
| 72 | +## 2. The dispatcher (universal dynamic dispatch) |
| 73 | + |
| 74 | +**Today:** role is a wid-partition branch taken once at entry (`occ_kernel_dsws.s` |
| 75 | +~L604–629); each role's `_follow` loop ends with an unconditional |
| 76 | +`s_branch .L<samerole>_follow` (L907/L951/compute equivalent). |
| 77 | + |
| 78 | +**New:** replace those three tail branches with one shared, scalar-only, |
| 79 | +wave-uniform trampoline: |
| 80 | + |
| 81 | +```asm |
| 82 | +.Ldispatch: // s59 = current role slot id (24/28/32) |
| 83 | + s_cmp_eq_u32 s59, NCOMP_SLOT |
| 84 | + s_cbranch_scc1 .Lcompute_follow |
| 85 | + s_cmp_eq_u32 s59, NAFEED_SLOT |
| 86 | + s_cbranch_scc1 .Lafeed_follow |
| 87 | + s_branch .Lbfeed_follow |
| 88 | +``` |
| 89 | + |
| 90 | +Three properties make this safe and cheap: |
| 91 | + |
| 92 | +1. **Lands on `_follow`, skipping `_alloc`/`_init`.** `conv_apply` already set |
| 93 | + the wave's VGPR footprint to the target role's size, and `_init` (INITFLAG |
| 94 | + wait) ran once at launch. Re-running `_alloc` would wrongly resize the wave; |
| 95 | + re-running `_init` would deadlock (INITFLAG already consumed). |
| 96 | +2. **Preserves the last-seen-epoch register** (`s35`). A re-dispatched wave |
| 97 | + therefore waits for the *next* epoch in its new role rather than |
| 98 | + re-processing the epoch it just converted in. |
| 99 | +3. **Entry and re-dispatch are the same mechanism.** The entry wid-branch is |
| 100 | + repurposed to *seed* `s59` (and grow the seed-compute waves), then fall into |
| 101 | + `.Ldispatch`. A non-converting wave reads back its own role each epoch and |
| 102 | + loops exactly as today. |
| 103 | + |
| 104 | +The claimer (wid-0) is **excluded**: it branches to `.Lclaimer` at entry (before |
| 105 | +the seed/`.Ldispatch` path), runs its own `.Lclaim_loop`, never converts, and is |
| 106 | +never counted among the `N_POOL−1` bailers. The dispatcher governs non-claimer |
| 107 | +waves only. |
| 108 | + |
| 109 | +Cost: initialize `s59` at each seed role (part of §3), the trampoline (~6 |
| 110 | +lines), replace 3 tail branches. Entirely additive; the proven `_follow` bodies |
| 111 | +are untouched. `s59` read + `s_branch` is scalar-only → **zero** new OOR |
| 112 | +exposure (§6). |
| 113 | + |
| 114 | +Rejected alternatives: per-bail inline role check (less DRY, no benefit); |
| 115 | +unified per-epoch dispatch loop that collapses the three `_follow` bodies |
| 116 | +(rewrites GPU-proven control flow for no functional gain). |
| 117 | + |
| 118 | +--- |
| 119 | + |
| 120 | +## 3. Launch & init (lean-start pool) |
| 121 | + |
| 122 | +All `N_POOL` waves `s_alloc_vgpr 32` at entry (already the case for feeds and |
| 123 | +claimer; new for compute). wid-0 → claimer (lean, stages B as today). |
| 124 | + |
| 125 | +**Seed assignment by wid** (under `DSWS2_CONV`), replacing the fat-launch |
| 126 | +partition: |
| 127 | + |
| 128 | +- wid-0 → claimer. |
| 129 | +- first `SEED_NCOMP` non-claimer wids → `s59 = NCOMP_SLOT`, grow to `NFV`. |
| 130 | +- next `SEED_NAFEED` wids → `s59 = NAFEED_SLOT` (stay lean). |
| 131 | +- remaining wids → `s59 = NBFEED_SLOT` (stay lean). |
| 132 | + |
| 133 | +Then every wave falls into `.Ldispatch`. |
| 134 | + |
| 135 | +**Defsyms (all empirically tuned, principled defaults):** |
| 136 | + |
| 137 | +- `N_POOL` — total launched waves (harness dims must match). Default: the |
| 138 | + feed-heaviest useful count for the target shapes; start 12, sweep. Hard |
| 139 | + constraint: `N_POOL × VLEAN ≤ BUDGET` (every wave always fits lean → no |
| 140 | + parking ever needed). |
| 141 | +- `SEED_NCOMP / SEED_NAFEED / SEED_NBFEED` — launch seed mix. Default: the |
| 142 | + shape's static-best guess (e.g. the current `4c2a2b` scaled to the pool). |
| 143 | + `SEED_NCOMP ≥ 1` (compute floor at launch). |
| 144 | +- `BUDGET` — the real per-SIMD VGPR ceiling (headroom for growth), **not** the |
| 145 | + zero-headroom launch-footprint default Task 5 inherited. This is the knob that |
| 146 | + makes feed→compute grows able to succeed. Validated by the RGA live-VGPR gate |
| 147 | + and a real dispatch — never a guessed constant. |
| 148 | +- `VRESV` seed = `SEED_NCOMP*NFV + (SEED_NAFEED+SEED_NBFEED)*VLEAN` (the live |
| 149 | + reservation the economy adjusts from). |
| 150 | + |
| 151 | +--- |
| 152 | + |
| 153 | +## 4. Conversion policy (sense → decide → commit → re-dispatch) |
| 154 | + |
| 155 | +At each terminal bail (`conv_apply` site), a wave runs Task 5's decision, plus |
| 156 | +two additions: |
| 157 | + |
| 158 | +- **Cooldown `K`** — a per-wave scalar counts down one per epoch; while `> 0` |
| 159 | + the wave skips the watermark check entirely (no ticket race, no conversion). |
| 160 | + On a committed conversion, reset to `K`. Default `K = 0` (spec-faithful; the |
| 161 | + storm gate sets 0 for maximal thrash). `K > 0` is a churn damper available if |
| 162 | + `s_alloc_vgpr` grow/shrink frequency starts bricking runs — its value is |
| 163 | + reducing the *number* of OOR windows per run. The cooldown counter joins the |
| 164 | + persistent scalar set (outside `s60–s65`, alongside `s57/s58/s59`). |
| 165 | +- **Force-convert bring-up hook** — a `DSWS2_FORCE` defsym path where a |
| 166 | + designated wid converts a chosen dir at a chosen epoch, watermarks bypassed; |
| 167 | + everything else static. The deterministic first-proof lever (§7). Gated, off |
| 168 | + by default, emits zero bytes when off. |
| 169 | + |
| 170 | +Direction semantics unchanged from Task 5: `occ_X < CTRL_LOW` → compute starved |
| 171 | +for X → shrink one compute → feed-X (delta −80, always wins); `occ_X > |
| 172 | +CTRL_HIGH_X` → feed-X over-serving → grow one feed-X → compute (delta +80, may |
| 173 | +abort over `BUDGET` → stay). Floor guard keeps every role ≥ 1 wave. |
| 174 | + |
| 175 | +After commit (or no-op), the wave bumps `QUIESCE_CNT` (Task 5 ordering) and |
| 176 | +falls into `.Ldispatch`. |
| 177 | + |
| 178 | +--- |
| 179 | + |
| 180 | +## 5. Quiesce reconciliation (the correctness crux) |
| 181 | + |
| 182 | +**Invariant that must hold every epoch:** *the claimer's per-epoch snapshot of |
| 183 | +the role mix equals the role mix the waves actually execute that epoch.* |
| 184 | + |
| 185 | +Re-dispatch closes Task 5's loop: |
| 186 | + |
| 187 | +1. A converting wave commits its slot swap in epoch *E* at bail time, **before** |
| 188 | + bumping `QUIESCE_CNT` (Task 5 ordering). |
| 189 | +2. The claimer waits for quiesce (all `N_POOL−1` non-claimer waves bailed **and** |
| 190 | + the snapshot sentinels), then snapshots the **live** (now-updated) slot counts |
| 191 | + into the *E+1* parity half of `SNAP_BASE`, resets `QUIESCE_CNT`, and bumps the |
| 192 | + epoch last. |
| 193 | +3. The converting wave, having flipped `s59`, enters its **new** role's `_follow` |
| 194 | + via `.Ldispatch` and wakes for *E+1*. |
| 195 | +4. In *E+1* the wave contributes to the claim counters **as its new role** — |
| 196 | + exactly the mix the claimer snapshotted for *E+1*. Sentinels match observed |
| 197 | + contributions. |
| 198 | + |
| 199 | +The `QUIESCE_CNT ≥ N_POOL−1` cross-check holds unchanged: **every wave bails |
| 200 | +exactly once per epoch regardless of role** (no parking), so the count is |
| 201 | +role-agnostic and invariant under conversion. The DIAG `occ[29]` flag asserts |
| 202 | +snapshot-sentinels ⟺ quiesce-count agree; with re-dispatch wired it reads |
| 203 | +*agree* even when conversions fire (whereas Task 5 alone would diverge — the |
| 204 | +divergence the flag was built to catch). Task 7's dynamic gate verifies this on |
| 205 | +silicon. |
| 206 | + |
| 207 | +--- |
| 208 | + |
| 209 | +## 6. OOR-poison window (SPEC §4, #1 brick risk) |
| 210 | + |
| 211 | +- **The dispatcher adds zero OOR exposure:** `.Ldispatch` is `s59` read + |
| 212 | + `s_branch`, scalar-only. A re-dispatched wave enters its new `_follow` with its |
| 213 | + footprint already correctly sized (`conv_apply` closed its grow window before |
| 214 | + the bump) and no pending grow. |
| 215 | +- **`conv_apply`'s `s_alloc_vgpr` grow** is unchanged from Task 5's audit: waves |
| 216 | + are lean-32 at every bail, every pre-grow LDS/atomic temp ≤ v15. |
| 217 | +- **One new grow site — seed-compute growing at launch init.** Cold path before |
| 218 | + any LDS racing, structurally identical to Phase-A compute's per-rowblk grow and |
| 219 | + equally lean-on-entry. Must carry the same pre-grow comment block and the |
| 220 | + ≤ v15 discipline. |
| 221 | + |
| 222 | +--- |
| 223 | + |
| 224 | +## 7. Failure modes, floors, and no-parking |
| 225 | + |
| 226 | +- **Role floors:** `conv_dec_floor` keeps compute ≥ 1 and each feed ≥ 1 — a role |
| 227 | + can never empty. |
| 228 | +- **Budget safety:** `N_POOL × VLEAN ≤ BUDGET` guarantees the lean floor always |
| 229 | + fits; `reserve_try` guarantees grows only commit within the real ceiling; the |
| 230 | + atomic-add-then-undo-on-abort path leaves `vgpr_reserved` exactly restored (no |
| 231 | + leaked reservation, Task 5-verified). |
| 232 | +- **No parking (YAGNI):** because the pool is sized so all waves stay |
| 233 | + active-lean, a wave never needs to go dormant. Surplus capacity does useful |
| 234 | + lean-feed work instead of sleeping. This is what preserves the simple |
| 235 | + `QUIESCE_CNT ≥ N_POOL−1` accounting and avoids a race-prone wake/sleep |
| 236 | + handshake on the brick-risk path. |
| 237 | + |
| 238 | +--- |
| 239 | + |
| 240 | +## 8. Validation ladder |
| 241 | + |
| 242 | +Offline (no GPU) → then supervised GPU gates, each needing an individual |
| 243 | +greenlight; any brick/hang/DMESG-fault/DIAG-mismatch = full STOP + bisect, never |
| 244 | +auto-advance. |
| 245 | + |
| 246 | +1. **Offline** — assemble all mixes `DSWS2_CONV=1 DIAG=1` → `ASSEMBLE_OK`; RGA |
| 247 | + `SGPR_SPILLS=0 VGPR_SPILLS=0` and live-VGPR within `BUDGET`; `DSWS2_CONV=0` |
| 248 | + sha256 byte-identical to pre-change; CPU control model `ALL PASS`; dry-print |
| 249 | + sanity. |
| 250 | +2. **Force-convert gate (SUPERVISED)** — `DSWS2_FORCE` one designated wave, one |
| 251 | + dir, one known epoch, watermarks bypassed. Expect oracle-CLEAN, `bad=0`, |
| 252 | + `occ[0]=0`, DIAG `occ[29]` agree, dmesg silent. The deterministic debut of a |
| 253 | + wave executing a converted footprint's role code. |
| 254 | +3. **Dynamic gate (SUPERVISED)** — watermark-driven conversions fire; oracle |
| 255 | + stays CLEAN as roles move; `occ[29]` agree across all mixes × tiers. |
| 256 | +4. **Storm gate (SUPERVISED)** — `K=0`, `EPOCH_SHIFT=0`, ×10 repeats — lock-free |
| 257 | + race-hunt under maximal conversion frequency. |
| 258 | + |
| 259 | +--- |
| 260 | + |
| 261 | +## 9. Open parameters (resolved during implementation, not guessed) |
| 262 | + |
| 263 | +- `N_POOL`, `SEED_*`, `BUDGET`, `K` — all empirically tuned defsyms with the |
| 264 | + principled defaults in §3–§4; final values set by RGA + real-dispatch |
| 265 | + measurement on the target ml8 shapes, not chosen a priori. |
| 266 | +- Persistent-scalar assignment for the cooldown counter (must sit outside |
| 267 | + `s60–s65` and every macro's clobber set; candidates alongside `s57/s58/s59`) — |
| 268 | + fixed at implementation time against the live register map. |
0 commit comments