|
1 | 1 | # TODO |
2 | 2 |
|
3 | | -## fp16 inference for Flash Attention |
4 | | - |
5 | | -The render decode loop runs at ~35ms/step (GPU-bound). The bottleneck is SDPA |
6 | | -over long sequences (~800 positions): each step attends over an 800×800-ish |
7 | | -matrix using PyTorch's materialized ("math") softmax kernel. |
8 | | - |
9 | | -**Why the math kernel is used:** PyTorch's SDPA dispatcher only enables Flash |
10 | | -Attention for fp16/bf16 inputs. All model weights and activations are currently |
11 | | -fp32, so every SDPA call falls back to the materialized kernel regardless of |
12 | | -whether an `attn_mask` is provided. |
13 | | - |
14 | | -**The opportunity:** Casting weights and activations to fp16 at inference time |
15 | | -(e.g., `.half()` on `CompiledHeadless._net`) would let SDPA dispatch to Flash |
16 | | -Attention on A100. Flash Attention is O(N) in memory and substantially faster |
17 | | -for long sequences — potentially 3–5× faster SDPA, which could bring decode |
18 | | -from ~35ms to ~15–20ms per step. |
19 | | - |
20 | | -The dynamic decode path (`forward_cached`, `is_causal=False`, no explicit mask) |
21 | | -already satisfies all Flash Attention preconditions except dtype. The static |
22 | | -KV cache path (added in this branch) uses a float `attn_mask` and would need |
23 | | -to switch to a boolean mask or be dropped in favour of the dynamic path. |
24 | | - |
25 | | -**Risk:** Weights are compiled in fp32 and encode precise numerical thresholds |
26 | | -(comparison results, attention argmin/argmax patterns). fp16 has ~3 significant |
27 | | -decimal digits vs ~7 for fp32. It's unknown whether the compiled logic survives |
28 | | -the precision loss without correctness failures. A test that compares fp32 vs |
29 | | -fp16 `step_frame` outputs would answer this. |
30 | | - |
31 | | -**Suggested experiment:** |
32 | | -1. `module._net.to(torch.float16)` after `compile_game()` |
33 | | -2. Cast inputs to fp16 in `_build_res_stream` |
34 | | -3. Run `make walkthrough ARGS="--frames 5"` and compare to fp32 reference |
35 | | -4. If pixel errors are within tolerance, fp16 is viable |
36 | | - |
37 | | -Files: `torchwright/compiler/export.py` (`_build_res_stream`, `CompiledHeadless`), |
38 | | -`torchwright/compiler/components/attn.py` (weights), `torchwright/doom/compile.py` |
39 | | - |
40 | | -## Move ceiling/floor into the transformer |
41 | | - |
42 | | -The host currently decides ceiling vs floor per pixel (`ceil if y < center_y |
43 | | -else floor_c`). This is computation on the host side. The transformer should |
44 | | -emit ceiling/floor pixels itself — either as part of each render chunk (fill |
45 | | -non-wall rows within the chunk) or as a separate post-render pass. |
46 | | - |
47 | | -Files: `torchwright/doom/compile.py` (step_frame render loop), |
48 | | -`torchwright/doom/game_graph.py` (render output) |
49 | | - |
50 | | -## Clip column iteration to screen bounds |
51 | | - |
52 | | -The state machine iterates col_lo..col_hi, which can include negative columns |
53 | | -and columns >= W (e.g., cols=[-2, 122) with W=120). The host safely skips |
54 | | -out-of-bounds columns, but each is still a full autoregressive step (~70ms). |
55 | | -At 120x100 with 4 full-screen walls, ~16 steps are wasted per frame. |
56 | | - |
57 | | -Fix: clamp col_lo/col_hi to [0, W) in the graph, or have the state machine |
58 | | -skip out-of-bounds columns via the feedback loop. |
59 | | - |
60 | | -Files: `torchwright/doom/game_graph.py` (state machine col iteration) |
61 | | - |
62 | | -## Make graph-side done_flag work when N < max_walls |
63 | | - |
64 | | -The graph checks `mask_sum > max_walls - 0.5`, which only fires when all |
65 | | -max_walls slots are masked. When the scene has fewer walls, the host |
66 | | -terminates via bit-count instead. The graph's done_flag computation runs on |
67 | | -every RENDER token but never triggers — wasted graph nodes. |
68 | | - |
69 | | -Options: |
70 | | -- Feed N as a graph input so the threshold is dynamic |
71 | | -- Detect sentinel wall data (score=99) after all real walls are masked |
72 | | -- Accept the current host-side fix and remove the graph-side done_flag entirely |
73 | | - |
74 | | -Files: `torchwright/doom/game_graph.py` (state_transitions), |
75 | | -`torchwright/doom/compile.py` (host-side termination) |
76 | | - |
77 | 3 | ## Tests that validate Asserts on the compiled graph |
78 | 4 |
|
79 | 5 | `check_asserts_on_compiled` (`torchwright/debug/probe.py:964`) already |
80 | 6 | runs each `Assert`'s predicate against the compiled transformer's |
81 | 7 | residual stream, but no tests currently call it. Asserts today only |
82 | 8 | fire during reference evaluation — the whole point of checking them |
83 | 9 | post-compile is to catch invariants that exact math satisfies but |
84 | | -compiled approximations violate (the class of bug Mode C was). |
| 10 | +compiled approximations violate. |
85 | 11 |
|
86 | | -Add tests that exercise it on the DOOM graph and on smaller stage-level |
87 | | -graphs. At minimum: collect the asserts via |
| 12 | +Add tests that exercise it on representative compiled graphs (the |
| 13 | +example transformers under `examples/`, plus stage-level synthetic |
| 14 | +graphs). At minimum: collect the asserts via |
88 | 15 | `torchwright.graph.asserts.collect_asserts(output_node)` before |
89 | 16 | `compile_headless`, compile, then call `check_asserts_on_compiled` on |
90 | 17 | a representative input battery and assert no violations. |
91 | 18 |
|
92 | 19 | Files: `torchwright/debug/probe.py` (`check_asserts_on_compiled`, |
93 | | -`collect_asserts`), new tests under `tests/doom/` and |
94 | | -`tests/doom/stages/`. |
| 20 | +`collect_asserts`), new tests under `tests/`. |
95 | 21 |
|
96 | 22 | ## Softmax hardness assertion on `attend_*` primitives |
97 | 23 |
|
|
0 commit comments