Skip to content

Commit c234ba4

Browse files
physicsrobclaude
andcommitted
Strip doom-only TODOs from TODO.md
TODO.md had six items, four of which were action items against doom code that no longer lives in this repo: - 'fp16 inference for Flash Attention' — render decode loop, targets `torchwright/doom/compile.py`, `step_frame`, `make walkthrough`. - 'Move ceiling/floor into the transformer' — the same Dumb-host- principle violation already deleted from CLAUDE.md, targets `torchwright/doom/compile.py` / `game_graph.py`. - 'Clip column iteration to screen bounds' — render state machine, targets `torchwright/doom/game_graph.py`. - 'Make graph-side done_flag work when N < max_walls' — render state machine, targets `torchwright/doom/game_graph.py` / `compile.py`. Step E moved the doom code to the sibling `torchwright_doom` repo; these TODOs belong with it, not here. Kept the two general compiler-work items, both of which target files that still exist here: - 'Tests that validate Asserts on the compiled graph' — calls for tests that exercise `check_asserts_on_compiled` on real compiled graphs. Lightly scrubbed: replaced 'on the DOOM graph and on smaller stage-level graphs' with 'representative compiled graphs (the example transformers under `examples/`, plus stage-level synthetic graphs)', dropped 'the class of bug Mode C was' (only appearance of 'Mode C' in the repo — its definition lived in a deleted postmortem), and pointed new tests at `tests/` rather than the deleted `tests/doom/` and `tests/doom/stages/`. - 'Softmax hardness assertion on attend_* primitives' — proposes an `assert_hardness_gt` kwarg across the `attend_*` family. All target files exist; kept verbatim. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 5702eac commit c234ba4

1 file changed

Lines changed: 5 additions & 79 deletions

File tree

TODO.md

Lines changed: 5 additions & 79 deletions
Original file line numberDiff line numberDiff line change
@@ -1,97 +1,23 @@
11
# TODO
22

3-
## fp16 inference for Flash Attention
4-
5-
The render decode loop runs at ~35ms/step (GPU-bound). The bottleneck is SDPA
6-
over long sequences (~800 positions): each step attends over an 800×800-ish
7-
matrix using PyTorch's materialized ("math") softmax kernel.
8-
9-
**Why the math kernel is used:** PyTorch's SDPA dispatcher only enables Flash
10-
Attention for fp16/bf16 inputs. All model weights and activations are currently
11-
fp32, so every SDPA call falls back to the materialized kernel regardless of
12-
whether an `attn_mask` is provided.
13-
14-
**The opportunity:** Casting weights and activations to fp16 at inference time
15-
(e.g., `.half()` on `CompiledHeadless._net`) would let SDPA dispatch to Flash
16-
Attention on A100. Flash Attention is O(N) in memory and substantially faster
17-
for long sequences — potentially 3–5× faster SDPA, which could bring decode
18-
from ~35ms to ~15–20ms per step.
19-
20-
The dynamic decode path (`forward_cached`, `is_causal=False`, no explicit mask)
21-
already satisfies all Flash Attention preconditions except dtype. The static
22-
KV cache path (added in this branch) uses a float `attn_mask` and would need
23-
to switch to a boolean mask or be dropped in favour of the dynamic path.
24-
25-
**Risk:** Weights are compiled in fp32 and encode precise numerical thresholds
26-
(comparison results, attention argmin/argmax patterns). fp16 has ~3 significant
27-
decimal digits vs ~7 for fp32. It's unknown whether the compiled logic survives
28-
the precision loss without correctness failures. A test that compares fp32 vs
29-
fp16 `step_frame` outputs would answer this.
30-
31-
**Suggested experiment:**
32-
1. `module._net.to(torch.float16)` after `compile_game()`
33-
2. Cast inputs to fp16 in `_build_res_stream`
34-
3. Run `make walkthrough ARGS="--frames 5"` and compare to fp32 reference
35-
4. If pixel errors are within tolerance, fp16 is viable
36-
37-
Files: `torchwright/compiler/export.py` (`_build_res_stream`, `CompiledHeadless`),
38-
`torchwright/compiler/components/attn.py` (weights), `torchwright/doom/compile.py`
39-
40-
## Move ceiling/floor into the transformer
41-
42-
The host currently decides ceiling vs floor per pixel (`ceil if y < center_y
43-
else floor_c`). This is computation on the host side. The transformer should
44-
emit ceiling/floor pixels itself — either as part of each render chunk (fill
45-
non-wall rows within the chunk) or as a separate post-render pass.
46-
47-
Files: `torchwright/doom/compile.py` (step_frame render loop),
48-
`torchwright/doom/game_graph.py` (render output)
49-
50-
## Clip column iteration to screen bounds
51-
52-
The state machine iterates col_lo..col_hi, which can include negative columns
53-
and columns >= W (e.g., cols=[-2, 122) with W=120). The host safely skips
54-
out-of-bounds columns, but each is still a full autoregressive step (~70ms).
55-
At 120x100 with 4 full-screen walls, ~16 steps are wasted per frame.
56-
57-
Fix: clamp col_lo/col_hi to [0, W) in the graph, or have the state machine
58-
skip out-of-bounds columns via the feedback loop.
59-
60-
Files: `torchwright/doom/game_graph.py` (state machine col iteration)
61-
62-
## Make graph-side done_flag work when N < max_walls
63-
64-
The graph checks `mask_sum > max_walls - 0.5`, which only fires when all
65-
max_walls slots are masked. When the scene has fewer walls, the host
66-
terminates via bit-count instead. The graph's done_flag computation runs on
67-
every RENDER token but never triggers — wasted graph nodes.
68-
69-
Options:
70-
- Feed N as a graph input so the threshold is dynamic
71-
- Detect sentinel wall data (score=99) after all real walls are masked
72-
- Accept the current host-side fix and remove the graph-side done_flag entirely
73-
74-
Files: `torchwright/doom/game_graph.py` (state_transitions),
75-
`torchwright/doom/compile.py` (host-side termination)
76-
773
## Tests that validate Asserts on the compiled graph
784

795
`check_asserts_on_compiled` (`torchwright/debug/probe.py:964`) already
806
runs each `Assert`'s predicate against the compiled transformer's
817
residual stream, but no tests currently call it. Asserts today only
828
fire during reference evaluation — the whole point of checking them
839
post-compile is to catch invariants that exact math satisfies but
84-
compiled approximations violate (the class of bug Mode C was).
10+
compiled approximations violate.
8511

86-
Add tests that exercise it on the DOOM graph and on smaller stage-level
87-
graphs. At minimum: collect the asserts via
12+
Add tests that exercise it on representative compiled graphs (the
13+
example transformers under `examples/`, plus stage-level synthetic
14+
graphs). At minimum: collect the asserts via
8815
`torchwright.graph.asserts.collect_asserts(output_node)` before
8916
`compile_headless`, compile, then call `check_asserts_on_compiled` on
9017
a representative input battery and assert no violations.
9118

9219
Files: `torchwright/debug/probe.py` (`check_asserts_on_compiled`,
93-
`collect_asserts`), new tests under `tests/doom/` and
94-
`tests/doom/stages/`.
20+
`collect_asserts`), new tests under `tests/`.
9521

9622
## Softmax hardness assertion on `attend_*` primitives
9723

0 commit comments

Comments
 (0)