Skip to content

Commit 57c2b5f

Browse files
authored
Merge init-harness into main (#41)
Sync main with the current init-harness baseline.
2 parents ccba80f + 9a9a1e2 commit 57c2b5f

25 files changed

Lines changed: 1662 additions & 115 deletions

CHANGELOG.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,11 +23,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
2323
- Reference implementations for all Week 0-2 operations
2424
- Comprehensive test suite with PyTorch reference validation
2525
- Environment check utility (`forge_cute_py.env_check`)
26-
- Documentation: README, DEVELOPMENT.md (#11), CLAUDE.md, CONTRIBUTING.md with contribution policy (#7)
26+
- Documentation: README, DEVELOPMENT.md (#11), CONTRIBUTING.md with contribution policy (#7)
2727
- Pre-commit hooks installation instructions in CONTRIBUTING.md (#7)
2828
- Support for float16, bfloat16, and float32 dtypes
29+
- `FORGE_SOFTMAX_IMPL` softmax mode selection (`auto`, `ref`, `kernel`)
2930

3031
### Changed
32+
- Modal benchmarks now use strict GPU matching (`!` suffix) for consistent hardware
3133
- **BREAKING:** Renamed `tile` parameter to `tile_size` in `copy_transpose` API (#5)
3234
- Modernized all ops to use two-function pattern: internal `_op_name` (mutates output) and public `op_name` (allocates + returns) (#12)
3335
- Updated `env_check` to use public API instead of `torch.ops` for testing
@@ -36,6 +38,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
3638
- Enhanced error messages with device information for better debugging
3739
- Set tolerance to 0 (exact comparison) for transpose correctness tests
3840
- Updated documentation to reflect current three-layer architecture and modern PyTorch patterns
41+
- Added `--impl` to `bench/benchmark_online_softmax.py` and updated benchmark handling for strict kernel mode
3942

4043
### Infrastructure
4144
- Package scaffolding with `pyproject.toml` and proper Python 3.13+ support

CONTRIBUTING.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,8 @@ A change that affects kernels or harness behavior should include:
8686

8787
The project is currently in `v0.1.0` development:
8888
- Focused on harness stabilization and Week 0-2 kernel implementations
89-
- Not yet tagged - will be tagged when v0.1.0 milestones are complete
89+
- Release candidate tag exists: `v0.1.0-rc1`
90+
- Final `v0.1.0` tag is pending completion of remaining milestones
9091
- See [ROADMAP.md](ROADMAP.md) for detailed progress tracking
9192

9293
### Release process (for maintainers)

DEVELOPMENT.md

Lines changed: 24 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -31,16 +31,17 @@ pattern:
3131
2. **Public API** (`op_name`): Allocates output tensor, calls internal op, returns
3232
result. For ops with autograd, wraps a `torch.autograd.Function`.
3333

34-
This enables usage via both `forge_cute_py.op_name()` and
34+
This enables usage via both `forge_cute_py.ops.op_name()` and
3535
`torch.ops.forge_cute_py._op_name()`.
3636

3737
### Current implementation status (v0.1)
3838

3939
| Op | Kernel | Status |
4040
|----|--------|--------|
4141
| `copy_transpose` | CuTe DSL | Fully implemented with tile-based shared memory |
42-
| `reduce_sum` | Reference | Stub using PyTorch reference (kernel TODO) |
43-
| `softmax_online` | Reference | Stub with autograd support (kernel TODO) |
42+
| `reduce` | CuTe DSL | Placeholder (sum only) |
43+
| `reduce_sum` | Reference | Placeholder (awaiting benchmarking) |
44+
| `softmax_online` | Reference | Stub with autograd support (kernel TODO, mode-gated via `FORGE_SOFTMAX_IMPL`) |
4445

4546
### Development flow
4647

@@ -126,8 +127,28 @@ non-zero tolerance in tests.
126127
```bash
127128
uv run python bench/run.py --suite smoke
128129
uv run python bench/benchmark_copy_transpose.py --tile-size 16
130+
uv run python bench/benchmark_reduce.py
131+
uv run python bench/benchmark_online_softmax.py --impl auto
132+
uv run python bench/benchmark_online_softmax.py --impl kernel
133+
modal run bench/modal_bench.py --suite smoke --out results.json
134+
modal run bench/modal_bench.py --suite smoke --op reduce_sum --out results.json
129135
```
130136

137+
`softmax_online` backend mode is controlled by `FORGE_SOFTMAX_IMPL`:
138+
- `auto` (default): try kernel if present, otherwise fallback to reference.
139+
- `ref`: force reference path.
140+
- `kernel`: require kernel path and fail fast if missing/incomplete.
141+
142+
> **Warning:** Modal benchmarks incur GPU costs. Review `bench/modal_bench.py`
143+
> and verify timeout/GPU settings before running. Start with `--suite smoke`
144+
> to validate your setup. You are responsible for any credits consumed.
145+
146+
Modal benchmarks run on B200 GPUs using CUDA 13.1 and PyTorch 2.9.1.
147+
148+
Modal GPU types and CLI usage:
149+
https://modal.com/docs/guide/gpu
150+
https://modal.com/docs/reference/cli/run
151+
131152
Add new benchmark cases to `bench/suites.yaml` and keep outputs reproducible.
132153

133154
## Profiling

README.md

Lines changed: 27 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
`forge-cute-py` is a project for developing and evaluating **CuTe DSL** kernels in Python.
22

3-
As initially planned, it should provides a workflow to **run kernels, validate correctness against PyTorch references, benchmark performance, and profile**.
3+
As initially planned, it provides a workflow to **run kernels, validate correctness against PyTorch references, benchmark performance, and profile**.
44

55
## Current scope (v0.1)
66

@@ -30,7 +30,7 @@ Not currently in scope for v0: FlashAttention kernels (FA1+), decode/KV-cache, F
3030

3131
```bash
3232
uv sync
33-
````
33+
```
3434

3535
If you need an editable/dev install, use your normal `uv` workflow (project is expected to be runnable via `uv run ...`).
3636

@@ -84,6 +84,8 @@ Run a standalone benchmark:
8484

8585
```bash
8686
uv run python bench/benchmark_copy_transpose.py --tile-size 16
87+
uv run python bench/benchmark_online_softmax.py --impl auto
88+
uv run python bench/benchmark_online_softmax.py --impl kernel
8789
```
8890

8991
Profile a kernel with helper script:
@@ -106,8 +108,9 @@ ncu --set full -o profiles/copy_transpose uv run python bench/benchmark_copy_tra
106108
| Op | Status | Variants | Notes |
107109
| --- | --- | --- | --- |
108110
| copy_transpose | Implemented | tile_size=16/32 | CuTe DSL kernel with tiled shared memory |
109-
| reduce_sum | Stub (ref) | naive/improved/shfl | Uses PyTorch reference; kernel to be implemented |
110-
| softmax_online | Stub (ref) | single-pass | Uses PyTorch reference with autograd support; kernel to be implemented |
111+
| reduce | Implemented (sum only) | - | Placeholder (awaiting benchmarking) |
112+
| reduce_sum | Stub (ref) | - | Placeholder (awaiting benchmarking) |
113+
| softmax_online | Stub (ref) | single-pass | Reference-backed by default; `FORGE_SOFTMAX_IMPL` can force `ref` or strict `kernel` mode |
111114

112115
---
113116

@@ -149,8 +152,28 @@ uv run pre-commit run --all-files # Run linting/formatting
149152
uv run python bench/run.py --suite smoke # Run benchmark suite
150153
uv run python bench/run.py --suite smoke --out out.json # Save results
151154
uv run python bench/benchmark_copy_transpose.py # Standalone benchmark
155+
uv run python bench/benchmark_reduce.py # Standalone benchmark
156+
uv run python bench/benchmark_online_softmax.py --impl auto # softmax fwd+bwd (fallback allowed)
157+
uv run python bench/benchmark_online_softmax.py --impl kernel # softmax fwd+bwd (hard fail if kernel missing)
158+
modal run bench/modal_bench.py --suite smoke --out results.json # Remote run on B200 via Modal
159+
```
160+
161+
`softmax_online` mode is controlled by `FORGE_SOFTMAX_IMPL`:
162+
- `auto` (default): try kernel first, then fallback to reference.
163+
- `ref`: force reference implementation.
164+
- `kernel`: require kernel implementation and fail fast if unavailable.
165+
166+
### Modal setup (remote benchmarks)
167+
```bash
168+
uv pip install modal
169+
modal token new
170+
modal run bench/modal_bench.py --suite smoke --out results.json
171+
modal run bench/modal_bench.py --suite smoke --op reduce_sum --out results.json
152172
```
153173

174+
Modal benchmarks run on B200 GPUs using CUDA 13.1 and PyTorch 2.9.1.
175+
See https://modal.com/docs/guide/gpu for more info.
176+
154177
### Profiling
155178
```bash
156179
./scripts/profile.sh ncu -- uv run python bench/benchmark_copy_transpose.py # Nsight Compute

ROADMAP.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ Target: Harness infrastructure + Week 0-2 kernel implementations aligned to Kern
2323
- [x] Profiling documentation and helper scripts
2424
- [x] CI/CD with ruff linting and formatting
2525
- [x] Pre-commit hooks configuration
26-
- [x] Documentation (README, CONTRIBUTING, CLAUDE.md)
26+
- [x] Documentation (README, CONTRIBUTING, DEVELOPMENT)
2727

2828
### Copy/Transpose Kernel
2929
- [x] CuTe DSL kernel implementation (`forge_cute_py/kernels/copy_transpose.py`)

bench/__init__.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
"""Benchmark suite for forge-cute-py kernels."""

bench/benchmark_online_softmax.py

Lines changed: 170 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,170 @@
1+
"""Benchmark softmax_online op against torch.softmax and torch.compile(torch.softmax)."""
2+
3+
import argparse
4+
import os
5+
6+
import torch
7+
8+
from forge_cute_py.ops.softmax_online import softmax_fwd, softmax_bwd
9+
from forge_cute_py.util.bench import do_bench, estimate_bandwidth, summarize_times
10+
11+
SHORT_M = [128, 512, 2048, 8192]
12+
SHORT_N = [1024, 2048, 4096, 8192]
13+
14+
LONG_M = [64, 128, 256]
15+
LONG_N = [16384, 32768, 65536, 131072]
16+
17+
DEFAULT_DTYPES = ["float16", "bfloat16", "float32"]
18+
19+
20+
def parse_int_list(s: str) -> list[int]:
21+
return [int(x.strip()) for x in s.split(",")]
22+
23+
24+
def parse_str_list(s: str) -> list[str]:
25+
return [x.strip() for x in s.split(",")]
26+
27+
28+
def main():
29+
parser = argparse.ArgumentParser(description="Benchmark softmax_online op")
30+
parser.add_argument(
31+
"--long", action="store_true", help="Use long-N benchmark suite (small M, large N)"
32+
)
33+
parser.add_argument("--m-sizes", type=parse_int_list, default=None)
34+
parser.add_argument("--n-sizes", type=parse_int_list, default=None)
35+
parser.add_argument("--dtypes", type=parse_str_list, default=DEFAULT_DTYPES)
36+
parser.add_argument("--warmup", type=int, default=20)
37+
parser.add_argument("--iterations", type=int, default=100)
38+
parser.add_argument(
39+
"--impl",
40+
choices=["auto", "ref", "kernel"],
41+
default="auto",
42+
help="softmax_online backend mode (FORGE_SOFTMAX_IMPL)",
43+
)
44+
args = parser.parse_args()
45+
os.environ["FORGE_SOFTMAX_IMPL"] = args.impl
46+
47+
if args.m_sizes is None:
48+
args.m_sizes = LONG_M if args.long else SHORT_M
49+
if args.n_sizes is None:
50+
args.n_sizes = LONG_N if args.long else SHORT_N
51+
52+
if not torch.cuda.is_available():
53+
raise RuntimeError("CUDA required for benchmarking")
54+
55+
gpu_name = torch.cuda.get_device_name(0)
56+
suite = "long" if args.long else "short"
57+
print(f"softmax_online benchmarks [{suite}] ({gpu_name}) [impl={args.impl}]")
58+
print()
59+
60+
header = (
61+
f"{'M':>6} {'N':>6} {'Dtype':<10} {'Op':<18} {'Pass':<5} "
62+
f"{'p50 (ms)':>10} {'BW (GB/s)':>10} {'vs torch':>10}"
63+
)
64+
print(header)
65+
print("-" * len(header))
66+
67+
for m in args.m_sizes:
68+
for n in args.n_sizes:
69+
for dtype_str in args.dtypes:
70+
dtype = getattr(torch, dtype_str)
71+
x = torch.randn(m, n, device="cuda", dtype=dtype)
72+
elem = x.element_size()
73+
74+
# --- Forward bandwidth: read input + write output ---
75+
fwd_bytes = 2 * m * n * elem
76+
77+
# --- torch.softmax fwd baseline ---
78+
torch_fn = lambda: torch.softmax(x, dim=-1)
79+
torch_times = do_bench(torch_fn, warmup=args.warmup, rep=args.iterations)
80+
torch_stats = summarize_times(torch_times)
81+
torch_fwd_p50 = torch_stats["p50_ms"]
82+
torch_fwd_bw = estimate_bandwidth(fwd_bytes, torch_fwd_p50)
83+
print(
84+
f"{m:>6} {n:>6} {dtype_str:<10} {'torch.softmax':<18} {'fwd':<5} "
85+
f"{torch_fwd_p50:>10.4f} {torch_fwd_bw:>10.2f} {1.0:>10.2f}x"
86+
)
87+
88+
# --- torch.compile fwd ---
89+
try:
90+
compiled_ref = torch.compile(lambda t: torch.softmax(t, dim=-1))
91+
compiled_ref(x)
92+
fn = lambda: compiled_ref(x)
93+
compiled_times = do_bench(fn, warmup=args.warmup, rep=args.iterations)
94+
compiled_stats = summarize_times(compiled_times)
95+
compiled_p50 = compiled_stats["p50_ms"]
96+
compiled_bw = estimate_bandwidth(fwd_bytes, compiled_p50)
97+
ratio = compiled_p50 / torch_fwd_p50 if torch_fwd_p50 > 0 else float("inf")
98+
print(
99+
f"{m:>6} {n:>6} {dtype_str:<10} {'torch.compile':<18} {'fwd':<5} "
100+
f"{compiled_p50:>10.4f} {compiled_bw:>10.2f} {ratio:>10.2f}x"
101+
)
102+
except Exception as e:
103+
print(
104+
f"{m:>6} {n:>6} {dtype_str:<10} {'torch.compile':<18} {'fwd':<5} "
105+
f"{'ERROR':>10} {'':>10} {'':>10} {e}"
106+
)
107+
108+
# --- softmax_online fwd ---
109+
try:
110+
softmax_fwd(x, dim=-1)
111+
fn = lambda: softmax_fwd(x, dim=-1)
112+
times = do_bench(fn, warmup=args.warmup, rep=args.iterations)
113+
stats = summarize_times(times)
114+
p50 = stats["p50_ms"]
115+
bw = estimate_bandwidth(fwd_bytes, p50)
116+
ratio = p50 / torch_fwd_p50 if torch_fwd_p50 > 0 else float("inf")
117+
print(
118+
f"{m:>6} {n:>6} {dtype_str:<10} {'softmax_online':<18} {'fwd':<5} "
119+
f"{p50:>10.4f} {bw:>10.2f} {ratio:>10.2f}x"
120+
)
121+
except Exception as e:
122+
print(
123+
f"{m:>6} {n:>6} {dtype_str:<10} {'softmax_online':<18} {'fwd':<5} "
124+
f"{'ERROR':>10} {'':>10} {'':>10} {e}"
125+
)
126+
127+
# --- Backward pass benchmarks ---
128+
# Pre-compute softmax output y and fake upstream gradient dy
129+
y = torch.softmax(x, dim=-1)
130+
dy = torch.randn_like(y)
131+
132+
# Backward bandwidth: read dy + read y + write dx = 3 * M * N * elem
133+
bwd_bytes = 3 * m * n * elem
134+
135+
# --- torch backward baseline ---
136+
torch_bwd_fn = lambda: torch._softmax_backward_data(dy, y, -1, x.dtype)
137+
torch_bwd_times = do_bench(torch_bwd_fn, warmup=args.warmup, rep=args.iterations)
138+
torch_bwd_stats = summarize_times(torch_bwd_times)
139+
torch_bwd_p50 = torch_bwd_stats["p50_ms"]
140+
torch_bwd_bw = estimate_bandwidth(bwd_bytes, torch_bwd_p50)
141+
print(
142+
f"{m:>6} {n:>6} {dtype_str:<10} {'torch.softmax':<18} {'bwd':<5} "
143+
f"{torch_bwd_p50:>10.4f} {torch_bwd_bw:>10.2f} {1.0:>10.2f}x"
144+
)
145+
146+
# --- softmax_online bwd ---
147+
try:
148+
y_ours = softmax_fwd(x, dim=-1)
149+
softmax_bwd(dy, y_ours, dim=-1)
150+
fn = lambda: softmax_bwd(dy, y_ours, dim=-1)
151+
times = do_bench(fn, warmup=args.warmup, rep=args.iterations)
152+
stats = summarize_times(times)
153+
p50 = stats["p50_ms"]
154+
bw = estimate_bandwidth(bwd_bytes, p50)
155+
ratio = p50 / torch_bwd_p50 if torch_bwd_p50 > 0 else float("inf")
156+
print(
157+
f"{m:>6} {n:>6} {dtype_str:<10} {'softmax_online':<18} {'bwd':<5} "
158+
f"{p50:>10.4f} {bw:>10.2f} {ratio:>10.2f}x"
159+
)
160+
except Exception as e:
161+
print(
162+
f"{m:>6} {n:>6} {dtype_str:<10} {'softmax_online':<18} {'bwd':<5} "
163+
f"{'ERROR':>10} {'':>10} {'':>10} {e}"
164+
)
165+
166+
print()
167+
168+
169+
if __name__ == "__main__":
170+
main()

0 commit comments

Comments
 (0)