Hello! I created a custom INT8 dp4a kernel for Pascal architecture to bypass the slow PyTorch fp8 fallback. It doubles the generation speed of Flux on GTX 1080/Ti and Titan X (from ~70s to ~30s).
I built this testing on a community fork (Neo), but the core CUDA math and PyTorch routing logic should be applicable to your experimental backend as well.
Profiler traces and source code are available here: [Твоя ссылка на GitHub].
Just wanted to share this in case you find the dp4a implementation useful for supporting older hardware in the new Forge backend.
Hello! I created a custom INT8 dp4a kernel for Pascal architecture to bypass the slow PyTorch fp8 fallback. It doubles the generation speed of Flux on GTX 1080/Ti and Titan X (from ~70s to ~30s).
I built this testing on a community fork (Neo), but the core CUDA math and PyTorch routing logic should be applicable to your experimental backend as well.
Profiler traces and source code are available here: [Твоя ссылка на GitHub].
Just wanted to share this in case you find the dp4a implementation useful for supporting older hardware in the new Forge backend.