Skip to content

Add Flash-SAE: High-performance Triton kernels for SAE training - #3

Open
alepot55 wants to merge 1 commit into
koayon:mainfrom
alepot55:add-flash-sae
Open

Add Flash-SAE: High-performance Triton kernels for SAE training#3
alepot55 wants to merge 1 commit into
koayon:mainfrom
alepot55:add-flash-sae

Conversation

@alepot55

Copy link
Copy Markdown

Adds Flash-SAE to the Open Source Libraries section.

Flash-SAE provides high-performance Triton kernels for training Top-K Sparse Autoencoders:

  • 13.6x decoder speedup via sparse gather operations
  • 97% memory reduction by never materializing dense activations
  • Ghost Gradients for dead latent recovery
  • FP8 quantization support (Ada Lovelace+)
  • Full PyTorch autograd compatibility

Repository: https://github.com/alepot55/flash-sae

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant