Title: Show and Tell: Eco-Metal — 63 Modular Plugins for Advanced LLM Inference natively on MSL
Body:
Hey MLX team! First of all, huge thanks for the amazing work on MLX. Apple Silicon is absolutely game-changing for local AI.
I've spent the past few months heavily utilizing your framework to build Eco-Metal (https://github.com/helgklaizar/Eco-Metal), a production-ready ecosystem of 63 modular AI components fully optimized for Mac.
Our main focus was to eliminate slow Python overhead and CUDA-wrappers. We've successfully ported and hardened custom Metal Shading Language (MSL) kernels and native mx.fast paths for several SOTA algorithms:
- Paged Attention & KV Traversal: True zero-copy caching architectures taking advantage of the 800 GB/s Unified Memory.
- H2O Heavy-Hitters KV Predictors: Dynamically drops non-essential KV tokens for infinite contexts.
- Extreme Quantization: (1-2 bit multi-codebook lookups natively on Metal)
- Rocket Pruning & Tri-Attention: Custom Metal branches handling sparse retrieval.
We ensure 100% test coverage and native JIT execution via mx.fast.metal_kernel and mx.fast.scaled_dot_product_attention.
I would be incredibly honored if someone from the core AMLR team (or anyone else!) could check out the project. I'm completely open to submitting any of our high-performance Metal MSL kernels directly upstream to MLX if you feel they fit the primary framework requirements!
Ecosystem Repository: https://github.com/helgklaizar/Eco-Metal
Cheers, and keep up the brilliant work!
Title: Show and Tell: Eco-Metal — 63 Modular Plugins for Advanced LLM Inference natively on MSL
Body:
Hey MLX team! First of all, huge thanks for the amazing work on MLX. Apple Silicon is absolutely game-changing for local AI.
I've spent the past few months heavily utilizing your framework to build Eco-Metal (
https://github.com/helgklaizar/Eco-Metal), a production-ready ecosystem of 63 modular AI components fully optimized for Mac.Our main focus was to eliminate slow Python overhead and CUDA-wrappers. We've successfully ported and hardened custom Metal Shading Language (MSL) kernels and native
mx.fastpaths for several SOTA algorithms:We ensure 100% test coverage and native JIT execution via
mx.fast.metal_kernelandmx.fast.scaled_dot_product_attention.I would be incredibly honored if someone from the core AMLR team (or anyone else!) could check out the project. I'm completely open to submitting any of our high-performance Metal MSL kernels directly upstream to MLX if you feel they fit the primary framework requirements!
Ecosystem Repository: https://github.com/helgklaizar/Eco-Metal
Cheers, and keep up the brilliant work!