Feature/layer streaming#1440
Open
SashimiSaketoro wants to merge 7 commits into
Open
Conversation
Wire asymmetric TurboQuant caches through generate_step, stream_generate, BatchGenerator, and the HTTP server with CLI flags. Disable server request batching when TurboQuant is active since those caches do not support merge. Co-authored-by: Cursor <cursoragent@cursor.com>
Implement BatchAsymmetricTurboQuantCache with merge, extract, prepare, and finalize support so mlx_lm.server batching works with TurboQuant KV caches. Fix deepcopy for MLX dtype fields and restore batchability checks. Co-authored-by: Cursor <cursoragent@cursor.com>
Add rolling-window layer loader, split_model utility, and StreamingModelWrapper in mlx_lm/streaming for running models larger than available unified memory using mlx from PyPI. Includes load_streaming() entry point and mlx_lm.split_model CLI.
- Auto-split via ensure_streaming_layout in load_streaming - --streaming flags on mlx_lm.generate CLI - Fix StreamingModelWrapper recursion and layer weight keys - Add benchmark CLI, get_stats, generate_step e2e tests
SashimiSaketoro
force-pushed
the
feature/layer-streaming
branch
from
3af9a82 to
6e6655e
Compare
- --streaming / --max-memory-gb / --streaming-window on mlx_lm.server - --split-for-streaming on mlx_lm.convert - Forward and sampling mx.array tests for streaming path - scripts/verify_streaming_consumer.py for end-to-end MLX tensor checks
SashimiSaketoro
force-pushed
the
feature/layer-streaming
branch
from
6e6655e to
0786571
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.