Skip to content

Commit c680a47

Browse files
0xClandestine0xClandestine
committed
Merge remote-tracking branch 'upstream/main' into feat/add-dflash
2 parents 7d150f9 + 336c8a8 commit c680a47

10 files changed

Lines changed: 1368 additions & 79 deletions

File tree

‎.agents/workflows/review-github-pr.md‎

Lines changed: 216 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,216 @@
1+
---
2+
description: Review a GitHub Issue or PR for SharpAI/SwiftLM — fetch, analyze, implement fixes, address review comments, and push back to the correct branch
3+
---
4+
5+
# Review GitHub Issue / PR
6+
7+
This workflow guides end-to-end handling of a GitHub Issue or Pull Request for the
8+
`SharpAI/SwiftLM` repository: from fetching context, through implementing or
9+
reviewing code changes, to pushing a clean commit back to the correct fork branch.
10+
11+
---
12+
13+
## Prerequisites
14+
15+
- `gh` CLI path on macOS: **`/opt/homebrew/bin/gh`**
16+
```bash
17+
export PATH="/opt/homebrew/bin:$PATH"
18+
which gh # → /opt/homebrew/bin/gh
19+
```
20+
- `gh` must be authenticated (`gh auth status`)
21+
- Working directory: `/Users/simba/workspace/mlx-server`
22+
- Remote `fork` may need to be added if pushing to a contributor's fork:
23+
```bash
24+
git remote add fork https://github.com/<contributor>/SwiftLM.git
25+
```
26+
27+
---
28+
29+
## Steps
30+
31+
### 1. Fetch the Issue or PR
32+
33+
Determine whether the user supplied an **Issue number** or a **PR number**, then
34+
pull the full context using `gh`:
35+
36+
```bash
37+
# For a PR
38+
gh pr view <NUMBER> --repo SharpAI/SwiftLM \
39+
--json number,title,body,state,baseRefName,headRefName,headRepository,commits,files
40+
41+
# For an Issue
42+
gh issue view <NUMBER> --repo SharpAI/SwiftLM \
43+
--json number,title,body,state,labels,comments
44+
```
45+
46+
Note the **`headRepository`** field — if it is not `SharpAI/SwiftLM`, the PR comes
47+
from a fork. You must push back to the fork's branch (see Step 6).
48+
49+
---
50+
51+
### 2. Understand the Scope
52+
53+
Read the PR/Issue body and associated comments carefully. Identify:
54+
55+
- **Category** — bug fix, feature, test improvement, CI/CD, documentation.
56+
- **Files touched** — run `gh pr diff <NUMBER> --repo SharpAI/SwiftLM` or read
57+
the `files` field.
58+
- **CI status** — check the latest run:
59+
```bash
60+
gh run list --repo SharpAI/SwiftLM --branch <headRefName> --limit 3
61+
```
62+
- **Review comments** — if Copilot or a human left inline review comments, read
63+
them all before writing a single line of code:
64+
```bash
65+
gh pr view <NUMBER> --repo SharpAI/SwiftLM --comments
66+
```
67+
68+
---
69+
70+
### 3. Check Out the Branch Locally
71+
72+
```bash
73+
# If the PR is from SharpAI directly
74+
git fetch origin
75+
git checkout <headRefName>
76+
77+
# If the PR is from a fork
78+
git remote add fork https://github.com/<forkOwner>/SwiftLM.git # once only
79+
git fetch fork <headRefName>
80+
git checkout -b <headRefName> fork/<headRefName>
81+
```
82+
83+
Verify you are on the correct branch:
84+
```bash
85+
git status
86+
git log --oneline -5
87+
```
88+
89+
---
90+
91+
### 4. Triage Review Comments (for PRs)
92+
93+
For each Copilot or human review comment:
94+
95+
1. **Classify** the severity:
96+
- 🔴 **Must fix** — correctness bugs, resource leaks, race conditions, broken CI.
97+
- 🟡 **Should fix** — test coverage gaps, false-pass logic, missing imports.
98+
- 🟢 **Optional** — style, wording, architecture refactors beyond the PR scope.
99+
100+
2. **Implement** all 🔴 and 🟡 items. For 🟢 items, document them as follow-up
101+
work in a code comment or GitHub comment but do not expand the PR scope.
102+
103+
3. **Key patterns learned from SwiftLM history**:
104+
- Shell scripts use `set -euo pipefail` — every `grep`, `jq`, or pipeline that
105+
may produce no output **must** be guarded with `|| true` or placed inside an
106+
`if` condition to prevent silent script abort.
107+
- Heartbeat / background `Task` objects in Swift **must** be cancelled via
108+
`defer { task?.cancel() }` so all exit paths (including client disconnect)
109+
are covered — not just the happy path.
110+
- CORS-related shell tests must target the dedicated `--cors` server instance,
111+
not the main server started without the flag.
112+
- Concurrent-request tests must use `--parallel N` (N ≥ 2) to actually exercise
113+
parallel code paths.
114+
- When adding new Swift test files that use `Data` / `JSONSerialization`,
115+
always add `import Foundation` — XCTest does not re-export it in all SPM environments.
116+
117+
---
118+
119+
### 5. Verify Locally
120+
121+
Build and run the relevant test suite before pushing:
122+
123+
```bash
124+
# Swift unit tests
125+
swift test --filter SwiftLMTests
126+
127+
# Integration tests (server)
128+
./tests/test-server.sh .build/release/SwiftLM 15413
129+
130+
# OpenCode / SDK compatibility test
131+
./tests/test-opencode.sh .build/release/SwiftLM 15414
132+
```
133+
134+
If CI previously failed with a specific test number, reproduce it locally first:
135+
```bash
136+
gh run view <RUN_ID> --repo SharpAI/SwiftLM --log-failed 2>&1 | grep -E "FAIL|error|Test [0-9]+"
137+
```
138+
139+
---
140+
141+
### 6. Commit and Push to the Correct Remote
142+
143+
> [!IMPORTANT]
144+
> Always push to the **fork's branch** when updating a fork-originated PR.
145+
> Pushing to `origin` (SharpAI) creates a new branch and does NOT update the PR.
146+
147+
```bash
148+
git add <files>
149+
git commit -m "<type>(<scope>): <summary>
150+
151+
<body: what changed and why>"
152+
153+
# PR from a fork → push to fork
154+
git push fork <headRefName>:<headRefName>
155+
156+
# PR from SharpAI directly → push to origin
157+
git push origin <headRefName>
158+
```
159+
160+
Verify the PR was updated:
161+
```bash
162+
gh pr view <NUMBER> --repo SharpAI/SwiftLM --json commits --jq '.commits[].messageHeadline'
163+
```
164+
165+
---
166+
167+
### 7. Monitor CI
168+
169+
After pushing, monitor the triggered workflow:
170+
171+
```bash
172+
# List recent runs on the branch
173+
gh run list --repo SharpAI/SwiftLM --branch <headRefName> --limit 5
174+
175+
# Stream logs for the latest run
176+
gh run view <RUN_ID> --repo SharpAI/SwiftLM --log
177+
178+
# Pull only failed steps
179+
gh run view <RUN_ID> --repo SharpAI/SwiftLM --log-failed 2>&1 | grep -E "FAIL|error|exit code"
180+
```
181+
182+
If tests fail, go back to Step 4. Iterate until CI is green.
183+
184+
---
185+
186+
### 8. Respond to Reviewers (Optional)
187+
188+
If a human or Copilot reviewer left inline comments that you have addressed,
189+
leave a reply comment summarising what was changed and why each item was handled
190+
(or deferred):
191+
192+
```bash
193+
gh pr comment <NUMBER> --repo SharpAI/SwiftLM \
194+
--body "Addressed all 🔴/🟡 review comments in commit <SHA>:
195+
- heartbeat leak: added defer cleanup in both streaming handlers
196+
- import Foundation: added to ServerSSETests.swift
197+
- CORS test: redirected to CORS_PORT server
198+
- parallel test: dedicated --parallel 2 server on PORT+3
199+
- set -e trap: guarded grep/jq pipelines with || true"
200+
```
201+
202+
---
203+
204+
## Quick Reference
205+
206+
| Task | Command |
207+
|------|---------|
208+
| View PR | `gh pr view <N> --repo SharpAI/SwiftLM` |
209+
| View PR diff | `gh pr diff <N> --repo SharpAI/SwiftLM` |
210+
| View PR comments | `gh pr view <N> --repo SharpAI/SwiftLM --comments` |
211+
| View Issue | `gh issue view <N> --repo SharpAI/SwiftLM` |
212+
| List CI runs | `gh run list --repo SharpAI/SwiftLM --branch <branch>` |
213+
| Failed CI logs | `gh run view <ID> --repo SharpAI/SwiftLM --log-failed` |
214+
| Push to fork | `git push fork <branch>:<branch>` |
215+
| Push to SharpAI | `git push origin <branch>` |
216+
| Verify PR commits | `gh pr view <N> --repo SharpAI/SwiftLM --json commits --jq '.commits[].messageHeadline'` |

‎.github/workflows/ci.yml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,9 @@ jobs:
6262
- name: SwiftBuddy Tests (MemPalace & Lifecycle)
6363
run: swift test --skip-build --filter SwiftBuddyTests --disable-swift-testing
6464

65+
- name: SwiftLM Server Tests (Streaming & SSE)
66+
run: swift test --skip-build --filter SwiftLMTests --disable-swift-testing
67+
6568
- name: Upload Binary Artifact
6669
uses: actions/upload-artifact@v4
6770
with:
@@ -73,10 +76,11 @@ jobs:
7376
needs: build_and_unit_test
7477
runs-on: macos-15
7578
timeout-minutes: 30
79+
continue-on-error: ${{ matrix.modality == 'opencode' }}
7680
strategy:
7781
fail-fast: false
7882
matrix:
79-
modality: [server, vision, audio, graph, omni]
83+
modality: [server, vision, audio, graph, omni, opencode]
8084
steps:
8185
- uses: actions/checkout@v4
8286
with:

‎Package.swift‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,10 @@ let package = Package(
114114
.testTarget(
115115
name: "SwiftBuddyTests",
116116
dependencies: ["SwiftBuddy", "MLXInferenceCore"]
117+
),
118+
.testTarget(
119+
name: "SwiftLMTests",
120+
dependencies: ["SwiftLM"]
117121
)
118122
]
119123
)

‎README.md‎

Lines changed: 102 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -89,25 +89,76 @@ Benchmark results for `gemma-4-26b-a4b-it-4bit` (26B MoE, 4-bit) on M5 Pro 64 GB
8989

9090
---
9191

92-
## 🧠 Supported Models & Methodologies
92+
## 📡 Supported Models & Methodologies
9393

94-
`SwiftLM` dynamically maps Apple MLX primitives to standard HuggingFace architectures, enabling complete support for the latest frontier open-weights models across modalities (Text, Vision, Audio).
94+
`SwiftLM` dynamically maps Apple MLX primitives to standard HuggingFace architectures, enabling native Metal inference across the latest frontier open-weights models.
9595

96-
### Text (LLMs)
97-
- **Gemma 4**: Fully supports both Dense (`gemma-4-e4b`) and Sparse Mixture of Experts (MoE) architectures (`gemma-4-26b`, `gemma-4-31b`).
98-
- **Qwen 2.5 & 3**: Robust support for sliding window attention limits and custom RoPE scaling.
99-
- **Mistral & Mixtral**: Out-of-the-box structural mappings.
100-
- **Phi-3 & Phi-3.5**: Full 128k context parsing via Swift chunked-prefill.
96+
### 💬 Text (LLMs)
10197

102-
### Vision (VLMs)
98+
| Family | Models | Notes |
99+
|---|---|---|
100+
| **Gemma 4** | `gemma-4-e2b`, `gemma-4-e4b` (dense) · `gemma-4-26b-a4b`, `gemma-4-31b` (MoE) | Interleaved local + global attention; KV sharing; native quantized KV cache (issue #71 fix) |
101+
| **Gemma 3 / 3n** | `gemma-3-*`, `gemma-3n-*` | Google Gemma 3 and nano variants |
102+
| **Gemma / Gemma 2** | `gemma-*`, `gemma-2-*` | Original Gemma family |
103+
| **Qwen 3.5** | `Qwen3.5-7B`, `Qwen3.5-27B`, `Qwen3.5-122B-A10B`, `Qwen3.5-397B-A22B` | Dense + MoE; SSD streaming at 10× for 122B/397B |
104+
| **Qwen 3** | `Qwen3-*` (dense + MoE) | Sliding window + hybrid attention |
105+
| **Qwen 2.5** | `Qwen2.5-7B`, `Qwen2.5-14B`, `Qwen2.5-72B` | Robust RoPE scaling |
106+
| **Qwen 2** | `Qwen2-*` | Linear RoPE variants |
107+
| **Phi 4 / PhiMoE** | `phi-4-mlx`, `Phi-3.5-MoE` | Microsoft Phi family incl. MoE |
108+
| **Phi 3 / Phi** | `Phi-3`, `Phi-3.5-mini` | 128k context via chunked prefill |
109+
| **Mistral / Mixtral** | `Mistral-7B`, `Mistral-4`, `Mixtral-*` | GQA + sliding window variants |
110+
| **Llama / Llama 3** | `Llama-3.1-*`, `Llama-3.2-*`, `Llama-3.3-*` | YaRN + dynamic NTK RoPE scaling |
111+
| **GLM 4** | `GLM-4-*` | THUDM GLM-4 dense + MoE-Lite variants |
112+
| **DeepSeek V3** | `DeepSeek-V3-*` | MLA attention architecture |
113+
| **Falcon H1** | `Falcon-H1-*` | Falcon hybrid SSM+attention |
114+
| **LFM 2** | `LFM2-*`, `LFM2-MoE-*` | Liquid AI dense + MoE |
115+
| **OLMo 2 / OLMo 3 / OLMoE** | `OLMo-2-*`, `OLMo-3-*` | AllenAI open language models |
116+
| **Granite / GraniteMoE** | `Granite-*`, `GraniteMoE-Hybrid-*` | IBM Granite hybrid Mamba+attention |
117+
| **SmolLM 3** | `SmolLM3-*` | HuggingFace compact LM |
118+
| **MiniCPM** | `MiniCPM-*` | Lightweight efficient LM |
119+
| **InternLM 2** | `InternLM2-*` | Shanghai AI Lab series |
120+
| **Cohere / Command-R** | `Command-R-*`, `c4ai-*` | Cohere retrieval-tuned models |
121+
| **Jamba** | `Jamba-v0.1` | AI21 hybrid Mamba+attention |
122+
| **Exaone 4** | `EXAONE-4.0-*` | LG AI Research |
123+
| **MiMo / MiMo V2** | `MiMo-7B-*` | Xiaomi reasoning model |
124+
| **Ernie 4.5** | `ERNIE-4.5-*` | Baidu ERNIE series |
125+
| **Baichuan M1** | `Baichuan-M1-*` | Baichuan multimodal base |
126+
| **Bailing MoE** | `Ling-*` | Bailing/Ling MoE family |
127+
| **NemotronH** | `Nemotron-H-*` | NVIDIA Nemotron hybrid |
128+
| **Starcoder 2** | `starcoder2-*` | Code generation |
129+
| **OpenELM** | `OpenELM-*` | Apple on-device efficient LM |
130+
| **Apertus / AfMoE** | `Apertus-*` | Sparse MoE research models |
131+
| **BitNet** | `bitnet-*` | 1-bit weight quantization |
132+
| **MiniMax** | `MiniMax-Text-*` | Lightning attention architecture |
133+
| **Olmo3** | `Olmo3-*` | AllenAI Olmo3 series |
134+
135+
### 👁️ Vision (VLMs)
103136
*Run with `--vision` flag.*
104-
- **Qwen2-VL & Qwen3-VL**: Real-time positional bounding and Metal image scaling.
105-
- **PaliGemma / LFM2-VL / Pixtral**: Base64 spatial decomposition.
106137

107-
### Audio (ALMs)
108-
*Run with `--audio` flag.*
109-
- **Qwen2-Audio (7B-Instruct)**: Deep multi-modal spectrogram processing via Swift audio interleaving.
110-
- **Gemma-4 Audio Pipelines**: Ready for Audio-in/Text-out variants mapping `.audio_tower` extraction parameters natively off NVMe.
138+
| Family | Models | Notes |
139+
|---|---|---|
140+
| **Gemma 4** | `gemma-4-*` (VLM mode) | Native image tower via MLXVLM |
141+
| **Gemma 3** | `gemma-3-*` (VLM mode) | PaLiGemma-style image projection |
142+
| **Qwen3-VL / Qwen3.5-VL** | `Qwen3-VL-*`, `Qwen3.5-VL-*` | Dynamic resolution with native RoPE |
143+
| **Qwen2-VL / Qwen2.5-VL** | `Qwen2-VL-2B/7B`, `Qwen2.5-VL-*` | Real-time positional bounding + Metal image scaling |
144+
| **LFM2-VL** | `LFM2-VL-1.6B` | Liquid AI multimodal |
145+
| **Pixtral** | `pixtral-12b` | Mistral vision model |
146+
| **PaliGemma** | `paligemma-*` | Google vision-language |
147+
| **Idefics 3** | `Idefics3-*` | HuggingFace multimodal |
148+
| **Mistral 3** | `Mistral-Small-3.1-*` | Mistral vision variant |
149+
| **FastVLM** | `FastVLM-*` | Apple on-device VLM |
150+
| **SmolVLM 2** | `SmolVLM2-*` | HuggingFace compact VLM |
151+
| **GLM OCR** | `glm-4v-*` | THUDM vision+OCR |
152+
| **QwenVL** | `Qwen-VL-*` | Original Qwen VL |
153+
154+
### 🎧 Audio (ALMs)
155+
*Run with `--audio` flag. Only `gemma-4-e4b` variants include an audio tower.*
156+
157+
| Family | Models | Notes |
158+
|---|---|---|
159+
| **Gemma 4 Omni** | `gemma-4-e4b-it-4bit`, `gemma-4-e4b-it-8bit` | Audio-in via vDSP STFT → Mel spectrogram (16kHz, 128 bins); text-out |
160+
161+
111162

112163
---
113164

@@ -352,10 +403,46 @@ curl http://localhost:5413/v1/chat/completions \
352403
| `--min-p` | `0.0` | Default min-p sampling threshold relative to the highest probability token (0 disables) |
353404
| `--gpu-layers` | `model_default`| Restrict the amount of layers allocated to GPU hardware |
354405
| `--stream-experts` | `false` | Enable SSD expert streaming for MoE models (10x speedup) |
355-
| `--turbo-kv` | `false` | Enable TurboQuant 3-bit KV cache compression |
406+
| `--turbo-kv` | `false` | Enable TurboQuant 3-bit KV cache compression (activates after 2048 tokens, server-wide) |
356407
| `--draft-model` | (none) | Draft model path/ID for speculative decoding (in-RAM models only) |
357408
| `--num-draft-tokens` | `4` | Number of draft tokens per speculation round |
358409

410+
## 🔧 Per-Request API Parameters
411+
412+
In addition to the standard OpenAI fields (`temperature`, `top_p`, `max_tokens`, etc.), SwiftLM accepts the following **SwiftLM-specific** fields on `POST /v1/chat/completions`:
413+
414+
| Field | Type | Description |
415+
|---|---|---|
416+
| `kv_bits` | `int` (4 or 8) | Enable **MLX-native quantized KV cache** for this request. Uses `QuantizedKVCache` (standard group quantization) instead of `KVCacheSimple`. Separate from `--turbo-kv`. Reduces KV memory ~2–4× at mild quality cost. |
417+
| `enable_thinking` | `bool` | Force-enable or disable chain-of-thought thinking blocks for Gemma-4 / Qwen3. |
418+
| `kv_group_size` | `int` | Group size for `kv_bits` quantization (default: `64`). |
419+
| `top_k` | `int` | Per-request top-k sampling override (0 = disabled). |
420+
| `min_p` | `float` | Per-request min-p sampling threshold (0 = disabled). |
421+
| `repetition_penalty` | `float` | Token repetition penalty (e.g. `1.15`). |
422+
423+
### `kv_bits` vs `--turbo-kv` — What's the difference?
424+
425+
| | `kv_bits` (per-request) | `--turbo-kv` (server flag) |
426+
|---|---|---|
427+
| **Scope** | Per-request, sent in JSON body | Server-wide, set at startup |
428+
| **Algorithm** | MLX-native group quantization (4-bit / 8-bit) | Custom 3-bit PolarQuant + QJL Walsh-Hadamard |
429+
| **Activation** | From token 0 | After 2048 tokens |
430+
| **Memory savings** | ~2–4× vs FP16 | ~3.5× vs FP16 |
431+
| **Use case** | Targeted memory reduction per conversation | Extreme long-context (100K+) compression |
432+
433+
### Example: Enable 4-bit KV cache per request
434+
```bash
435+
curl http://localhost:5413/v1/chat/completions \\
436+
-H "Content-Type: application/json" \\
437+
-d '{
438+
"model": "gemma-4-26b-a4b-it-4bit",
439+
"kv_bits": 4,
440+
"messages": [
441+
{"role": "user", "content": "Summarize the history of computing in 3 sentences."}
442+
]
443+
}'
444+
```
445+
359446
## 📦 Requirements
360447

361448
- macOS 14.0+

0 commit comments

Comments
 (0)