fix(vllm-cpp): default the MLX GEMM provider OFF on darwin

This branch opened with VLLM_CPP_MLX=on, justified by an A/B that measured the
MLX provider at 1.88x to 2.19x against the native MSL GEMM. That measurement was
correct when taken and is now stale: vllm.cpp's own Metal kernels have improved
several-fold since, through mma prefill attention, a vectorised decode V
accumulation, vectorised attention staging, a fused qk-norm-RoPE preamble and a
simdgroup-per-row softmax. The native path MLX was compared against no longer
exists.

Re-measured on the same Apple M4, in the same binary, with the arms toggled by
VT_OP_PROVIDER_DISABLE=mlx, on Qwen3-1.7B-bf16 warm at p=512 g=128:

  MLX provider ON   prefill TTFT 1370 ms   warm throughput 11.98 tok/s
  MLX provider OFF  prefill TTFT 1400 ms   warm throughput 22.06 tok/s

Shipping the previous default would have halved Apple Silicon throughput.

MLX's steel GEMM is still about 20% faster than ours in isolation, but the
provider pays a per-op mx::eval synchronisation plus an output memcpy, because it
cannot write into our buffer. Across prefill's roughly 112 GEMMs that overhead
leaves a 2% gain; on decode, where the same synchronisation is paid once per
matmul per token, it costs 46%. The option is kept for prefill-dominated
workloads, where the margin is small but real.

The README section is rewritten rather than patched: it previously presented the
stale table as the reason for the default, so leaving it in place would have made
the new default look arbitrary.

Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Author: mudler

backend/go/vllm-cpp/Makefile

@@ -17,7 +17,12 @@ VLLM_CPP_VERSION?=9e1c9025ae61167a3335454d7cc0de6093c21845
 # Consumed as the prebuilt pip wheel: building MLX from source needs `xcrun
 # metal`, i.e. a full Xcode the macOS runners do not have, while the wheel ships
 # include/, lib/libmlx.dylib and the compiled mlx.metallib ready to link.
-VLLM_CPP_MLX?=on
+#
+# DEFAULT OFF. It was on when this branch opened, on the strength of an A/B that
+# had MLX at 1.88x. That measurement is stale: vllm.cpp's own Metal kernels have
+# since improved several-fold, and re-measured on the same M4 the provider is now
+# 46% SLOWER end to end (11.98 vs 22.06 warm tok/s). See the README.
+VLLM_CPP_MLX?=off
 MLX_VERSION?=0.29.3
 MLX_VENV?=$(abspath ./mlx-venv)
 # Resolved lazily (recursive `=`, not `:=`): the glob only matches once the venv

backend/go/vllm-cpp/README.md

@@ -41,30 +41,39 @@ options:
 - max_num_seqs:16
 ```
 
-## Apple Silicon: the MLX GEMM provider
-
-`BUILD_TYPE=metal` builds the Metal backend with vllm.cpp's optional MLX
-provider for the dense GEMM (`VLLM_CPP_MLX=on`, the default here). Upstream keeps
-it off because it costs a ~19 MB `libmlx.dylib` plus a ~105 MB `mlx.metallib`;
-this backend accepts that because the provider was measured to pay for it on an
-Apple M4, against the native MSL GEMM in the SAME binary (arms toggled with
-`VT_OP_PROVIDER_DISABLE=mlx`), Qwen3-1.7B-bf16 at p=512 g=128:
-
-| Concurrency | MLX agg tok/s | native agg tok/s | speedup |
-|--:|--:|--:|--:|
-| 1 | 5.79 | 3.08 | 1.88x |
-| 8 | 25.70 | 13.69 | 1.88x |
-| 16 | 38.65 | 17.69 | 2.19x |
-
-TTFT improves 2x to 3x, peak memory is unchanged, and the GEMM output is
-bit-identical to the native kernel on every parity shape. MLX serves the dense
-GEMM only: paged attention stays vllm.cpp's own kernel, because MLX has no
-paged-KV primitive. Full disposition in vllm.cpp `docs/BENCHMARKS.md`,
-"MLX GEMM provider A/B on Apple M4".
+## Apple Silicon: the MLX GEMM provider (OFF by default)
+
+`BUILD_TYPE=metal` can build vllm.cpp's optional MLX provider for the dense GEMM
+(`VLLM_CPP_MLX=on`). **It is OFF by default, because it is currently slower.**
+
+This branch originally shipped it ON, on the strength of an A/B that had MLX at
+1.88-2.19x against the native MSL GEMM. That measurement was honest when taken
+and is now stale: vllm.cpp's Metal kernels have since improved several-fold
+(mma prefill attention, vectorised decode V accumulation, a fused qk-norm-RoPE
+preamble and more), so the native path no longer resembles the one MLX was
+compared against.
+
+Re-measured on the same Apple M4, same binary, arms toggled with
+`VT_OP_PROVIDER_DISABLE=mlx`, Qwen3-1.7B-bf16 warm at p=512 g=128:
+
+| | prefill TTFT | warm throughput |
+|---|--:|--:|
+| MLX provider ON | 1370 ms | **11.98 tok/s** |
+| MLX provider OFF | 1400 ms | **22.06 tok/s** |
+
+MLX's steel GEMM is still ~20% faster than ours in isolation, but the provider
+pays a per-op `mx::eval` synchronisation plus an output `memcpy` (it cannot write
+into our buffer). On prefill's ~112 GEMMs that overhead leaves +2%; on decode,
+where the same sync is paid once per matmul per token, it costs 46%.
+
+Turning it on is therefore only sensible for prefill-dominated workloads, and
+even then the margin is small. Full disposition in vllm.cpp `docs/BENCHMARKS.md`,
+"The MLX provider verdict".
 
 Build knobs:
 
-- `VLLM_CPP_MLX=off` builds Metal without the provider: ~124 MB smaller, slower.
+- `VLLM_CPP_MLX=on` builds the provider in: ~19 MB `libmlx.dylib` plus a ~105 MB
+  `mlx.metallib`, and currently slower end to end. Off is the default.
 - `MLX_VERSION` pins the wheel (default `0.29.3`). MLX is consumed as the
   prebuilt pip wheel because building it from source needs `xcrun metal`, i.e. a
   full Xcode the macOS runners do not have.