Merge MPS 4-bit matmul sub-phase (M1-M4): native gemv + gemm on Metal

Brings the native Metal 4-bit matmul path onto main, following the earlier
quantize/dequantize phases.

- M1: benchmarked the unfused baseline; decided the design fork with numbers
  (gemv is dequant-bound -> fuse; large-M gemm -> MPSMatrixMultiplication).
- M2: fused native gemv_4bit MSL kernel (dequant in registers, no B_dq), 3.5-7x.
- M3: native gemm_4bit = chunked dequant + MPSMatrixMultiplication on one command
  buffer (bf16 falls back -- MPSMatMul has no bf16 on macOS 26.4.1).
- M4: measured the per-call sync tax (~0.15ms, architectural -- documented, not
  removed; queue-sharing rejected with evidence), verified the offset-0 clone is
  load-bearing, and updated docs honestly (README QLoRA-4bit row -> checkmark with
  a footnote on the bf16/large-M caveats).

328 parity tests green under BNB_MPS_REQUIRE_NATIVE=1; native paths asserted via
spy tests; graceful fallback preserved. LLM.int8 and 8-bit optimizers remain
out of scope on mps.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: eaglstun

README.md

@@ -182,12 +182,18 @@ bitsandbytes has the following minimum requirements for all platforms:
       <td>⬜ Metal <br><code>mps</code></td>
       <td>Apple M1+</td>
       <td>❌</td>
-      <td>🐢</td>
+      <td>✅ <sup>1</sup></td>
       <td>❌</td>
     </tr>
   </tbody>
 </table>
 
+<small><sup>1</sup> On <code>mps</code>, 4-bit matmul runs on native Metal kernels: a fused gemv for
+inference (M=1) and an <code>MPSMatrixMultiplication</code>-backed GEMM for fp16/fp32 batches. bf16
+batched matmul and other unsupported shapes use a slower dequantize+matmul fallback, and large-batch
+GEMM performs on par with dequantize+matmul. Details:
+<a href="./docs/apple_silicon/README.md">docs/apple_silicon</a>.</small>
+
 ## :book: Documentation
 
 - [Official Documentation](https://huggingface.co/docs/bitsandbytes/main)

_typos.toml

@@ -15,6 +15,7 @@ extend-ignore-re = [
 extend-ignore-identifiers-re = [
     ".*arange.*",
     ".*ARANGE.*",
+    "numer",  # mach_timebase_info_data_t.numer (csrc/mps_ops.mm)
 ]
 
 [type.py.extend-words]

benchmarks_wip/bench_gemm_baseline.py

@@ -0,0 +1,76 @@
+"""Phase M1 baseline + Phase M3 comparison for gemm_4bit.
+
+M1 question: as M grows, does the GEMM cost overtake dequant? (Answer: crossover near
+M~512; below it gemm is dequant-bound.)
+
+M3 question: what does the native path (chunked dequant -> scratch -> MPSMatrixMultiplication
+-> bias, ONE command buffer / ONE sync) buy over the dequant + F.linear fallback (native
+dequant wait + torch GEMM + second sync)? `gemm_4bit` routes native automatically when built,
+so `native` here is just the op; `fallback` reproduces the old tail verbatim.
+"""
+
+import time
+
+import torch
+
+import bitsandbytes.backends.mps.ops as mps_ops
+import bitsandbytes.functional as F
+
+DEV = "mps"
+ITERS = 30
+WARMUP = 8
+
+
+def sync():
+    torch.mps.synchronize()
+
+
+def timed(fn):
+    for _ in range(WARMUP):
+        fn()
+    sync()
+    t0 = time.perf_counter()
+    for _ in range(ITERS):
+        fn()
+    sync()
+    return (time.perf_counter() - t0) / ITERS * 1e3
+
+
+def bench(M, N, K, dtype, quant_type="nf4", blocksize=64):
+    A = torch.randn(1, M, K, dtype=dtype, device=DEV)
+    B = torch.randn(N, K, dtype=dtype, device=DEV)
+    B_q, qs = F.quantize_4bit(B, blocksize=blocksize, quant_type=quant_type)
+
+    def native():
+        # Routes through bnb_mps_gemm_4bit when the native library is built (fp32/fp16).
+        return torch.ops.bitsandbytes.gemm_4bit(A, B_q, list(B.shape), qs.absmax, blocksize, quant_type)
+
+    def dequant_only():
+        return torch.ops.bitsandbytes.dequantize_4bit(B_q, qs.absmax, blocksize, quant_type, list(B.shape), dtype)
+
+    def fallback():
+        # The pre-M3 tail: native dequant (its own sync) + torch F.linear (torch's queue).
+        B_dq = mps_ops._dequantize_4bit_impl(B_q, qs.absmax, blocksize, quant_type, list(B.shape), dtype)
+        return torch.nn.functional.linear(A, B_dq)
+
+    B_dq = dequant_only()
+
+    def linear_only():
+        return torch.nn.functional.linear(A, B_dq)
+
+    t_nat, t_fb, t_deq, t_lin = timed(native), timed(fallback), timed(dequant_only), timed(linear_only)
+    print(
+        f"  M={M:>4} N={N:>5} K={K:>5} {str(dtype).replace('torch.', ''):>8}  "
+        f"native={t_nat:7.3f}ms  fallback={t_fb:7.3f}ms  ({t_fb / t_nat:4.2f}x)  "
+        f"[fallback = dequant {t_deq:6.3f} + linear {t_lin:6.3f}]"
+    )
+
+
+if __name__ == "__main__":
+    native = "native" if mps_ops._native_available() else "FALLBACK-ONLY (no native build)"
+    print(f"iters={ITERS} warmup={WARMUP} device={DEV} lib={native}\n")
+    for dtype in (torch.float16, torch.float32):
+        print(f"=== gemm_4bit, N=K=4096, {str(dtype).replace('torch.', '')} ===")
+        for M in (8, 64, 512, 2048):
+            bench(M, 4096, 4096, dtype)
+        print()

benchmarks_wip/bench_gemv_fused.py

@@ -0,0 +1,79 @@
+"""Phase M2: fused native gemv_4bit vs the Phase-M1 baseline (dequant -> F.linear).
+
+Per shape, times:
+  - fused    torch.ops.bitsandbytes.gemv_4bit (routes to the fused Metal kernel)
+  - baseline native dequant of B + F.linear (what gemv_4bit did before Phase M2)
+and reports ms/iter plus the fused kernel's effective read bandwidth
+(packed B + absmax + A, i.e. the memory the fused kernel actually touches).
+"""
+
+import time
+
+import torch
+
+from bitsandbytes.backends.mps import ops as mps_ops
+import bitsandbytes.functional as F
+
+DEV = "mps"
+ITERS = 50
+WARMUP = 10
+
+assert mps_ops._native_available(), "native MPS library required for this benchmark"
+assert hasattr(mps_ops._mps_native._lib, "bnb_mps_gemv_4bit"), "fused gemv kernel missing"
+
+
+def sync():
+    torch.mps.synchronize()
+
+
+def timed(fn):
+    for _ in range(WARMUP):
+        fn()
+    sync()
+    t0 = time.perf_counter()
+    for _ in range(ITERS):
+        fn()
+    sync()
+    return (time.perf_counter() - t0) / ITERS * 1e3  # ms/iter
+
+
+def bench(N, K, dtype, quant_type="nf4", blocksize=64):
+    A = torch.randn(1, 1, K, dtype=dtype, device=DEV)
+    B = torch.randn(N, K, dtype=dtype, device=DEV)
+    B_q, absmax = torch.ops.bitsandbytes.quantize_4bit(B, blocksize, quant_type, torch.uint8)
+    code = F.get_4bit_type(quant_type, device=DEV, blocksize=blocksize)
+
+    def fused():
+        return torch.ops.bitsandbytes.gemv_4bit(A, B_q, B.shape, absmax, code, blocksize)
+
+    def baseline():
+        B_dq = torch.ops.bitsandbytes.dequantize_4bit(B_q, absmax, blocksize, quant_type, list(B.shape), dtype)
+        return torch.nn.functional.linear(A, B_dq)
+
+    # Sanity: fused output matches the baseline it replaces.
+    ref = baseline()
+    got = fused()
+    max_err = (got.float() - ref.float()).abs().max().item()
+
+    t_fused = timed(fused)
+    t_base = timed(baseline)
+
+    # Memory the fused kernel reads: packed B (N*K/2 bytes) + absmax (N*K/blocksize fp32)
+    # + A (K fp32); writes out (N fp32).
+    bytes_moved = N * K // 2 + (N * K // blocksize) * 4 + K * 4 + N * 4
+    gbps = bytes_moved / (t_fused * 1e-3) / 1e9
+
+    print(
+        f"  N={N:>6} K={K:>6} {str(dtype).replace('torch.', ''):>8}  "
+        f"fused={t_fused:7.3f}ms  baseline={t_base:7.3f}ms  "
+        f"speedup={t_base / t_fused:5.1f}x  fused-read={gbps:6.1f} GB/s  max|err|={max_err:.2e}"
+    )
+
+
+if __name__ == "__main__":
+    print(f"iters={ITERS} warmup={WARMUP} device={DEV}\n")
+    for dtype in (torch.float16, torch.bfloat16, torch.float32):
+        print(f"=== gemv_4bit (M=1), {str(dtype).replace('torch.', '')} ===")
+        for N, K in [(4096, 4096), (11008, 4096), (4096, 11008)]:
+            bench(N, K, dtype)
+        print()

benchmarks_wip/bench_matmul_baseline.py

@@ -0,0 +1,70 @@
+"""Phase M1 baseline: how slow is today's unfused gemv_4bit (dequant -> F.linear) on MPS?
+
+Times, per shape:
+  - total  gemv_4bit (native dequant of B + torch F.linear)
+  - dequant-only (native _dequantize_4bit_impl of B)  -> the isolable cost
+  - F.linear on an already-materialized B_dq          -> the GEMM cost
+so we can see where the wall-clock actually goes and whether fusing dequant into
+the matmul (Option B) or just moving the GEMM on-device (Option A) is the win.
+"""
+
+import time
+
+import torch
+
+import bitsandbytes.functional as F
+
+DEV = "mps"
+ITERS = 50
+WARMUP = 10
+
+
+def sync():
+    torch.mps.synchronize()
+
+
+def timed(fn):
+    for _ in range(WARMUP):
+        fn()
+    sync()
+    t0 = time.perf_counter()
+    for _ in range(ITERS):
+        fn()
+    sync()
+    return (time.perf_counter() - t0) / ITERS * 1e3  # ms/iter
+
+
+def bench(N, K, dtype, quant_type="nf4", blocksize=64):
+    A = torch.randn(1, 1, K, dtype=dtype, device=DEV)
+    B = torch.randn(N, K, dtype=dtype, device=DEV)
+    B_q, absmax = torch.ops.bitsandbytes.quantize_4bit(B, blocksize, quant_type, torch.uint8)
+    code = F.get_4bit_type(quant_type, device=DEV, blocksize=blocksize)
+
+    def full():
+        return torch.ops.bitsandbytes.gemv_4bit(A, B_q, B.shape, absmax, code, blocksize)
+
+    def dequant_only():
+        return torch.ops.bitsandbytes.dequantize_4bit(B_q, absmax, blocksize, quant_type, list(B.shape), dtype)
+
+    B_dq = dequant_only()
+
+    def linear_only():
+        return torch.nn.functional.linear(A, B_dq)
+
+    t_full = timed(full)
+    t_deq = timed(dequant_only)
+    t_lin = timed(linear_only)
+    print(
+        f"  N={N:>6} K={K:>6} {str(dtype).replace('torch.', ''):>8}  "
+        f"total={t_full:7.3f}ms  dequant={t_deq:7.3f}ms  linear={t_lin:7.3f}ms  "
+        f"(dequant is {100 * t_deq / t_full:4.1f}% of total)"
+    )
+
+
+if __name__ == "__main__":
+    print(f"iters={ITERS} warmup={WARMUP} device={DEV}\n")
+    for dtype in (torch.float16, torch.bfloat16):
+        print(f"=== gemv_4bit (M=1), {str(dtype).replace('torch.', '')} ===")
+        for N, K in [(4096, 4096), (11008, 4096), (4096, 11008)]:
+            bench(N, K, dtype)
+        print()