Feature request: run MoE expert weights from host RAM via PCIe DMA (no H2D copy) — 23GB MoE on 1.6GB VRAM, 81GB Qwen3-235B on 8GB cards

[yalun753]

TL;DR

For MoE models, keep expert weights in host memory (pinned buffer) and let cuBLAS read them directly over PCIe DMA — never copy experts into VRAM. Non-expert layers stay in VRAM as usual. Result (RTX 4090, real measurements):

Model	Standard (all-in-VRAM)	This approach	VRAM saved	Speed
DeepSeek-V2-Lite Q2_K (23.3GB)	23.3GB VRAM, 65 t/s	1.6GB VRAM, 37.5 t/s	−93%	58% retained
Qwen3.6-35B-A3B (11GB)	OOM on 8GB cards	2.1GB VRAM, 46.8 t/s	—	≈ parity
Qwen3-235B-A22B Q2_K (81.7GB)	impossible on consumer GPUs	8GB VRAM (verification in progress)	—	TBD

Motivation

The biggest real-world MoE use case is running a large model on a small GPU. With all experts in VRAM, an 8GB card tops out around a 7-8B dense or a ~11GB MoE. But MoE experts are only activated sparsely (2-8 of 256 per token) — the full weight set doesn't need to live on the GPU. The bottleneck is that llama.cpp's offload path copies expert weights H2D per layer (see #25859), and the --n-cpu-moe path computes experts on CPU (slow).

Proposed approach (high level)

1. Load expert weights into a host pinned buffer (not GPU VRAM, not anonymous mmap that gets evicted).
2. Run the expert matmuls on GPU via cuBLAS operating directly on host pointers — the CUDA driver DMA-reads the expert weights from host RAM over PCIe on demand, per activated expert. No H2D staging copy, no per-layer streaming.
3. VRAM holds only non-expert layers + KV cache. For DS-V2-Lite that's ~1.6GB total.
4. Optional: a small expert cache (a few slots, LRU) that D2D-copies the hot experts into VRAM once and reuses them — helps when the same experts fire repeatedly (e.g. long prompts / codegen patterns).

Measured results (RTX 4090, 2026-08-02)

• DeepSeek-V2-Lite Q2_K: VRAM 23.3GB → 1.6GB; speed 37.5 t/s (vs 65 t/s fully offloaded — reasonable PCIe price for −93% VRAM). Expert cache at 0.25: prompt throughput 99 → 308 t/s (+211%), gen +5%.
• Qwen3.6-35B-A3B: VRAM 2.1GB, 46.8 t/s — an 11GB MoE that OOMs an 8GB card normally, now runs on ~2GB with near-full speed.
• Qwen3-235B-A22B Q2_K (81.7GB): download in progress; target: run on 8GB VRAM (experts ~80GB in host RAM + PCIe DMA).

Why this is different from existing work

• llama-hot-experts: pin hottest MoE experts in RAM via --pin-hot-experts #26414 (pin hot experts in RAM via mlock): keeps experts in system RAM but still computes them on CPU (or pages them back) — this keeps expert compute on GPU with DMA reads.
• Offloaded-MoE prefill leaves the GPU idle waiting on serial expert H2D copies (single-GPU -ncmoe) #25859 (-ncmoe H2D copies serial): that path still copies weights into VRAM per layer; this approach reads them in place from host memory — the copy disappears entirely.
• Feature Request: Allow "spill-over KV cache", "KV-cache GPUs" and "MoE GPUs" #23855 (spill-over / MoE GPUs): multi-GPU layout question; orthogonal.

This also composes with existing features: --n-cpu-moe-style flag to select the expert storage mode, GGML_OP_OFFLOAD_MIN_BATCH to control when offload kicks in, and the expert cache as an optional accelerator.

Ask

• Would the maintainers accept a host-pinned expert buffer + DMA-direct expert matmul path as an official load mode (e.g. --expert-mode host-dma)?
• Happy to open a PR with the CUDA backend changes (it's a focused patch: model loader + CUDA expert matmul path + optional small expert cache).
• Open to discussion on trade-offs: PCIe bandwidth vs VRAM, multi-GPU, pageable vs pinned, and whether a --host-ram-budget knob makes sense.

Context

• All numbers above are from a single RTX 4090 box, measured with llama.cpp-based server (this is a custom fork, patch described at high level above).
• Env: Linux x86_64, CUDA 12, PCIe 4.0 (gen4 single-GPU).
• Happy to provide full benchmark methodology / logs on request.

[PmNz8]

I think what you propose is somehow in the idea of https://gitlab.com/IsolatedOctopi/greenboost - but without the part of VRAM extension, just compute on GPU over pcie. The main issue I see is that is probably CUDA limited, and it would be good to have it supported on Vulkan backend also.

[yalun753]

Thanks for the greenboost pointer — similar spirit, different layer: it extends VRAM via DMA-BUF pages, while we keep expert weights mmap'd in host RAM and stream only the activated experts to the GPU per step (with an LRU hot-expert cache). That's exactly the "compute on GPU over PCIe" part.

On Vulkan: agreed it's a reasonable extension; we're focused on validating the CUDA path with real benchmarks first, Vulkan is on the roadmap.

Measured on RTX 4090 (experts in host RAM, GPU compute):

• Qwen3.6-35B-A3B (32B MoE, 256 experts, top-8): 46.8 t/s gen, ~2.1 GB VRAM — 32B MoE on an 8 GB card
• DeepSeek-V2-Lite (16B MoE): 37.5 t/s gen, ~1.6 GB VRAM

Goal: consumer GPUs run MoE models far larger than their VRAM. Open source: https://github.com/yalun753/moe-l2 (benchmark reports in references/)