Port DeepSeek-v4 to TQ

[giveen]

Overview

Adds DeepSeek V4 model support to the turboquant fork. V4 introduces a new architecture with lightweight indexer-based sparse attention, hyper-connection (HC) layers for cross-block communication, and compressed-state KV cache storage.

Key Changes

Model Architecture (src/models/deepseek4.cpp, src/llama-arch.{cpp,h})

• New LLM_ARCH_DEEPSEEK4 arch with V4-specific hparams: indexer head count/size/top-k, output group count, output LoRA rank, compress RoPE frequency base, hyper-connection multiplier, sinkhorn iterations, SWIGLU clamp per expert/shared-expert arrays.
• New models/templates/deepseek-ai-DeepSeek-V4.jinja chat template.

KV Cache (src/llama-kv-cache-dsv4.{cpp,h}, src/llama-kv-cache-iswa.{cpp,h})

• DSV4 compressed-state KV cache with configurable compression ratio (CSA-4 for K, HCA-128 for V) and stream-based tensor ops.
• ISWA (Importance-based Sliding Window Attention) cache improvements — variable window sizes, per-head importance tracking.

CUDA Kernels (ggml/src/ggml-cuda/)

• dsv4-hc: Hyper-connection layer fused kernel (sinkhorn normalization, attention-biased routing).
• lightning-indexer: Sparse top-k indexer — computes per-head relevance scores, selects top-k positions, gathers scatter KV blocks.
• set-rows.cu: Extended for V4's compressed-state scatter/gather.

CPU Ops (ggml/src/ggml-cpu/)

• dsv4-ops.cpp: Reference implementations for DSV4 tensor operations (compressed-state packing, hyper-connection).
• ops.{cpp,h}: New CPU op entries.

GGML Infra (ggml/src/ggml.c, ggml/include/ggml.h)

• New tensor ops for indexer and hyper-connection primitives.
• RPC header updates for V4 op forwarding.

Conversion Scripts (conversion/)

• deepseek.py: V4 parameter mapping, hyper-connection weight layout, indexer config export.
• constants.py / gguf_writer.py: New V4 keys — attention.indexer.head-count, attention.indexer.top-k, hyper-connection.count, attention.output.group-count, attention.output.lora-rank, attention.compress-rope.freq-base, etc.

Removed

• build-xcframework.sh: Stale Apple framework script, not used in this fork.

---

Additional Information

Requires a DeepSeek V4 GGUF checkpoint — not compatible with V2/V3 or other architectures. The indexer CUDA kernel is optimized for NVIDIA GPUs with sm_90+; CPU fallback is available for all ops but significantly slower for inference.

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes, the full port.

[giveen]

Right now, it runs about 95% of the equivelent speed as on main llama.cpp
Main - 230pp/12tks gen
TQ Build - 180pp/10tk gen

I'm trying to do a bit more optimization
I have tested using q8_0/turbo4 , and it works, but being VRAM poor makes the model slow anyways.

./llama-server \
   --host 0.0.0.0 \
   --port 8080 \
   --alias deepseek \
   --parallel 1 \
   --cont-batching \
   -m /mnt/storage/models/deepseek-v4-flash/Q3/UD-Q3_K_XL/DeepSeek-V4-Flash-UD-Q3_K_XL-00001-of-00004.gguf \
   --ctx-size 262000 \
   -fa 1 \
   --fit off \
   --cache-type-k q8_0 \
   --cache-type-v q8_0 \
   --cache-reuse 4096 \
   -ctxcp 32 \
   -cms 8192 \
   --no-mmap \
   --mlock \
   --numa isolate \
   --n-cpu-moe 46 \
   --threads 12 \
   --threads-batch 8 \
   --batch-size 2048 \
   --ubatch-size 256 \
   --jinja \
   --no-warmup \
   --chat-template-kwargs '{"reasoning_effort":"max"}' \
   --presence-penalty 0.0 \
   --repeat-penalty 1.05

   ```

   

[giveen]

Performance changes in ds4 branch (2 new commits)
Because it touched things outside of the DS4 direct port , I also tested on Qwen3.6-27B and Gemma4-26

4f5cb54 — Warp-parallel HC_COMB kernel
Rewrote the DSV4 hyper-connection comb kernel:
- 16 threads per 4×4 sinkhorn matrix (one per element) instead of 1 thread per token.
- Uses __shfl_xor_sync for row/column reductions, eliminating the serial softmax loop.
- Improves occupancy from ~4/256 to ~32/128 threads on small batches (BS 1–4).
- Added 15 test configurations (n_tokens=1..64 × n_iter=1..5) verified against CPU reference on RTX 5090.

95a7c30 — PDL + restrict fix for flash attention
- Port of upstream b820cc8: Consistent use of restrict + PDL for FA.
- Fixes a compiler race condition between PDL and restrict on Hopper/Blackwell GPUs
that causes incorrect codegen in flash_attn_mask_to_KV_max.
- Enables PDL enrollment for this kernel via ggml_cuda_kernel_launch().
- Scope: All models using flash attention on sm_90+ GPUs.

Additional Changes:

• Build fix for RTX 5090: Added -DCMAKE_CUDA_ARCHITECTURES="90-real;100-real;120-real".
• Without this, the 5090 (sm_120) would silently fallback to CPU compute—a classic
  case of "working, but not actually working."

[giveen]

Okay speeds are the same as main llama.cpp and it holds steady even across long context, just like main, I'm still working on a few ideas.