Merge remote-tracking branch 'origin/master' into sync/upstream-2026-06-09

Author: kmbandy

common/chat.cpp

@@ -1670,7 +1670,15 @@ static common_chat_params common_chat_params_init_lfm2(const common_chat_templat
 
         auto reasoning = p.eps();
         if (extract_reasoning) {
-            reasoning = p.optional(THINK_START + p.reasoning(p.until(THINK_END)) + THINK_END);
+            // LFM2.5's template does not prefill an opening <think> into the generation prompt,
+            // so the model must emit it itself each turn. It sometimes skips the opening tag and
+            // dives straight into reasoning, closing only with </think>. Without forced-open
+            // handling that raw reasoning bleeds into content. Make the opening tag
+            // optional so a leading reasoning block terminated by </think> is captured whether or
+            // not <think> was emitted. A pure-content turn has no </think>, so until() fails to
+            // find the terminator, the sequence fails, and the outer optional collapses -> the
+            // whole output falls through to content exactly as before.
+            reasoning = p.optional(p.optional(p.literal(THINK_START)) + p.reasoning(p.until(THINK_END)) + p.literal(THINK_END));
         }
 
         if (!has_tools || inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_NONE) {

docs/paged-attn-debug.md

@@ -170,3 +170,90 @@ exercised).
 * Reverting the multi-seq commits — they aren't the cause.
 * Rewriting from scratch — the existing kernel is mostly right; it's
   one or two layout / shape assumption bugs.
+
+---
+
+## 2026-06-15 session — non-WMMA prefill, gfx803 perf, checkpoint+paged, tool bleed
+
+Investigation on **mad-lab-2026** (GTX 1070 / sm_61 / CUDA `build-army`;
+RX 480 / gfx803 / HIP `build-rocm-gfx803` in docker `rx480-army`).
+
+### A. Paged PREFILL was on the slow scalar kernel for ALL non-WMMA GPUs (fixed)
+
+`nsys` on a 2501-tok paged prefill (1070): `mt_paged_attention_kernel`
+= **84.7% of GPU time, 623 ms/launch**, grid `n_heads×num_seqs` (NO query
+parallelism). Root cause: the fast query-tiled prefill kernel
+(`mt_pagedattn_tile.cu`, grid `n_heads×num_seqs×n_q_tiles`) is gated at
+`mt_pagedattn.cu` behind `tile_gate_on = wmma_ok && tile_env_on`, and
+`wmma_ok = amd_wmma_available(cc)` is **RDNA3/4-only**. So Pascal (1070)
+and GCN/gfx803 (480) both fall through to the scalar O(n_q·n_kv) kernel.
+Decode was unaffected (`launch_paged_attn_decode` is not WMMA-gated).
+
+**Fix (uncommitted, working tree):**
+1. Added a portable FMA `#else` body to `mt_paged_attention_tile_kernel`
+   (was `NO_DEVICE_CODE`): QK^T as smem dot-products, scores→`smem_s`,
+   scores·V as smem GEMM. Same lane layout as the WMMA path; shfl
+   reductions pass explicit `WARP_SIZE` (correct on gfx803 64-lane waves).
+   Footprint ~13 KiB @ HS128 (fits Pascal's 48 KiB).
+2. Dispatch: `tile_gate_on = tile_env_on` (drop `wmma_ok`); force
+   `mw_on=false` on non-WMMA (multi-warp tile still WMMA-only).
+3. **Also fixed a latent grid-sizing bug** (same class as the decode
+   `93a1e6a11` fix): the tile launches sized `n_q_tiles` from
+   `avg_q_len = total_q_tokens/num_seqs`, which floors with idle
+   `--parallel` slots (1 active + N idle) → high Q-tiles never launch →
+   uncomputed rows → token-salad. Now use `total_q_tokens`. **This also
+   fixes the WMMA path and is very likely the core of MAD-288.**
+
+**Verified:** 1070 (CUDA) paged prefill **160 → 328 t/s** (non-paged ref
+395), output byte-matches the scalar oracle (`GGML_PAGED_TILE=0`) for
+Qwen3.5-9B (HS128) and LFM2.5-8B-A1B (HS64 **padded to 128** in the paged
+cache → uses the tile path; no separate HS64 work needed — clarifies
+MAD-298). 480 (gfx803): output correct, no crash. Revert: `GGML_PAGED_TILE=0`.
+
+### B. gfx803 (RX 480) ROCm is ~10× slower than Vulkan — and it's NOT the attn kernel
+
+Same card, Qwen3.5-9B, `-p512 -n128`:
+
+| Path | Prefill | Decode |
+|---|---|---|
+| gfx803 ROCm/HIP, paged tile | 16.6 t/s | — |
+| gfx803 ROCm/HIP, paged scalar | 13.8 t/s | — |
+| **480 via Vulkan0 (RADV, f16 KV)** | **162 t/s** | 20.4 t/s |
+| 1070 via Vulkan | 181 t/s | — |
+
+Tile ≈ scalar on gfx803 (both ~16) → the bottleneck is **gfx803 ROCm/HIP
+itself**, not the attention kernel and not the Polaris silicon. Suspects:
+turbo4 per-element dequant in `stage_k/v_tile` (`ops::k_load`), paged-block
+gather, or `gated_delta_net` hybrid layers under HIP. **Decision (Kurtis):
+keep the custom paged/turbo4/tiered features on gfx803, find the ROCm perf
+bug — do NOT port to Vulkan** (Vulkan lacks all custom kernels; only viable
+for standard dense models that fit VRAM). NEXT: profile the gfx803 paged
+prefill (rocprof not installed in `rx480-army` — needs a profiling path).
+
+### C. Checkpoint reuse vs paged cache — block-alignment crash (NOT fixed)
+
+`--ctx-checkpoints>0` would let hybrid/recurrent models reuse the prompt
+prefix (the restore path at `server-context.cpp` ~3397 already exists, and
+with paged the checkpoints are tiny — **0.282 MiB**, blocks referenced not
+copied). But restore sets `n_past` to the checkpoint pos (e.g. 1364), and
+`llama_kv_cache_paged::seq_rm` requires **block-aligned (×16)** ranges →
+unaligned trim leaves stale partial blocks → `GGML_ASSERT compute_slot_mapping
+failed` abort. This is why LFM2.5 runs `--ctx-checkpoints 0` and full-
+reprocesses every turn (~2.5 min for ~3700 tok). **Two fix attempts failed:**
+round-at-restore is wrong (recurrent state can't be partially rolled back —
+must reuse exactly `pos_max+1`); block-aligning the checkpoint batch-break at
+creation crashed turn-1 prefill. **Reverted.** Needs a deeper design (paged
+`seq_rm`/`compute_slot_mapping` tolerant of the boundary, or a hybrid+paged-
+aware checkpoint path). Upstream unmerged references: PRs #20955, #21099, #20428.
+
+### D. Tool-schema bleed (LFM2.5)
+
+"hello!" → model regurgitates tool-schema JSON fragments then hallucinates.
+The LFM2.5 jinja template renders the tool list as a **plain-text system
+message** `List of tools: [{json}]` (faithful to the official template;
+`common/chat.cpp:2226` detects this variant, vs LFM2's native
+`<|tool_list_start|>` tokens at :622). Tool *calls* are correct
+(grammar-constrained `<|tool_call_start|>`). With `thinking=1` the model
+also reasons verbosely about the tools. Fix TBD (limit/disable reasoning on
+tool turns, or revisit tool presentation). Single/multi-turn tool calls
+otherwise correct on 1070.

ggml/include/ggml.h

@@ -439,7 +439,8 @@ extern "C" {
         GGML_TYPE_F8_E4M3 = 49, // MAD-223 Phase G: fp8 e4m3 1-byte storage, used as sidecar dtype for ml8 centroids
         GGML_TYPE_ML8_4_SOA = 50, // MAD-223 Phase G.7 / MAD-244: ml8-4 with stored-as-repacked SOA layout (b_packed bytes followed by b_scale fp32 per expert row). Same numerics as ML8_4; eliminates the runtime MoE repack cache.
         GGML_TYPE_ML8_FP8 = 51,  // MAD Task 9: ml8 fp8 weight quant, 32-element blocks, fp16 per-block scale + 32 e4m3 bytes (34 bytes/block). On-disk id=51; matches gguf-py GGML_QUANT_SIZES[ML8_FP8]=(32,34).
-        GGML_TYPE_COUNT   = 52,
+        GGML_TYPE_TURBO4_64 = 52, // MAD-301C Lever B: native head_dim-64 turbo4 KV cache, 64-element block (34 bytes), no 64->128 pad. Runtime KV-cache-only type (never serialized to GGUF).
+        GGML_TYPE_COUNT   = 53,
     };
 
     // precision

ggml/src/ggml-common.h

@@ -329,6 +329,19 @@ static_assert(sizeof(block_turbo4_0) == 2*sizeof(ggml_half) + QK_TURBO4*3/8 + QK
 
 static_assert(QK_TURBO4 == 128, "turbo4 kernels assume QK_TURBO4 == 128");
 
+// MAD-301C Lever B: native head_dim-64 turbo4 KV cache block.
+// Identical 4-bit PolarQuant + per-block norm as block_turbo4_0, but a 64-element
+// block so a head_dim-64 head (LFM2.5, gpt-oss) is exactly one block with NO
+// 64->128 zero-padding. The paged path skips RHT, so dequant returns
+// centroid*norm and dot products match the padded-128 path bit-for-bit at ~half
+// the storage (34 vs 68 bytes/head). 4-bit centroid layout only.
+#define QK_TURBO4_64 64
+typedef struct {
+    ggml_half  norm;                       //  2 bytes: per-block scale
+    uint8_t    qs[QK_TURBO4_64 / 2];        // 32 bytes: 4-bit PolarQuant indices (nibble packed)
+} block_turbo4_64;                          // 34 bytes total
+static_assert(sizeof(block_turbo4_64) == 2 + QK_TURBO4_64/2, "wrong turbo4_64 block size");
+
 // TurboQuant 2-bit: 2-bit PolarQuant indices only (no QJL)
 // Per block: norm(fp16) + 2-bit indices (8 bytes) = 10 bytes per 32 values
 // = 2.5 bits/value → 6.4× compression vs fp16

ggml/src/ggml-cuda/ggml-cuda.cu

@@ -2930,7 +2930,12 @@ static void ggml_cuda_mul_mat_id(ggml_backend_cuda_context & ctx, ggml_tensor *
             }
         }
 
-        if (ggml_cuda_should_use_mmq(src0->type, cc, ne12, /*n_experts=*/ne02)) {
+        // routing_active forces MMQ: it is the only mul_mat_id path that honors the
+        // weight-pager expert pointers here (MMVQ handled above for small batch). On
+        // gfx80x ggml_cuda_should_use_mmq() now returns false for large batches (no
+        // hardware dp4a -> route to dequant+hipBLAS), so without this OR the routing
+        // case would fall through to the GGML_ABORT below.
+        if (routing_active || ggml_cuda_should_use_mmq(src0->type, cc, ne12, /*n_experts=*/ne02)) {
             ggml_cuda_mul_mat_q(ctx, src0, src1, ids, dst);
             return;
         }
@@ -5940,7 +5945,8 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g
                     || op->src[1]->type == GGML_TYPE_Q8_0
                     || op->src[1]->type == GGML_TYPE_TURBO4_0
                     || op->src[1]->type == GGML_TYPE_TURBO3_0
-                    || op->src[1]->type == GGML_TYPE_TURBO4_FP8_BS256)  // MAD-214
+                    || op->src[1]->type == GGML_TYPE_TURBO4_FP8_BS256   // MAD-214
+                    || op->src[1]->type == GGML_TYPE_TURBO4_64)         // MAD-301C Lever B
                 && op->src[6]                          // k_cur (fused scatter)
                 && op->src[6]->type == GGML_TYPE_F16
                 && op->src[7]                          // v_cur (fused scatter)

ggml/src/ggml-cuda/mmq.cu

@@ -477,5 +477,13 @@ bool ggml_cuda_should_use_mmq(enum ggml_type type, int cc, int64_t ne11, int64_t
         return true;
     }
 
+    // gfx80x (Tonga/Fiji/Polaris, GGML_CUDA_CC_GCN4 and earlier) have no hardware dp4a
+    // (v_dot4): ggml_cuda_dp4a falls back to fully-scalar byte emulation, which makes MMQ
+    // ~8x slower than dequantization + hipBLAS for prefill-sized batches (rocBLAS ships
+    // gfx803 fp16/fp32 Tensile kernels). Restrict MMQ to small (decode) batches only.
+    if (GGML_CUDA_CC_IS_GCN(cc) && cc <= GGML_CUDA_CC_GCN4) {
+        return ne11 < MMQ_DP4A_MAX_BATCH_SIZE;
+    }
+
     return (!GGML_CUDA_CC_IS_CDNA(cc)) || ne11 < MMQ_DP4A_MAX_BATCH_SIZE;
 }