ggml-ve : Q4_K N>1 — HBM-aware cache upload (no more SEGV)

ROOT CAUSE of the N>1 SEGV:

  - With GGML_VE_Q4K_N_GT_1=1 the scheduler decides Q4_K weights are
    best placed on VE_HBM (because our backend now consumes them in
    batched form too), so w->data becomes an HBM pointer.
  - The host-side canonical-pack in get_or_upload_q4k_canon reads
    src_blocks via memcpy. That SEGVs on an HBM pointer.
  - Earlier sessions only ever saw CPU_Mapped Q4_K weights (N=1 path),
    so the cache had never been exposed to HBM-resident inputs.

Fix: before calling the cache, check ggml_backend_buffer_is_host on
the weight buffer. If non-host (= HBM), download the weight to a host
bounce buffer with vedaMemcpyDtoH first, then pass that to the cache.
One-time cost per unique weight at first lookup; subsequent calls hit
the cache.

Confirmed working with GGML_VE_Q4K_N_GT_1=1 on MiniCPM5-1B-Q4_K_M:
  - prompt eval (N=5):  prints "Q4K-DEVICE-BOUNCE" once per weight,
    then no crashes; correct output
  - decode (N=1):       hits the now-populated canon cache, no
    bounce, runs the existing 8-row tile kernel

Perf right now (1B, full N>1 path enabled):
  | path                          | pp t/s   | tg t/s |
  | ----------------------------- | -------- | ------ |
  | default (N>1 → CPU)           | 21.9     | 14.4   |
  | GGML_VE_Q4K_N_GT_1=1 (this)   |  7.8     |  8.5   |

VE path slower than CPU because each N>1 op is LOOPED as N sequential
matvecs (1.3 ms × N), where CPU's AVX2 Q4_K does a true batched
SGEMM. So this commit gets us to "N>1 works on VE" but not "N>1 is
fast on VE".

Real win comes from the next commit: tile-batched matmul. Dequant a
row tile of weights to F32 once, then run N matvecs against that
cached tile (or one cblas_sgemm). Amortizes dequant across N x-columns.

For now N>1 stays GATED behind GGML_VE_Q4K_N_GT_1=1 because the
sequential path is slower than CPU. Will flip default when batched
matmul lands.

Author: wmeddie

ggml/src/ggml-ve/ops/mul_mat_q.cpp

@@ -14,6 +14,8 @@
 
 #include "ggml.h"
 
+#include <memory>
+
 namespace ggml_ve {
 namespace ops {
 
@@ -219,8 +221,38 @@ bool mul_mat_q(backend_context * ctx, ggml_tensor * dst) {
             fprintf(stderr, "[Q4K-DEBUG] name=%s M=%ld K=%ld N=%ld\n",
                     name ? name : "?", (long)M, (long)K, (long)N);
         }
+        // The host-side canonical-pack reads from src_blocks via memcpy. If
+        // the weight buffer is device-resident (VE_HBM), w->data is an HBM
+        // pointer and host memcpy SEGVs. Download to a host bounce buffer
+        // first.
+        //
+        // This happens with GGML_VE_Q4K_N_GT_1: once our backend accepts
+        // Q4_K MUL_MATs in batched form, the scheduler decides Q4_K weights
+        // are best placed on VE_HBM (the buffer they're consumed on). The
+        // cache then needs to read them BACK to host to do the canonical
+        // pack + pre-decode. One-time cost per weight at first lookup.
+        const void * src_for_cache = w->data;
+        std::unique_ptr<uint8_t[]> bounce;
+        const int64_t weight_bytes = (int64_t) M * (K / 256) * 144;
+        if (w->buffer && !ggml_backend_buffer_is_host(w->buffer)) {
+            bounce.reset(new uint8_t[weight_bytes]);
+            if (vedaMemcpyDtoH(bounce.get(),
+                               (VEDAdeviceptr)(uintptr_t) w->data,
+                               weight_bytes) != VEDA_SUCCESS) {
+                if (std::getenv("GGML_VE_Q4K_DEBUG")) {
+                    fprintf(stderr, "[Q4K-FAIL] DtoH bounce for %s failed\n",
+                            name ? name : "?");
+                }
+                return false;
+            }
+            src_for_cache = bounce.get();
+            if (std::getenv("GGML_VE_Q4K_DEBUG")) {
+                fprintf(stderr, "[Q4K-DEVICE-BOUNCE] %s: %ld bytes downloaded for canon pack\n",
+                        name ? name : "?", (long) weight_bytes);
+            }
+        }
         if (!ctx->cache().get_or_upload_q4k_canon(
-                name, w->data, (uint64_t) M, (uint64_t) K, &qs_v, &hdr_v)) {
+                name, src_for_cache, (uint64_t) M, (uint64_t) K, &qs_v, &hdr_v)) {
             return false;
         }
         /* N>1 (batched matmul): loop the matvec, advancing y by M and x by K