Skip to content

Commit fa59546

Browse files
CISCggerganov
andauthored
graph : handle non-contiguous Q/K/V in mul_mat_aux (#22630)
* qkv may not always be contiguous * cont : make the cont conditional --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
1 parent a817a22 commit fa59546

1 file changed

Lines changed: 5 additions & 1 deletion

File tree

src/llama-graph.cpp

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,11 @@ static ggml_tensor * ggml_mul_mat_aux(
6565

6666
ggml_tensor * res;
6767

68-
res = ggml_reshape_2d(ctx, cur, n, ggml_nelements(cur)/n);
68+
if (!ggml_is_contiguous(cur)) {
69+
res = ggml_cont_2d (ctx, cur, n, ggml_nelements(cur)/n);
70+
} else {
71+
res = ggml_reshape_2d(ctx, cur, n, ggml_nelements(cur)/n);
72+
}
6973
res = ggml_mul_mat (ctx, rot, res);
7074
ggml_mul_mat_set_hint(res, GGML_HINT_SRC0_IS_HADAMARD);
7175
res = ggml_reshape_4d(ctx, res, cur->ne[0], cur->ne[1], cur->ne[2], cur->ne[3]);

0 commit comments

Comments
 (0)