Gemma4 e4b bug

[serj-47]

Describe the Issue
I tried two models of the latest gemma4: E2B-it-Q4_K_M.gguf and E4B-it-Q4_K_M.gguf. There are no problems with the E2 model, everything is working fine. The E4 model responds in most cases with emoticons or punctuation marks. I tried to include jinja / jinja for tools, but as I understand it, it corrects the reasoning process more.

Additional Information:
Hardware=Steam deck , backend=Vulkan, UMA in bios=4gb

[henk717]

Let me know on which version of KoboldCpp you tested this, UI as well as your general settings, its a very fresh model.

[serj-47]

1.111.1-release. I only selected Vulkan and changed the number of gpu layers, but it seems to me that with the -1 parameter, the number of layers is determined normally, also tried to turn on jinja.

[serj-47]

I also noticed that when loading any, even small, model, such as the previous 2.3GB gemma3-4b, it uses both gpu memory and main memory, but I expected the model to use only gpu memory because it is sufficient. For example, before launching the model, the SteamOS uses 3.8GB of regular memory and 800 MB of dedicated video memory, but after loading the model, SteamOS uses 6.2Gb of regular memory and 3.8Gb of video memory, it feels like the model is loaded twice into both types of memory. Because of this, it is impossible to use large models, because the memory runs out quickly. Of course, I can reduce the amount of allocated memory for the gpu in bios 4gb-->1gb, but it seems to me this would be the wrong approach.

[henk717]

It a known issue that autofit is inaccurate for gemma4, since this is a feature from llamacpp itself we just have to wait for them to fix it.

[LostRuins]

Please try the rolling builds and see if the issue is solved.

[serj-47]

I checked releases 1.111.2 and experimental and the same problems, gemma 4 e4b responds in most cases with symbols and uses allocated and shared memory more than expected

[LostRuins]

Okay may be a backend issue then. Can you try the pure CPU "use CPU" option

[serj-47]

It works fine on the CPU. The only problem is with Vulkan, but I get fewer tokens/sec.

[wbruna]

It works fine on the CPU. The only problem is with Vulkan, but I get fewer tokens/sec.

In some systems, Vulkan can behave weirdly when there is too little available VRAM (which may include offloading to system/shared RAM). And depending on the configuration, Gemma 4 may need a lot of VRAM for the context. Can you reproduce this behavior with a much smaller context, and/or with SWA on?

Better yet: could you post a full debug log, with the model loading + a small first generation?

Of course, I can reduce the amount of allocated memory for the gpu in bios 4gb-->1gb, but it seems to me this would be the wrong approach.

My own AMD iGPU (Ryzen 3400G) benefits from using only shared memory for inference. If you want to give it a try, lower the dedicated VRAM like this, and run Koboldcpp with the environment variable GGML_VK_PREFER_HOST_MEMORY=1.

[Vaitek]

I’m not sure whether to open a new issue, but I ran into a crash today with the gemma-4 model…
Maybe the context was too large... After I reduced the context size from 128K to 64K, the model loaded.

Details

user@homepc:~/Programs/KoboldCPP$ ./koboldcpp-linux-x64
***
Welcome to KoboldCpp - Version 1.111.2
For command line arguments, please refer to --help
***
No such schema “org.gnome.mutter”
Auto Selected CUDA Backend (flag=0)

Loading Chat Completions Adapter: /tmp/_MEIxZgYVF/kcpp_adapters/AutoGuess.json
Chat Completions Adapter Loaded
Forced autofit is selected, moecpu and overridetensors will be set automatically.
Auto Recommended GPU Layers: 0
System: Linux #20~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Mar 19 01:28:37 UTC 2 x86_64 x86_64
Detected Available GPU Memory: 6144 MB
Detected Available RAM: 22826 MB
Initializing dynamic library: koboldcpp_cublas.so
==========
Namespace(admin=False, admindir='', adminpassword='', adminunloadtimeout=0, analyze='', autofit=True, autofitpadding=1024, autoswapmode=False, batchsize=512, benchmark=None, blasthreads=None, chatcompletionsadapter='AutoGuess', cli=False, config=None, contextsize=131072, debugmode=0, defaultgenamt=1024, device='', downloaddir='', draftamount=8, draftgpulayers=999, draftgpusplit=None, draftmodel=None, embeddingsgpu=False, embeddingsmaxctx=0, embeddingsmodel='', enableguidance=False, exportconfig='', exporttemplate='', failsafe=False, flashattention=False, forceversion=False, foreground=False, gendefaults='', gendefaultsoverwrite=False, genlimit=0, gpulayers=0, highpriority=False, hordeconfig=None, hordegenlen=0, hordekey='', hordemaxctx=0, hordemodelname='', hordeworkername='', host='', ignoremissing=False, jinja=False, jinja_kwargs='', jinja_tools=False, launch=False, lora=None, loramult=1.0, lowvram=False, maingpu=-1, maxrequestsize=32, mcpfile=None, mmproj='/media/user/Raid0/260404.koboldcpp.mmproj.gemma4-E4B-mmproj-BF16.gguf', mmprojcpu=False, model=[], model_param='/media/user/Raid0/260407.mradermacher.gemma-4-E4B-it-heretic-i1-GGUF.gemma-4-E4B-it-heretic.i1-Q6_K.gguf', moecpu=0, moeexperts=-1, multiplayer=False, multiuser=1, musicdiffusion='', musicembeddings='', musicllm='', musiclowvram=False, musicvae='', noavx2=False, noblas=False, nobostoken=False, nocertify=False, nofastforward=False, noflashattention=False, nommap=False, nomodel=False, nopipelineparallel=False, noshift=True, onready='', overridekv=None, overridenativecontext=0, overridetensors='', password=None, pipelineparallel=False, port=5001, port_param=5001, preloadstory=None, prompt='', proxy_port=None, quantkv=0, quiet=True, ratelimit=0, remotetunnel=False, ropeconfig=[0.0, 10000.0], routermode=False, savedatafile=None, sdclamped=0, sdclampedsoft=0, sdclip1='', sdclip2='', sdclipgpu=False, sdconfig=None, sdconvdirect='off', sdflashattention=False, sdgendefaults=False, sdlora=[], sdloramult=[1.0], sdmaingpu=-1, sdmodel='', sdnotile=False, sdoffloadcpu=False, sdphotomaker='', sdquant=0, sdt5xxl='', sdthreads=7, sdtiledvae=768, sdupscaler='', sdvae='', sdvaeauto=False, sdvaecpu=False, showgui=False, singleinstance=False, skiplauncher=False, smartcache=0, smartcontext=False, ssl=None, tensor_split=None, testmemory=False, threads=7, ttsdir='', ttsgpu=False, ttsmaxlen=4096, ttsmodel='', ttsthreads=0, ttswavtokenizer='', unpack='', usecpu=False, usecuda=['normal', '0', 'mmq'], usemlock=False, usemmap=True, useswa=True, usevulkan=None, version=False, visionmaxres=1024, websearch=False, whispermodel='')
==========
Loading Text Model: /media/user/Raid0/260407.mradermacher.gemma-4-E4B-it-heretic-i1-GGUF.gemma-4-E4B-it-heretic.i1-Q6_K.gguf

The reported GGUF Arch is: gemma4
Arch Category: 49

---
Identified as GGUF model.
Attempting to Load...
---

SWA Mode is ENABLED!
Note that using SWA Mode cannot be used with Context Shifting, and can lead to degraded recall when combined with Fast Forwarding!
Using automatic RoPE scaling for GGUF. If the model has custom RoPE settings, they'll be used directly instead!
System Info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | AMX_INT8 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | 
CUDA MMQ: True
---
Initializing CUDA/HIP, please wait, the following step may take a few minutes (only for first launch)...
---
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 5748 MiB):
  Device 0: NVIDIA GeForce GTX 1660 Ti, compute capability 7.5, VMM: yes, VRAM: 5748 MiB
The following devices will have suboptimal performance due to a lack of tensor cores:
  Device 0: NVIDIA GeForce GTX 1660 Ti
Consider compiling with CMAKE_CUDA_ARCHITECTURES=61-virtual;80-virtual and DGGML_CUDA_FORCE_MMQ to force the use of the Pascal code for Turing.

Attempting to use llama.cpp's automating fitting code. This will override all your layer configs, may or may not work!
Autofit Reserve Space: 2442 MB
ggml/src/ggml-backend.cpp:1283: GGML_ASSERT(n_inputs < GGML_SCHED_MAX_SPLIT_INPUTS) failed
[New LWP 52490]
[New LWP 52489]
[New LWP 52488]
[New LWP 52484]
[New LWP 50132]

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.ubuntu.com>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f8757110813 in __GI___wait4 (pid=52495, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30	../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#0  0x00007f8757110813 in __GI___wait4 (pid=52495, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30	in ../sysdeps/unix/sysv/linux/wait4.c
#1  0x00007f873e69629b in ggml_print_backtrace () from /tmp/_MEIxZgYVF/koboldcpp_cublas.so
#2  0x00007f873e6963f5 in ggml_abort () from /tmp/_MEIxZgYVF/koboldcpp_cublas.so
#3  0x00007f873ed0cfc3 in ggml_backend_sched_split_graph () from /tmp/_MEIxZgYVF/koboldcpp_cublas.so
#4  0x00007f873e88bd1f in llama_context::graph_reserve(unsigned int, unsigned int, unsigned int, llama_memory_context_i const*, bool, unsigned long*) () from /tmp/_MEIxZgYVF/koboldcpp_cublas.so
#5  0x00007f873e88cdd5 in llama_context::sched_reserve() () from /tmp/_MEIxZgYVF/koboldcpp_cublas.so
#6  0x00007f873e8b8ced in llama_context::llama_context(llama_model const&, llama_context_params) () from /tmp/_MEIxZgYVF/koboldcpp_cublas.so
#7  0x00007f873e8b999e in llama_init_from_model () from /tmp/_MEIxZgYVF/koboldcpp_cublas.so
#8  0x00007f873e999f86 in ?? () from /tmp/_MEIxZgYVF/koboldcpp_cublas.so
#9  0x00007f873e99a77d in ?? () from /tmp/_MEIxZgYVF/koboldcpp_cublas.so
#10 0x00007f873e99d2f5 in ?? () from /tmp/_MEIxZgYVF/koboldcpp_cublas.so
#11 0x00007f873e99de7e in llama_params_fit () from /tmp/_MEIxZgYVF/koboldcpp_cublas.so
#12 0x00007f873e9a025c in gpttype_load_model(load_model_inputs, FileFormat, FileFormatExtraMeta) () from /tmp/_MEIxZgYVF/koboldcpp_cublas.so
#13 0x00007f873e7bf7f7 in load_model () from /tmp/_MEIxZgYVF/koboldcpp_cublas.so
#14 0x00007f8756f5602a in ffi_call_unix64 () from /tmp/_MEIxZgYVF/libffi.so.8
#15 0x00007f8756f554a9 in ffi_call_int () from /tmp/_MEIxZgYVF/libffi.so.8
#16 0x00007f8756f55bbd in ffi_call () from /tmp/_MEIxZgYVF/libffi.so.8
#17 0x00007f8756a0097f in _call_function_pointer (argtypecount=<optimized out>, argcount=1, resmem=0x7ffff2f2cfe0, restype=<optimized out>, atypes=<optimized out>, avalues=<optimized out>, pProc=0x7f873e7bf380 <load_model>, flags=<optimized out>) at /usr/local/src/conda/python-3.8.20/Modules/_ctypes/callproc.c:921
warning: 921	/usr/local/src/conda/python-3.8.20/Modules/_ctypes/callproc.c: No such file or directory
#18 _ctypes_callproc (pProc=0x7f873e7bf380 <load_model>, argtuple=<optimized out>, flags=<optimized out>, argtypes=<optimized out>, restype=0x588d96b57480, checker=0x0) at /usr/local/src/conda/python-3.8.20/Modules/_ctypes/callproc.c:1264
1264	in /usr/local/src/conda/python-3.8.20/Modules/_ctypes/callproc.c
#19 0x00007f87569fffcf in PyCFuncPtr_call (self=<optimized out>, inargs=<optimized out>, kwds=0x0) at /usr/local/src/conda/python-3.8.20/Modules/_ctypes/_ctypes.c:4201
warning: 4201	/usr/local/src/conda/python-3.8.20/Modules/_ctypes/_ctypes.c: No such file or directory
#20 0x00007f8756d1bbf4 in _PyObject_MakeTpCall (callable=0x7f8754763700, args=<optimized out>, nargs=<optimized out>, keywords=<optimized out>) at /usr/local/src/conda/python-3.8.20/Objects/call.c:159
warning: 159	/usr/local/src/conda/python-3.8.20/Objects/call.c: No such file or directory
#21 0x00007f8756d18eb0 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f875403cf10, callable=<optimized out>) at /usr/local/src/conda/python-3.8.20/Include/cpython/abstract.h:125
warning: 125	/usr/local/src/conda/python-3.8.20/Include/cpython/abstract.h: No such file or directory
#22 _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f875403cf10, callable=<optimized out>) at /usr/local/src/conda/python-3.8.20/Include/cpython/abstract.h:115
115	in /usr/local/src/conda/python-3.8.20/Include/cpython/abstract.h
#23 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x588d96adea80) at /usr/local/src/conda/python-3.8.20/Python/ceval.c:4963
warning: 4963	/usr/local/src/conda/python-3.8.20/Python/ceval.c: No such file or directory
#24 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.8.20/Python/ceval.c:3469
3469	in /usr/local/src/conda/python-3.8.20/Python/ceval.c
#25 0x00007f8756d2158b in function_code_fastcall (globals=<optimized out>, nargs=1, args=<optimized out>, co=<optimized out>) at /usr/local/src/conda/python-3.8.20/Objects/call.c:284
warning: 284	/usr/local/src/conda/python-3.8.20/Objects/call.c: No such file or directory
#26 _PyFunction_Vectorcall (func=<optimized out>, stack=0x588d97b31968, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.8.20/Objects/call.c:411
411	in /usr/local/src/conda/python-3.8.20/Objects/call.c
#27 0x00007f8756d14936 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x588d97b31968, callable=0x7f8755012d30) at /usr/local/src/conda/python-3.8.20/Include/cpython/abstract.h:127
warning: 127	/usr/local/src/conda/python-3.8.20/Include/cpython/abstract.h: No such file or directory
#28 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x588d96adea80) at /usr/local/src/conda/python-3.8.20/Python/ceval.c:4963
warning: 4963	/usr/local/src/conda/python-3.8.20/Python/ceval.c: No such file or directory
#29 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.8.20/Python/ceval.c:3500
3500	in /usr/local/src/conda/python-3.8.20/Python/ceval.c
#30 0x00007f8756d13711 in _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x588d96f21310, argcount=argcount@entry=3, kwnames=<optimized out>, kwargs=0x588d96f21328, kwcount=<optimized out>, kwstep=1, defs=0x7f87551d0918, defcount=2, kwdefs=0x0, closure=0x0, name=0x7f87551be3f0, qualname=0x7f87551be3f0) at /usr/local/src/conda/python-3.8.20/Python/ceval.c:4298
4298	in /usr/local/src/conda/python-3.8.20/Python/ceval.c
#31 0x00007f8756d2161d in _PyFunction_Vectorcall (func=<optimized out>, stack=0x588d96f21310, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.8.20/Objects/call.c:436
warning: 436	/usr/local/src/conda/python-3.8.20/Objects/call.c: No such file or directory
#32 0x00007f8756d14936 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x588d96f21310, callable=0x7f8755017e50) at /usr/local/src/conda/python-3.8.20/Include/cpython/abstract.h:127
warning: 127	/usr/local/src/conda/python-3.8.20/Include/cpython/abstract.h: No such file or directory
#33 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x588d96adea80) at /usr/local/src/conda/python-3.8.20/Python/ceval.c:4963
warning: 4963	/usr/local/src/conda/python-3.8.20/Python/ceval.c: No such file or directory
#34 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.8.20/Python/ceval.c:3500
3500	in /usr/local/src/conda/python-3.8.20/Python/ceval.c
#35 0x00007f8756d13711 in _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x588d96e6fb10, argcount=argcount@entry=0, kwnames=<optimized out>, kwargs=0x588d96e6fb10, kwcount=<optimized out>, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x7f87569ba6b0, qualname=0x7f87569ba6b0) at /usr/local/src/conda/python-3.8.20/Python/ceval.c:4298
4298	in /usr/local/src/conda/python-3.8.20/Python/ceval.c
#36 0x00007f8756d2161d in _PyFunction_Vectorcall (func=<optimized out>, stack=0x588d96e6fb10, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.8.20/Objects/call.c:436
warning: 436	/usr/local/src/conda/python-3.8.20/Objects/call.c: No such file or directory
#37 0x00007f8756d15633 in _PyObject_Vectorcall (kwnames=0x7f87551d08c0, nargsf=<optimized out>, args=<optimized out>, callable=0x7f8755017ca0) at /usr/local/src/conda/python-3.8.20/Include/cpython/abstract.h:127
warning: 127	/usr/local/src/conda/python-3.8.20/Include/cpython/abstract.h: No such file or directory
#38 call_function (kwnames=0x7f87551d08c0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=<optimized out>) at /usr/local/src/conda/python-3.8.20/Python/ceval.c:4963
warning: 4963	/usr/local/src/conda/python-3.8.20/Python/ceval.c: No such file or directory
#39 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.8.20/Python/ceval.c:3515
3515	in /usr/local/src/conda/python-3.8.20/Python/ceval.c
#40 0x00007f8756d13711 in _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=argcount@entry=0, kwnames=<optimized out>, kwargs=0x0, kwcount=<optimized out>, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at /usr/local/src/conda/python-3.8.20/Python/ceval.c:4298
4298	in /usr/local/src/conda/python-3.8.20/Python/ceval.c
#41 0x00007f8756d8b97a in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=args@entry=0x0, argcount=argcount@entry=0, kws=kws@entry=0x0, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0) at /usr/local/src/conda/python-3.8.20/Python/ceval.c:4327
4327	in /usr/local/src/conda/python-3.8.20/Python/ceval.c
#42 0x00007f8756d8b92c in PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, locals=<optimized out>) at /usr/local/src/conda/python-3.8.20/Python/ceval.c:718
718	in /usr/local/src/conda/python-3.8.20/Python/ceval.c
#43 0x0000588d5a24429a in ?? ()
#44 0x0000588d5a244c57 in ?? ()
#45 0x00007f875702a1ca in __libc_start_call_main (main=main@entry=0x588d5a243050, argc=argc@entry=1, argv=argv@entry=0x7ffff2f2fd28) at ../sysdeps/nptl/libc_start_call_main.h:58
warning: 58	../sysdeps/nptl/libc_start_call_main.h: No such file or directory
#46 0x00007f875702a28b in __libc_start_main_impl (main=0x588d5a243050, argc=1, argv=0x7ffff2f2fd28, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffff2f2fd18) at ../csu/libc-start.c:360
warning: 360	../csu/libc-start.c: No such file or directory
#47 0x0000588d5a243091 in ?? ()
[Inferior 1 (process 49961) detached]
Aborted (core dumped)
user@homepc:~/Programs/KoboldCPP$ 

[yann-amd]

Looking at the output, this looks like the same issue tracked in ggml-org#21516. It's been reproduced on NVIDIA too, in both Vulkan single-GPU (RTX 3090) and CUDA multi-GPU configurations (2× A6000, RTX 3090 + 2070), so it's not AMD or Steam Deck specific. It's an upstream Gemma 4 inference precision issue that hits multiple backends.

Worth noting that E2B working in your case might be specific to your setup — in ggml-org#21516 E2B was also affected on other configurations, so it's not a clean size-dependent split.

For now, running on CPU backend should give correct output. An F32 MoE precision fix was explored upstream in PR ggml-org#21506 but was closed without merging, so the issue is still open. On the Steam Deck, some users have had luck with GGML_VK_PREFER_HOST_MEMORY=1 to avoid GPU memory pressure, though I haven't verified that specifically with Gemma4.