Eval bug: Very slow inference of Q1_0 Bonsai model

[nbkgroup]

Name and Version

version: 8685 (0988acc)
built with GNU 13.3.0 for Linux x86_64

Operating systems

Linux

GGML backends

CUDA, CPU, Vulkan

Hardware

Ryzen 7 5700X + GeForce RTX 5060 Ti Driver Version: 580.126.20 CUDA Version: 13.0

Models

Bonsai-8B.gguf

Problem description & steps to reproduce

After adding support for Bonsai models and Q1_0 quantization, the Bonsai-8B model inference speed on all the new versions of llama.cpp is about 0.3 - 0.5 t/s on CPU/Vulkan/NVidia.

Tested on various hardware, the speed is the same. It appears no hardware acceleration used.

The inference speed of the original fork is about 130-140 t/s on the reported hardware.

First Bad Commit

No response

Relevant log output

Logs

llama-cli -m Bonsai-8B.gguf -ngl 99 -p "Tell me your name"
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 15825 MiB):
  Device 0: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes, VRAM: 15825 MiB

Loading model...  


▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀

build      : b8685-0988accf8
model      : Bonsai-8B.gguf
modalities : text

available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read <file>        add a text file
  /glob <pattern>     add text files using globbing pattern


> Tell me your name

My name is Bonsai. I am an AI assistant developed by PrismML. How can I help you today?

[ Prompt: 0,3 t/s | Generation: 0,3 t/s ]

> 

[pwilkin]

There's no CUDA backend yet.

[iplayfast]

I'm getting really good speeds from this repo https://github.com/PrismML-Eng/Bonsai-demo.git Which is based on llama.cpp

[pwilkin]

There will be a CUDA backend. There just isn't one yet :) the support just got merged.

[iplayfast]

I wasn't complaining, just pointing out a working system if help was needed. :)

[BarfingLemurs]

the merged cpu inference isn't optimized for x86, further cpu speedup are those pull requests mentioned from here.
PrismML-Eng/Bonsai-demo#17 (comment)

[pwilkin]

Try #21584 and please tell me if it works OK.

[khosravipasha]

For CPU x86 now this PR should speed things up after its merged:
#21636

[VexilonHacker]

Having same problem in Vulkan.
Slow token generation compared to other larger models.

[nbkgroup]

I'm getting really good speeds from this repo https://github.com/PrismML-Eng/Bonsai-demo.git Which is based on llama.cpp

Yep, I compared the performance decline exactly against it.

[nbkgroup]

There's no CUDA backend yet.

As I understand, there is not ANY backed yet. It's surprising, for the work already been accomplished in the fork. Now I see the work is in progress, thanks to all involved developers.

[khosravipasha]

@nbkgroup CUDA backend was merged 1-2 days ago: #21629

and x86 optimized one is almost there too: #21636

[emansom]

Having same problem in Vulkan. Slow token generation compared to other larger models.

@VexilonHacker Integrated graphics, be realistic. Are you testing with a build that includes #21539?

Update: Yeah you're not, you are running b8694. Update to latest and try again.

[nbkgroup]

Having same problem in Vulkan. Slow token generation compared to other larger models.

@VexilonHacker Integrated graphics, be realistic. Are you testing with a build that includes #21539?

Update: Yeah you're not, you are running b8694. Update to latest and try again.

I have same speed for CUDA and Vulkan backends on same hardware, about 130-140 t/s.