Feature Request: support 1 bit Bonsai models

[laurentpayot]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

The new 1 bit Bonsai 8B model has just been released.:

1-bit Bonsai 8B implements a proprietary 1-bit model design across the entire network: embeddings, attention layers, MLP layers, and the LM head are all 1-bit. There are no higher-precision escape hatches. It is a true 1-bit model, end to end, across 8.2 billion parameters.
Despite being 14x smaller than the 8B (16-bit) full-precision models in its parameter-count class, it performs competitively on standard benchmarks while operating at radically higher efficiency.

Too bad a fork of llama.cpp is needed.

Related feature request #12598 was unfortunately closed. Could Bonsai models support be added?

Motivation

Performances of this model are impressive. Reactions are very positive.

Possible Implementation

No response

[webda2l]

Related #21273

[henk717]

Considering #21273 is the developer of said fork trying to upstream it I'd say the feature request is already in progress in that PR.

[laurentpayot]

I’m now using release b8685 (Ubuntu 25.10 Vulkan), post release b8682 where #21273 was merged.
Now no Llama-server startup errors with the Bonsai 8B model, but when a request is done it is stuck forever, with the CPU (not GPU) usage at 50%.

Here are the logs:

srv  update_slots: all slots are idle
srv  params_from_: Chat format: peg-native
slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1
srv  get_availabl: updating prompt cache
srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 65536 tokens, 8589934592 est)
srv  get_availabl: prompt cache update took 0.01 ms
slot launch_slot_: id  3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist 
slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
slot update_slots: id  3 | task 0 | new prompt, n_ctx_slot = 65536, n_keep = 0, task.n_tokens = 1627
slot update_slots: id  3 | task 0 | n_tokens = 0, memory_seq_rm [0, end)
slot init_sampler: id  3 | task 0 | init sampler, took 0.20 ms, tokens: text = 1627, total = 1627
slot update_slots: id  3 | task 0 | prompt processing done, n_tokens = 1627, batch.n_tokens = 1627

Here is my script:

#!/bin/bash
llama-server -m ~/models/Bonsai-8B.gguf \
  --gpu-layers 37 \
  --predict 256 \
  --temp 0.5 \
  --top-p 0.85 \
  --top-k 20 \
  --port 8012 \
  --host 0.0.0.0

[foxbg]

It looks vulkan support was not yet added. CPU is working, but really slow on Ryzen5 5500 i get ~ 5-6 tps

 .\llama-bench.exe -dev none -m Bonsai-8B.gguf
load_backend: loaded RPC backend from C:\llama-b8705-bin-win-cpu-x64\ggml-rpc.dll
load_backend: loaded CPU backend from C:\llama-b8705-bin-win-cpu-x64\ggml-cpu-haswell.dll
| model                          |       size |     params | backend    | threads | dev          |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------ | --------------: | -------------------: |
| qwen3 8B unknown, may not work |   1.07 GiB |     8.19 B | CPU        |       6 | none         |           pp512 |          6.35 ± 0.12 |
| qwen3 8B unknown, may not work |   1.07 GiB |     8.19 B | CPU        |       6 | none         |           tg128 |          5.59 ± 0.02 |

build: 09343c019 (8705)

[Disonantemus]

• Last release already has vulkan support.
• Previously was painfully slow, now is ok to use!.
• I think was this commit: vulkan: Support Q1_0 #21539

$ llama-bench -m Bonsai-8B.gguf
load_backend: loaded RPC backend from /home/user1/bin5/libggml-rpc.so
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce GTX 1660 SUPER (NVIDIA) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from /home/user1/bin5/libggml-vulkan.so
load_backend: loaded CPU backend from /home/user1/bin5/libggml-cpu-haswell.so
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen3 8B Q1_0                  |   1.07 GiB |     8.19 B | Vulkan     |  99 |           pp512 |        412.15 ± 1.68 |
| qwen3 8B Q1_0                  |   1.07 GiB |     8.19 B | Vulkan     |  99 |           tg128 |         48.58 ± 0.18 |

build: d7ff074c8 (8744)

Only the markdown table:

model	size	params	backend	ngl	test	t/s
qwen3 8B Q1_0	1.07 GiB	8.19 B	Vulkan	99	pp512	412.15 ± 1.68
qwen3 8B Q1_0	1.07 GiB	8.19 B	Vulkan	99	tg128	48.58 ± 0.18

---

System

distro: Arch Linux x86_64
kernel: 6.12.33-1-lts
ram: 32GB (DDR3)
cpu: Intel i7-4790 (8) @ 3.600GHz
gpu: NVIDIA GeForce GTX 1660 SUPER (6GB VRAM)

[emansom]

distro: Arch Linux x86_64
kernel: 6.12.33-1-lts
ram: 32GB (DDR3)
cpu: Intel i7-4790 (8) @ 3.600GHz
gpu: NVIDIA GeForce GTX 1660 SUPER (6GB VRAM)

@Disonantemus You are aware that you are testing on ancient hardware? Not really representative nor useful.

Several advancements in instructions per clock and x86 architecture and GPGPU architecture have been made the past 12 years, leading to multiple tens of factors of performance improvements. On a modern Zen 5 architecture paired with a RTX 5000 series graphics processor, the t/s should be in the ten folds higher.

It's cool that it works at all on such hardware, at that token/s rate. However, it should not be taken as representation of the absolute performance Q1 is capable of achieving.

[laurentpayot]

Using release b8747, I’m getting around 100 tokens/s on my Strix Halo after a "hi" prompt with the pi agent. It’s definitively working. Thanks to the Llama.cpp team! 👍

[foxbg]

Confirming ~10 fold increase after vulkan support was added.

 .\llama-bench.exe -m .\Bonsai-8B.gguf
load_backend: loaded RPC backend from C:\llama-b8748-bin-win-vulkan-x64\ggml-rpc.dll
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 6600 (AMD proprietary driver) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from C:\llama-b8748-bin-win-vulkan-x64\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\llama-b8748-bin-win-vulkan-x64\ggml-cpu-haswell.dll
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen3 8B unknown, may not work |   1.07 GiB |     8.19 B | Vulkan     |  99 |           pp512 |        432.32 ± 0.52 |
| qwen3 8B unknown, may not work |   1.07 GiB |     8.19 B | Vulkan     |  99 |           tg128 |         68.26 ± 0.10 |

build: 5dd102539 (8748)