v0.3.0

Author: Anbeeld

CHANGELOG.md

@@ -2,42 +2,27 @@
 
 ## v0.3.0
 
-- Rebased BeeLlama onto a much newer llama.cpp upstream while preserving the fork-specific DFlash, CopySpec, DDTree, TurboQuant/TCQ, Tom TQ, Gemma, Qwen, multimodal, reasoning, and server changes that were still production-ready.
-- Merged post-rebase llama.cpp upstream through `35c9b1f39`, preserving upstream history while adapting conflicts around Bee's DFlash/MTP server paths, checkpointing, CUDA/Metal/TurboQuant surfaces, UI state, and CI split workflows.
-- Adopted the upstream speculative-decoding framework shape for Bee features, including the newer MTP support, parallel drafting infrastructure, backend sampling path, and upstream spec argument naming.
-- Preserved DFlash explicit selection and DFlash draft-GGUF auto-detection, including support for both Bee/buun `dflash-draft` and upstream-style `dflash` draft metadata.
-- Kept upstream raw `--spec-draft-n-max` default at `3`, but applies DFlash's effective omitted draft max of `16` only when DFlash is explicit or auto-detected.
-- Restored upstream target batch defaults for DFlash: `-b` remains `2048` and `-ub` remains `512` unless the user overrides them.
-- Kept DFlash's effective omitted draft context default at `-cd 256`, including the server-side auto-detect path where DFlash is only known after loading the draft model.
-- Removed stale or misleading speculative CLI surface: `--spec-dflash-default`, `--draft`, `--draft-n`, `--draft-max`, `--draft-min`, `--draft-n-min`, `--tree-budget`, `--dflash-max-slots`, `--draft-topk`, `--draft-model`, `--spec-draft-replace`, and `--spec-replace`.
-- Kept upstream-compatible draft aliases such as `--model-draft`, `--draft-p-split`, and `--draft-p-min`; canonical Bee DFlash controls now use `--spec-*` names.
-- Made DFlash-only args warning/no-op for MTP and other non-DFlash modes, while preserving them long enough for server-side DFlash draft-model auto-detection when a draft model is supplied without an explicit `--spec-type`.
-- Gated adaptive Draft-Max to actual DFlash slots so MTP and other non-DFlash modes do not inherit DFlash adaptive depth behavior.
-- Separated pure upstream MTP from Bee fork speculators after the rebase: MTP now uses upstream shared speculative state and explicit backup rollback, while DFlash/CopySpec/suffix/recycle keep Bee per-slot state and DFlash-specific recurrent handling.
-- Restored `--spec-draft-temp` for sampled DFlash, including greedy `0`, positive sampled/Gumbel drafter temperature, and `auto` mirroring of target temperature.
-- Preserved CopySpec, suffix-tree speculation, recycle speculation, and upstream ngram speculative modes as explicit opt-in backends.
-- Preserved branch-only DDTree verification through `--spec-branch-budget`, restored Qwen DDTree conv and GDN paths, and removed the old public-buun total-node `--tree-budget` CLI spelling.
-- Restored Qwen per-layer KV heads, Qwen final-layer output gather, and Qwen MTP draft-context stability after the upstream merge.
-- Preserved DFlash hidden-state capture, the 4096-token per-layer CPU ring, configurable DFlash cross-attention window, GPU ring path, and DFlash draft graph integration.
-- Added DFlash GPU hidden-ring span guards, discarded stale DFlash ring checkpoints, skipped checkpoints for uncached prompts, and avoided raw prompt-logit output in DFlash paths that should not request it.
-- Restored DFlash tool-call drafting and stale drafter-state cleanup so long tool-call content can use DFlash without corrupting syntax or lazy-grammar behavior.
-- Fixed DFlash multi-slot serving issues: slots now default to server parallelism, mixed speculative/non-spec target batches are avoided, uneven `-np` target batches are split safely, and shared prefill capture state is restored.
-- Enabled device-aware multi-GPU DFlash capture/tape/replay by default on v0.3.0, with `GGML_DFLASH_MULTI_GPU_TAPE=0` and `GGML_DFLASH_ALLOW_MULTI_GPU_TAPE=0` kill switches.
-- Added per-layer device placement for DFlash hidden capture, prefill capture, recurrent tape, conv replay, direct GDN replay, recurrent backup copy plans, and multi-device synchronization.
-- Hardened DFlash CUDA helpers by preserving the caller CUDA device, enabling peer D2D copies where available, validating pointer-device placement before direct replay, and falling back to CPU/eval-callback paths when split placement is unsupported.
-- Kept DFlash draft model placement single-device by default unless the user explicitly passes draft-device placement.
-- Restored and preserved adaptive Draft-Max fixes: profit-controller cold start, premature demotion prevention, per-request state reset, continuation-state preservation, and profiling-gated timing logs.
-- Preserved TurboQuant and TCQ cache backends (`turbo2`, `turbo3`, `turbo4`, `turbo2_tcq`, `turbo3_tcq`), buun-compatible cache enum layout, and 128-value `turbo2`/`turbo3` block sizes from Tom's public TurboQuant work.
-- Preserved Tom's `TQ3_1S` and `TQ4_1S` model weight formats with non-conflicting GGML type IDs.
-- Kept HIP TCQ attention on the native vector path, preserved the fused GDN 4D state fast path, restored the D512 flash-attention selector, and added CPU f16 `out_prod` fallback support.
-- Preserved Gemma3 full image chunks, Gemma multimodal precision/profiling hooks where still relevant, and flat-DFlash multimodal compatibility while keeping tree and non-DFlash speculation disabled under multimodal where unsupported.
-- Suppressed leading reasoning/thinking syntax in streamed title generation so session titles do not surface stray tags such as closing thinking markers.
-- Preserved the reasoning loop guard, malformed tool-call filtering, request-level speculative overrides, and DFlash/mmproj server guardrails on the newer upstream server.
-- Adopted upstream MTP/cache and server fixes after the v0.3.0 rebase: MTP draft contexts now use draft KV cache types instead of inheriting target `--cache-type-k/v`, draft/MTP fit memory is reserved before target fitting, MTP draft hidden state is aligned to the verified `n_past` boundary, and prompt checkpoints are placed around the latest user turn with `--checkpoint-min-step`.
-- Brought in upstream server/runtime improvements after v0.2.0, including draft/MTP resource cleanup on sleep, prompt token counts in `/slots`, router and model metadata fixes, API compatibility updates, and newer built-in server tooling.
-- Brought in upstream post-rebase runtime/backend updates including NVFP4 MTP scale tensors, GGUF callback/buffer initialization APIs, CUDA FWHT handling for Hadamard-hinted matmuls, parallel quant LUT initialization, Metal device IDs, cpp-httplib 0.45.1, and UI continue/agentic-loop fixes.
-- Brought in upstream conversion, UI/app, Docker, CI, backend, model, tokenizer, and documentation updates from the newer llama.cpp base while keeping BeeLlama-specific docs and workflow metadata.
-- Expanded parser and DFlash plumbing tests for removed args, DFlash default normalization, DFlash-only no-op behavior, auto-detect preservation, multi-GPU policy helpers, per-layer capture/tape allocation, device-aware replay, CUDA device restoration, and adaptive Draft-Max behavior.
+- Updated to a much newer llama.cpp base. The upstream refresh brings native MTP speculative decoding, parallel drafting and backend sampling work, the unified `llama` app, newer server/API behavior, the `tools/ui` Web UI restructure, model and converter additions, multimodal improvements, and backend gains across CUDA/HIP, Metal, Vulkan, SYCL, OpenCL, WebGPU, Hexagon, ZenDNN, and SpacemiT.
+- Added usable MTP serving on the Bee tree. `draft-mtp`/`mtp` now has its own speculative path instead of being mixed with DFlash state, uses draft KV cache types, cleans up draft resources on sleep, works with target pre-norm hidden-state capture, and handles text requests when an `mmproj` is loaded.
+- Cleaned up DFlash command-line behavior around canonical `--spec-*` arguments. DFlash can still be selected with `--spec-type dflash` or auto-detected from compatible draft GGUF metadata, but stale aliases such as `--spec-dflash-default`, `--draft*`, `--draft-topk`, `--draft-model`, `--tree-budget`, `--dflash-max-slots`, `--spec-draft-replace`, and `--spec-replace` were removed.
+- Made DFlash defaults match the new upstream speculative surface without changing the practical DFlash defaults. Raw `--spec-draft-n-max` stays at upstream `3`, DFlash raises the effective omitted draft max to `16` only after explicit or auto-detected DFlash, omitted `--spec-draft-ctx-size` still becomes `256`, and DFlash no longer lowers target `-b 2048` / `-ub 512` unless the user asks for smaller batches.
+- Made DFlash-only arguments safe around non-DFlash modes. DFlash-only controls warn and no-op for MTP or other speculative types, while still surviving long enough for server-side DFlash draft-model auto-detection when no explicit `--spec-type dflash` was passed.
+- Expanded multi-slot DFlash serving. DFlash slots now default to server parallelism, `--spec-dflash-max-slots` caps them below `-np` when needed, uneven `-np` target batches are split safely, mixed speculative/non-speculative target batches are avoided, and flat multi-slot DFlash can use shared drafter batching with `GGML_DFLASH_SHARED_DRAFT_BATCH=0` as the fallback switch.
+- Reworked adaptive DFlash draft depth for live serving. The default `profit` controller now seeds and periodically remeasures a no-spec baseline, probes shallow/mid/full positive depths, backs off failed wake probes, avoids premature demotion, resets per-request state correctly, preserves compatible continuation state, and gates timing logs behind DFlash profiling.
+- Added default-on device-aware DFlash GPU capture/tape/replay for split CUDA/ROCm target placement. Hidden capture, prefill capture, recurrent tape, conv replay, direct GDN replay, rollback copies, and synchronization now follow each layer's backend device, with CPU/eval-callback fallback and `GGML_DFLASH_MULTI_GPU_TAPE=0` / `GGML_DFLASH_ALLOW_MULTI_GPU_TAPE=0` kill switches.
+- Hardened DFlash device placement. Explicit single draft-device placement pins the target output tensor before target load, auto-detected DFlash drafters stay single-device by default unless draft devices are explicit, iGPU backends are accepted for GPU paths, tensor-split/meta placement is guarded, CUDA helper calls preserve the caller device, and peer D2D copies are used where available.
+- Hardened DFlash hidden-ring and prompt-cache state. The server guards GPU hidden-ring spans, discards stale DFlash ring checkpoints, skips DFlash checkpoints for uncached prompts, restores shared prefill capture state, and avoids requesting raw prompt logits from DFlash paths that should not read them.
+- Reduced DFlash accept and verification overhead while failing closed on bad reduced-logit state. The accept path defers single-slot rollback sync and drafter KV maintenance, keeps the fused GDN 4D state fast path, adds CPU f16 `out_prod` fallback, and disables DFlash drafting after repeated invalid reduced-logits drafts instead of looping or corrupting state.
+- Improved DFlash tool-call and reasoning behavior. DFlash can keep drafting before a lazy tool-call grammar actually constrains output, stale drafter KV state is cleared before long tool-call continuations, stable partial tool-call headers stream earlier, streaming reasoning deltas stay isolated, and streamed title generation suppresses leading thinking syntax.
+- Kept flat DFlash available with multimodal serving while making unsupported combinations explicit. With `--mmproj`, Bee keeps flat DFlash usable, forces DFlash tree branch budget to `0`, disables non-DFlash speculative modes, disables unsupported context-shift/cache-reuse paths, and fixes MTP text-only requests when an `mmproj` is present.
+- Updated Qwen 3.5/3.6 speculative paths. Qwen gets per-layer KV heads, final-layer output gathering, stable MTP draft context behavior, and Qwen DDTree conv/GDN paths for tree verification.
+- Removed legacy DDTree total-node semantics from the public CLI. Tree DFlash now uses the branch-only `--spec-branch-budget` model consistently, with `--spec-draft-top-k` controlling candidates per draft position and flat DFlash forcing top-k back to `1`.
+- Added new cache and quantization surface beyond v0.2.0. `q6_0` is available as a KV/cache type, and Tom's `TQ3_1S` / `TQ4_1S` model weight formats are exposed through `llama-quantize` with non-conflicting serialized GGML type IDs; existing `turbo2`, `turbo3`, `turbo4`, `turbo2_tcq`, and `turbo3_tcq` cache types remain available on the newer base.
+- Hardened backend behavior used by Bee features. HIP TCQ attention stays on the native vector path, ROCm can probe fused GDN support, D512 Flash Attention selection is available again, unsupported CPU BF16 scale and CPU Flash Attention types are rejected instead of misrunning, CUDA fatal-warning cases were fixed, and the build now requires C++17 for the common base.
+- Improved server and API behavior inherited from upstream and Bee integration. The server reports prompt token counts in `/slots`, supports SSE ping intervals and HTTP ETags, exposes real-time reasoning interruption, handles router/model metadata fixes, adds built-in tools such as datetime, and keeps malformed tool-looking text out of final responses.
+- Updated packaging and release workflows. Docker image links and labels were aligned for BeeLlama, SYCL package/release images were added and documented, release builds were split and cached more reliably, stale package builds are cancelled, and obsolete CUDA architecture options are rejected at configure time.
+- Updated user documentation for the new release state, including DFlash args/defaults, removed aliases, adaptive Draft-Max, multi-GPU DFlash behavior, Turbo/TQ cache and weight formats, Docker images, and the upstream multi-GPU guide.
+- Expanded regression coverage around DFlash default normalization, removed arg aliases, DFlash-only no-op behavior, DFlash auto-detection, multi-GPU policy helpers, per-layer capture/tape allocation, device-aware replay, CUDA device restoration, adaptive Draft-Max, reduced-logit failure handling, and tool-call/speculative boundaries.
 
 ## v0.2.0
 

README.md

@@ -18,6 +18,7 @@ BeeLlama.cpp (or just Bee) is a performance-focused llama.cpp fork for squeezing
 - **DFlash speculative decoding**: `--spec-type dflash` drives a DFlash draft GGUF alongside the target model. The target captures hidden states into a ring buffer, the drafter cross-attends to the most recent `--spec-dflash-cross-ctx` hidden-state tokens and proposes drafts for target verification.
 - **Adaptive draft-max control**: The server adjusts the active draft horizon at runtime instead of using a fixed `--spec-draft-n-max`. The default `profit` controller compares speculative throughput against a no-spec baseline, the `fringe` alternative maps acceptance-rate bands to draft depth.
 - **Full multimodal support**: When `--mmproj` is active, the server keeps DFlash available for text generation. The model can be fully offloaded to CPU with no problems to reduce VRAM pressure.
+- **Up to date with upstream llama.cpp** — MTP speculative decoding, parallel drafting, the unified llama app, updated server/API behavior, backend improvements across CUDA, HIP, Metal, Vulkan, and more.
 - **TurboQuant / TCQ KV-cache compression**: Five cache types (`turbo2`, `turbo3`, `turbo4`, `turbo2_tcq`, `turbo3_tcq`) spanning from 4x to 7.5x compression, with TCQ types offering good precision for their size. Set independently with `--cache-type-k` and `--cache-type-v`.
 - **Tom TQ3/TQ4 model weight formats**: `TQ3_1S` and `TQ4_1S` are available through `llama-quantize` with non-conflicting GGML type IDs `47` and `48`. These are GGUF weight formats, not KV-cache types; backend acceleration should be validated before claiming it for a deployment.
 - **Reasoning-loop protection**: The server detects repeated hidden reasoning output and intervenes. Default mode is `force-close`, with `--reasoning-loop-window` and `--reasoning-loop-max-period` tuning available.
@@ -291,27 +292,42 @@ K and V cache types are set independently with `--cache-type-k` and `--cache-typ
 | turbo2_tcq | fork | 2.25 | 7.11× | Last resort, 54% precision at KLD 99.9%. CUDA-only |
 | turbo2 | fork | 2.125 | 7.53× | Extreme quality risk. Use only when TCQ is unavailable |
 
-### Model Weight Quantization
-
-Tom's `TQ3_1S` and `TQ4_1S` are model weight formats exposed through `llama-quantize`:
-
-```sh
-llama-quantize model.f16.gguf model.tq3_1s.gguf TQ3_1S
-llama-quantize model.f16.gguf model.tq4_1s.gguf TQ4_1S
-```
-
-These formats use GGML type IDs `47` and `48` so they do not collide with Bee/Buun TCQ cache IDs `45` and `46`. They are separate from `--cache-type-k` and `--cache-type-v`.
-
 ## Installation
 
 ### Plug-and-Play Setups
 
 - [Qwen 3.6 27B Q5_K_S + DFlash + vision + 160k context in 24 GB VRAM](docs/quickstart-qwen36-dflash.md)
 - [Gemma 4 31B Q4_K_S + DFlash + vision + 140k context in 24 GB VRAM](docs/quickstart-gemma-4-31b-dflash.md)
 
-### Prebuilt (Windows)
-
-Download the release archive for your CUDA version (12.4 or 13.1) from the [releases page](https://github.com/Anbeeld/beellama.cpp/releases). Extract it. The server binary is `llama-server.exe`. Don't forget to download a separate archive with CUDA libraries and place it in the same folder!
+### Prebuilt
+
+Binaries for every backend are on the [releases page](https://github.com/Anbeeld/beellama.cpp/releases):
+
+| Platform | Backend | Archive |
+| --- | --- | --- |
+| macOS | Apple Silicon Metal | `bin-macos-arm64.tar.gz` |
+| Linux x64 | CPU | `bin-ubuntu-x64.tar.gz` |
+| Linux x64 | Vulkan | `bin-ubuntu-vulkan-x64.tar.gz` |
+| Linux x64 | ROCm 7.2 | `bin-ubuntu-rocm-7.2-x64.tar.gz` |
+| Linux x64 | SYCL | `bin-ubuntu-sycl-x64.tar.gz` |
+| Windows x64 | CPU | `bin-win-cpu-x64.zip` |
+| Windows x64 | SYCL | `bin-win-sycl-x64.zip` |
+| Windows x64 | CUDA 12 | `bin-win-cuda-12.4-x64.zip` |
+| Windows x64 | CUDA 13 | `bin-win-cuda-13.1-x64.zip` |
+| Windows x64 | HIP/Radeon | `bin-win-hip-radeon-x64.zip` |
+
+Windows CUDA archives contain a `ggml-cuda.dll` backend; download the matching `cudart-win-cuda-*-x64.zip` runtime archive and extract it into the same folder. Windows SYCL and HIP archives ship as standalone packages with all required runtime DLLs bundled.
+
+Docker images are published to `ghcr.io/anbeeld/beellama.cpp`:
+
+| Image | Acceleration | Platforms |
+| --- | --- | --- |
+| `server-cpu` | CPU | linux/amd64, linux/arm64 |
+| `server-cuda12` | CUDA 12.4 | linux/amd64 |
+| `server-cuda13` | CUDA 13.1 | linux/amd64 |
+| `server-rocm` | ROCm | linux/amd64 |
+| `server-vulkan` | Vulkan | linux/amd64 |
+| `server-sycl` | SYCL | linux/amd64 |
 
 Building from source with `-DGGML_NATIVE=ON` *may* result in a *tiny* bit better performance, so it might still be a good idea to do that if/when you decide to use this fork long-term.