Merge pull request #4 from countzero/develop

Release v1.38.0

Author: countzero

.github/FUNDING.yml

@@ -0,0 +1,2 @@
+github: countzero
+ko_fi: finnkumkar

AGENTS.md

@@ -29,6 +29,7 @@ Binaries land in `./vendor/llama.cpp/build/bin/Release/`. Conda env `llama.cpp`
 - **Each build wipes the submodule** back to `origin/master` then checks out the requested `-version` / PR. Any local edits under `vendor/llama.cpp/` are lost by design. The reset/`--remote` step is **scoped to `vendor/llama.cpp` only** — other submodules (e.g. `vendor/Qwen-Fixed-Chat-Templates`, default branch `main`) stay at the SHA pinned in the superproject and are never advanced by the build script. To bump them, do it manually: `git -C vendor/Qwen-Fixed-Chat-Templates fetch && git -C vendor/Qwen-Fixed-Chat-Templates checkout <sha> && git add vendor/Qwen-Fixed-Chat-Templates && git commit`. Once the pin is committed, the next `rebuild_llama.cpp.ps1` mirrors it into the working tree (auto-discovered from `.gitmodules`, `--force`); hand-edits inside the submodule do not survive a rebuild.
 - **`ml64.exe` (MASM) must be passed as `-DCMAKE_ASM_COMPILER`.** Upstream `ggml/CMakeLists.txt` sets `cmake_policy(SET CMP0194 NEW)` and declares `project(... ASM)`; on CMake 4.1+ with the VS generator this rejects `cl.exe` as the ASM compiler. The script locates `ml64.exe` via `vswhere.exe`. Don't remove.
 - **CUDA is selected iff *both* `nvidia-smi` and `nvcc` are on PATH.** Missing either silently falls back to OpenBLAS.
+- **CUDA builds pass `-DGGML_CUDA_FA_ALL_QUANTS=ON`.** Without it the CUDA flash-attention path compiles only four *symmetric* KV kernels — `f16/f16`, `q4_0/q4_0`, `q8_0/q8_0`, `bf16/bf16` (`vendor/llama.cpp/ggml/src/ggml-cuda/CMakeLists.txt:119-124`) — and the dispatcher returns `BEST_FATTN_KERNEL_NONE` -> `GGML_ABORT` for any other K type or any mismatched K/V pair (`ggml-cuda/fattn.cu:424-446`). The flag pulls in the full `fattn-vec*.cu` set so asymmetric / q5 / q4_1 caches work (the presets use `q5_0` K + `q4_1` V). Costs extra `nvcc` compile time.
 - **Build parallelism is SMT-aware.** `cmake --build --parallel` is fed a count derived from `Win32_Processor`. Upstream's `UseMultiToolTask=true` + `EnforceProcessCountAcrossBuilds=true` (`vendor/llama.cpp/CMakeLists.txt:92-93`) makes this the single cap on concurrent `cl.exe`/`nvcc` — no per-project `/MP` multiplication. On SMT CPUs it uses physical cores (`Sum(NumberOfCores)`): dropping the logical siblings avoids starving the scheduler / ~doubling peak `nvcc` RAM (no throughput gain) and leaves them free so the machine stays usable. On non-SMT CPUs where physical == logical (hybrid Arrow/Lunar Lake; e.g. Core Ultra 9 285HX = 8P+16E, 24 threads) using all cores would peg the box at 100%, so it backs off to 80% of physical (`floor(cores * 0.8)` = 19 on the 285HX) to keep the machine usable during builds. Override with `-parallelJobs N`.
 - **`requirements_override.txt` layers on top of upstream `vendor/llama.cpp/requirements.txt`.** It pins `torch` to a CUDA 12.6 wheel, adds `tiktoken` (missing upstream, required for GLM), pins `transformers==5.3.0`, and narrows `numpy` to resolve an `opencv-python-headless` conflict. When bumping any of these, verify both constraints still hold.
 - **`server.ps1` reads GGUF metadata** by shelling out to `vendor/llama.cpp/gguf-py/gguf/scripts/gguf_dump.py`. Upstream has moved this path before (CHANGELOG 1.24.0) — if server startup fails with "Failed to extract model details", check the path first.

CHANGELOG.md

@@ -8,6 +8,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 
+## [1.38.0] - 2026-06-30
+
+### Added
+- [Documentation] Add MIT license
+- [Documentation] Add status badges and Support section to the README
+- [Documentation] Add GitHub Sponsors and Ko-fi funding configuration
+
+### Changed
+- [Build] Enable GGML_CUDA_FA_ALL_QUANTS on CUDA builds for all KV cache quant combinations
+- [Build] Pin GGML_SCHED_MAX_COPIES=1 on CUDA builds
+- [Presets] Switch dual-GPU 16GB+8GB KV cache to q5_0 K / q4_1 V
+- [Presets] Add tensor-split 1,2 to all dual-GPU 16GB+8GB entries
+- [Presets] Switch dual-GPU 16GB+8GB gemma-4-31B draft model to q4_0
+- [Presets] Retune dual-GPU 16GB+8GB ctx sizes (gemma-4-31B 102400, Qwen3.6-27B 200000)
+- [Presets] Lower gemma-4-31B 24GB ctx-size to 100000
+- [Presets] Lower Qwen3.6-27B IQ3_XXS 16GB ctx-size to 100000
+- [Presets] Switch 16GB tier from fit-target to fit=off
+- [Vendor] Bump llama.cpp submodule to b3fed31
+
+### Removed
+- [Presets] Drop gemma-4-26B-A4B from dual-GPU 16GB+8GB tier
+
+
 ## [1.37.0] - 2026-06-15
 
 ### Added

LICENSE.md

@@ -0,0 +1,32 @@
+# MIT License
+
+Copyright (c) 2023-2026 Finn Kumkar
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+## Third-party components
+
+This repository is a wrapper around third-party software that is licensed
+separately under its own terms:
+
+- `vendor/llama.cpp` (git submodule) - MIT License, Copyright (c) 2023-2026 The ggml authors
+- `vendor/Qwen-Fixed-Chat-Templates` (git submodule) - Apache License 2.0, froggeric
+- OpenBLAS (downloaded during build, not distributed in this repository) - BSD 3-Clause License
+
+Each component's full license text is provided within its respective source tree.

README.md

@@ -1,5 +1,12 @@
 # Windows llama.cpp
 
+[![Latest release](https://img.shields.io/github/v/release/countzero/windows_llama.cpp)](https://github.com/countzero/windows_llama.cpp/releases/latest)
+[![Last commit](https://img.shields.io/github/last-commit/countzero/windows_llama.cpp)](https://github.com/countzero/windows_llama.cpp/commits/main)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE.md)
+[![PowerShell 5.0+](https://img.shields.io/badge/PowerShell-5.0%2B-5391FE)](https://github.com/PowerShell/PowerShell)
+[![GitHub Sponsors](https://img.shields.io/github/sponsors/countzero?label=Sponsor&logo=GitHub)](https://github.com/sponsors/countzero)
+[![Ko-fi](https://img.shields.io/badge/Ko--fi-Tip-FF5E5B?logo=ko-fi&logoColor=white)](https://ko-fi.com/finnkumkar)
+
 A PowerShell automation to rebuild [llama.cpp](https://github.com/ggerganov/llama.cpp) for a Windows environment. It automates the following steps:
 
 1. Fetching and extracting a specific release of [OpenBLAS](https://github.com/xianyi/OpenBLAS/releases)
@@ -269,3 +276,12 @@ You can build a specific version of llama.cpp by specifying a git tag, commit or
 | `./rebuild_llama.cpp.ps1 -version "b1138"`     | The tag `b1138`          |
 | `./rebuild_llama.cpp.ps1 -version "1d16309"`   | The commit `1d16309`     |
 | `./rebuild_llama.cpp.ps1 -pullRequest "18675"` | The pull request `18675` |
+
+## Support
+
+If this project saves you time, consider supporting future work:
+
+- [GitHub Sponsors](https://github.com/sponsors/countzero): recurring or one-time.
+- [Ko-fi](https://ko-fi.com/finnkumkar): one-time tip, no signup required.
+
+[![Support on Ko-fi](https://storage.ko-fi.com/cdn/kofi2.png?v=6)](https://ko-fi.com/finnkumkar)

presets/models_16GB_8GB_VRAM.ini

@@ -1,56 +1,22 @@
 [*]
 threads = 24
 split-mode = layer
-tensor-split = 1,2
 main-gpu = 1
 models-max = 1
 
 [gemma-4-31B-it-qat-q4_0.IQ4_XS.gguf]
 alias = gemma-4-31B-it-qat-q4_0.IQ4_XS.gguf
 model = D:\AI\LLM\gguf\gemma-4-31B-it-qat-q4_0-unquantized\gemma-4-31B-it-qat-q4_0-unquantized.IQ4_XS.gguf
 mmproj = D:\AI\LLM\gguf\gemma-4-31B-it-qat-q4_0-unquantized\mmproj.gemma-4-31B-it-qat-q4_0-unquantized.BF16.gguf
-spec-draft-model = D:\AI\LLM\gguf\gemma-4-31B-it-qat-q4_0-unquantized-assistant\mtp-gemma-4-31B-it-qat-q4_0-unquantized-assistant.Q8_0.gguf
-no-mmproj-offload = true
-ctx-size = 100000
-parallel = 1
-n-gpu-layers = -1
-cache-type-k = q4_0
-cache-type-v = q4_0
-cache-type-k-draft = q8_0
-cache-type-v-draft = q8_0
-flash-attn = true
-reasoning = on
-jinja = true
-chat-template-file = vendor\llama.cpp\models\templates\google-gemma-4-31B-it.jinja
-kv-unified = true
-ctx-checkpoints = 32
-cache-ram = 51200
-min-p = 0.0
-top-p = 0.95
-top-k = 64
-temp = 1.0
-spec-type = draft-mtp,ngram-mod
-spec-draft-n-max = 3
-spec-ngram-mod-n-match = 24
-spec-ngram-mod-n-min = 48
-spec-ngram-mod-n-max = 64
-direct-io = true
-no-mmap = true
-fit = off
-
-[gemma-4-26B-A4B-it-qat-q4_0.IQ4_XS.gguf]
-alias = gemma-4-26B-A4B-it-qat-q4_0.IQ4_XS.gguf
-model = D:\AI\LLM\gguf\gemma-4-26B-A4B-it-qat-q4_0-unquantized\gemma-4-26B-A4B-it-qat-q4_0-unquantized.IQ4_XS.gguf
-mmproj = D:\AI\LLM\gguf\gemma-4-26B-A4B-it-qat-q4_0-unquantized\mmproj.gemma-4-26B-A4B-it-qat-q4_0-unquantized.BF16.gguf
-spec-draft-model = D:\AI\LLM\gguf\gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant\mtp-gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant.Q8_0.gguf
+spec-draft-model = D:\AI\LLM\gguf\gemma-4-31B-it-qat-q4_0-unquantized-assistant\mtp-gemma-4-31B-it-qat-q4_0-unquantized-assistant.Q4_0.gguf
 no-mmproj-offload = true
-ctx-size = 262144
+ctx-size = 102400
 parallel = 1
 n-gpu-layers = -1
-cache-type-k = q8_0
-cache-type-v = q8_0
-cache-type-k-draft = q8_0
-cache-type-v-draft = q8_0
+cache-type-k = q5_0
+cache-type-v = q4_1
+cache-type-k-draft = q5_0
+cache-type-v-draft = q4_1
 flash-attn = true
 reasoning = on
 jinja = true
@@ -69,20 +35,21 @@ spec-ngram-mod-n-min = 48
 spec-ngram-mod-n-max = 64
 direct-io = true
 no-mmap = true
+tensor-split = 1,2
 fit = off
 
 [Qwen3.6-27B.IQ4_XS.gguf]
 alias = Qwen3.6-27B.IQ4_XS.gguf
 model = D:\AI\LLM\gguf\Qwen3.6-27B\Qwen3.6-27B.IQ4_XS.gguf
 mmproj = D:\AI\LLM\gguf\Qwen3.6-27B\mmproj.Qwen3.6-27B.BF16.gguf
 no-mmproj-offload = true
-ctx-size = 162144
+ctx-size = 200000
 parallel = 1
 n-gpu-layers = -1
-cache-type-k = q4_0
-cache-type-v = q4_0
-cache-type-k-draft = q4_0
-cache-type-v-draft = q4_0
+cache-type-k = q5_0
+cache-type-v = q4_1
+cache-type-k-draft = q5_0
+cache-type-v-draft = q4_1
 flash-attn = true
 reasoning = on
 jinja = true
@@ -102,6 +69,7 @@ spec-ngram-mod-n-min = 48
 spec-ngram-mod-n-max = 64
 direct-io = true
 no-mmap = true
+tensor-split = 1,2
 fit = off
 
 [Qwen3.6-35B-A3B.IQ4_XS.gguf]
@@ -112,10 +80,10 @@ no-mmproj-offload = true
 ctx-size = 262144
 parallel = 1
 n-gpu-layers = -1
-cache-type-k = q4_0
-cache-type-v = q4_0
-cache-type-k-draft = q4_0
-cache-type-v-draft = q4_0
+cache-type-k = q5_0
+cache-type-v = q4_1
+cache-type-k-draft = q5_0
+cache-type-v-draft = q4_1
 flash-attn = true
 reasoning = on
 jinja = true
@@ -135,4 +103,5 @@ spec-ngram-mod-n-min = 48
 spec-ngram-mod-n-max = 64
 direct-io = true
 no-mmap = true
+tensor-split = 1,2
 fit = off

presets/models_16GB_VRAM.ini

@@ -34,7 +34,7 @@ alias = Qwen3.6-27B.IQ3_XXS.gguf
 model = D:\AI\LLM\gguf\Qwen3.6-27B\Qwen3.6-27B.IQ3_XXS.gguf
 mmproj = D:\AI\LLM\gguf\Qwen3.6-27B\mmproj.Qwen3.6-27B.BF16.gguf
 no-mmproj-offload = true
-ctx-size = 262144
+ctx-size = 100000
 threads = 16
 parallel = 1
 n-gpu-layers = -1
@@ -61,7 +61,7 @@ spec-ngram-mod-n-min = 48
 spec-ngram-mod-n-max = 64
 direct-io = true
 no-mmap = true
-fit-target = 1024
+fit = off
 
 [Qwen3.6-27B.IQ4_XS.gguf]
 alias = Qwen3.6-27B.IQ4_XS.gguf
@@ -120,7 +120,6 @@ top-p = 0.95
 top-k = 20
 temp = 0.6
 presence-penalty = 0
-fit-target = 512
 spec-type = draft-mtp,ngram-mod
 spec-draft-n-max = 4
 spec-ngram-mod-n-match = 24

presets/models_24GB_VRAM.ini

@@ -8,7 +8,7 @@ model = D:\AI\LLM\gguf\gemma-4-31B-it-qat-q4_0-unquantized\gemma-4-31B-it-qat-q4
 mmproj = D:\AI\LLM\gguf\gemma-4-31B-it-qat-q4_0-unquantized\mmproj.gemma-4-31B-it-qat-q4_0-unquantized.BF16.gguf
 spec-draft-model = D:\AI\LLM\gguf\gemma-4-31B-it-qat-q4_0-unquantized-assistant\mtp-gemma-4-31B-it-qat-q4_0-unquantized-assistant.Q4_0.gguf
 no-mmproj-offload = true
-ctx-size = 131072
+ctx-size = 100000
 parallel = 1
 n-gpu-layers = -1
 cache-type-k = q4_0

rebuild_llama.cpp.ps1

@@ -304,9 +304,20 @@ switch ($blasAccelerator) {
     }
 
     "CUDA" {
+        # Pin pipeline-parallel staging to a single copy (ggml default is 4).
+        # On multi-GPU layer-split, ggml pre-allocates GGML_SCHED_MAX_COPIES
+        # copies of the compute buffer per device. Single-stream decode gains
+        # nothing from the extra copies.
+        #
+        # GGML_CUDA_FA_ALL_QUANTS compiles the full set of flash-attention
+        # K/V quant kernels. Without it only f16/f16, q4_0/q4_0, q8_0/q8_0,
+        # bf16/bf16 are built; any mixed or q5/q4_1 KV cache aborts at runtime.
+        # Unlocks q5_0/q4_1 at the cost of extra nvcc compile time.
         cmake `
             -DCMAKE_ASM_COMPILER="$ml64" `
             -DGGML_CUDA=ON `
+            -DGGML_SCHED_MAX_COPIES=1 `
+            -DGGML_CUDA_FA_ALL_QUANTS=ON `
             -DLLAMA_CURL=OFF `
             ..
     }