|
1 | 1 | --- |
| 2 | +- name: "gemma-4-26b-a4b-it-qat" |
| 3 | + url: "github:mudler/LocalAI/gallery/virtual.yaml@master" |
| 4 | + urls: |
| 5 | + - https://huggingface.co/unsloth/gemma-4-26B-A4B-it-qat-GGUF |
| 6 | + description: | |
| 7 | + Hugging Face | |
| 8 | + GitHub | |
| 9 | + Launch Blog | |
| 10 | + Documentation |
| 11 | + |
| 12 | + License: Apache 2.0 | Authors: Google DeepMind |
| 13 | + |
| 14 | + > [!Note] |
| 15 | + > This model card is for the new versions of the Gemma 4 family optimized with Quantization-Aware Training (QAT), which allows preserving similar quality to bfloat16 while dramatically reducing the memory requirements to load the model. |
| 16 | + > Four versions of the QAT checkpoints are available: |
| 17 | + > * **Unquantized QAT checkpoints** (Q4_0): Half-precision weights extracted from the QAT pipeline, ideal for custom downstream compilation and research. Available for Gemma 4 E2B, E4B, 12B, 26B A4B, and 31B, and their drafter models. |
| 18 | + > * **GGUF** (Q4_0): Ready-to-deploy formats for broad ecosystem compatibility. Available for Gemma 4 E2B, E4B, 12B, 26B A4B, and 31B. |
| 19 | + > * **Mobile-optimized** (wNa8o8): A custom schema engineered explicitly for mobile hardware efficiency. It features targeted 2-bit decoding layers, optimized KV caches, and static activations to maximize VRAM savings. Available for Gemma 4 E2B and E4B. |
| 20 | + > * **Compressed Tensors** (w4a16): QAT checkpoints serialized in the compressed-tensors format for native, optimized inference with vLLM. Available for Gemma 4 E2B, E4B, 12B |
| 21 | + |
| 22 | + ... |
| 23 | + license: "apache-2.0" |
| 24 | + tags: |
| 25 | + - llm |
| 26 | + - gguf |
| 27 | + - gemma |
| 28 | + icon: https://ai.google.dev/gemma/images/gemma4_banner.png |
| 29 | + overrides: |
| 30 | + backend: llama-cpp |
| 31 | + function: |
| 32 | + automatic_tool_parsing_fallback: true |
| 33 | + grammar: |
| 34 | + disable: true |
| 35 | + known_usecases: |
| 36 | + - chat |
| 37 | + mmproj: llama-cpp/mmproj/gemma-4-26B-A4B-it-qat-GGUF/mmproj-F32.gguf |
| 38 | + options: |
| 39 | + - use_jinja:true |
| 40 | + parameters: |
| 41 | + min_p: 0 |
| 42 | + model: llama-cpp/models/gemma-4-26B-A4B-it-qat-GGUF/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf |
| 43 | + repeat_penalty: 1 |
| 44 | + temperature: 1 |
| 45 | + top_k: 64 |
| 46 | + top_p: 0.95 |
| 47 | + template: |
| 48 | + use_tokenizer_template: true |
| 49 | + files: |
| 50 | + - filename: llama-cpp/models/gemma-4-26B-A4B-it-qat-GGUF/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf |
| 51 | + sha256: dcf179a91153e3a7ece792e48ef872180d9d6ef9b7677f0a0bd3e83cfe624d5e |
| 52 | + uri: https://huggingface.co/unsloth/gemma-4-26B-A4B-it-qat-GGUF/resolve/main/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf |
| 53 | + - filename: llama-cpp/mmproj/gemma-4-26B-A4B-it-qat-GGUF/mmproj-F32.gguf |
| 54 | + sha256: ef269e294502d6ee3722cbf129681b2586c2e6ceb79d0507963c92146e058cd4 |
| 55 | + uri: https://huggingface.co/unsloth/gemma-4-26B-A4B-it-qat-GGUF/resolve/main/mmproj-F32.gguf |
2 | 56 | - name: "gemma-4-12b-it-qat-q4_0" |
3 | 57 | url: "github:mudler/LocalAI/gallery/virtual.yaml@master" |
4 | 58 | urls: |
|
0 commit comments