Skip to content

Commit 8006883

Browse files
NexesenexNexesenex
committed
Add a readme.
1 parent b4df7f2 commit 8006883

1 file changed

Lines changed: 105 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,108 @@
1+
# Croco.Cpp (CCPP), a KoboldCPP mod, empowered by IK_Llama quants and Esobold.
2+
3+
Branch Esocrok :
4+
- Llama.cpp mainline compliant branch -> No IK MulMat, not IK quants second gen or Trellis.
5+
6+
Readme to be updated :
7+
8+
<details>
9+
<summary>Unroll DISCLAIMER:</summary>
10+
11+
Croco.Cpp is a fork/mod of KoboldCPP, already known as KoboldCPP Frankenstein, Frankenfork, or shortened in KCPP-F.
12+
The namechange is due to my boredom with the Frankenstein marker I myself initiated a year ago.
13+
As usual, the Croco.Cpp builds are NOT supported by the KoboldCPP (KCPP) team, Github, or Discord channel.
14+
They are for greedy-test and amusement only.
15+
Any potential support found them is a courtesy, not a due.
16+
My CCPP version number bumps as soon as the version number in the official experimental branch bumps in the following way x.xxx (ex : 1.80.1) : (KCPP)x.xxx.(CCPP)xx.
17+
They are not "upgrades" over the official version. And they might be bugged at time: only the official KCPP releases are to be considered correctly numbered, reliable and "fixed".
18+
The LllamaCPP version + the Esbold version in my CCPP versioning in the title, so everybody knows what version they deal with.
19+
</details>
20+
21+
Presentation :
22+
23+
Croco.Cpp (CCPP) is a fork/mod of the experimental branch of KoboldCPP (KCPP) enhanced with Esobold, mainly aimed at NVidia Cuda users (I'm myself using Ampere GPUs and it doesn't support the other backends as of now except a CPU failsafe for the mainline Llama.cpp quants (for now) since v1.93040.. it might also support Hipblas/ROCm, but it's not tested for a year now..), this mod having a few modifications accordingly to my own needs, among which :
24+
- More context steps in GUI, as well as more Blas Batch Size (supports MMVQ 1-8 for example).
25+
- 22 or so different modes of quantization for the context cache (F16, around 15 KV modes with Flash Attention, BF16, 7 quantum legacy K cache modes without Flash Attention for models like Gemma).
26+
- KV cache supports IQ4_NL and Q6_0 (except for Gemma), thanks to Ikawrakow. IQ4_NL gives 2% perplexity gain over q4_0, and q6_0 0.1-0.2% over Q5_1.
27+
- Configurable KV cache for the draft model in case of speculative decoding.
28+
- Shrunk Blas Batch size on the draft model compared to the main model BBS, due to the draft's logically smaller size and thus higher PP.
29+
- Supports CPU and CUDA inference for q6_0 and the IQ_K quants (first generation) made by Ikawrakow (Q6_0 legacy for irregularly shaped tensors ; IQ_2K, IQ_3K, IQ_4K, IQ_5K, IQ_6K).
30+
- Supports in Cuda MMQ Mode: Mainline, Q6_0, IQ2_K, IQ3_K, IQ4_K, IQ5_K, IQ6_K.
31+
- A few commits taken from Ikawrakow's IK_Llama.CPP for performances or quant quality.
32+
- 64 Stories slots instead of 10 or so in the web-interface (KLite).
33+
- More recent dependencies (that's nerdy, isn't it?).
34+
35+
Recommanded settings for Commande Line Interface / GUI :
36+
```
37+
--flashattention (except for Gemma?)
38+
--blastbatchsize 128 (256 for Gemma)
39+
--usecublas mmq (for NVidia users, MMQ mode is faster)
40+
```
41+
Check the help section (koboldcpp.exe --help or python koboldcpp.py --help) for more infos.
42+
43+
## Croco.Cpp specifics :
44+
45+
<details>
46+
<summary>Unroll the 26 KV cache options (all should be considered experimental except F16, KV Q8_0, and KV Q4_0)</summary>
47+
48+
With Flash Attention :
49+
- F16 -> Fullproof (the usual KV quant since the beginning of LCPP/KCPP)
50+
- BF16 (experimental)
51+
- K F16 with : V Q8_0, Q6_0 (experimental), Q5_0, iq4_nl.
52+
- K Q8_0 with : V Q8_0 (stable, part of the LCPP/KCPP main triplet), Q6_0 (experimental), Q5_0 (maybe unstable), iq4_nl (maybe stable).
53+
- K Q6_0 with : V Q6_0, Q5_0, iq4_nl.
54+
- K Q5_1 with : V Q5_0, iq4_nl.
55+
- K Q5_0 with : V iq4_nl.
56+
- KV Q4_0 (quite stable, if we consider that it's part of the LCPP/KCPP main triplet)
57+
Works in command line, normally also via the GUI, and normally saves on .KCPPS config files.
58+
- KV iq4_nl (with -2% perplexity compared to Q4_0).
59+
60+
Without Flash Attention nor MMQ (for models like Gemma) :
61+
- V F16 with K Q8_0, Q5_1, Q5_0, Q4_1, and Q4_0.
62+
- K Q6_0 and IQ4_NL to be tested, might not work.
63+
</details>
64+
65+
<details>
66+
<summary>Unroll the options to set KV Quants (obsolete)</summary>
67+
68+
KCPP official KV quantized modes (modes 1 and 2 require Flash Attention) :
69+
70+
0 = 1616/F16 (16 BPW),
71+
1 = FA8080/KVq8_0 (8.5 BPW),
72+
2 = FA4040/KVq4_0 (4.5BPW),
73+
74+
CCPP unofficial KV quantized modes (require flash attention) :
75+
76+
"1 - q8_0 - (8.5BPW) - FA",
77+
"2 - q4_0 - (4.5BPW) - FA - possibly faulty on some models",
78+
"3* - K F16 - V q8_0 (12.25BPW) - FA",
79+
"4* - K F16 - V q6_0 (11.25BPW) - FA. Doesn't work on Gemma 2 FA.",
80+
"5 - K q8_0 - V q6_0 (7.5BPW) - FA. Doesn't work on Gemma 2 FA.",
81+
"6* - K q8_0 - V q5_0 (7BPW) - FA",
82+
"7 - K q8_0 - V iq4_nl (6.5BPW) - FA. Doesn't work on Gemma 2 FA.",
83+
"8* - K q6_0 - V q6_0 (6.5BPW) - FA. Doesn't work on Gemma 2 FA.",
84+
"9 - K q6_0 - V q5_0 (6BPW) - FA, best game in FA town. Doesn't work on Gemma 2 FA.",
85+
"10* - K q6_0 - V iq4_nl (5.5BPW) - FA - faulty on some models (Gemma 2 FA. Qwen 2.5 1.5b?)",
86+
"11 - K q5_1 - V q5_0 (5.5BPW) - FA - possibly faulty on some models (Qwen 2.5 1.5b?)",
87+
"12* - K q5_1 - V iq4_nl (5.25BPW) - FA",
88+
"13 - K q5_0 - V iq4_nl (5BPW) - FA - possibly faulty on some models (Qwen 2.5 1.5b?)",
89+
"14 - K iq4_nl - V iq4_nl (4.5BPW) - FA",
90+
"15 - BF16 (16BPW) - no FA, experimental for Cuda, not tested on other backends.",
91+
"16 - K q8_0 - V F16 (12.25BPW) - NO FA, slower",
92+
"17 - K q6_0 - V F16 (11.25BPW) - NO FA, slower, best game in non-FA town.",
93+
"18 - K q5_1 - V F16 (11BPW) - NO FA, slower - possibly faulty on some models (Qwen 2.5 1.5b?)",
94+
"19 - K q5_0 - V F16 (11.75BPW) - NO FA, slower - possibly faulty on some models (Qwen 2.5 1.5b?)",
95+
"20 - K q4_1 - V F16 (10.5BPW) - NO FA, slower - possibly faulty on some models (Qwen 2.5 1.5b?)",
96+
"21 - K q4-0 - V F16 (10.25BPW) - NO FA, slower - possibly faulty on some models (Qwen 2.5 1.5b?)",
97+
"22 - K iq4_nl - V F16 (10.25BPW) - NO FA, slower"]
98+
99+
choices=[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22], default=0)
100+
</details>
101+
102+
103+
=====
104+
105+
1106
# Esobold (Esolithe's fork of KoboldCPP)
2107

3108
![Kobold](https://github.com/user-attachments/assets/f5cb4087-bc72-47fc-91a5-29a4d6495648)

0 commit comments

Comments
 (0)