Is there a way to use llamacpp flags that aren't in the kobold launcher? #2160
-
|
Mainly --spec-type ngram-mod. Or do I just add those to the kcpps? |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 13 replies
-
|
No, not by adding arbitrary llama.cpp flags to the KoboldCpp has its own CLI/config surface. In the current launcher/parser I can see the exposed speculative/draft options such as So if the llama.cpp option you want is not represented by a KoboldCpp flag, the practical options are to use the closest exposed KoboldCpp draft-model settings, or request/add support for that llama.cpp option in KoboldCpp itself. A raw extra flag in |
Beta Was this translation helpful? Give feedback.
-
|
Stumbled over this thread reseearching n-gram availability in kobold. Is this something that is being considered/will be implemented in the future into kcpp? |
Beta Was this translation helpful? Give feedback.
-
|
I tried many values, but too low generates many false-positives and too high generates no matches at all. I just couldn't get any useful value. |
Beta Was this translation helpful? Give feedback.
-
|
(the parameter is the minimun len of the substring that the ngram identifies as a probable repetition) |
Beta Was this translation helpful? Give feedback.
-
|
So it's an all or nothing deal? I thought it was smart enough to reuse the relevant part of the available cache, not be an exact match or nothing. |
Beta Was this translation helpful? Give feedback.
-
|
It's something like: when it finds a previous substring that is valid as a probable repetition, it checks in parallel during generation if the whole substring is expected as a result. It generates until failure, it can 0, 1, 2 or more characters. If it can match initial 2 or more characters, there is a performance gain. But f it matches 0 characters, it takes longer because it needs to do the normal generation again. To be useful it needs to repeat long substrings often, but I was able to find no use case worth the extra overhead. And as longer the context gets, there is more CPU effort to find the probable substrings. |
Beta Was this translation helpful? Give feedback.
-
|
Ah gotcha, then maybe this is the reason why there wasn't more traction/interest to get this feature implemented into KCPP already. |
Beta Was this translation helpful? Give feedback.
No, not by adding arbitrary llama.cpp flags to the
.kcpps.KoboldCpp has its own CLI/config surface. In the current launcher/parser I can see the exposed speculative/draft options such as
--draftmodel,--draftamount,--draftgpulayers, and--draftgpusplit, and the.kcppssave/load path stores those known fields. I do not see--spec-typeorngram-modas accepted KoboldCpp options.So if the llama.cpp option you want is not represented by a KoboldCpp flag, the practical options are to use the closest exposed KoboldCpp draft-model settings, or request/add support for that llama.cpp option in KoboldCpp itself. A raw extra flag in
.kcppswill likely be ignored or rejected rather than passed thr…