Summarizer

[3inary]

Describe the bug

Thanks for your awesome work!

Summarizer fails, chat is trunctated but no summary is generated.

Steps to reproduce

Chatbot Demo

1. Context window: 1024
2. Batch/Predict: 512
3. Overflow Strat: Summarize
4. (Set a Save File)
5. Ask bot to "tell a absurd sifi story"
6. Ask bot to "continue" until it forgets what you where talking about

LLMUnity version

latest

Operating System

macOs

[amakropoulos]

If the summary exceeds the overflow target ratio e.g. 50%*1024=512, then the summary gets erased.
Can you check if a summary is generated using the "Debug Prompt"?

[3inary]

I think so, since the response takes considerably longer.

[3inary]

End-User wise the chat history should not be deleted, maybe just feed the last N messages to llama.cpp?

[amakropoulos]

yes you are right, I'll implement that!