Honor minimum common prefix for prompt cache reuse#1854
Conversation
firecoperana
left a comment
firecoperana
left a comment
There was a problem hiding this comment.
Please check my comment.
| ret->cache_tokens = ret->server_cached_prompt.tokens.clone(); // recover cache tokens | ||
| ret->n_discarded_prompt = ret->server_cached_prompt.n_discarded_prompt; | ||
| ret->n_kept_prompt = ret->server_cached_prompt.n_kept_prompt; | ||
| if (prompt_loaded) { |
There was a problem hiding this comment.
This whole part is not needed because if no prompt is loaded, there is no need to change cache_tokens, but you are erasing all the cache tokens.
| int32_t back = 4; | ||
| if (prefix.second >= back && prefix.first >= back) { | ||
| print_tokens(slot.prompt_tokens, slot.cache_tokens, prefix.second - back, prefix.first - back, 30); | ||
| const size_t n_common = std::min(prefix.first, prefix.second); |
There was a problem hiding this comment.
No change is needed in this part.
| options.push_back({ "*", "-cram, --cache-ram N", "set the maximum cache size in MiB (default: %d, -1 - no limit, 0 - disable)",params.cache_ram_mib }); | ||
| options.push_back({ "*", "-crs, --cache-ram-similarity N", "max of similarity of prompt tokens to cache tokens that triggers prompt cache (default: %.2f).",params.cache_ram_similarity }); | ||
| options.push_back({ "*", "-cram-n-min --cache-ram-n-min N", "minimum number of the cached tokens that triggers prompt cache (default: %d).", params.cache_ram_n_min }); | ||
| options.push_back({ "*", "-cram-n-min --cache-ram-n-min N", "minimum number of common prompt tokens required to reuse or save prompt cache (default: %d).", params.cache_ram_n_min }); |
There was a problem hiding this comment.
The use case of cache-ram-n-min is different. It is for the whole length of the prompt, not just the common prefix. Create a new parameter instead.
Hm, that is a better idea, I will see to explore it, thanks a lot 😄 ! |
|
It's not that easy after I thought about it. When you retrieve the prompt A and run some inference, it becomes B. The next prompt C is a different prompt, so it saves prompt B. Now A and B are very similar, but prompt B may not fully contain A. You now have two long prompts A and B. You need some unique id to know that prompt B is from prompt A, so you can delete A and keep B. |
|
Yup and there is also a case where it alternates between two caches that are almost identical because they are both in memory (A and B) 🙈 And the question should B always replace A if there is eviction based on lineage 🤔 I will see what can be done, interesting problem to solve anyway 😄 |
|
Closing in favor of #1877 |
2 participants
Summary
This fixes prompt-cache behavior when a single server slot alternates between large prompts with different early tool/schema sections.
In that case the prompts may share only a small prefix, but the server could still try to reuse live slot state or RAM prompt-cache state. With checkpointing enabled this can lead to stale checkpoint reuse, checkpoint invalidation, and repeated full prompt reprocessing.
The change makes
--cache-ram-n-minact as a minimum common-prefix requirement for prompt-cache reuse, not only as a size gate for saving.Changes
cache_ram_n_minwhen selecting RAM prompt-cache entries.cache_ram_n_min.n_commonto prompt-cache logs to make reuse/skip decisions visible.cache_ram_n_minbehavior.Why
This was observed with agent workloads where a main coding prompt and a background/tooling prompt share only about 7k initial tokens while each full prompt is 70k-80k tokens.
Before this change, a low-prefix request could poison the single live slot and cause the next matching request to reprocess the entire prompt. After this change, low-prefix requests run from scratch while the hot prompt can be preserved in RAM prompt cache and restored when the matching prompt shape returns.
Validation
cmake --build build --target llama-server -j 24--cache-ram-n-min 32768are skipped