Skip to content

Commit fc360e5

Browse files
howard0suCopilot
andcommitted
gemma4: fix prefix cache miss — add snapshot_save in generate/restore paths
The Gemma4 backend was missing the snapshot_save(req.snap_slot) call after prefill. The prefix cache registered entries (confirm_inline_snap), but the actual KV data was never stored. On subsequent requests, restore_and_generate found snapshots_[slot].ctx == nullptr → 'bad slot' error → 0 output tokens. Fix: add inline snapshot save after prefill in both generate() and restore_and_generate(), matching the pattern used by Qwen3/Qwen35 backends. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 1090d51 commit fc360e5

1 file changed

Lines changed: 20 additions & 0 deletions

File tree

dflash/src/gemma4/gemma4_backend.cpp

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -511,6 +511,16 @@ GenerateResult Gemma4Backend::generate(const GenerateRequest & req,
511511
return result;
512512
}
513513

514+
// Inline snapshot at snap_pos for prefix cache
515+
if (req.snap_slot >= 0 && req.snap_pos > 0 && req.snap_pos <= committed) {
516+
cache_.cur_pos = req.snap_pos;
517+
if (snapshot_save(req.snap_slot)) {
518+
std::fprintf(stderr, "[gemma4] inline-snap slot=%d cur_pos=%d\n",
519+
req.snap_slot, req.snap_pos);
520+
}
521+
cache_.cur_pos = committed;
522+
}
523+
514524
if (req.n_gen > 0) {
515525
// Try speculative decode if draft is available and temp==0
516526
const bool can_spec = dflash_target_
@@ -651,6 +661,16 @@ GenerateResult Gemma4Backend::restore_and_generate(int slot,
651661
}
652662
// else: prompt_len == snap_pos → no delta, committed stays at snap_pos
653663

664+
// Inline snapshot at snap_pos for prefix cache (new snap from this request)
665+
if (req.snap_slot >= 0 && req.snap_pos > 0 && req.snap_pos <= committed) {
666+
cache_.cur_pos = req.snap_pos;
667+
if (snapshot_save(req.snap_slot)) {
668+
std::fprintf(stderr, "[gemma4] inline-snap slot=%d cur_pos=%d\n",
669+
req.snap_slot, req.snap_pos);
670+
}
671+
cache_.cur_pos = committed;
672+
}
673+
654674
// Generate
655675
if (req.n_gen > 0) {
656676
const bool can_spec = dflash_target_

0 commit comments

Comments
 (0)