Is anyone using Speculative Decoding? And if so does it work with thinking?

[Manimap]

Hi everyone,

Is anyone using Gemma 4 with speculative decoding? If so, do you happen to know if thinking still works with it?

I'm sending --chat-template-kwargs '{"enable_thinking":true}' with just Gemma 31B, and it works, but if I do the same thing and use Gemma 26B as a draft model, thinking doesn't work anymore.

[Manimap]

Nevermind I don't think the draft model receives any parameter like "chat_template_kwargs" either through the api or as a starting flag in the launcher.
I solved the issue by simply adding <|think|> as a system message, which perfectly worked.

[LostRuins]

It does, the draft model should see the same input as the main model.

[Manimap]

After many tests, you are right, thanks for the clarification.

It was not an issue about what kwargs I was sending, but it was linked to how max context is behaving relative to how many tokens are actually sent. My requests were overbudget so it was probably trimming the beginning of my prompt which ate the <|think|>part.