Skip to content

fix(server): SEGV when --mtp-head + --mmproj are both passed #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
WillowOneVision wants to merge 1 commit into
base: feature/turboquant-kv-cache
Choose a base branch
from
Open

fix(server): SEGV when --mtp-head + --mmproj are both passed #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions common/speculative.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1288,6 +1288,15 @@ bool common_speculative_is_compat(llama_context * ctx_tgt) {
common_speculative * common_speculative_init(
common_params_speculative & params,
llama_context * ctx_tgt) {
// Defensive: if speculative was disabled upstream (e.g. server disables it when mmproj
// is loaded), bail out before any impl construction. Without this guard, a caller that
// sets params.type=NONE but leaves params.mparams_dft.path (set via --mtp-head/--model-draft)
// would still trigger the DRAFT config below with ctx_dft=nullptr, crashing in the
// common_speculative_state_draft ctor at llama_n_batch(ctx_dft).
if (params.type == COMMON_SPECULATIVE_TYPE_NONE) {
return nullptr;
}

llama_context * ctx_dft = nullptr;
// Gemma4 MTP loads the assistant into the target model (llama_model_load_mtp_from_file); no second context.
if (params.model_dft && params.type != COMMON_SPECULATIVE_TYPE_MTP) {
Expand Down
5 changes: 5 additions & 0 deletions tools/server/server-context.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -735,6 +735,11 @@ struct server_context_impl {

if (params_base.speculative.type != COMMON_SPECULATIVE_TYPE_NONE) {
params_base.speculative.type = COMMON_SPECULATIVE_TYPE_NONE;
// Also clear the draft model path so common_speculative_init does not
// observe an orphan has_draft=true with type=NONE (would build a DRAFT
// config and crash on ctx_dft=nullptr). See common/speculative.cpp init.
params_base.speculative.mparams_dft.path.clear();
params_base.speculative.model_dft = nullptr;
SRV_WRN("%s\n", "speculative decoding is not supported by multimodal, it will be disabled");
}
}
Expand Down