Skip to content

Commit b817f00

Browse files
authored
feat(lemonade): serve models by their canonical Hugging Face name (#80)
* Serve canonical Hugging Face names via Lemonade `rocm serve --engine lemonade <owner/repo:variant>` previously failed: the name was passed straight to `lemonade load`, which only resolves built-in/registered models. Lemonade's model registry cannot preserve an `owner/repo:variant` name either — a bare pull derives a `repo-variant` name, custom registration forces a `user.` prefix, and the resulting entries load inconsistently. Serve such a checkpoint by bypassing the registry: pull it once to download the GGUF, then run a packaged llama.cpp `llama-server` on the file directly with `--alias <canonical>`, which serves it under exactly that name (the same id vLLM already uses). The server binary is discovered under `bin/llamacpp/<backend>/` so this works with whatever backend is installed — `vulkan` (e.g. WSL2) or `rocm-stable` (native ROCm). The GGUF is located in the HF hub cache with Lemonade-style variant matching (exact filename, else a quantization-label match). Built-in models are unaffected; the direct-serve path is generalized and reused for them. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> * Address review: GPU-only direct serve, explicit variant, direct-serve readiness - Drop `cpu` from the direct-serve backend allowlist so an HF checkpoint is never silently served on CPU under gpu_required (AGENTS.md §6). - Require an explicit `:variant` for HF checkpoints and refuse an ambiguous variant (one matching multiple cached GGUFs) instead of picking arbitrarily. - Recognize the direct-serve stock `llama-server` as ready: its `/v1/models` entry has no `recipe_options`, so accept it by name (that path is GPU-only) while still requiring a ROCm/expected backend for Lemonade-router entries. Applied in both the engine healthcheck and `apps/rocm` readiness. - Kill the child and mark the service failed if the direct server never becomes ready, instead of leaking it and leaving state `running`. - Add tests (variant ambiguity, readiness shapes, backend discovery, GGUF collection) and doc fixes. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com> --------- Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
1 parent a73900f commit b817f00

2 files changed

Lines changed: 721 additions & 46 deletions

File tree

apps/rocm/src/main.rs

Lines changed: 43 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15152,11 +15152,23 @@ fn model_list_ready_for_model(
1515215152
.into_iter()
1515315153
.filter_map(|field| model.get(field).and_then(serde_json::Value::as_str))
1515415154
.any(|loaded| service_model_names_match(loaded, canonical_model_id));
15155-
name_matches && (!require_rocm_backend || service_model_reports_rocm_backend(model))
15155+
name_matches && (!require_rocm_backend || service_model_gpu_ready(model))
1515615156
})
1515715157
})
1515815158
}
1515915159

15160+
/// Whether a lemonade `/v1/models` entry is served on GPU. A stock `llama-server`
15161+
/// (direct-serve) entry has no `recipe_options` and is accepted (that path only runs
15162+
/// GPU backends); a Lemonade-router entry carries `recipe_options`, so it must report a
15163+
/// ROCm backend — which keeps a registered-but-unloaded model (empty `recipe_options`)
15164+
/// from reading as ready.
15165+
fn service_model_gpu_ready(model: &serde_json::Value) -> bool {
15166+
match model.get("recipe_options") {
15167+
None => true,
15168+
Some(_) => service_model_reports_rocm_backend(model),
15169+
}
15170+
}
15171+
1516015172
fn service_model_reports_rocm_backend(model: &serde_json::Value) -> bool {
1516115173
model
1516215174
.get("recipe_options")
@@ -15605,6 +15617,36 @@ mod tests {
1560515617
));
1560615618
}
1560715619

15620+
#[test]
15621+
fn lemonade_direct_serve_model_reads_ready_without_recipe_options() {
15622+
// The HF direct-serve path runs a stock llama-server whose `/v1/models` entry
15623+
// has no `recipe_options`. It must read as ready by name (that path is GPU-only),
15624+
// while a registered-but-unloaded lemonade entry (empty `recipe_options`) must not.
15625+
let direct = json!({
15626+
"data": [{ "id": "LiquidAI/LFM2.5-230M-GGUF:Q4_0", "object": "model" }]
15627+
})
15628+
.to_string();
15629+
assert!(service_http_readiness_response_ready(
15630+
"lemonade",
15631+
"/v1/models",
15632+
200,
15633+
&direct,
15634+
"LiquidAI/LFM2.5-230M-GGUF:Q4_0"
15635+
));
15636+
15637+
let registered = json!({
15638+
"data": [{ "id": "LiquidAI/LFM2.5-230M-GGUF:Q4_0", "recipe_options": {} }]
15639+
})
15640+
.to_string();
15641+
assert!(!service_http_readiness_response_ready(
15642+
"lemonade",
15643+
"/v1/models",
15644+
200,
15645+
&registered,
15646+
"LiquidAI/LFM2.5-230M-GGUF:Q4_0"
15647+
));
15648+
}
15649+
1560815650
#[test]
1560915651
fn lemonade_stop_unloads_selected_model_over_http() -> Result<()> {
1561015652
use std::io::{Read, Write};

0 commit comments

Comments
 (0)