Commit b817f00
authored
feat(lemonade): serve models by their canonical Hugging Face name (#80)
* Serve canonical Hugging Face names via Lemonade
`rocm serve --engine lemonade <owner/repo:variant>` previously failed:
the name was passed straight to `lemonade load`, which only resolves
built-in/registered models. Lemonade's model registry cannot preserve an
`owner/repo:variant` name either — a bare pull derives a `repo-variant`
name, custom registration forces a `user.` prefix, and the resulting
entries load inconsistently.
Serve such a checkpoint by bypassing the registry: pull it once to
download the GGUF, then run a packaged llama.cpp `llama-server` on the
file directly with `--alias <canonical>`, which serves it under exactly
that name (the same id vLLM already uses). The server binary is discovered
under `bin/llamacpp/<backend>/` so this works with whatever backend is
installed — `vulkan` (e.g. WSL2) or `rocm-stable` (native ROCm). The GGUF
is located in the HF hub cache with Lemonade-style variant matching
(exact filename, else a quantization-label match). Built-in models are
unaffected; the direct-serve path is generalized and reused for them.
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
* Address review: GPU-only direct serve, explicit variant, direct-serve readiness
- Drop `cpu` from the direct-serve backend allowlist so an HF checkpoint is
never silently served on CPU under gpu_required (AGENTS.md §6).
- Require an explicit `:variant` for HF checkpoints and refuse an ambiguous
variant (one matching multiple cached GGUFs) instead of picking arbitrarily.
- Recognize the direct-serve stock `llama-server` as ready: its `/v1/models`
entry has no `recipe_options`, so accept it by name (that path is GPU-only)
while still requiring a ROCm/expected backend for Lemonade-router entries.
Applied in both the engine healthcheck and `apps/rocm` readiness.
- Kill the child and mark the service failed if the direct server never
becomes ready, instead of leaking it and leaving state `running`.
- Add tests (variant ambiguity, readiness shapes, backend discovery, GGUF
collection) and doc fixes.
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
---------
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>1 parent a73900f commit b817f00
2 files changed
Lines changed: 721 additions & 46 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15152 | 15152 | | |
15153 | 15153 | | |
15154 | 15154 | | |
15155 | | - | |
| 15155 | + | |
15156 | 15156 | | |
15157 | 15157 | | |
15158 | 15158 | | |
15159 | 15159 | | |
| 15160 | + | |
| 15161 | + | |
| 15162 | + | |
| 15163 | + | |
| 15164 | + | |
| 15165 | + | |
| 15166 | + | |
| 15167 | + | |
| 15168 | + | |
| 15169 | + | |
| 15170 | + | |
| 15171 | + | |
15160 | 15172 | | |
15161 | 15173 | | |
15162 | 15174 | | |
| |||
15605 | 15617 | | |
15606 | 15618 | | |
15607 | 15619 | | |
| 15620 | + | |
| 15621 | + | |
| 15622 | + | |
| 15623 | + | |
| 15624 | + | |
| 15625 | + | |
| 15626 | + | |
| 15627 | + | |
| 15628 | + | |
| 15629 | + | |
| 15630 | + | |
| 15631 | + | |
| 15632 | + | |
| 15633 | + | |
| 15634 | + | |
| 15635 | + | |
| 15636 | + | |
| 15637 | + | |
| 15638 | + | |
| 15639 | + | |
| 15640 | + | |
| 15641 | + | |
| 15642 | + | |
| 15643 | + | |
| 15644 | + | |
| 15645 | + | |
| 15646 | + | |
| 15647 | + | |
| 15648 | + | |
| 15649 | + | |
15608 | 15650 | | |
15609 | 15651 | | |
15610 | 15652 | | |
| |||
0 commit comments