Skip to content

rocm serve --engine lemonade: interactive variant selection for owner/repo #82

Description

@rominf

Motivation

rocm serve --engine lemonade <owner/repo:variant> serves a Hugging Face GGUF under its canonical name. Today a variant is required — a bare owner/repo, or a partial/ambiguous variant like :Q4 that matches several quantizations, is rejected with an actionable error rather than guessed (to avoid silently serving the wrong quantization). See PR #80.

A nicer UX would be to enumerate the available variants and let the user pick when the variant is missing or ambiguous, instead of erroring.

Proposed approach

  • Enumerate a repo's GGUF variants without downloading (Lemonade exposes a variants endpoint used by its own pull flow).
  • In the foreground/interactive serve path (which has a TTY, during the serve-plan phase in apps/rocm), present the variants and let the user choose, then proceed with the chosen owner/repo:variant.
  • Keep the current non-interactive behavior as the fallback: with no TTY (managed/background serves, scripts, CI), list the available variants in the error message so the user can specify one.

Notes / constraints

  • Interactive selection only applies to the foreground path; managed/background serves must keep the clear error.
  • Belongs in the apps/rocm foreground layer (TTY + serve plan), not the engine's detached serve_http.
  • Invariant to preserve: never silently pick a variant.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions