Stop inference testing on macOS and adjust the "hot" models

[jeremyfowers]

Closes #1674
Closes #1525

This PR unblocks the automated llamacpp upgrade workflow in two ways.

macOS

GitHub-hosted macOS runners are virtualized and fail Metal kernel execution on newer llama.cpp builds. This PR disables any test that would load or inference a model on macOS.

Also bumps llamacpp metal b8460 -> b8766 to match vulkan/cpu.

Hot Models

Removes qwen3.5-122b from the hot models list because:

1. I don't think its actually that hot
2. it's huge and kinda borderline for fitting into memory on Windows, which makes CI flakey

[Geramy]

I think this works, has the CI ran successfully, I'm not able to find backend_versions.json on Mac. Once O do I'll test the latest llama.

[Geramy]

@claude on macOS where is backend_versions.JSON install path?

[github-actions]

Claude finished @Geramy's task in 55s —— View job

---

backend_versions.json on macOS

• Read source files to find the macOS installation path

The file is loaded by BackendManager via get_resource_path("resources/backend_versions.json") (backend_manager.cpp:19). The path resolution logic in path_utils.cpp:137-173 checks these locations in order:

1. Next to the lemond executable (dev builds / local build): <path-to-lemond>/resources/backend_versions.json
2. User local install: ~/.local/share/lemonade-server/resources/backend_versions.json
3. System install (macOS): /Library/Application Support/Lemonade/resources/backend_versions.json
4. /usr/local/share/lemonade-server/resources/backend_versions.json
5. /opt/share/lemonade-server/resources/backend_versions.json

So for a standard macOS installation, it will be at:

/Library/Application Support/Lemonade/resources/backend_versions.json

If you built from source, it will be in your build output directory next to the lemond binary (e.g. build/resources/backend_versions.json).

Note: The file in this PR (src/cpp/resources/backend_versions.json) is the source file that gets copied into the install location at build/package time. Editing it there and rebuilding is how you'd test a version bump locally.
|

[Geramy]

I would rename this too Unblock MacOS CI Pipeline by using CPU and not metal :)

[Geramy]

Looks good.

[Geramy]

@claude where does llama-server install to on macOS? Where can I find it locally when it gets unzipped.

[github-actions]

Claude finished @Geramy's task in 56s —— View job

---

llama-server install path on macOS

• Read source files to find where llama-server is installed/extracted on macOS

---

llama-server is downloaded from GitHub Releases and extracted into the Lemonade cache directory. On macOS, the path is:

~/.cache/lemonade/bin/llamacpp/metal/llama-server

(If running as root, it goes to /Library/Application Support/lemonade/.cache/bin/llamacpp/metal/llama-server instead.)

What gets downloaded

The artifact pulled for macOS is a .tar.gz from the llama.cpp GitHub releases:

https://github.com/ggml-org/llama.cpp/releases/download/b8766/llama-b8766-bin-macos-arm64.tar.gz

It's extracted with tar --strip-components=1 into the directory above, so the binary lands at:

~/.cache/lemonade/bin/llamacpp/metal/llama-server

A version.txt file is also written there to track the installed version.

Installation flow

1. On first model load, LlamaCppServer::install() is called
2. It invokes BackendUtils::get_install_directory("llamacpp", "metal") → ~/.cache/lemonade/bin/llamacpp/metal
3. Downloads the .tar.gz to a temp dir
4. Extracts it into the install directory
5. Sets chmod 0755 on the binary
6. Cleans up the temp archive

Key source files

• src/cpp/server/backends/backend_utils.cpp — download + extract logic
• src/cpp/server/utils/path_utils.cpp — get_cache_dir() path resolution
• src/cpp/resources/backend_versions.json — current pinned version (b8766 on this branch)

[Geramy]

@claude if I want to run llama-server manually and load a model where do I put the folder path to load the model?

[github-actions]

Claude finished @Geramy's task in 59s —— View job

---

Running llama-server manually with a model

• Found how llama-server is invoked and where model files live

---

The key flag is -m (or --model), which takes the full path to a .gguf file.

Basic command

~/.cache/lemonade/bin/llamacpp/metal/llama-server \
  -m /path/to/your/model.gguf \
  --port 8080 \
  --ctx-size 4096 \
  --jinja

Where Lemonade stores downloaded models

Models downloaded via lemonade pull land in the HuggingFace hub cache — the same place the huggingface_hub Python library uses:

~/.cache/huggingface/hub/models--<org>--<model-name>/snapshots/<hash>/<file>.gguf

For example, after pulling llama3.2-3b-instruct, the GGUF would be somewhere like:

~/.cache/huggingface/hub/models--amd--Llama-3.2-3B-Instruct-GGUF/snapshots/<hash>/Llama-3.2-3B-Instruct.Q4_K_M.gguf

You can find the exact path by running:

find ~/.cache/huggingface/hub -name "*.gguf"

What Lemonade actually passes

From llamacpp_server.cpp:277, the full command Lemonade builds is roughly:

llama-server \
  -m <resolved_gguf_path> \
  --ctx-size <ctx_size> \
  --port <port> \
  --jinja \
  --keep 16 \
  --reasoning-format auto \
  --no-webui

So for a quick manual test, the minimum you need is:

~/.cache/lemonade/bin/llamacpp/metal/llama-server -m ~/.cache/huggingface/hub/.../model.gguf --port 8080

[Geramy]

@claude how does ggml-org/llama.cpp#21936 effect upgrading to the latest llama.cpp revision on Mac?