From 95025b4edcc0e8e31e3e52dffecf753954f9d598 Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Wed, 24 Jun 2026 20:09:07 +0100 Subject: [PATCH 1/2] Use `llama serve` in Jobs serving guide (llama.cpp unified CLI) Switches the llama.cpp section of the Jobs serving guide from the standalone `llama-server` binary to the unified `llama serve` command, per review feedback on #2554. `llama serve` and `llama-server` are the same server (the unified `llama` CLI, ggml-org/llama.cpp#23875). The unified binary ships in the `full-cuda` image; the `server-cuda` image only contains `/app/llama-server`, so the image changes too. Tested on live Jobs (full-cuda + a10g-small): both the `-hf` and the volume-mounted `--model` forms reached ready and returned completions through the exposed-port proxy. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/hub/jobs-serving.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/hub/jobs-serving.md b/docs/hub/jobs-serving.md index a64c156d8..70a84c8ac 100644 --- a/docs/hub/jobs-serving.md +++ b/docs/hub/jobs-serving.md @@ -64,17 +64,17 @@ Because the token travels in the `Authorization` header, these URLs work from sc ## Serve GGUF models with llama.cpp -The same pattern works with any server that speaks HTTP. llama.cpp's `llama-server` pulls GGUF files straight from the Hub with the `-hf` flag and serves the same OpenAI-compatible API. For example, to serve [Gemma 4 E4B](https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF) with Gemma's recommended sampling settings: +The same pattern works with any server that speaks HTTP. llama.cpp's `llama serve` pulls GGUF files straight from the Hub with the `-hf` flag and serves the same OpenAI-compatible API. For example, to serve [Gemma 4 E4B](https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF) with Gemma's recommended sampling settings: ```bash >>> hf jobs run --detach --expose 8080 --flavor a10g-small -s HF_TOKEN \ -... ghcr.io/ggml-org/llama.cpp:server-cuda -- \ -... /app/llama-server -hf ggml-org/gemma-4-E4B-it-GGUF \ +... ghcr.io/ggml-org/llama.cpp:full-cuda -- \ +... /app/llama serve -hf ggml-org/gemma-4-E4B-it-GGUF \ ... --host 0.0.0.0 --port 8080 -ngl 99 \ ... --temp 1.0 --top-p 0.95 --top-k 64 ``` -The `--` separates the job's command from `hf jobs run`'s own options — needed here because `llama-server`'s flags would otherwise be parsed by the CLI itself. +The `--` separates the job's command from `hf jobs run`'s own options — needed here because `llama serve`'s flags would otherwise be parsed by the CLI itself. > [!TIP] > You can skip the model download entirely by mounting the model repo as a read-only volume and pointing the server at the file directly: @@ -82,15 +82,15 @@ The `--` separates the job's command from `hf jobs run`'s own options — needed > ```bash > >>> hf jobs run --detach --expose 8080 --flavor a10g-small -s HF_TOKEN \ > ... -v hf://ggml-org/gemma-4-E4B-it-GGUF:/model:ro \ -> ... ghcr.io/ggml-org/llama.cpp:server-cuda -- \ -> ... /app/llama-server --model /model/gemma-4-E4B-it-Q4_K_M.gguf \ +> ... ghcr.io/ggml-org/llama.cpp:full-cuda -- \ +> ... /app/llama serve --model /model/gemma-4-E4B-it-Q4_K_M.gguf \ > ... --host 0.0.0.0 --port 8080 -ngl 99 > ``` > > The server starts much faster since there is nothing to download — the model streams from the mounted repo as it loads. > [!WARNING] -> Your server must listen on `0.0.0.0`. `llama-server` binds to `127.0.0.1` by default, which the jobs proxy can't reach — pass `--host 0.0.0.0` explicitly. +> Your server must listen on `0.0.0.0`. `llama serve` binds to `127.0.0.1` by default, which the jobs proxy can't reach — pass `--host 0.0.0.0` explicitly. The same applies to any other OpenAI-compatible server (SGLang, ...): start the server on an exposed port, listen on `0.0.0.0`, and connect with your HF token as the API key. From 3f9c7ace83c5dee41da5a444ac9e8268bd862eaf Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Mon, 29 Jun 2026 13:43:21 +0100 Subject: [PATCH 2/2] Use lighter server-cuda image now that llama ships there (ggml-org/llama.cpp#25035) Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/hub/jobs-serving.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hub/jobs-serving.md b/docs/hub/jobs-serving.md index 70a84c8ac..6202d4e29 100644 --- a/docs/hub/jobs-serving.md +++ b/docs/hub/jobs-serving.md @@ -68,7 +68,7 @@ The same pattern works with any server that speaks HTTP. llama.cpp's `llama serv ```bash >>> hf jobs run --detach --expose 8080 --flavor a10g-small -s HF_TOKEN \ -... ghcr.io/ggml-org/llama.cpp:full-cuda -- \ +... ghcr.io/ggml-org/llama.cpp:server-cuda -- \ ... /app/llama serve -hf ggml-org/gemma-4-E4B-it-GGUF \ ... --host 0.0.0.0 --port 8080 -ngl 99 \ ... --temp 1.0 --top-p 0.95 --top-k 64 @@ -82,7 +82,7 @@ The `--` separates the job's command from `hf jobs run`'s own options — needed > ```bash > >>> hf jobs run --detach --expose 8080 --flavor a10g-small -s HF_TOKEN \ > ... -v hf://ggml-org/gemma-4-E4B-it-GGUF:/model:ro \ -> ... ghcr.io/ggml-org/llama.cpp:full-cuda -- \ +> ... ghcr.io/ggml-org/llama.cpp:server-cuda -- \ > ... /app/llama serve --model /model/gemma-4-E4B-it-Q4_K_M.gguf \ > ... --host 0.0.0.0 --port 8080 -ngl 99 > ```