Skip to content

Commit cad45fc

Browse files
davanstrienclaude
andauthored
Use llama serve in Jobs serving guide (llama.cpp unified CLI) (#2593)
* Use `llama serve` in Jobs serving guide (llama.cpp unified CLI) Switches the llama.cpp section of the Jobs serving guide from the standalone `llama-server` binary to the unified `llama serve` command, per review feedback on #2554. `llama serve` and `llama-server` are the same server (the unified `llama` CLI, ggml-org/llama.cpp#23875). The unified binary ships in the `full-cuda` image; the `server-cuda` image only contains `/app/llama-server`, so the image changes too. Tested on live Jobs (full-cuda + a10g-small): both the `-hf` and the volume-mounted `--model` forms reached ready and returned completions through the exposed-port proxy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Use lighter server-cuda image now that llama ships there (ggml-org/llama.cpp#25035) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 8a8282a commit cad45fc

1 file changed

Lines changed: 5 additions & 5 deletions

File tree

docs/hub/jobs-serving.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -64,17 +64,17 @@ Because the token travels in the `Authorization` header, these URLs work from sc
6464

6565
## Serve GGUF models with llama.cpp
6666

67-
The same pattern works with any server that speaks HTTP. llama.cpp's `llama-server` pulls GGUF files straight from the Hub with the `-hf` flag and serves the same OpenAI-compatible API. For example, to serve [Gemma 4 E4B](https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF) with Gemma's recommended sampling settings:
67+
The same pattern works with any server that speaks HTTP. llama.cpp's `llama serve` pulls GGUF files straight from the Hub with the `-hf` flag and serves the same OpenAI-compatible API. For example, to serve [Gemma 4 E4B](https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF) with Gemma's recommended sampling settings:
6868

6969
```bash
7070
>>> hf jobs run --detach --expose 8080 --flavor a10g-small -s HF_TOKEN \
7171
... ghcr.io/ggml-org/llama.cpp:server-cuda -- \
72-
... /app/llama-server -hf ggml-org/gemma-4-E4B-it-GGUF \
72+
... /app/llama serve -hf ggml-org/gemma-4-E4B-it-GGUF \
7373
... --host 0.0.0.0 --port 8080 -ngl 99 \
7474
... --temp 1.0 --top-p 0.95 --top-k 64
7575
```
7676

77-
The `--` separates the job's command from `hf jobs run`'s own options — needed here because `llama-server`'s flags would otherwise be parsed by the CLI itself.
77+
The `--` separates the job's command from `hf jobs run`'s own options — needed here because `llama serve`'s flags would otherwise be parsed by the CLI itself.
7878

7979
> [!TIP]
8080
> You can skip the model download entirely by mounting the model repo as a read-only volume and pointing the server at the file directly:
@@ -83,14 +83,14 @@ The `--` separates the job's command from `hf jobs run`'s own options — needed
8383
> >>> hf jobs run --detach --expose 8080 --flavor a10g-small -s HF_TOKEN \
8484
> ... -v hf://ggml-org/gemma-4-E4B-it-GGUF:/model:ro \
8585
> ... ghcr.io/ggml-org/llama.cpp:server-cuda -- \
86-
> ... /app/llama-server --model /model/gemma-4-E4B-it-Q4_K_M.gguf \
86+
> ... /app/llama serve --model /model/gemma-4-E4B-it-Q4_K_M.gguf \
8787
> ... --host 0.0.0.0 --port 8080 -ngl 99
8888
> ```
8989
>
9090
> The server starts much faster since there is nothing to download — the model streams from the mounted repo as it loads.
9191
9292
> [!WARNING]
93-
> Your server must listen on `0.0.0.0`. `llama-server` binds to `127.0.0.1` by default, which the jobs proxy can't reach — pass `--host 0.0.0.0` explicitly.
93+
> Your server must listen on `0.0.0.0`. `llama serve` binds to `127.0.0.1` by default, which the jobs proxy can't reach — pass `--host 0.0.0.0` explicitly.
9494
9595
The same applies to any other OpenAI-compatible server (SGLang, ...): start the server on an exposed port, listen on `0.0.0.0`, and connect with your HF token as the API key.
9696

0 commit comments

Comments
 (0)