Skip to content

Commit a11c6c7

Browse files
davanstrienclaude
davanstrien
and
claude
authored
Add Serve Models guide for Jobs (vLLM + llama.cpp via exposed ports) (#2554)
* Add Serve Models guide for Jobs (vLLM + llama.cpp via exposed ports) All commands and client snippets verified against live jobs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Switch llama.cpp example to Gemma 4 E4B for a faster demo Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add volume-mount tip for faster server startup Mounting the model repo read-only skips the download (verified: 31s vs 4m15s to listening for the same model). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 484434e commit a11c6c7

2 files changed

Lines changed: 108 additions & 0 deletions

File tree

‎docs/hub/_toctree.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -454,6 +454,8 @@
454454
title: Configuration
455455
- local: jobs-popular-images
456456
title: Popular Images
457+
- local: jobs-serving
458+
title: Serve Models
457459
- local: jobs-examples
458460
title: Examples & Tutorials
459461
- local: jobs-schedule

‎docs/hub/jobs-serving.md‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# Serve Models on Jobs
2+
3+
With [exposed ports](./jobs-configuration#expose-ports), a Job can act as a temporary inference server: start vLLM on a GPU flavor, point any OpenAI-compatible client at the job's URL, and cancel the job when you are done. You pay per minute while the job runs, and the endpoint disappears with the job.
4+
5+
This is a good fit when the endpoint is a means rather than the product: an evaluation run, a data labelling session, iterating on prompts against a hot model, or a demo that only needs to live for an afternoon.
6+
7+
> [!TIP]
8+
> If you want a more permanent endpoint that won't disappear, you want [Inference Endpoints](https://huggingface.co/docs/inference-endpoints), which provides managed infrastructure with autoscaling, monitoring, and stable URLs.
9+
10+
## Start a vLLM server
11+
12+
The `vllm/vllm-openai` image has everything pre-installed. One command starts an OpenAI-compatible server:
13+
14+
```bash
15+
>>> hf jobs run --detach --expose 8000 --flavor a10g-small -s HF_TOKEN \
16+
... vllm/vllm-openai \
17+
... vllm serve LiquidAI/LFM2.5-8B-A1B --max-model-len 8192
18+
✓ Job started
19+
id: 6a2b137a59bbdade52d4a58c
20+
url: https://huggingface.co/jobs/davanstrien/6a2b137a59bbdade52d4a58c
21+
Hint: Exposed ports are reachable at (requires an HF token with read access to the job):
22+
https://6a2b137a59bbdade52d4a58c--8000.hf.jobs
23+
```
24+
25+
`-s HF_TOKEN` forwards your Hugging Face token to the job as a secret, so the model download is authenticated — required for gated or private models, and gives you higher rate limits and faster downloads for everything else.
26+
27+
> [!NOTE]
28+
> Jobs runs the command you provide directly — it does not pass arguments to the image's entrypoint like `docker run` does. Always spell out the full command (`vllm serve ...`, not just `--model ...`).
29+
30+
The server takes a few minutes to become ready (image pull, model download, model load). Follow progress with `hf jobs logs -f <job_id>`; the server is ready when the logs show `Application startup complete`.
31+
32+
## Connect a client
33+
34+
Exposed ports require an HF token with `read` access to the job's namespace, passed as a Bearer token. For an OpenAI-compatible server this slots directly into the client's API key — the base URL is the exposed port URL plus `/v1`:
35+
36+
```python
37+
import os
38+
from openai import OpenAI
39+
40+
client = OpenAI(
41+
base_url="https://6a2b137a59bbdade52d4a58c--8000.hf.jobs/v1",
42+
api_key=os.environ["HF_TOKEN"], # any token with read access to the job's namespace
43+
)
44+
45+
response = client.chat.completions.create(
46+
model="LiquidAI/LFM2.5-8B-A1B",
47+
messages=[{"role": "user", "content": "Write a haiku about ephemeral compute."}],
48+
)
49+
print(response.choices[0].message.content)
50+
```
51+
52+
LFM2.5 is a reasoning model, so the response includes its reasoning inside `<think>` tags before the final answer.
53+
54+
Or with `curl`:
55+
56+
```bash
57+
>>> curl https://6a2b137a59bbdade52d4a58c--8000.hf.jobs/v1/chat/completions \
58+
... -H "Authorization: Bearer $HF_TOKEN" \
59+
... -H "Content-Type: application/json" \
60+
... -d '{"model": "LiquidAI/LFM2.5-8B-A1B", "messages": [{"role": "user", "content": "Hello!"}]}'
61+
```
62+
63+
Because the token travels in the `Authorization` header, these URLs work from scripts, notebooks, and agents — anywhere you'd use an OpenAI-compatible API. They can't be opened directly in a browser.
64+
65+
## Serve GGUF models with llama.cpp
66+
67+
The same pattern works with any server that speaks HTTP. llama.cpp's `llama-server` pulls GGUF files straight from the Hub with the `-hf` flag and serves the same OpenAI-compatible API. For example, to serve [Gemma 4 E4B](https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF) with Gemma's recommended sampling settings:
68+
69+
```bash
70+
>>> hf jobs run --detach --expose 8080 --flavor a10g-small -s HF_TOKEN \
71+
... ghcr.io/ggml-org/llama.cpp:server-cuda -- \
72+
... /app/llama-server -hf ggml-org/gemma-4-E4B-it-GGUF \
73+
... --host 0.0.0.0 --port 8080 -ngl 99 \
74+
... --temp 1.0 --top-p 0.95 --top-k 64
75+
```
76+
77+
The `--` separates the job's command from `hf jobs run`'s own options — needed here because `llama-server`'s flags would otherwise be parsed by the CLI itself.
78+
79+
> [!TIP]
80+
> You can skip the model download entirely by mounting the model repo as a read-only volume and pointing the server at the file directly:
81+
>
82+
> ```bash
83+
> >>> hf jobs run --detach --expose 8080 --flavor a10g-small -s HF_TOKEN \
84+
> ... -v hf://ggml-org/gemma-4-E4B-it-GGUF:/model:ro \
85+
> ... ghcr.io/ggml-org/llama.cpp:server-cuda -- \
86+
> ... /app/llama-server --model /model/gemma-4-E4B-it-Q4_K_M.gguf \
87+
> ... --host 0.0.0.0 --port 8080 -ngl 99
88+
> ```
89+
>
90+
> The server starts much faster since there is nothing to download — the model streams from the mounted repo as it loads.
91+
92+
> [!WARNING]
93+
> Your server must listen on `0.0.0.0`. `llama-server` binds to `127.0.0.1` by default, which the jobs proxy can't reach — pass `--host 0.0.0.0` explicitly.
94+
95+
The same applies to any other OpenAI-compatible server (SGLang, ...): start the server on an exposed port, listen on `0.0.0.0`, and connect with your HF token as the API key.
96+
97+
## Stop the server
98+
99+
The job — and its billing — stops when you cancel it or when its timeout is reached (30 minutes by default; set `--timeout` explicitly for longer sessions):
100+
101+
```bash
102+
>>> hf jobs cancel <job_id>
103+
```
104+
105+
> [!NOTE]
106+
> Exposed ports require `huggingface_hub` >= 1.19.0 and are billed on top of the job's hardware price — see [Jobs pricing](./jobs-pricing).

0 commit comments

Comments
 (0)