From 9592593a2ec284c11dc18e42748e72633b697e37 Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Tue, 2 Jun 2026 09:15:52 +0100 Subject: [PATCH 1/5] Document reusing image-baked libraries with `hf jobs uv run` Add a "Reusing image-baked libraries with uv run" section to the Popular Images page. By default `hf jobs uv run` resolves a script's dependencies from PyPI and does not use the libraries already in the image; for images that ship prebuilt, CUDA-matched builds (vLLM, PyTorch, FlashInfer) you usually want to reuse them via `--python` + `PYTHONPATH`. The section covers: - the two-flag recipe, with vLLM as the worked example - a cpu-basic probe to discover the interpreter / site-packages paths per image - the `RuntimeError: Could not find nvcc` symptom (FlashInfer JIT) as one example - that this applies only to `uv run`, not generic `hf jobs run`, plus the upstream uv issue (astral-sh/uv#7999) Also adds a short cross-link tip in the vLLM section. --- docs/hub/jobs-popular-images.md | 53 +++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/docs/hub/jobs-popular-images.md b/docs/hub/jobs-popular-images.md index 98f79671a9..57ea92c268 100644 --- a/docs/hub/jobs-popular-images.md +++ b/docs/hub/jobs-popular-images.md @@ -4,6 +4,54 @@ Here is the list of ready-to-use Docker images from popular frameworks that you These Docker images already have uv installed but if you want to use an image + uv for an image without uv installed you’ll need to make sure uv is installed first. This will work well in many cases but for LLM inference libraries which can have quite specific requirements, it can be useful to use a specific image that has the library installed. +## Reusing image-baked libraries with `uv run` + +When you run a script against one of these images with `hf jobs uv run`, UV builds its own +isolated environment and installs your `# /// script` dependencies from PyPI — it does +**not** reuse the libraries already baked into the image. For images that ship prebuilt, +CUDA-matched builds (vLLM, PyTorch, FlashInfer, and so on) you usually want the opposite: +reuse the image's copies for faster cold starts and to avoid CUDA/ABI mismatches with the +image's driver. Two flags do that — point UV at the image's interpreter, and add its +site-packages to the import path: + +```bash +hf jobs uv run \ + --image vllm/vllm-openai \ + --flavor l4x4 \ + --python /usr/bin/python3 \ + -e PYTHONPATH=/usr/local/lib/python3.12/dist-packages \ + -s HF_TOKEN \ + generate-responses.py +``` + +- `--python` runs the script with the **image's** interpreter instead of one UV provisions + itself, keeping ABI compatibility with the image's compiled extensions. +- `-e PYTHONPATH=...` adds the image's site-packages to the import path **for that UV run**, + so `import vllm` resolves to the image's prebuilt build rather than a fresh PyPI install. + +The exact paths differ per image. Find them with a one-off probe on `cpu-basic` — it prints +the interpreter path to pass as `--python` and the package directory to pass as `PYTHONPATH`: + +```bash +hf jobs run --flavor cpu-basic vllm/vllm-openai bash -c ' + which python3 + python3 -c "import importlib.util, os; print(os.path.dirname(os.path.dirname(importlib.util.find_spec(\"vllm\").origin)))" +' +``` + +(Swap `vllm` for whichever library you're reusing.) + +> [!TIP] +> Symptoms of *not* reusing the image's build vary by library. With vLLM, for example, a +> PyPI-installed copy can fail at startup with `RuntimeError: Could not find nvcc` — its +> FlashInfer sampler tries to JIT-compile a CUDA kernel the runtime has no compiler for, +> while the image already ships the prebuilt kernels. + +This only applies to `hf jobs uv run`. With generic `hf jobs run python ...`, the +image's libraries import directly — no `--python` or `PYTHONPATH` needed. There's an +[open issue in `uv`](https://github.com/astral-sh/uv/issues/7999) to make this less manual +in future. + ## vLLM vLLM is a very well known and heavily used inference engine. It is known for its ability to scale inference for LLMs. @@ -15,6 +63,11 @@ Use the `--image` argument to use this Docker image: >>> hf jobs uv run --image vllm/vllm-openai --flavor l4x4 generate-responses.py ``` +> [!TIP] +> The `vllm/vllm-openai` image ships prebuilt vLLM and FlashInfer kernels. To use those +> rather than letting UV reinstall vLLM from PyPI, follow [Reusing image-baked libraries +> with `uv run`](#reusing-image-baked-libraries-with-uv-run) above. + You can find more information on vLLM batch inference on Jobs in [Daniel Van Strien's blog post](https://danielvanstrien.xyz/posts/2025/hf-jobs/vllm-batch-inference.html). ## TRL From 8746d80b54ec6e73303f2ad81dfb722dc6a51f32 Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Wed, 3 Jun 2026 17:53:24 +0100 Subject: [PATCH 2/5] Move image-baked libraries section to bottom, add top warning Address review: relocate the detailed "Reusing image-baked libraries with uv run" section to the end of the page, and add a warning callout near the top so the naive `hf jobs uv run` failure mode is flagged before the framework list. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/hub/jobs-popular-images.md | 63 ++++++++++++++++++--------------- 1 file changed, 35 insertions(+), 28 deletions(-) diff --git a/docs/hub/jobs-popular-images.md b/docs/hub/jobs-popular-images.md index 57ea92c268..20aa08841a 100644 --- a/docs/hub/jobs-popular-images.md +++ b/docs/hub/jobs-popular-images.md @@ -4,6 +4,41 @@ Here is the list of ready-to-use Docker images from popular frameworks that you These Docker images already have uv installed but if you want to use an image + uv for an image without uv installed you’ll need to make sure uv is installed first. This will work well in many cases but for LLM inference libraries which can have quite specific requirements, it can be useful to use a specific image that has the library installed. +> [!WARNING] +> By default, `hf jobs uv run` installs your script's dependencies from PyPI and does **not** +> reuse the libraries already baked into these images. For CUDA-heavy inference libraries +> (vLLM, FlashInfer, and so on) the naive command can fail at startup. See +> [Reusing image-baked libraries with `uv run`](#reusing-image-baked-libraries-with-uv-run) +> below for the workaround. + +## vLLM + +vLLM is a very well known and heavily used inference engine. It is known for its ability to scale inference for LLMs. +They provide the `vllm/vllm-openai` Docker image with vLLM and UV ready. This image is ideal to run batch inference. + +Use the `--image` argument to use this Docker image: + +```bash +>>> hf jobs uv run --image vllm/vllm-openai --flavor l4x4 generate-responses.py +``` + +> [!TIP] +> The `vllm/vllm-openai` image ships prebuilt vLLM and FlashInfer kernels. To use those +> rather than letting UV reinstall vLLM from PyPI, follow [Reusing image-baked libraries +> with `uv run`](#reusing-image-baked-libraries-with-uv-run) below. + +You can find more information on vLLM batch inference on Jobs in [Daniel Van Strien's blog post](https://danielvanstrien.xyz/posts/2025/hf-jobs/vllm-batch-inference.html). + +## TRL + +TRL is a library designed for post-training models using techniques like Supervised Fine-Tuning (SFT), Group Relative Policy Optimization (GRPO), and Direct Preference Optimization (DPO). An up-to-date Docker image with UV and all TRL dependencies is available at `huggingface/trl` and can be used directly with Hugging Face Jobs. + +Use the `--image` argument to use this Docker image: + +```bash +>>> hf jobs uv run --image huggingface/trl --flavor a100-large -s HF_TOKEN train.py +``` + ## Reusing image-baked libraries with `uv run` When you run a script against one of these images with `hf jobs uv run`, UV builds its own @@ -51,31 +86,3 @@ This only applies to `hf jobs uv run`. With generic `hf jobs run python image's libraries import directly — no `--python` or `PYTHONPATH` needed. There's an [open issue in `uv`](https://github.com/astral-sh/uv/issues/7999) to make this less manual in future. - -## vLLM - -vLLM is a very well known and heavily used inference engine. It is known for its ability to scale inference for LLMs. -They provide the `vllm/vllm-openai` Docker image with vLLM and UV ready. This image is ideal to run batch inference. - -Use the `--image` argument to use this Docker image: - -```bash ->>> hf jobs uv run --image vllm/vllm-openai --flavor l4x4 generate-responses.py -``` - -> [!TIP] -> The `vllm/vllm-openai` image ships prebuilt vLLM and FlashInfer kernels. To use those -> rather than letting UV reinstall vLLM from PyPI, follow [Reusing image-baked libraries -> with `uv run`](#reusing-image-baked-libraries-with-uv-run) above. - -You can find more information on vLLM batch inference on Jobs in [Daniel Van Strien's blog post](https://danielvanstrien.xyz/posts/2025/hf-jobs/vllm-batch-inference.html). - -## TRL - -TRL is a library designed for post-training models using techniques like Supervised Fine-Tuning (SFT), Group Relative Policy Optimization (GRPO), and Direct Preference Optimization (DPO). An up-to-date Docker image with UV and all TRL dependencies is available at `huggingface/trl` and can be used directly with Hugging Face Jobs. - -Use the `--image` argument to use this Docker image: - -```bash ->>> hf jobs uv run --image huggingface/trl --flavor a100-large -s HF_TOKEN train.py -``` From f1e8dac0537f3b3705975dc65feba6ddbd1ad572 Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Thu, 4 Jun 2026 09:22:08 +0100 Subject: [PATCH 3/5] Reframe around using a framework image; demote Python-package reuse MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit GPU-tested the claims on HF Jobs (l4x1/a100, Qwen2.5-0.5B): - Reproduced `Could not find nvcc` ONLY on the default uv image (no CUDA toolkit). With --image vllm/vllm-openai it works, because the image provides nvcc + the CUDA system stack. - So the real fix is using the framework image, not the --python + PYTHONPATH recipe; "usually all you need" is just --image. - Reuse via PYTHONPATH does not skip the PyPI install (same ~3.5GB downloaded), so it's not a speed-up — demoted to an optional sub-section for ABI/CUDA-mismatch cases. Lead the section with the system-stack point and the nvcc repro; keep the probe (paths/uv vary per image, verified across vLLM, sglang, unsloth, trl, transformers, axolotl). Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/hub/jobs-popular-images.md | 88 +++++++++++++++++++-------------- 1 file changed, 51 insertions(+), 37 deletions(-) diff --git a/docs/hub/jobs-popular-images.md b/docs/hub/jobs-popular-images.md index 20aa08841a..476916305e 100644 --- a/docs/hub/jobs-popular-images.md +++ b/docs/hub/jobs-popular-images.md @@ -4,12 +4,10 @@ Here is the list of ready-to-use Docker images from popular frameworks that you These Docker images already have uv installed but if you want to use an image + uv for an image without uv installed you’ll need to make sure uv is installed first. This will work well in many cases but for LLM inference libraries which can have quite specific requirements, it can be useful to use a specific image that has the library installed. -> [!WARNING] -> By default, `hf jobs uv run` installs your script's dependencies from PyPI and does **not** -> reuse the libraries already baked into these images. For CUDA-heavy inference libraries -> (vLLM, FlashInfer, and so on) the naive command can fail at startup. See -> [Reusing image-baked libraries with `uv run`](#reusing-image-baked-libraries-with-uv-run) -> below for the workaround. +> [!TIP] +> For GPU inference libraries, pass `--image` so the run gets a matching CUDA system stack +> (toolkit, `nvcc`, libraries) — see [Using framework images for GPU +> libraries](#using-framework-images-for-gpu-libraries) below. ## vLLM @@ -23,9 +21,9 @@ Use the `--image` argument to use this Docker image: ``` > [!TIP] -> The `vllm/vllm-openai` image ships prebuilt vLLM and FlashInfer kernels. To use those -> rather than letting UV reinstall vLLM from PyPI, follow [Reusing image-baked libraries -> with `uv run`](#reusing-image-baked-libraries-with-uv-run) below. +> The `vllm/vllm-openai` image bundles the CUDA toolkit and prebuilt vLLM/FlashInfer kernels. +> Using it is usually all you need; you can also reuse its prebuilt Python builds — see [Using +> framework images for GPU libraries](#using-framework-images-for-gpu-libraries) below. You can find more information on vLLM batch inference on Jobs in [Daniel Van Strien's blog post](https://danielvanstrien.xyz/posts/2025/hf-jobs/vllm-batch-inference.html). @@ -39,15 +37,35 @@ Use the `--image` argument to use this Docker image: >>> hf jobs uv run --image huggingface/trl --flavor a100-large -s HF_TOKEN train.py ``` -## Reusing image-baked libraries with `uv run` +## Using framework images for GPU libraries + +GPU libraries like vLLM need more than their Python package — they need a matching system +environment: the CUDA toolkit (including `nvcc`), system libraries like NCCL and cuDNN, and so +on. If you omit `--image`, `hf jobs uv run` uses the default uv image +(`ghcr.io/astral-sh/uv:python3.12-bookworm`), a bare Python base with no CUDA toolkit. Your +dependencies still install from PyPI, but at runtime a library that needs the toolkit can fail — +for example FlashInfer's sampler JIT-compiles a kernel and aborts with: + +```text +RuntimeError: Could not find nvcc and default cuda_home='/usr/local/cuda' doesn't exist +``` + +Passing the framework image fixes this — it provides the CUDA toolkit, `nvcc`, and matched +libraries — and **this is usually all you need**: + +```bash +hf jobs uv run --image vllm/vllm-openai --flavor l4x4 -s HF_TOKEN generate-responses.py +``` + +UV still reinstalls your `# /// script` dependencies from PyPI, but they now run against the +image's system stack, so the framework works. -When you run a script against one of these images with `hf jobs uv run`, UV builds its own -isolated environment and installs your `# /// script` dependencies from PyPI — it does -**not** reuse the libraries already baked into the image. For images that ship prebuilt, -CUDA-matched builds (vLLM, PyTorch, FlashInfer, and so on) you usually want the opposite: -reuse the image's copies for faster cold starts and to avoid CUDA/ABI mismatches with the -image's driver. Two flags do that — point UV at the image's interpreter, and add its -site-packages to the import path: +### Optionally: reuse the image's prebuilt Python builds + +These images also ship prebuilt, CUDA-matched builds of the framework itself. To import those +instead of UV's fresh PyPI install — handy if a PyPI build and the image's CUDA stack disagree +(an ABI mismatch, or a kernel that won't build) — point UV at the image's interpreter and add +its site-packages to the import path: ```bash hf jobs uv run \ @@ -59,30 +77,26 @@ hf jobs uv run \ generate-responses.py ``` -- `--python` runs the script with the **image's** interpreter instead of one UV provisions - itself, keeping ABI compatibility with the image's compiled extensions. -- `-e PYTHONPATH=...` adds the image's site-packages to the import path **for that UV run**, - so `import vllm` resolves to the image's prebuilt build rather than a fresh PyPI install. +- `--python` uses the **image's** interpreter, keeping ABI compatibility with its compiled extensions. +- `-e PYTHONPATH=...` makes `import vllm` resolve to the image's prebuilt build for that run. -The exact paths differ per image. Find them with a one-off probe on `cpu-basic` — it prints -the interpreter path to pass as `--python` and the package directory to pass as `PYTHONPATH`: +Paths differ per image, so probe them on `cpu-basic` rather than hardcoding: ```bash -hf jobs run --flavor cpu-basic vllm/vllm-openai bash -c ' - which python3 - python3 -c "import importlib.util, os; print(os.path.dirname(os.path.dirname(importlib.util.find_spec(\"vllm\").origin)))" -' +hf jobs run --flavor cpu-basic vllm/vllm-openai bash -c 'which python3; which uv; python3 -m pip show vllm | grep Location' +``` + +```text +/usr/bin/python3 # pass to --python +/usr/local/bin/uv # uv is present, so `uv run` works +Location: /usr/local/lib/python3.12/dist-packages # pass to PYTHONPATH ``` -(Swap `vllm` for whichever library you're reusing.) +Swap `vllm` for whichever library you're reusing. Layouts vary — `vllm/vllm-openai` and +`lmsysorg/sglang` use the system `dist-packages` above, while `unsloth/unsloth` uses a +virtualenv (`/opt/venv/...`). > [!TIP] -> Symptoms of *not* reusing the image's build vary by library. With vLLM, for example, a -> PyPI-installed copy can fail at startup with `RuntimeError: Could not find nvcc` — its -> FlashInfer sampler tries to JIT-compile a CUDA kernel the runtime has no compiler for, -> while the image already ships the prebuilt kernels. - -This only applies to `hf jobs uv run`. With generic `hf jobs run python ...`, the -image's libraries import directly — no `--python` or `PYTHONPATH` needed. There's an -[open issue in `uv`](https://github.com/astral-sh/uv/issues/7999) to make this less manual -in future. +> This pins imports to the image's build; UV still installs your declared dependencies, so it +> isn't a speed-up. A `uv run --system-site-packages` that would reuse the image's packages and +> skip the reinstall is [requested upstream](https://github.com/astral-sh/uv/issues/7999). From b737fb3acc436a6ce08418396007161b311705fd Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Thu, 4 Jun 2026 09:34:59 +0100 Subject: [PATCH 4/5] Apply suggestion from @davanstrien --- docs/hub/jobs-popular-images.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hub/jobs-popular-images.md b/docs/hub/jobs-popular-images.md index 476916305e..5b740bac7d 100644 --- a/docs/hub/jobs-popular-images.md +++ b/docs/hub/jobs-popular-images.md @@ -51,7 +51,7 @@ RuntimeError: Could not find nvcc and default cuda_home='/usr/local/cuda' doesn' ``` Passing the framework image fixes this — it provides the CUDA toolkit, `nvcc`, and matched -libraries — and **this is usually all you need**: +libraries — and this is usually all you need: ```bash hf jobs uv run --image vllm/vllm-openai --flavor l4x4 -s HF_TOKEN generate-responses.py From 1191e76e924a6c3216d519c2d0ec615ba1c0111e Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Thu, 4 Jun 2026 09:35:33 +0100 Subject: [PATCH 5/5] Apply suggestion from @davanstrien --- docs/hub/jobs-popular-images.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hub/jobs-popular-images.md b/docs/hub/jobs-popular-images.md index 5b740bac7d..ba6475b8bd 100644 --- a/docs/hub/jobs-popular-images.md +++ b/docs/hub/jobs-popular-images.md @@ -57,7 +57,7 @@ libraries — and this is usually all you need: hf jobs uv run --image vllm/vllm-openai --flavor l4x4 -s HF_TOKEN generate-responses.py ``` -UV still reinstalls your `# /// script` dependencies from PyPI, but they now run against the +UV still reinstalls your script dependencies from PyPI, but they now run against the image's system stack, so the framework works. ### Optionally: reuse the image's prebuilt Python builds