Skip to content

feat(vllm-tensorizer): Bump vLLM to v0.20.2 on CUDA 13.2 / Ubuntu 24.04 #160

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JustinPerlman merged 13 commits into from
May 15, 2026
+25 −17
Merged

feat(vllm-tensorizer): Bump vLLM to v0.20.2 on CUDA 13.2 / Ubuntu 24.04 #160

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
1161cbf
feat(vllm-tensorizer): Upgrade vLLM to v0.20.2
JustinPerlman May 11, 2026
d2d91eb
feat(vllm-tensorizer): Upgrade vLLM to v0.20.2
JustinPerlman May 11, 2026
1530550
Merge branch 'jperlman/vllm0.20.2' of https://github.com/coreweave/ml…
JustinPerlman May 11, 2026
6ff9df5
feat(vllm-tensorizer): Bump CUDA to 13.2
JustinPerlman May 11, 2026
8bb3975
fix(vllm-tensorizer): Use major.minor for cuda-python version spec
JustinPerlman May 11, 2026
c6e25b4
feat(vllm-tensorizer): Bump Ubuntu to 24.04
JustinPerlman May 11, 2026
d20edb0
fix(vllm-tensorizer): Fix pip install for Ubuntu 24.04
JustinPerlman May 11, 2026
3fc0de1
fix(vllm-tensorizer): Install wheel for lmcache-builder
JustinPerlman May 11, 2026
3a87962
fix(vllm-tensorizer): Restore wheel to builder-base pip install
JustinPerlman May 11, 2026
81b22b2
fix(vllm-tensorizer): Remove EXTERNALLY-MANAGED in final image stage
JustinPerlman May 11, 2026
1305884
fix(vllm-tensorizer): Purge system python3-jwt before pip installs
JustinPerlman May 12, 2026
cc65ad3
chore(vllm-tensorizer): Fix minor style inconsistencies
JustinPerlman May 12, 2026
202ef09
feat(vllm-tensorizer): Build matrix for CUDA 12.9 and CUDA 13.2
JustinPerlman May 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 13 additions & 10 deletions .github/configurations/vllm-tensorizer.yml
View file
Open in desktop
Copy link
Copy Markdown
Collaborator

@Eta0 Eta0 May 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd personally suggest to not repeat yourself as much in this config file and to construct more parts of this dynamically, like the tag suffix, but not a hard requirement.

Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
vllm-commit:
- 'v0.20.0'
flashinfer-commit:
- 'v0.6.8'
lmcache-commit:
- 'v0.4.2'
builder-base-image:
- 'ghcr.io/coreweave/ml-containers/torch:bc8c66e-nccl-cuda12.9.1-ubuntu22.04-nccl2.30.4-1-torch2.11.0-vision0.26.0-audio2.11.0-abi1'
final-base-image:
- 'ghcr.io/coreweave/ml-containers/torch:bc8c66e-nccl-cuda12.9.1-ubuntu22.04-nccl2.30.4-1-torch2.11.0-vision0.26.0-audio2.11.0-abi1'
include:
- vllm-commit: 'v0.20.2'
flashinfer-commit: 'v0.6.8'
lmcache-commit: 'v0.4.2'
builder-base-image: 'ghcr.io/coreweave/ml-containers/torch:bc8c66e-nccl-cuda13.2.1-ubuntu24.04-nccl2.30.4-1-torch2.11.0-vision0.26.0-audio2.11.0-abi1'
final-base-image: 'ghcr.io/coreweave/ml-containers/torch:bc8c66e-nccl-cuda13.2.1-ubuntu24.04-nccl2.30.4-1-torch2.11.0-vision0.26.0-audio2.11.0-abi1'
tag-suffix: 'v0.20.2-cuda13.2.1-ubuntu24.04'
- vllm-commit: 'v0.20.2'
flashinfer-commit: 'v0.6.8'
lmcache-commit: 'v0.4.2'
builder-base-image: 'ghcr.io/coreweave/ml-containers/torch:bc8c66e-nccl-cuda12.9.1-ubuntu24.04-nccl2.30.4-1-torch2.11.0-vision0.26.0-audio2.11.0-abi1'
final-base-image: 'ghcr.io/coreweave/ml-containers/torch:bc8c66e-nccl-cuda12.9.1-ubuntu24.04-nccl2.30.4-1-torch2.11.0-vision0.26.0-audio2.11.0-abi1'
tag-suffix: 'v0.20.2-cuda12.9.1-ubuntu24.04'
2 changes: 1 addition & 1 deletion .github/workflows/vllm-tensorizer.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ jobs:
with:
image-name: vllm-tensorizer
folder: vllm-tensorizer
tag-suffix: ${{ matrix.vllm-commit }}
tag-suffix: ${{ matrix.tag-suffix }}
build-contexts: |
common=common
object-storage-secrets: true
Expand Down
17 changes: 11 additions & 6 deletions vllm-tensorizer/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# syntax=docker/dockerfile:1.10
ARG BUILDER_BASE_IMAGE="ghcr.io/coreweave/ml-containers/torch:17ad6db-nccl-cuda12.9.1-ubuntu22.04-nccl2.29.2-1-torch2.10.0-vision0.25.0-audio2.10.0-abi1"
ARG FINAL_BASE_IMAGE="ghcr.io/coreweave/ml-containers/torch:17ad6db-nccl-cuda12.9.1-ubuntu22.04-nccl2.29.2-1-torch2.10.0-vision0.25.0-audio2.10.0-abi1"
ARG BUILDER_BASE_IMAGE="ghcr.io/coreweave/ml-containers/torch:bc8c66e-nccl-cuda13.2.1-ubuntu24.04-nccl2.30.4-1-torch2.11.0-vision0.26.0-audio2.11.0-abi1"
ARG FINAL_BASE_IMAGE="ghcr.io/coreweave/ml-containers/torch:bc8c66e-nccl-cuda13.2.1-ubuntu24.04-nccl2.30.4-1-torch2.11.0-vision0.26.0-audio2.11.0-abi1"
ARG SCCACHE_VERSION="0.14.0"

FROM alpine/curl:8.17.0 AS sccache-downloader
Expand All @@ -22,9 +22,10 @@ RUN ldconfig

RUN apt-get -qq update && \
apt-get -qq install -y --no-install-recommends \
python3-pip git ninja-build cmake gcc-12 g++-12 && \
git ninja-build cmake gcc-12 g++-12 && \
apt-get clean && \
pip3 install -U --no-cache-dir pip packaging 'setuptools>=77.0.3,<81.0.0' wheel setuptools_scm regex build
rm -f /usr/lib/python3.*/EXTERNALLY-MANAGED && \
python3 -m pip install -U --no-cache-dir pip packaging 'setuptools>=77.0.3,<81.0.0' wheel setuptools_scm regex build
Comment on lines +27 to +28
Copy link
Copy Markdown
Collaborator

@Eta0 Eta0 May 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rm -f /usr/lib/python3.*/EXTERNALLY-MANAGED and pip/setuptools installation and upgrading is already handled by the torch image, so you don't need to repeat those bits here.


# Create the /wheels directory
WORKDIR /wheels
Expand Down Expand Up @@ -164,7 +165,7 @@ RUN --mount=type=bind,from=flashinfer-downloader,source=/git/flashinfer,target=/
python3 -m pip install --no-cache-dir \
requests nvidia-ml-py ninja tqdm filelock \
'nvidia-cudnn-frontend>=1.13.0,<1.19.0' \
"cuda-python~=${CUDA_VERSION}" \
"cuda-python~=${CUDA_VERSION%.*}" \
"nvidia-nvshmem-cu${CUDA_VERSION%%.*}<3.6" \
'apache-tvm-ffi==0.1.9' && \
export FLASHINFER_LOCAL_VERSION="$(sed -E 's@([[:digit:]]+)\.([[:digit:]]+).*$@cu\1\2@')" \
Expand Down Expand Up @@ -264,7 +265,11 @@ FROM ${FINAL_BASE_IMAGE} AS base

WORKDIR /workspace

RUN apt-get -qq update && apt-get install -y --no-install-recommends curl libsodium23 libnuma-dev && apt-get clean
RUN apt-get -qq update && \
apt-get install -y --no-install-recommends curl libsodium23 libnuma-dev && \
apt-get purge -y python3-jwt && \
Copy link
Copy Markdown
Collaborator

@Eta0 Eta0 May 15, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's that apt-get purge -y python3-jwt for? 👀

apt-get clean && \
rm -f /usr/lib/python3.*/EXTERNALLY-MANAGED
Copy link
Copy Markdown
Collaborator

@Eta0 Eta0 May 15, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as before: this rm is already handled by the base image.


RUN --mount=type=bind,from=freezer,target=/tmp/frozen \
/tmp/frozen/freeze.sh torch torchaudio torchvision xformers > /tmp/constraints.txt && \
Expand Down
Loading