Skip to content

Commit 4e154b5

Browse files
localai-botmudler
andauthored
fix(ci): unbreak rerankers (torch bump) and vllm-omni on aarch64 (#9688)
Two unrelated CI breakages bundled together since both are one-liners: - rerankers: bump torch 2.4.1 -> 2.7.1 on cpu/cublas12. The unpinned transformers resolves to 5.x, whose moe.py registers a custom_op with string-typed `'torch.Tensor'` annotations that torch 2.4.1's infer_schema rejects, blocking the gRPC server from starting and failing all 5 backend tests with "Connection refused" on :50051. Matches the version used by the transformers backend. - vllm-omni: strip fa3-fwd from the upstream requirements/cuda.txt before resolving on aarch64. fa3-fwd 0.0.3 ships only an x86_64 wheel and has no sdist, making the cuda profile unsatisfiable on Jetson/SBSA. fa3-fwd is a soft runtime dep — vllm-omni's attention backends fall back to FA2 then SDPA when it's missing. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
1 parent 969005b commit 4e154b5

3 files changed

Lines changed: 10 additions & 2 deletions

File tree

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
transformers
22
accelerate
3-
torch==2.4.1
3+
torch==2.7.1
44
rerankers[transformers]
Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
transformers
22
accelerate
3-
torch==2.4.1
3+
torch==2.7.1
44
rerankers[transformers]

backend/python/vllm-omni/install.sh

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,14 @@ fi
7979

8080
cd vllm-omni/
8181

82+
# fa3-fwd ships no aarch64 wheels and there is no source distribution, so on
83+
# aarch64 (e.g. l4t13 / SBSA cu130) the upstream requirements/cuda.txt is
84+
# unsatisfiable. Drop it before resolving — vllm-omni does not hard-require
85+
# the fused FA3 kernel at import time on Jetson/SBSA targets.
86+
if [ "$(uname -m)" = "aarch64" ] && [ -f requirements/cuda.txt ]; then
87+
sed -i '/^fa3-fwd[[:space:]]*==/d' requirements/cuda.txt
88+
fi
89+
8290
if [ "x${USE_PIP}" == "xtrue" ]; then
8391
pip install ${EXTRA_PIP_INSTALL_FLAGS:-} -e .
8492
else

0 commit comments

Comments
 (0)