Skip to content

Commit 8d6fdf2

Browse files
localai-botmudler
localai-bot
and
mudler
authored
fix(backends): derive the protoc generator from the protobuf runtime, regenerate stubs after late installs (#11057)
* fix(backends): choose the protoc generator from the protobuf runtime, and regenerate stubs after late installs The vLLM backends still crash on startup with VersionError: Detected incompatible Protobuf Gencode/Runtime versions when loading backend.proto: gencode 7.35.0 runtime 6.33.6 despite #10735 and #10944. Three separate defects kept it alive. 1. runProtogen picked the generator from the installed *grpcio* version. grpcio-tools' version tracks grpcio, but the gencode its bundled protoc emits tracks *protobuf*, and the two move independently: grpcio-tools 1.82.1 (the version #10735 pins to, matching grpcio 1.82.1) requires protobuf>=7.35.1 and stamps gencode 7.35.0. Pinning to grpcio could therefore never constrain the gencode. Constrain the install to the protobuf already in the venv instead and let the resolver pick the newest compatible grpcio-tools. That both selects a generator the runtime accepts and stops protogen from moving the runtime under the backend's other deps. This is self-correcting, so the hardcoded GRPCIO_TOOLS_VERSION=1.78.0 escape hatch from #10944 is no longer needed and is removed. 2. The stubs were generated too early. Most branches of vllm/install.sh (and vllm-omni) install vllm *after* installRequirements, and vllm re-resolves the protobuf runtime as it lands. Stubs generated against the pre-vllm runtime can end up newer than the runtime that finally ships, which is the ROCm failure exactly. Regenerate once the dependency set is final. 3. rm -f of the .py sources left __pycache__ behind. CPython validates a .pyc against source mtime and size, both of which can be unchanged across a regeneration (the gencode triple is the same width whether it reads 7.35.0 or 6.33.5), so a stale backend_pb2.pyc could shadow the stub just written. Also fail the build when the generated stub cannot be imported, so a gencode/runtime mismatch surfaces at image build time instead of reaching users as an opaque "grpc service not ready". Verified by driving the real runProtogen through the ROCm install sequence in a venv harness: before, gencode 7.35.0 against runtime 6.33.6 (reproducing the reported error verbatim); after, gencode 6.33.5 against runtime 6.33.6 and the stub imports cleanly. Closes #10940 Closes #10718 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit] * fix(backends): regenerate protobuf stubs in the other backends that install after installRequirements Same defect as the vllm change: installRequirements generates the stubs at the end of its own run, so any backend that installs further packages afterwards can have the protobuf runtime moved out from under stubs that were already written. The gencode stamped into backend_pb2.py then exceeds the runtime that ships and the backend dies at model load with "grpc service not ready". fish-speech already had this bug and worked around the symptom: it forces protobuf>=5.29.0 after installRequirements precisely because "transitive deps (wandb, tensorboard) may downgrade protobuf to 3.x but our generated backend_pb2.py requires protobuf 5+". Regenerating after the pin addresses the cause rather than propping up the runtime to match stale stubs. Applied to the backends whose post-installRequirements step resolves a dependency graph and can therefore move protobuf: fish-speech -e . plus an explicit protobuf install vibevoice pip install . (with deps) llama-cpp-quantization gguf / GGUF_PIP_SPEC trl gguf / GGUF_PIP_SPEC Deliberately not applied to ace-step and chatterbox (both --no-deps, so the dependency graph cannot change) or voxcpm (pins setuptools only). gguf does not depend on protobuf today, but it resolves dependencies, and "this package does not touch protobuf right now" is exactly the assumption that made the earlier fix ineffective. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit] * fix(backends): resolve the protoc generator in a throwaway env so it cannot edit the backend's pinned deps Installing grpcio-tools into the backend's own venv to generate the stubs also drags its dependencies in: grpcio-tools 1.82.1 requires grpcio>=1.82.1, so a backend that pinned grpcio==1.78.1 silently shipped 1.82.1 instead. Caught by building the llama-cpp-quantization image and reading the versions back out of the artifact: before grpcio 1.82.1 (requirements.txt pins grpcio==1.78.1) after grpcio 1.78.1 grpcio-tools absent from the venv entirely Resolve the generator in a throwaway environment instead, still constrained to the protobuf the backend ships so the gencode stays compatible. The backend's dependency set is then exactly what its requirements files declared. protoc's output is plain Python and carries no dependency on the interpreter that produced it, so generating from a different env is safe; the import check still runs under the backend's python, since that is the interpreter that has to load the stubs at model load. Verified on the rebuilt image: gencode 7.35.0, runtime protobuf 7.35.1, grpcio back at its pinned 1.78.1, and the shipped stub imports cleanly against 7.35.1. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit] * fix(backends): bound the protoc generator by BOTH the installed grpcio and protobuf The generated stubs impose two independent constraints, and every fix so far, including the previous commit on this branch, satisfied one while violating the other: backend_pb2.py needs protobuf runtime >= gencode backend_pb2_grpc.py needs installed grpcio >= grpcio-tools Resolving the generator against protobuf alone picked grpcio-tools 1.82.1 for a backend holding grpcio at 1.78.1, so the gencode was fine but the gRPC stub was not: RuntimeError: The grpc package installed is at version 1.78.1, but the generated code in backend_pb2_grpc.py depends on grpcio>=1.82.1. That is also why installing grpcio-tools into the backend venv appeared to work earlier: it dragged grpcio up to match, which was load-bearing rather than the regression it looked like. Isolating the generator removed the accidental fix and exposed the missing constraint. Bound grpcio-tools from both sides instead and let the resolver find the newest version satisfying both. The protobuf ceiling makes it back off to an older generator when the runtime trails, bounding the gencode; the grpcio ceiling keeps the _grpc stub loadable. Resolved against the four real runtime pairs observed in built images: grpcio 1.78.1 / protobuf 7.35.1 -> grpcio-tools 1.78.0, gencode 6.31.1 OK grpcio 1.78.0 / protobuf 6.33.6 -> grpcio-tools 1.78.0, gencode 6.31.1 OK grpcio 1.82.1 / protobuf 6.33.6 -> grpcio-tools 1.81.1, gencode 6.33.5 OK grpcio 1.82.1 / protobuf 7.35.1 -> grpcio-tools 1.82.1, gencode 7.35.0 OK Also restore the import check to cover backend_pb2_grpc as well as backend_pb2. Narrowing it to backend_pb2 is why the image build passed while CI failed: the guard could not see the constraint that was actually broken. Verified by running the CI sequence locally for llama-cpp-quantization, the backend whose test failed: make -C backend/python/llama-cpp-quantization -> exit 0 make -C backend/python/llama-cpp-quantization test -> exit 0, OK Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
1 parent 1919e29 commit 8d6fdf2

7 files changed

Lines changed: 112 additions & 34 deletions

File tree

‎backend/python/common/libbackend.sh‎

Lines changed: 67 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -398,40 +398,80 @@ function ensureVenv() {
398398
function runProtogen() {
399399
ensureVenv
400400

401-
# Match grpcio-tools to the grpcio already installed by the backend's
402-
# requirements. grpcio and grpcio-tools are released in lockstep, and the
403-
# protoc that grpcio-tools bundles stamps a Protobuf "gencode" version into
404-
# backend_pb2.py. Left unpinned, `uv pip install grpcio-tools` pulls the
405-
# newest release, whose newer gencode (e.g. 7.35.0) trips Protobuf's
406-
# runtime >= gencode guarantee at import time when a backend caps the
407-
# protobuf runtime lower (vLLM pins it to 6.33.6), crashing the backend with
408-
# "grpc service not ready" before it ever loads a model. Pinning
409-
# grpcio-tools to the installed grpcio version keeps the gencode in step with
410-
# the runtime. Backends whose protobuf runtime trails grpcio can set
411-
# GRPCIO_TOOLS_VERSION to the newest generator their runtime accepts. Falls
412-
# back to unpinned when grpcio isn't installed yet.
413-
# See mudler/LocalAI#10718.
414-
local grpcio_tools_spec="grpcio-tools"
415-
local grpcio_version
416-
if [ -n "${GRPCIO_TOOLS_VERSION:-}" ]; then
417-
grpcio_tools_spec="grpcio-tools==${GRPCIO_TOOLS_VERSION}"
418-
else
419-
grpcio_version="$(python -c 'import importlib.metadata as m; print(m.version("grpcio"))' 2>/dev/null || true)"
420-
fi
421-
if [ "${grpcio_tools_spec}" = "grpcio-tools" ] && [ -n "${grpcio_version}" ]; then
422-
grpcio_tools_spec="grpcio-tools==${grpcio_version}"
423-
fi
401+
# The protoc that grpcio-tools bundles stamps a Protobuf "gencode" version
402+
# into backend_pb2.py, and Protobuf refuses to import a stub whose gencode is
403+
# newer than the installed runtime. Left unpinned, grpcio-tools resolves to
404+
# the newest release (1.82.1, gencode 7.35.0) which a backend holding the
405+
# runtime lower (vLLM resolves protobuf to 6.33.6) then cannot import,
406+
# crashing with "grpc service not ready" before it ever loads a model.
407+
#
408+
# Resolve the generator in a THROWAWAY environment so the backend's own venv
409+
# keeps exactly the dependency set its requirements files declared. The
410+
# generator is a build tool and has no business editing the runtime deps: the
411+
# spec below is chosen to fit the venv as it stands, never to change it.
412+
# Falls back to unpinned when neither version can be detected.
413+
# See mudler/LocalAI#10718, #10940.
414+
local protobuf_version grpcio_version protogen_env protogen_python
415+
protobuf_version="$(python -c 'import importlib.metadata as m; print(m.version("protobuf"))' 2>/dev/null || true)"
416+
grpcio_version="$(python -c 'import importlib.metadata as m; print(m.version("grpcio"))' 2>/dev/null || true)"
417+
418+
# The stubs impose TWO independent constraints on the generator, and both
419+
# have to hold or the backend dies on import:
420+
#
421+
# backend_pb2.py needs protobuf runtime >= gencode
422+
# backend_pb2_grpc.py needs installed grpcio >= grpcio-tools
423+
#
424+
# Bound grpcio-tools from both sides and let the resolver find the newest
425+
# version that satisfies them. The protobuf ceiling makes it back off to an
426+
# older grpcio-tools when the runtime is behind, which is what bounds the
427+
# gencode; the grpcio ceiling keeps the _grpc stub loadable. Pinning only one
428+
# side is what made the earlier attempts fail, in both directions.
429+
local -a protogen_spec=("grpcio-tools")
430+
if [ -n "${grpcio_version}" ]; then
431+
protogen_spec=("grpcio-tools<=${grpcio_version}")
432+
fi
433+
if [ -n "${protobuf_version}" ]; then
434+
protogen_spec+=("protobuf<=${protobuf_version}")
435+
fi
436+
437+
protogen_env="$(mktemp -d)"
424438

425439
if [ "x${USE_PIP}" == "xtrue" ]; then
426-
pip install "${grpcio_tools_spec}"
440+
python -m venv "${protogen_env}"
441+
"${protogen_env}/bin/pip" install "${protogen_spec[@]}"
427442
else
428-
uv pip install "${grpcio_tools_spec}"
443+
uv venv "${protogen_env}"
444+
VIRTUAL_ENV="${protogen_env}" uv pip install "${protogen_spec[@]}"
429445
fi
446+
protogen_python="${protogen_env}/bin/python"
447+
430448
pushd "${EDIR}" >/dev/null
449+
# Drop the cached bytecode along with the sources. CPython validates a
450+
# .pyc against the source's mtime *and size*, both of which can be
451+
# unchanged across a regeneration (the gencode triple is the same width
452+
# whether it reads 7.35.0 or 6.33.5), so a stale backend_pb2.pyc can be
453+
# reused in place of the stub we just generated.
431454
rm -f backend_pb2.py backend_pb2.pyi backend_pb2_grpc.py
432-
# use the venv python (ensures correct interpreter & sys.path)
433-
python -m grpc_tools.protoc -I../../ -I./ --python_out=. --grpc_python_out=. backend.proto
455+
rm -rf __pycache__
456+
# Generate with the throwaway toolchain; the output is plain Python and
457+
# carries no dependency on the interpreter that produced it.
458+
"${protogen_python}" -m grpc_tools.protoc -I../../ -I./ --python_out=. --grpc_python_out=. backend.proto
459+
460+
# Verify with the BACKEND's python, which is the interpreter that has to
461+
# import these at model load. Fail the build rather than ship a backend
462+
# that cannot import its own stubs: otherwise the gencode/runtime
463+
# mismatch only surfaces in a released image, as an opaque
464+
# "grpc service not ready".
465+
# Check BOTH stubs: backend_pb2 catches a gencode ahead of the protobuf
466+
# runtime, backend_pb2_grpc catches generated code ahead of the installed
467+
# grpcio. Checking only the former lets the latter reach CI, or users.
468+
if ! python -c 'import backend_pb2, backend_pb2_grpc' >/dev/null; then
469+
echo "runProtogen: generated stubs are not importable by the backend venv (protobuf ${protobuf_version:-unknown}, grpcio ${grpcio_version:-unknown})" >&2
470+
exit 1
471+
fi
434472
popd >/dev/null
473+
474+
rm -rf "${protogen_env}"
435475
}
436476

437477

‎backend/python/fish-speech/install.sh‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,3 +60,9 @@ if [ "x${USE_PIP}" == "xtrue" ]; then
6060
else
6161
uv pip install "protobuf>=5.29.0"
6262
fi
63+
64+
# Regenerate the stubs against the protobuf runtime settled on just above. The
65+
# pin exists because transitive deps drag protobuf down; generating before that
66+
# pin is applied is what stamps a gencode the runtime then rejects.
67+
# See mudler/LocalAI#10718.
68+
runProtogen

‎backend/python/llama-cpp-quantization/install.sh‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,3 +68,11 @@ if [ ! -x "${QUANTIZE_BIN}" ] && ! command -v llama-quantize &>/dev/null; then
6868
echo "Warning: cmake not found — llama-quantize will not be available. Install cmake or provide llama-quantize on PATH."
6969
fi
7070
fi
71+
72+
# The stubs generated at the end of installRequirements were built against the
73+
# protobuf runtime as it stood before the installs above, which resolve their own
74+
# dependencies and can move that runtime. Regenerate now that the dependency set
75+
# is final, so the gencode stamped into backend_pb2.py cannot be newer than the
76+
# runtime that ships. Same failure mode as mudler/LocalAI#10718; runProtogen
77+
# clears the previous stubs first, so this is idempotent.
78+
runProtogen

‎backend/python/trl/install.sh‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,3 +41,11 @@ else
4141
uv pip install "gguf>=0.16.0"
4242
}
4343
fi
44+
45+
# The stubs generated at the end of installRequirements were built against the
46+
# protobuf runtime as it stood before the installs above, which resolve their own
47+
# dependencies and can move that runtime. Regenerate now that the dependency set
48+
# is final, so the gencode stamped into backend_pb2.py cannot be newer than the
49+
# runtime that ships. Same failure mode as mudler/LocalAI#10718; runProtogen
50+
# clears the previous stubs first, so this is idempotent.
51+
runProtogen

‎backend/python/vibevoice/install.sh‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,4 +38,12 @@ if [ ! -d VibeVoice ]; then
3838
else
3939
uv pip install ${EXTRA_PIP_INSTALL_FLAGS:-} .
4040
fi
41-
fi
41+
fi
42+
43+
# The stubs generated at the end of installRequirements were built against the
44+
# protobuf runtime as it stood before the installs above, which resolve their own
45+
# dependencies and can move that runtime. Regenerate now that the dependency set
46+
# is final, so the gencode stamped into backend_pb2.py cannot be newer than the
47+
# runtime that ships. Same failure mode as mudler/LocalAI#10718; runProtogen
48+
# clears the previous stubs first, so this is idempotent.
49+
runProtogen

‎backend/python/vllm-omni/install.sh‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,3 +87,9 @@ else
8787
fi
8888

8989
cd ..
90+
91+
# vllm-omni lands after installRequirements has already generated the protobuf
92+
# stubs, and it re-resolves the protobuf runtime as it installs. Regenerate now
93+
# that the dependency set is final, so the stubs cannot be newer than the runtime
94+
# that ships. Same failure mode as mudler/LocalAI#10718.
95+
runProtogen

‎backend/python/vllm/install.sh‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,6 @@ set -e
33

44
EXTRA_PIP_INSTALL_FLAGS="--no-build-isolation"
55

6-
# grpcio 1.82 bundles protoc 7.x, while vLLM's resolved protobuf runtime is
7-
# still 6.33.x. Generate the backend stubs with the newest grpcio-tools release
8-
# that emits protobuf 6.x gencode; the newer grpcio runtime can consume those
9-
# stubs, but protobuf 6.x cannot import 7.x gencode.
10-
export GRPCIO_TOOLS_VERSION="1.78.0"
11-
126
# Avoid to overcommit the CPU during build
137
# https://github.com/vllm-project/vllm/issues/20079
148
# https://docs.vllm.ai/en/v0.8.3/serving/env_vars.html
@@ -265,3 +259,11 @@ elif [ "x${BUILD_TYPE}" == "x" ] && [ "x${FROM_SOURCE:-}" == "xtrue" ]; then
265259
else
266260
installRequirements
267261
fi
262+
263+
# installRequirements generates the protobuf stubs at the end of its own run, but
264+
# most branches above install vllm *after* it, and vllm re-resolves the protobuf
265+
# runtime when it lands. Stubs generated against the pre-vllm runtime can end up
266+
# newer than the runtime that finally ships, which is exactly the gencode 7.35.0 /
267+
# runtime 6.33.6 crash in mudler/LocalAI#10718. Regenerate once the dependency set
268+
# is final; runProtogen clears the previous stubs first, so this is idempotent.
269+
runProtogen

0 commit comments

Comments
 (0)