vllm: 0.16.0 -> 0.19.0

[CertainLach]

Diff: vllm-project/vllm@releases/v0.16.0...v0.17.0
Changelog: https://github.com/vllm-project/vllm/releases/tag/v0.17.0

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

Only tested on rocm, strix halo

[CertainLach]

opentelemetry-api updated because opentelemetry-semantic-conventions-ai wants newer versions

[github-actions]

The PR's base branch is set to master, but this PR causes 4377 rebuilds.
It is therefore considered a mass rebuild.
Please change the base branch to the right base branch for your changes (probably staging).

[CertainLach]

Oh, maybe I should split opentelemetry changes into another PR. Relaxing dependencies for opentelemetry-semantic-conventions-ai doesn't help, as it fails at runtime otherwise.

[CertainLach]

For opentelemetry changes, might switch to depend on #489017 instead

[GaetanLepage]

Thanks for the PR. However, I would prefer if its scope would be narrower.

[CertainLach]

I agree, I only aggregated all the required changes for now, and will split them into PRs

[GaetanLepage]

I agree, I only aggregated all the required changes for now, and will split them into PRs

Thanks!

[CertainLach]

Dependency graph for this PR:

Will create a separate PR for amd-quark once #498053 and #498052 are merged

Opentelemetry PRs: #498050 #498051

Also kaldi-native-fbank: #498056

[CertainLach]

amd-quark pr: #498069

[pkieltyka]

as I recall I was hitting tool call warnings and errors and needed to upgrade to package in order to resolve those.

I think upgrading in this PR is a good idea

[d-goldin]

So, this is how I got it to work with CUDA for now. PR briefly explains why I think it's safe enough to drop those deps for now.

@CertainLach: CertainLach#3

At the same time though, I must say that 0.19 seems to work a bit worse for me, startup times are longer despite already pre-compiled/filled caches. Also sometimes during startup some background worker seems to die after waiting for something else. But I have no reason to believe this has anything to do with our PR here, nothing in the logs points to it. Rather some new memory profiling features or similar. Also sometimes doesn't react well to clean SIGTERM in my llama-swap setup.

So I'll have to dig around a bit deeper to see, for my personal setup and use.

[stefanboca]

For mistral-common, I've opened #512667. vllm 0.19.0 depends on mistral-common>=1.10.0, and It looks like vllm 0.20.0 is about to be release which depends on mistral-common>=1.11.0.

@d-goldin @CertainLach might I suggest using pythonRemoveDeps instead of patching the requirements files? see here. ex:

  pythonRemoveDeps = [
    "flashinfer-cubin"
    "nvidia-cudnn-frontend"

    # QuACK and Cutlass DSL seem to be added only for FA4
    # which in our case handles its own deps
    "nvidia-cutlass-dsl"
    "quack-kernels"
  ];

[CertainLach]

Oof. The HEAD of this branch is correct for vllm deployment, and I have also included mistral-common as a parent for this request, but github doesn't seem to like to see this pointed to staging instead of master, and we can't point it to master since we have opentelemetry changes unmerged. Thus lots of unnecessary commits

Should I point it to master for now?.. It should be pointed to master anyway, the only reason it is being pointed to staging is that we have opentelemetry changes here, and those do require staging

What a mess...

This PR being pointed to master is not correct, but since it is stacked, pointing it to staging makes github very not happy and it might call lots of reviewers here. Should I mark it as a draft until we sort out the situation with python opentelemetry packages?

Review dismissed automatically

[d-goldin]

@d-goldin @CertainLach might I suggest using pythonRemoveDeps instead of patching the requirements files? see here. ex:

  pythonRemoveDeps = [
    "flashinfer-cubin"
    "nvidia-cudnn-frontend"

    # QuACK and Cutlass DSL seem to be added only for FA4
    # which in our case handles its own deps
    "nvidia-cutlass-dsl"
    "quack-kernels"
  ];

Will try to switch to it. Not sure why I did it the other way around, maybe it didn't work with the structure they have with the requirements/* folder.

Edit: Tried, seems to work fine. So yeah, cleaner to do it that way. Will add a commit later.

[leo60228]

For mistral-common, I've opened #512667. vllm 0.19.0 depends on mistral-common>=1.10.0, and It looks like vllm 0.20.0 is about to be release which depends on mistral-common>=1.11.0.

It seems like vllm 0.16.0 already has this dependency. I have the same issue on nixos-unstable.

[peperunas]

Is there anything I can help with to speed-up the PR merge?

[CertainLach]

Is there anything I can help with to speed-up the PR merge?

This PR merging is blocked on opentelemetry python packages update anyway, and I have no idea what to do with them