[ROCm] Enable bitsandbytes quantization support on ROCm

[Abdennacer-Badaoui]

Description:

Summary

• Enable bitsandbytes quantization on ROCm GPUs (including gfx9 architectures)
• This is made possible by the upstream bitsandbytes PR that enables blocksize=64 for 4bit quantization on ROCm: Add blocksize=64 4-bit quantization support for ROCm CDNA (warp64) GPUs bitsandbytes-foundation/bitsandbytes#1856
• Remove the gfx9 warp size 64 limitation guard in vllm/platforms/rocm.py

Test plan

• pytest tests/models/test_transformers.py::test_quantization passes locally on MI325X (gfx942)

[mergify]

Documentation preview: https://vllm--34688.org.readthedocs.build/en/34688/

[gemini-code-assist]

Code Review

This pull request enables bitsandbytes quantization support on ROCm by updating the bitsandbytes dependency to a version that supports it, removing test skips, and adjusting version checks in the code. The changes look good and are consistent with the goal of the PR. I've identified one area for improvement regarding code duplication in the version checking logic, which should be refactored to improve maintainability.

[gemini-code-assist]

This version check logic is duplicated from BitsAndBytesLinearMethod.__init__ (lines 186-200). To improve maintainability and avoid future inconsistencies, consider extracting this logic into a shared helper function at the module level.

For example:

def _check_bitsandbytes_version():
    min_version = "0.49.2" if current_platform.is_rocm() else "0.46.1"
    try:
        import bitsandbytes
        from packaging import version

        if version.parse(bitsandbytes.__version__) < version.parse(min_version):
            raise ImportError(
                "bitsandbytes version is wrong. Please "
                f"install bitsandbytes>={min_version}."
            )
    except ImportError as err:
        raise ImportError(
            f"Please install bitsandbytes>={min_version} via "
            f"`pip install bitsandbytes>={min_version}` to use "
            "bitsandbytes quantizer."
        ) from err

Then both __init__ methods can simply call _check_bitsandbytes_version().

[hmellor]

I like Gemini's suggestion, could you extract the check to a method as described in its review comment?

[Abdennacer-Badaoui]

Yes of course :)

[hmellor]

cc @AndreasKaratzas can you confirm if any of the AMD failures are caused by this PR or if they already existed?

[Titus-von-Koeller]

Hey all, thanks to everyone!

Lgtm for me as well.

[AndreasKaratzas]

cc @AndreasKaratzas can you confirm if any of the AMD failures are caused by this PR or if they already existed?

Sry for delay. Will look try and into it today.

[AndreasKaratzas]

@Abdennacer-Badaoui Could you please rebase before I take a look into AMD CI failures?

[AndreasKaratzas]

Also, can we add a test that runs a bits and bytes model if bits and bytes package is found? This will immediately give us a feedback regarding correctness of bitsandbytes on ROCm.

Also, lets add the package requirement on rocm-test.txt as well in this PR.

EDIT: Oops missed the transformers test there with bitsandbytes. Do you think we need any other bitsandbytes correctness test?

[hmellor]

Could we add this as a parametrisation to the test_quantization test instead of creating a new one? Then we have a case for online quantisation and pre-quantised for bnb

[Abdennacer-Badaoui]

Yes, it would be cleaner. Thanks

[Abdennacer-Badaoui]

@AndreasKaratzas
test_quantization (in the transformers tests) now covers both inflight quantization and pre-quantized 4-bit checkpoints for bitsandbytes, comparing auto vs transformers backends for logprob consistency. I think this gives us a good coverage for now.

[AndreasKaratzas]

This appears to have been done already

@hmellor Version is there, but not pinned.

[hmellor]

This appears to have been done already

@hmellor Version is there, but not pinned.

Oh my mistake, I misunderstood what you meant

[dosubot]

Related Documentation

Checked 0 published document(s) in 1 knowledge base(s). No updates required.

^{How did I do? Any feedback?}