Add GemLite quantization backend support for pre-quantized checkpoints

[gabe-engineers]

What does this PR do?

This PR adds GemLite as a quantization backend in Diffusers. It's scoped to only loading already pre-quantized checkpoints.

GemLite provides Triton kernels for low-bit matrix operations and supports packed low-bit weight formats, including binary and ternary weights.

The integration enables Diffusers models with pre-quantized GemLite weights to be loaded and executed directly through the standard Diffusers quantization interface.

Motivation

Several low-bit image-generation models, including the binary and ternary variants of Bonsai Image, distribute their weights using GemLite’s packed representation.

Diffusers currently cannot load these checkpoints directly. Users must instead rely on custom model-loading code or maintain separate forks and patches.

With this backend, models using GemLite low-bit packing can be loaded and run through Diffusers without model-specific loading logic. For the Bonsai Image series of models this PR will not be sufficient but it would be a part 1 of 2. With it you will be able to load the transformer checkpoints which are packed in GemLite format, however the encoder is quantized and packed with HQQ, and cannot be loaded yet (a good follow up but wanted to keep the PR scoped).

Fixes partially #13925 missing working HQQ support afterwards

Last self-review report:

## Self-review — NEEDS CHANGES

  ### Blocking issues

  1. GemLite 0.6.0 is not installable through the configured public install path. src/diffusers/quantizers/gemlite/gemlite_quantizer.py:23 requires gemlite>=0.6.0, while
     nightly installs uv pip install -U gemlite at .github/workflows/nightly_tests.yml:379. PyPI currently serves 0.5.1.post1, despite an upstream v0.6.0 GitHub release.
     Consequently, normal users and nightly CI install 0.5.1, then fail validation; most functional tests are skipped by their >=0.6.0 guard. PyPI
     (https://pypi.org/project/gemlite/), [v0.6.0 release](https://github.com/dropbox/gemlite/releases) (https://github.com/dropbox/gemlite/releases)

     Fix by publishing 0.6.0 to PyPI, or temporarily install the pinned upstream tag in CI and document that same installation route.

  ### Non-blocking issues

  None found.

  ### Dead code advisory

   Path                                                         Status    Reason
  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
   src/diffusers/quantizers/gemlite/gemlite_quantizer.py:26     Used      Applies modules_to_not_convert during replacement.
  ───────────────────────────────────────────────────────────  ────────  ────────────────────────────────────────────────────────────────────────────────────
   src/diffusers/quantizers/gemlite/gemlite_quantizer.py:30     Used      Normalizes loader device-map values before tensor placement.
  ───────────────────────────────────────────────────────────  ────────  ────────────────────────────────────────────────────────────────────────────────────
   src/diffusers/quantizers/gemlite/gemlite_quantizer.py:301    Used      Called from ModelMixin.save_pretrained to preserve GemLite serialization metadata.

  ### Suggestions / additional info

  - Add a persisted pre-quantized checkpoint → from_pretrained → forward/save/reload test. Current tests exercise the pieces, but not the complete new loader/finalization
    ordering.

  - Consider adding a short GemLite loading example to the quantization docs once the install path is settled.

The blocking issue, turned out to be a stale metadata result from codex's search tool. The needed gemlite 0.6.0 version is live in PyPI

I took both suggestions since they were good ones, added 2 end to end tests one with the Krea2 transformer and another with mini-flux's pipeline. Then I added an example to the documentation, I tested the example with a L40 GPU and verified the image. One note on the example is we are using a bonsai model with an unpacked encoder, this is because the encoder is quantized and packed using HQQ, and the HQQ support seems to be broken in transformers (huggingface/transformers#45147) and not available yet natively in diffusers (again a good follow up perhaps, but I wanted to keep this PR scoped).

Testing

• Ran the gemlite tests with RUN_NIGHTLY=1 python -m pytest tests/quantization/gemlite/test_gemlite.py. Output here.

• Also manually ran inference with:

import torch
from diffusers import DiffusionPipeline

model_id = "gabe-engineers/bonsai-image-ternary-4B-gemlite-2bit-unpacked-encoder" # Unpacked encoder here because diffusers cant load an HQQ checkpoint yet

pipe = DiffusionPipeline.from_pretrained(
    model_id,
    dtype=torch.float16,
    device_map="cuda",
)
image = pipe(
    prompt="A bonsai tree in a quiet ceramic studio, soft morning light",
    height=1024,
    width=1024,
    num_inference_steps=4,
    guidance_scale=1.0,
).images[0]
image.save("bonsai-gemlite.png")

Resulted in this image

Before submitting

• Did you use an AI agent (Claude Code, Codex, Cursor, etc.) to help with this PR? If so:
  • Did you read the Coding with AI agents guide?
  • Did you run the self-review skill on the diff?
  • Did you share the final self-review notes in the PR description or a comment?
• Did you read the contributor guideline?
• Did you read our philosophy doc? (important for complex PRs)
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
  • It was not, there is a stale open issue here (1 bit, ternary image generation support #13925) but decide to just bias for action and open the PR to generate discussion. Open to changes and more discussions of course.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?
• Are you the author (or part of the team) of the model/pipeline (only applicable for model/pipeline related PRs)?

Who can review?

Primary: @sayakpaul
Secondary: @DN6
Optional: @mobicham

[mobicham]

Don't do this manually, the helper functions do all of that and manage some edge cases. Also using the helper functions makes it easier to upgrade without breaking things.
You can simply use: https://github.com/dropbox/gemlite/blob/master/gemlite/helper.py#L35