Raise on quantization_config + already-instantiated model in DPOTrainer/ KTOTrainer

[qgallouedec]

DPOTrainer accepts a quantization_config argument that is only meaningful when the model is loaded from a model identifier (str). That's the only point where from_pretrained runs and can apply it. When the policy is passed already instantiated, the config can't be applied to it, and the trainer only logged a warning saying it would be ignored.

But the warning didn't tell the whole story. The reference model is auto-created from self.model.config via from_pretrained, and that path did inject quantization_config. So in this scenario you silently ended up with:

• policy → full precision (config "ignored", as warned)
• reference → 4/8-bit (config quietly applied)

Reproducer (previous behavior)

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from trl import DPOConfig, DPOTrainer
from datasets import load_dataset

# User builds the policy themselves (full precision) ...
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2-0.5B-Instruct")

dataset = load_dataset("trl-internal-testing/zen", "standard_preference", split="train")

# ... and also passes a quantization_config, expecting it to be ignored (as the warning says).
trainer = DPOTrainer(
    model=model,
    train_dataset=dataset,
    quantization_config=BitsAndBytesConfig(load_in_4bit=True),
)

[RANK 0] You passed `quantization_config` to the trainer, but your model is already instantiated. The `quantization_config` will be ignored.

What this PR does

Turn the silent, misleading warning into a hard error: passing quantization_config with an already-instantiated model is a contradiction, so reject it up front.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    trainer = DPOTrainer(
        model=model,
    ...<2 lines>...
        quantization_config=BitsAndBytesConfig(load_in_4bit=True),
    )
  File "/fsx/qgallouedec/trl/trl/trainer/dpo_trainer.py", line 605, in __init__
    raise ValueError(
    ...<3 lines>...
    )
ValueError: You passed `quantization_config` to the trainer, but your model is already instantiated. Quantization can only be applied when the model is loaded from a model identifier (`str`). Either pass the model as a model identifier, or omit `quantization_config`.

---

Note

Low Risk
Small validation change at trainer init; may break callers who relied on the old warn-and-continue behavior, but prevents incorrect mixed-precision DPO/KTO runs.

Overview
DPOTrainer and KTOTrainer now fail at construction with a ValueError if you pass quantization_config while model is already a loaded PreTrainedModel, instead of only logging that the config will be ignored on the policy.

That closes a footgun where the warning suggested quantization was fully ignored, but the auto-loaded reference model could still be created with quantization_config, leaving policy full-precision and reference 4/8-bit. The error tells callers to pass a model id string (so both loads can quantize consistently) or drop quantization_config.

^{Reviewed by Cursor Bugbot for commit 012735f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 80a6dab06d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.