Fix lora_only bias serialization

[loginov-kirill]

I noticed a bug when using LoraConfig(bias="lora_only"). Even though the base layer's bias is correctly marked as trainable during training, it gets dropped when calling get_peft_model_state_dict() or save_pretrained().

This happens because the bias parameter of the wrapped layer is named {prefix}.{module_name}.base_layer.bias in the state dict. However, the serialization code in get_peft_model_state_dict() splits the LoRA weight key on "lora_" and appends "bias", looking for {prefix}.{module_name}.bias. Since it doesn't find this key, it skips the bias completely.

This PR adds a fallback check for {prefix}.{module_name}.base_layer.bias in the bias == "lora_only" logic so that these trained biases are saved correctly.

Fixes #3306

[BenjaminBossan]

Thanks for tackling that issue. For us to be sure that this fixes the bug and to prevent future regressions, we need to add a unit test though.

I checked if any of the existing tests can be extended, and indeed we have test_parameters_after_loading_model here:

peft/tests/test_custom_models.py

Line 2613 in 036abd2

def test_parameters_after_loading_model(self, test_name, model_id, config_cls, config_kwargs):

Here we could either 1) create a similar test and check LoRA with bias="lora_only" or 2) extend the existing TEST_CASES (this would need to be for this test only, as other tests won't work with bias enabled).

[BenjaminBossan]

Let's add a comment to explain why this name needs to be checked.

[BenjaminBossan]

Oh, I just saw that we already have a PR for this, #3307.