[BUG] peft 0.19 diffusers LoRA Index Error.

[galaxygliese]

System Info

I tried SDXL LoRA, but the following error appeared. It didn't appear until upgrade the peft. I trained my model before upgrading the peft.

    r = lora_alpha = list(rank_dict.values())[0]
                     ~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

My env:

diffusers==0.38.0
transformers==5.8.0
peft==0.19.1

For SD3.5, the following warning showed up.

Key                                                                           | Status     | 
------------------------------------------------------------------------------+------------+-
encoder.layers.{0...11}.self_attn.k_proj.lora_B.default_0.weight              | UNEXPECTED | 
encoder.layers.{0...11}.self_attn.k_proj.lora_A.default_0.weight              | UNEXPECTED | 
encoder.layers.{0...11}.self_attn.out_proj.lora_A.default_0.weight            | UNEXPECTED | 
encoder.layers.{0...11}.self_attn.v_proj.lora_A.default_0.weight              | UNEXPECTED | 
encoder.layers.{0...11}.self_attn.out_proj.lora_B.default_0.weight            | UNEXPECTED | 
encoder.layers.{0...11}.self_attn.q_proj.lora_A.default_0.weight              | UNEXPECTED | 
encoder.layers.{0...11}.self_attn.v_proj.lora_B.default_0.weight              | UNEXPECTED | 
encoder.layers.{0...11}.self_attn.q_proj.lora_B.default_0.weight              | UNEXPECTED | 
text_model.encoder.layers.{0...11}.self_attn.q_proj.lora_B.default_0.weight   | MISSING    | 
text_model.encoder.layers.{0...11}.self_attn.v_proj.lora_A.default_0.weight   | MISSING    | 
text_model.encoder.layers.{0...11}.self_attn.q_proj.lora_A.default_0.weight   | MISSING    | 
text_model.encoder.layers.{0...11}.self_attn.out_proj.lora_B.default_0.weight | MISSING    | 
text_model.encoder.layers.{0...11}.self_attn.k_proj.lora_A.default_0.weight   | MISSING    | 
text_model.encoder.layers.{0...11}.self_attn.k_proj.lora_B.default_0.weight   | MISSING    | 
text_model.encoder.layers.{0...11}.self_attn.out_proj.lora_A.default_0.weight | MISSING    | 
text_model.encoder.layers.{0...11}.self_attn.v_proj.lora_B.default_0.weight   | MISSING    | 

Who can help?

No response

Reproduction

The code:

model_id = "stabilityai/stable-diffusion-xl-base-1.0"
pipeline = AutoPipelineForInpainting.from_pretrained(
         model_id, torch_dtype=torch.float16, variant="fp16", use_safetensors=True
 )
pipeline.enable_model_cpu_offload()
pipeline.enable_xformers_memory_efficient_attention()
pipeline.load_lora_weights(args.lora_checkpoint, weight_name="pytorch_lora_weights.safetensors")

Expected behavior

Maybe the text encoder LoRA parameter keys have changed in the latest update.

[dparikh79]

Hi @galaxygliese, thanks for the report. Quick redirect: the actual failure path here is in diffusers, not in peft. The IndexError you saw is raised at src/diffusers/utils/peft_utils.py:158 in get_peft_kwargs, which crashes on list(rank_dict.values())[0] when the LoRA rank-discovery loop in the caller (e.g. _load_lora_into_text_encoder) does not find any {module}.lora_B.weight keys in the converted state_dict.

I opened huggingface/diffusers#13759 against diffusers to surface this as an actionable ValueError instead of a cryptic IndexError. The new message names the common underlying causes (missing/extra prefix, adapter-name infix, unconverted format) and includes a sample of the state_dict keys, so your specific SDXL state_dict shape will be visible in the error.

That PR is intentionally a defensive surface-area fix only; the root cause for SDXL/SD3.5 specifically (the text_model. prefix mismatch and the UNEXPECTED/MISSING keys in your second example) needs separate investigation against your exact state_dict. Once huggingface/diffusers#13759 lands, a follow-up issue against diffusers with the new error message will give the maintainers concrete data to triage from.

(Disclosure: analysis assisted by Claude; I am the human contributor.)

[BenjaminBossan]

@galaxygliese Thanks for reporting. It looks like the prefix is missing, as you can tell here:

Key                                                                           | Status     | 
------------------------------------------------------------------------------+------------+-
encoder.layers.{0...11}.self_attn.k_proj.lora_B.default_0.weight              | UNEXPECTED | 
[...]
text_model.encoder.layers.{0...11}.self_attn.q_proj.lora_B.default_0.weight   | MISSING    | 

Could you try passing prefix="text_model." to load_lora_weights and see if it fixes the issue? If not, could you please paste the full error message?

@dparikh79 You are right that it's a good idea to check for an empty state dict and raise a helpful error instead of just failing with an IndexError.

[galaxygliese]

@BenjaminBossan
Thank you for your message. I tried adding prefix="text_model." but still the same error appeared for both models.

For SDXL:

  File "generate.py", line 123, in main
    pipeline.load_lora_weights(args.lora_checkpoint, weight_name="pytorch_lora_weights.safetensors", prefix="text_model.")
  File "/home/user/.dev_venv/lib/python3.12/site-packages/diffusers/loaders/lora_pipeline.py", line 657, in load_lora_weights
    self.load_lora_into_text_encoder(
  File "/home/user/.dev_venv/lib/python3.12/site-packages/diffusers/loaders/lora_pipeline.py", line 901, in load_lora_into_text_encoder
    _load_lora_into_text_encoder(
  File "/home/user/.dev_venv/lib/python3.12/site-packages/diffusers/loaders/lora_base.py", line 390, in _load_lora_into_text_encoder
    lora_config = _create_lora_config(state_dict, network_alphas, metadata, rank, is_unet=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.dev_venv/lib/python3.12/site-packages/diffusers/utils/peft_utils.py", line 355, in _create_lora_config
    lora_config_kwargs = get_peft_kwargs(
                         ^^^^^^^^^^^^^^^^
  File "/home/user/.dev_venv/lib/python3.12/site-packages/diffusers/utils/peft_utils.py", line 158, in get_peft_kwargs
    r = lora_alpha = list(rank_dict.values())[0]
                     ~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

For SD3.5:

Notes:
- UNEXPECTED:	can be ignored when loading from different task/architecture; not ok if you expect identical arch.
- MISSING:	those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.
Loading weights: 0it [00:00, ?it/s]
[transformers] CLIPTextModelWithProjection LOAD REPORT from: None
Key                                                                           | Status     | 
------------------------------------------------------------------------------+------------+-
encoder.layers.{0...31}.self_attn.q_proj.lora_B.default_0.weight              | UNEXPECTED | 
encoder.layers.{0...31}.self_attn.out_proj.lora_B.default_0.weight            | UNEXPECTED | 
encoder.layers.{0...31}.self_attn.out_proj.lora_A.default_0.weight            | UNEXPECTED | 
encoder.layers.{0...31}.self_attn.v_proj.lora_B.default_0.weight              | UNEXPECTED | 
encoder.layers.{0...31}.self_attn.k_proj.lora_A.default_0.weight              | UNEXPECTED | 
encoder.layers.{0...31}.self_attn.q_proj.lora_A.default_0.weight              | UNEXPECTED | 
encoder.layers.{0...31}.self_attn.v_proj.lora_A.default_0.weight              | UNEXPECTED | 
encoder.layers.{0...31}.self_attn.k_proj.lora_B.default_0.weight              | UNEXPECTED | 
text_model.encoder.layers.{0...31}.self_attn.v_proj.lora_A.default_0.weight   | MISSING    | 
text_model.encoder.layers.{0...31}.self_attn.q_proj.lora_B.default_0.weight   | MISSING    | 
text_model.encoder.layers.{0...31}.self_attn.out_proj.lora_B.default_0.weight | MISSING    | 
text_model.encoder.layers.{0...31}.self_attn.k_proj.lora_A.default_0.weight   | MISSING    | 
text_model.encoder.layers.{0...31}.self_attn.k_proj.lora_B.default_0.weight   | MISSING    | 
text_model.encoder.layers.{0...31}.self_attn.v_proj.lora_B.default_0.weight   | MISSING    | 
text_model.encoder.layers.{0...31}.self_attn.q_proj.lora_A.default_0.weight   | MISSING    | 
text_model.encoder.layers.{0...31}.self_attn.out_proj.lora_A.default_0.weight | MISSING    | 

Notes:
- UNEXPECTED:	can be ignored when loading from different task/architecture; not ok if you expect identical arch.
- MISSING:	those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.

[BenjaminBossan]

Thanks for trying @galaxygliese. It's strange that passing the prefix seems to have no effect at all. For me to investigate further, I'd need have a full reproducer. That means the full code required to get to the error, as well as access to the LoRA checkpoint that causes the error (ideally uploaded to HF as a safetensors file).

[galaxygliese]

@BenjaminBossan
Hi, sorry for the late reply. I uploaded my test code here.
You can try the same error message with tiny-stable-diffusion-xl-pipe.

When I downgraded to peft==0.18.0 the following error appeared:

    from peft.utils.save_and_load import _maybe_shard_state_dict_for_tp
ImportError: cannot import name '_maybe_shard_state_dict_for_tp' from 'peft.utils.save_and_load' (/home/user/lib/python3.12/site-packages/peft/utils/save_and_load.py)

[BenjaminBossan]

@galaxygliese Thanks for posting a reproducer. I haven't run your example yet as it's quite involved. Since the error occurs during loading, wouldn't it be enough to have the loading code + a checkpoint file on HF to reproduce the error? I don't want to train a whole model each time I need to try something.

Independently of that, I worked in a PR that might actually solve your original issue: #3262. If you have the opportunity, could you please check out that branch and see if it resolves your problem?