Can you load LoRAs with lmd, or lmd_plus?

[KristianMischke]

I've been loving this tool but have been wanting to use it with some LoRAs I've created.

Is it possible to:

• load LoRAs based on runwayml/stable-diffusion-v1-5 when generating with lmd?
• load LoRAs based on CompVis/stable-diffusion-v1-4 when generating with lmd_plus?
• finetune or create LoRAs based on longlian/lmd_plus?
  • I tried to base off of it using https://github.com/bmaltais/kohya_ss to train, but get an issue:

  Unexpected key(s) in state_dict: "position_net.null_positive_feature", "position_net.null_position_feature", 
  "position_net.linears.0.weight", "position_net.linears.0.bias", "position_net.linears.2.weight", "position_net.linears.2.bias", 
  "position_net.linears.4.weight", "position_net.linears.4.bias", "down_blocks.0.attentions.0.transformer_blocks.0.fuser.alpha_attn", 
  "down_blocks.0.attentions.0.transformer_blocks.0.fuser.alpha_dense", 
  "down_blocks.0.attentions.0.transformer_blocks.0.fuser.linear.weight", 
  "down_blocks.0.attentions.0.transformer_blocks.0.fuser.linear.bias", 
  "down_blocks.0.attentions.0.transformer_blocks.0.fuser.attn.to_q.weight", 
  "down_blocks.0.attentions.0.transformer_blocks.0.fuser.attn.to_k.weight", 
  "down_blocks.0.attentions.0.transformer_blocks.0.fuser.attn.to_v.weight", 
  "down_blocks.0.attentions.0.transformer_blocks.0.fuser.attn.to_out.0.weight", 
  "down_blocks.0.attentions.0.transformer_blocks.0.fuser.attn.to_out.0.bias", 
  "down_blocks.0.attentions.0.transformer_blocks.0.fuser.ff.net.0.proj.weight", 
  "down_blocks.0.attentions.0.transformer_blocks.0.fuser.ff.net.0.proj.bias", 
  "down_blocks.0.attentions.0.transformer_blocks.0.fuser.ff.net.2.weight", 
  "down_blocks.0.attentions.0.transformer_blocks.0.fuser.ff.net.2.bias", 
  "down_blocks.0.attentions.0.transformer_blocks.0.fuser.norm1.weight", 
  "down_blocks.0.attentions.0.transformer_blocks.0.fuser.norm1.bias"
  ...
  

  It seems the attentions from GLIGEN are messing up the training
• So far with my attempts at loading a LoRA for lmd_plus with the pipeline on huggingface https://github.com/huggingface/diffusers/blob/main/examples/community/README.md#llm-grounded-diffusion have been unsuccessful. I get 'UNet2DConditionModel' object has no attribute 'attn_processors' after I load LoRAs on the pipeline. It seems that loading a LoRA clears our the attention processors from GLIGEN which causes issues down stream. Is there a way to preserve them? Does that even make sense, I'm not super familiar with how the GLIGEN attention processors work and if updating the unet layers with LoRAs would mess up the attentions.

So I guess is it possible? and are there any examples of it that I can reference? any additional help would be appreciated