What is the maximum value of lora weights?

[CaledoniaProject]

I have a standard pipeline,

pipe = DiffusionPipeline.from_pretrained("SG161222/RealVisXL_V4.0", torch_dtype=torch.float32)
pipe.load_lora_weights("working.safetensors", adapter_name="myTest", weight_name="myTest")
pipe.set_adapters(['myTest'], adapter_weights=[1.1])

My questions are,

1. Do I have to specify adapter_name and weight_name in load_lora_weights()?
2. What's the value range of adapter_weights?
3. When should I call pipe.fuse_lora(lora_scale=1.1)? and will lora_scale affect the previously specified weight?

[asomoza]

Hi, here's the answers:

1. no, if you don't specify one, it would just assign a default name, this will prevent you to be able to change the scale later though.
2. we don't have a range, usually a good lora works with 1.0 and you can lower the scale so the effect is less evident, there's also some loras commonly called "sliders" which can go as high as 10.0 and as low as -10, since they are trained to reduce or enhance certain details/effects. I remember I saw one that had a range like -30 and 30 but that's really not needed.
3. fuse_lora as the name says, it's for merging the lora with the model, this is needed only if you don't want to have it as a separate adapter, the scale you pass to that function is the scale on which the lora will be merged to the model, in this case, the scale used in set_adapters is not relevant.

[CaledoniaProject]

Hi @alozowski for question 2, how do I know the best recommended value range? In ComfyUI, some nodes have upper and bottom value limits.

[asomoza]

seems you're tagging another person by mistake.

how do I know the best recommended value range?

usually is 1.0 for the good loras, when they're overtrained or undertrained, usually the model owner posts in the page where you downloaded it the recommended settings. Probably you missed that since you have to also know and check the trigger words for the loras to work.

In ComfyUI, some nodes have upper and bottom value limits.

Those numbers are just whatever number the developer chose to put, there isn't a universal recommended range (for loras), this is also made because comfyui has the "ui sliders", developers are forced to use a range so it can display them, usually the UIs where you just type a number don't have the range restriction.

Just to be clear, ComfyUI doesn't have a custom node for each lora, is the same node with the same range for all of them.

[CaledoniaProject]

Thanks, I understood this part now. Just being curious, by it would just assign a default name, what are the default value of adapter_name, weight_name and adapter_weights?

[asomoza]

you can look into this function for the defaults.

I don't know where you got the adapter_weights but the other two are:

weight_name (`str`, *optional*, defaults to None):
                Name of the serialized state dict file.

adapter_name (`str`, *optional*):
                Adapter name to be used for referencing the loaded adapter model. If not specified, it will use
                `default_{i}` where i is the total number of adapters being loaded.

[CaledoniaProject]

also checked adapter_weights, the default value will be a list of None

[asomoza]

oh yes, I missed that since I always just pass the lists and don't use the keyword args, just to be complete with the answer for future references:

diffusers/src/diffusers/loaders/lora_base.py

Lines 659 to 662 in 5d2d239

	def set_adapters(
	self,
	adapter_names: Union[List[str], str],
	adapter_weights: Optional[Union[float, Dict, List[float], List[Dict]]] = None,

This is where the set_adapter function and default args can be read.