miss update

[Joluck]

I’m very happy to share that MiSS has been accepted to ICLR 2026. Here are a few small updates.

[BenjaminBossan]

Thanks for updating the MiSS paper and examples. I have two small comments, please check.

[BenjaminBossan]

I think adding dropout of 0.01 here can be confusing. It may appear like that is the recommended setting, but skimming the paper, this doesn't appear to be the case.

[Joluck]

We observe that many papers do not apply dropout to MiSS, while LoRA variants typically use dropout, which is not a fair comparison. We would like to clarify that MiSS also supports a dropout parameter.

[BenjaminBossan]

I see. I think that's fair, I'd just not put 0.01 here, which I assume has no noticeable effect. Is there a rate you'd recommend as it works best?

[Joluck]

Yes, 0.01 is not necessarily the optimal setting, but researchers often use it as a default. However, it can be effective in preventing overfitting in RL training.

By the way, is the official PEFT arena still being updated? Currently, there is no unified training and evaluation setup for fair comparison, so I believe maintaining a well-designed PEFT arena is important. A good benchmark can significantly promote the development of PEFT methods. I’d be happy to help if needed.

[BenjaminBossan]

Yes, 0.01 is not necessarily the optimal setting, but researchers often use it as a default. However, it can be effective in preventing overfitting in RL training.

I was not aware.

By the way, is the official PEFT arena still being updated?

Yes. We recently merged some new PEFT methods and will rerun it sometime soon. But generally, unless we have any indication that something important changed (say, a bug in PEFT that affects performance), we won't rerun the old experiments, just the new ones.

Currently, there is no unified training and evaluation setup for fair comparison, so I believe maintaining a well-designed PEFT arena is important. A good benchmark can significantly promote the development of PEFT methods. I’d be happy to help if needed.

We see it the same way and definitely want to expand our support in that area. Not sure if you saw that, but we're currently adding an image generation (#3082) and a reinforcement learning (#3078) benchmark. We always welcome help with that. Easiest is to contribute new experiments, but other forms of contribution are also welcome (ideally discussed with us first to be aligned).

Besides that, we added a feature to convert non-LoRA adapaters into LoRA adapters (yet unreleased, see https://huggingface.co/docs/peft/main/en/package_reference/lora_conversion). This should allow methods like MiSS to be more easily adapted in downstream packages that only support LoRA, like Diffusers or vLLM. The conversion is not perfect, it's lossy (using SVD) and right now, only supports "easy conversion" targets of type: W' = W_0 + dW (so bat works but mini doesn't). This is also an area that we welcome support.

[Joluck]

Yes. We recently merged some new PEFT methods and will rerun it sometime soon. But generally, unless we have any indication that something important changed (say, a bug in PEFT that affects performance), we won't rerun the old experiments, just the new ones.

My intention is not to rerun the experiments, but rather to build a more complete PEFT pipeline.

We see it the same way and definitely want to expand our support in that area. Not sure if you saw that, but we're currently adding an image generation (#3082) and a reinforcement learning (#3078) benchmark. We always welcome help with that. Easiest is to contribute new experiments, but other forms of contribution are also welcome (ideally discussed with us first to be aligned).

^^

Besides that, we added a feature to convert non-LoRA adapaters into LoRA adapters (yet unreleased, see https://huggingface.co/docs/peft/main/en/package_reference/lora_conversion). This should allow methods like MiSS to be more easily adapted in downstream packages that only support LoRA, like Diffusers or vLLM. The conversion is not perfect, it's lossy (using SVD) and right now, only supports "easy conversion" targets of type: W' = W_0 + dW (so bat works but mini doesn't). This is also an area that we welcome support.

MiSS can be converted into the LoRA form without any loss. I will add this section to the paper as soon as possible. Meanwhile, I would also like to integrate it into PEFT—how should I proceed?

[BenjaminBossan]

My intention is not to rerun the experiments, but rather to build a more complete PEFT pipeline.

If you have a suggestion, LMK. Ideally, let's open a new issue for better visibility.

MiSS can be converted into the LoRA form without any loss. I will add this section to the paper as soon as possible. Meanwhile, I would also like to integrate it into PEFT—how should I proceed?

If exact conversion is possible, we definitely want to support that. The main logic lives here:

peft/src/peft/tuners/lora/conversion.py

Line 81 in 2d48882

def convert_to_lora(

I think we could add a check for the PEFT type and then, instead of running this loop:

peft/src/peft/tuners/lora/conversion.py

Lines 234 to 269 in 2d48882

	state_dict = {}
	for name, module in tqdm(
	model.named_modules(), disable=not progressbar, desc="Converting to LoRA", total=num_modules_total
	):
	if not isinstance(module, BaseTunerLayer):
	continue
	if not hasattr(module, "get_delta_weight"):
	# if we arrive here, it means that the layer actually does not support LoRA conversion, which should not
	# happen
	raise TypeError(
	f"Module of type {type(module)} does not have a get_delta_weight method, which is required for "
	"conversion. Please open an issue: https://github.com/huggingface/peft/issues"
	)
	lora_A, lora_B, effective_rank = convert_module_to_lora(module, rank=rank, adapter_name=adapter_name)
	if effective_rank == 0:
	# This shouldn't really happen, as we ensure that the rank is greater than 0 (int) or, for thresholds
	# (float), at least one SV is included. But better be safe than sorry, as, in principle, it is fine to
	# exclude some layers. Also makes this more future proof.
	lora_config.exclude_modules.add(name.removeprefix(peft_prefix))
	continue
	# the rank was dynamically adjusted, store it in rank and alpha pattern
	if (effective_rank != rank) or isinstance(lora_config.target_modules, str):
	# we need to add an entry to rank/alpha pattern iff:
	# 1) The effective rank differs from the general rank
	# 2) target modules is a string, as we cannot simply append the name to target_modules regex
	lora_config.rank_pattern[name.removeprefix(peft_prefix)] = effective_rank
	lora_config.alpha_pattern[name.removeprefix(peft_prefix)] = effective_rank
	else:
	# effective rank is the same and target_modules are a set, just add the name
	lora_config.target_modules.add(name.removeprefix(peft_prefix))
	# don't include adapter_name in key
	state_dict[f"{name}.lora_A.weight"] = lora_A
	state_dict[f"{name}.lora_B.weight"] = lora_B

we dispatch to a MiSS-specific function for exact conversion. If you want, feel free to start a (draft) PR and I'll help you with the implementation details.

[Joluck]

ty

[BenjaminBossan]

I see. I think that's fair, I'd just not put 0.01 here, which I assume has no noticeable effect. Is there a rate you'd recommend as it works best?

[BenjaminBossan]

Thanks for the additional info. PR LGTM.

[BenjaminBossan]

Yes, 0.01 is not necessarily the optimal setting, but researchers often use it as a default. However, it can be effective in preventing overfitting in RL training.

I was not aware.

By the way, is the official PEFT arena still being updated?

Yes. We recently merged some new PEFT methods and will rerun it sometime soon. But generally, unless we have any indication that something important changed (say, a bug in PEFT that affects performance), we won't rerun the old experiments, just the new ones.

Currently, there is no unified training and evaluation setup for fair comparison, so I believe maintaining a well-designed PEFT arena is important. A good benchmark can significantly promote the development of PEFT methods. I’d be happy to help if needed.

We see it the same way and definitely want to expand our support in that area. Not sure if you saw that, but we're currently adding an image generation (#3082) and a reinforcement learning (#3078) benchmark. We always welcome help with that. Easiest is to contribute new experiments, but other forms of contribution are also welcome (ideally discussed with us first to be aligned).

Besides that, we added a feature to convert non-LoRA adapaters into LoRA adapters (yet unreleased, see https://huggingface.co/docs/peft/main/en/package_reference/lora_conversion). This should allow methods like MiSS to be more easily adapted in downstream packages that only support LoRA, like Diffusers or vLLM. The conversion is not perfect, it's lossy (using SVD) and right now, only supports "easy conversion" targets of type: W' = W_0 + dW (so bat works but mini doesn't). This is also an area that we welcome support.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.