Skip to content

Use _cast_input_dtype in FourierFTLinear.forward#3170

Open
Chessing234 wants to merge 1 commit intohuggingface:mainfrom
Chessing234:fix/fourierft-forward-cast-input-dtype
Open

Use _cast_input_dtype in FourierFTLinear.forward#3170
Chessing234 wants to merge 1 commit intohuggingface:mainfrom
Chessing234:fix/fourierft-forward-cast-input-dtype

Conversation

@Chessing234
Copy link
Copy Markdown
Contributor

Bug

FourierFTLinear.forward in src/peft/tuners/fourierft/layer.py casts the input to the adapter's dtype with a raw x.to(delta_w.dtype):

for active_adapter in self.active_adapters:
    if active_adapter not in self.fourierft_spectrum.keys():
        continue

    delta_w = self.get_delta_weight(active_adapter)
    x = x.to(delta_w.dtype)                        # <- raw cast
    result = result + F.linear(x, delta_w)

This bypasses BaseTunerLayer._cast_input_dtype, which is the method that consults cast_input_dtype_enabled (toggled by the peft.helpers.disable_lora_input_dtype_casting context manager introduced in #2353 and generalized to non-LoRA tuners in #2433).

Root cause

#2433 consolidated dtype handling across additive tuners onto self._cast_input_dtype. Every comparable tuner's forward uses the helper:

  • lora/layer.py:967x = self._cast_input_dtype(x, lora_A.weight.dtype)
  • waveft/layer.py:283x = self._cast_input_dtype(x, delta_w.dtype)
  • hra/layer.py:260, 445x = self._cast_input_dtype(x, new_weight.dtype)
  • psoft/layer.py:447x_cast = self._cast_input_dtype(x, A.dtype)

FourierFT was missed in that sweep, so toggling disable_lora_input_dtype_casting silently has no effect on FourierFT layers — the cast still happens unconditionally.

Fix

Route the cast through self._cast_input_dtype to match the established pattern. Behavior is identical when the casting context manager is not used (both call x.to(dtype)); the difference is that users can now opt out as documented.

FourierFTLinear.forward casts the input with a raw 'x.to(delta_w.dtype)'
call, which bypasses BaseTunerLayer._cast_input_dtype and therefore
ignores the 'disable_input_dtype_casting' context manager introduced in
huggingface#2353 / huggingface#2433.

All other comparable additive tuners (LoRA layer.py:967, WaveFT
layer.py:283, HRA layer.py:260/445, pSoFT layer.py:447) route the cast
through self._cast_input_dtype so users can opt out globally. FourierFT
was missed when huggingface#2433 consolidated that helper.

Swap the raw cast for self._cast_input_dtype(x, delta_w.dtype) to match
the established pattern and restore the intended opt-out behavior.
Chessing234 added a commit to Chessing234/peft that referenced this pull request Apr 17, 2026
`vera/layer.py` Linear.forward casts the input to the adapter's dtype
with a raw `x.to(lambda_d.dtype)` call, bypassing
`BaseTunerLayer._cast_input_dtype` — the helper that consults
`cast_input_dtype_enabled` (toggled by
`peft.helpers.disable_lora_input_dtype_casting`, introduced in huggingface#2353
and generalized to non-LoRA tuners in huggingface#2433).

Every other additive tuner routes the cast through `self._cast_input_dtype`:

- `lora/layer.py:967` — `x = self._cast_input_dtype(x, lora_A.weight.dtype)`
- `fourierft/layer.py:183` (after huggingface#3170)
- `waveft/layer.py:283` — `x = self._cast_input_dtype(x, delta_w.dtype)`
- `psoft/layer.py:447` — `x_cast = self._cast_input_dtype(x, A.dtype)`
- `hra/layer.py:260,445` — `x = self._cast_input_dtype(x, new_weight.dtype)`

VeRA was missed when huggingface#2433 consolidated that helper. Switch the raw
cast to `self._cast_input_dtype` to match the established pattern so
users can opt out via the documented context manager. Behavior is
unchanged when the context manager isn't used (both code paths call
`x.to(dtype)`).
@BenjaminBossan
Copy link
Copy Markdown
Member

@Chessing234 Thanks for the PR, the change is correct. Instead of opening a separate PR for each PEFT method, could you please group all changes into a single PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants