System Info

• transformers: 5.8.1
• Python: 3.14.0
• OS: macOS (reproduced locally; same class of failure reported with vLLM + Qwen3.5 merged HF checkpoints on Linux eval jobs)
• Model family: Qwen3_5TextConfig / Qwen3.5

Who can help?

@ArthurZucker @Cyrilvallez

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Description

RotaryEmbeddingConfigMixin.convert_rope_params_to_dict() raises a TypeError when ignore_keys_at_rope_validation is a list (e.g. deserialized from JSON in a config.json) because the union with {"partial_rotary_factor"} is performed without normalizing the operand to a set first:

TypeError: unsupported operand type(s) for |: 'list' and 'set'

In 5.8.1, modeling_rope_utils.py line 722:

self.ignore_keys_at_rope_validation = self.ignore_keys_at_rope_validation | {"partial_rotary_factor"}

ignore_keys_at_rope_validation is a class attribute on RotaryEmbeddingConfigMixin (set() by default; configs like Qwen3_5TextConfig set it to {"mrope_section", "mrope_interleaved"}). When a config.json contains this field as a JSON array, from_dict / __init__ sets it as an instance attribute (list), which shadows the class-level set. The next call to convert_rope_params_to_dict then evaluates list | set and crashes.

Reproduction

import transformers
from transformers import Qwen3_5TextConfig

print(f"transformers version: {transformers.__version__}")  # 5.8.1

cfg = Qwen3_5TextConfig.from_dict({
    "model_type": "qwen3_5_text",
    "vocab_size": 100,
    "hidden_size": 64,
    "num_hidden_layers": 2,
    "num_attention_heads": 2,
    "num_key_value_heads": 2,
    "ignore_keys_at_rope_validation": ["mrope_section", "mrope_interleaved"],  # list, as from JSON
    "partial_rotary_factor": 0.25,
    "rope_parameters": {"rope_type": "default", "rope_theta": 10_000_000},
})

print(type(cfg.ignore_keys_at_rope_validation).__name__)  # 'list' (shadows class-level set)

cfg.convert_rope_params_to_dict(partial_rotary_factor=0.25)

Traceback:

  File ".../transformers/modeling_rope_utils.py", line 722, in convert_rope_params_to_dict
    self.ignore_keys_at_rope_validation = self.ignore_keys_at_rope_validation | {"partial_rotary_factor"}
                                          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~
TypeError: unsupported operand type(s) for |: 'list' and 'set'

Expected behavior

convert_rope_params_to_dict should accept list or set (or any iterable of strings) and normalize before union.

Actual behavior

TypeError: unsupported operand type(s) for |: 'list' and 'set' whenever ignore_keys_at_rope_validation is a list (as JSON forces) and partial_rotary_factor is not None.

Downstream impact (vLLM / merged checkpoints)

We hit this in production when serving merged Qwen3.5 Hugging Face checkpoints with vLLM. LoRA merge / export tools (e.g. ms-swift) write ignore_keys_at_rope_validation into config.json:

"ignore_keys_at_rope_validation": ["mrope_section", "mrope_interleaved"]

JSON has no set type, so the field becomes a list at load time. During serving stack startup, config / RoPE initialization can hit convert_rope_params_to_dict with that list-typed instance attribute → TypeError → the server never becomes healthy and batch eval jobs fail before any inference.

This is a Transformers robustness issue (callers should not crash on list input); vLLM is the serving stack where we observe it. Current workarounds: strip ignore_keys_at_rope_validation from checkpoint JSON before vllm serve, or monkey-patch modeling_rope_utils.py to wrap with set() before |.

Suggested fix

Coerce to set before every | union involving ignore_keys_at_rope_validation. The single-line change in RotaryEmbeddingConfigMixin.convert_rope_params_to_dict covers the reported path.

I'm happy to open a PR against main that:

1. Adds the set(...) coercion on the union line in RotaryEmbeddingConfigMixin.convert_rope_params_to_dict (and any other union sites I find).
2. Adds a regression test covering JSON-list input for ignore_keys_at_rope_validation — both via direct call and via from_dict round-trip — so this can't silently regress again across refactors.

Just let me know if you'd prefer a different shape (e.g. normalizing in __setattr__, or hardening validate_rope directly) and I'll match that.