feat - Migrate to Transformers 5.5.4 (invoke-ai#9248)

* Update to Transformers 5.1.0

* remove extra stuff

* chore(deps): compel fork + transformers>=5.9.0 + remove override

Switches compel from PyPI 2.1.1 to invoke-ai/compel@main fork which supports
transformers 5.x. Bumps transformers floor to 5.9.0. Removes the
transformers>=5.1.0 uv override that was only needed to bypass compel 2.1.1's
<5.0 constraint.

NOTE: compel fork pulls notebook dep (full Jupyter stack); flag to maintainer for cleanup.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(z_image): resolve rope_theta from rope_parameters for transformers 5.x

transformers 5.x no longer exposes rope_theta as a top-level attribute on
Qwen3Config; the value is stored in the rope_parameters (and rope_scaling)
dict instead. Read it from there with a getattr fallback so the inv_freq
buffer is computed from the configured base (1e6 / 256) instead of raising
AttributeError. Applies to both the safetensors and GGUF Qwen3 encoder paths.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(model_manager): replace removed hf_hub get_token_permission with whoami

huggingface_hub 1.x removed get_token_permission(). HFTokenHelper.get_status()
now validates the token via whoami(), which returns user info for a valid token
and raises HfHubHTTPError for an invalid one. Preserves the original three-way
status: VALID on success, INVALID on HfHubHTTPError (e.g. 401), UNKNOWN on any
other error (e.g. network failure).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore(deps): regenerate uv.lock after upstream merge

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(sd3): resolve merge conflict marker, drop T5TokenizerFast

The upstream merge left an unresolved conflict marker in _t5_encode and
reintroduced T5TokenizerFast. Keep our v5 assertion (T5Tokenizer only) plus
upstream's new t5_device logic, and drop the now-dead T5TokenizerFast
monkeypatch in the test (the name no longer exists in the module).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* style: ruff fixes on merge-resolved files

- flux_text_encoder.py: drop unused typing.Union (F401) left by v5 import merge
- huggingface.py: ruff format (wrap append(SimpleNamespace(...)))

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore(deps): pin transformers <5.6 (diffusers single-file CLIP incompat)

transformers 5.6 flattened CLIPTextModel (removed the self.text_model wrapper,
hoisted embeddings/encoder/final_layer_norm to the top level). diffusers' single-file
checkpoint loader (create_diffusers_clip_model_from_ldm) still assumes the nested
layout, so loading SD1.5 .safetensors checkpoints fails on 5.6+ with
'CLIPTextModel object has no attribute text_model' and, once that read is shimmed,
'Cannot copy out of meta tensor' (weights never populate the flattened model).

Pin to >=5.5,<5.6 (last pre-flattening release) which keeps both the single-file
and from_pretrained paths working. The invoke-ai/compel fork accepts any 5.x.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* @
chore(deps): replace compel fork with official compel 2.4.0

compel 2.4.0 (released 2026-05-30) merges the transformers-5 support that
the invoke-ai fork carried (both descend from upstream PR invoke-ai#129), plus the
maintainer-reviewed padding rework and added diffusers/T5 smoke coverage.
Switch from the git fork to the PyPI release.

- pyproject: compel git+main -> compel>=2.4.0,<3
- uv.lock: compel 2.3.1 (git 8f404b45) -> 2.4.0 (pypi)
- transformers stays 5.5.4 (satisfies compel >=5,<6 and our <5.6 pin)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@

* chore(uv): update uv.lock

---------

Co-authored-by: Your Name <you@example.com>
Co-authored-by: 4pointoh <97913726+4pointoh@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Alexander Eichhorn <alex@eichhorn.dev>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>

Author: 6 people

invokeai/app/invocations/flux_text_encoder.py

@@ -1,8 +1,8 @@
 from contextlib import ExitStack
-from typing import Iterator, Literal, Optional, Tuple, Union
+from typing import Iterator, Literal, Optional, Tuple
 
 import torch
-from transformers import CLIPTextModel, CLIPTokenizer, T5EncoderModel, T5Tokenizer, T5TokenizerFast
+from transformers import CLIPTextModel, CLIPTokenizer, T5EncoderModel, T5Tokenizer
 
 from invokeai.app.invocations.baseinvocation import BaseInvocation, invocation
 from invokeai.app.invocations.fields import (
@@ -86,7 +86,7 @@ def _t5_encode(self, context: InvocationContext) -> torch.Tensor:
             ExitStack() as exit_stack,
         ):
             assert isinstance(t5_text_encoder, T5EncoderModel)
-            assert isinstance(t5_tokenizer, (T5Tokenizer, T5TokenizerFast))
+            assert isinstance(t5_tokenizer, T5Tokenizer)
 
             # Determine if the model is quantized.
             # If the model is quantized, then we need to apply the LoRA weights as sidecar layers. This results in
@@ -186,7 +186,7 @@ def _t5_lora_iterator(self, context: InvocationContext) -> Iterator[Tuple[ModelP
     def _log_t5_tokenization(
         self,
         context: InvocationContext,
-        tokenizer: Union[T5Tokenizer, T5TokenizerFast],
+        tokenizer: T5Tokenizer,
     ) -> None:
         """Logs the tokenization of a prompt for a T5-based model like FLUX."""
 

invokeai/app/invocations/sd3_text_encoder.py

@@ -8,7 +8,6 @@
     CLIPTokenizer,
     T5EncoderModel,
     T5Tokenizer,
-    T5TokenizerFast,
 )
 
 from invokeai.app.invocations.baseinvocation import BaseInvocation, invocation
@@ -103,7 +102,7 @@ def _t5_encode(self, context: InvocationContext, max_seq_len: int) -> torch.Tens
         ):
             context.util.signal_progress("Running T5 encoder")
             assert isinstance(t5_text_encoder, T5EncoderModel)
-            assert isinstance(t5_tokenizer, (T5Tokenizer, T5TokenizerFast))
+            assert isinstance(t5_tokenizer, T5Tokenizer)
             t5_device = get_effective_device(t5_text_encoder)
 
             text_inputs = t5_tokenizer(

invokeai/backend/image_util/safety_checker.py

@@ -9,7 +9,7 @@
 import numpy as np
 from diffusers.pipelines.stable_diffusion.safety_checker import StableDiffusionSafetyChecker
 from PIL import Image, ImageFilter
-from transformers import AutoFeatureExtractor
+from transformers import AutoImageProcessor
 
 import invokeai.backend.util.logging as logger
 from invokeai.app.services.config.config_default import get_config
@@ -36,14 +36,14 @@ def _load_safety_checker(cls):
         try:
             model_path = get_config().models_path / CHECKER_PATH
             if model_path.exists():
-                cls.feature_extractor = AutoFeatureExtractor.from_pretrained(model_path)
+                cls.feature_extractor = AutoImageProcessor.from_pretrained(model_path)
                 cls.safety_checker = StableDiffusionSafetyChecker.from_pretrained(model_path)
             else:
                 model_path.mkdir(parents=True, exist_ok=True)
-                cls.feature_extractor = AutoFeatureExtractor.from_pretrained(repo_id)
-                cls.feature_extractor.save_pretrained(model_path, safe_serialization=True)
+                cls.feature_extractor = AutoImageProcessor.from_pretrained(repo_id)
+                cls.feature_extractor.save_pretrained(model_path)
                 cls.safety_checker = StableDiffusionSafetyChecker.from_pretrained(repo_id)
-                cls.safety_checker.save_pretrained(model_path, safe_serialization=True)
+                cls.safety_checker.save_pretrained(model_path)
         except Exception as e:
             logger.warning(f"Could not load NSFW checker: {str(e)}")
 

invokeai/backend/model_manager/load/model_loaders/flux.py

@@ -13,7 +13,7 @@
     CLIPTextModel,
     CLIPTokenizer,
     T5EncoderModel,
-    T5TokenizerFast,
+    T5Tokenizer,
 )
 
 from invokeai.app.services.config.config_default import get_config
@@ -442,7 +442,7 @@ def _load_model(
             )
         match submodel_type:
             case SubModelType.Tokenizer2 | SubModelType.Tokenizer3:
-                return T5TokenizerFast.from_pretrained(
+                return T5Tokenizer.from_pretrained(
                     Path(config.path) / "tokenizer_2", max_length=512, local_files_only=True
                 )
             case SubModelType.TextEncoder2 | SubModelType.TextEncoder3:
@@ -470,8 +470,11 @@ def _load_state_dict_into_t5(cls, model: T5EncoderModel, state_dict: dict[str, t
         missing_keys, unexpected_keys = model.load_state_dict(state_dict, strict=False, assign=True)
         assert len(unexpected_keys) == 0
         assert set(missing_keys) == {"encoder.embed_tokens.weight"}
-        # Assert that the layers we expect to be shared are actually shared.
-        assert model.encoder.embed_tokens.weight is model.shared.weight
+        # Re-tie shared weights. In transformers 5.x, weight tying is implemented at the
+        # parameter level (via _tie_weights / tie_weights) rather than as a Python object
+        # alias.  load_state_dict(assign=True) replaces parameters in-place, which severs
+        # the parameter-level tie.  Calling tie_weights() re-establishes it.
+        model.tie_weights()
 
 
 @ModelLoaderRegistry.register(base=BaseModelType.Any, type=ModelType.T5Encoder, format=ModelFormat.T5Encoder)
@@ -488,7 +491,7 @@ def _load_model(
 
         match submodel_type:
             case SubModelType.Tokenizer2 | SubModelType.Tokenizer3:
-                return T5TokenizerFast.from_pretrained(
+                return T5Tokenizer.from_pretrained(
                     Path(config.path) / "tokenizer_2", max_length=512, local_files_only=True
                 )
             case SubModelType.TextEncoder2 | SubModelType.TextEncoder3:

invokeai/backend/model_manager/load/model_loaders/z_image.py

@@ -770,7 +770,13 @@ def _load_from_singlefile(
                 # For rotary embeddings, this is inv_freq which is computed from config
                 if buffer_name == "inv_freq":
                     # Compute inv_freq from config (same logic as Qwen3RotaryEmbedding.__init__)
-                    base = qwen_config.rope_theta
+                    # NB: transformers 5.x moved rope_theta into the rope_parameters/rope_scaling dict
+                    rope_params = (
+                        getattr(qwen_config, "rope_parameters", None)
+                        or getattr(qwen_config, "rope_scaling", None)
+                        or {}
+                    )
+                    base = rope_params.get("rope_theta") or getattr(qwen_config, "rope_theta", 1000000.0)
                     inv_freq = 1.0 / (base ** (torch.arange(0, head_dim, 2, dtype=torch.float32) / head_dim))
                     parent.register_buffer(buffer_name, inv_freq.to(model_dtype), persistent=False)
                 else:
@@ -986,7 +992,13 @@ def _load_from_gguf(
 
                 if buffer_name == "inv_freq":
                     # Compute inv_freq from config - keep on CPU, cache system will move to GPU as needed
-                    base = qwen_config.rope_theta
+                    # NB: transformers 5.x moved rope_theta into the rope_parameters/rope_scaling dict
+                    rope_params = (
+                        getattr(qwen_config, "rope_parameters", None)
+                        or getattr(qwen_config, "rope_scaling", None)
+                        or {}
+                    )
+                    base = rope_params.get("rope_theta") or getattr(qwen_config, "rope_theta", 1000000.0)
                     inv_freq = 1.0 / (base ** (torch.arange(0, head_dim, 2, dtype=torch.float32) / head_dim))
                     parent.register_buffer(buffer_name, inv_freq.to(dtype=compute_dtype), persistent=False)
                 else:

invokeai/backend/model_manager/load/model_util.py

@@ -10,7 +10,7 @@
 import torch
 from diffusers.pipelines.pipeline_utils import DiffusionPipeline
 from diffusers.schedulers.scheduling_utils import SchedulerMixin
-from transformers import CLIPTokenizer, PreTrainedTokenizerBase, T5Tokenizer, T5TokenizerFast
+from transformers import CLIPTokenizer, PreTrainedTokenizerBase, T5Tokenizer
 
 from invokeai.backend.image_util.depth_anything.depth_anything_pipeline import DepthAnythingPipeline
 from invokeai.backend.image_util.grounding_dino.grounding_dino_pipeline import GroundingDinoPipeline
@@ -64,10 +64,7 @@ def calc_model_size_by_data(logger: logging.Logger, model: AnyModel) -> int:
             return 0
     elif isinstance(
         model,
-        (
-            T5TokenizerFast,
-            T5Tokenizer,
-        ),
+        T5Tokenizer,
     ):
         # HACK(ryand): len(model) just returns the vocabulary size, so this is blatantly wrong. It should be small
         # relative to the text encoder that it's used with, so shouldn't matter too much, but we should fix this at some

invokeai/backend/model_manager/metadata/fetch/huggingface.py

@@ -16,10 +16,12 @@
 import json
 import re
 from pathlib import Path
+from types import SimpleNamespace
 from typing import Optional
 
 import requests
-from huggingface_hub import hf_hub_url
+from huggingface_hub import HfApi, hf_hub_url
+from huggingface_hub.errors import RepositoryNotFoundError, RevisionNotFoundError
 from pydantic.networks import AnyHttpUrl
 from requests.sessions import Session
 
@@ -46,28 +48,39 @@ def __init__(self, session: Optional[Session] = None):
         this module without an internet connection.
         """
         self._requests = session or requests.Session()
+        self._has_custom_session = session is not None
 
     @classmethod
     def from_json(cls, json: str) -> HuggingFaceMetadata:
         """Given the JSON representation of the metadata, return the corresponding Pydantic object."""
         metadata = HuggingFaceMetadata.model_validate_json(json)
         return metadata
 
-    def _fetch_model_info(self, repo_id: str, variant: Optional[ModelRepoVariant] = None) -> dict:
-        """Fetch model info from HuggingFace API using self._requests session.
-
-        This allows the session to be mocked in tests via requests_testadapter.
-        """
+    def _model_info_via_session(self, repo_id: str, variant: Optional[ModelRepoVariant] = None) -> SimpleNamespace:
+        """Fetch model info using the injected requests session (for testing/custom backends)."""
+        params = {"blobs": "true"}
         url = f"https://huggingface.co/api/models/{repo_id}"
-        params: dict[str, str] = {"blobs": "True"}
         if variant is not None:
-            params["revision"] = str(variant)
-
-        response = self._requests.get(url, params=params)
-        if response.status_code == 404:
-            raise UnknownMetadataException(f"'{repo_id}' not found.")
-        response.raise_for_status()
-        return response.json()
+            url += f"/revision/{variant}"
+        resp = self._requests.get(url, params=params)
+        if resp.status_code == 404:
+            error_code = resp.headers.get("X-Error-Code", "")
+            if error_code == "RevisionNotFound" or (variant is not None):
+                raise RevisionNotFoundError(f"Revision '{variant}' not found for repo '{repo_id}'.")
+            raise RepositoryNotFoundError(f"Repository '{repo_id}' not found.")
+        resp.raise_for_status()
+        data = resp.json()
+        # Convert siblings dicts to SimpleNamespace objects matching HfApi.model_info() shape
+        siblings = []
+        for s in data.get("siblings", []):
+            siblings.append(
+                SimpleNamespace(
+                    rfilename=s.get("rfilename"),
+                    size=s.get("size") or (s.get("lfs", {}) or {}).get("size"),
+                    lfs=s.get("lfs"),
+                )
+            )
+        return SimpleNamespace(id=data["id"], siblings=siblings)
 
     def from_id(self, id: str, variant: Optional[ModelRepoVariant] = None) -> AnyModelRepoMetadata:
         """Return a HuggingFaceMetadata object given the model's repo_id."""
@@ -81,10 +94,15 @@ def from_id(self, id: str, variant: Optional[ModelRepoVariant] = None) -> AnyMod
         repo_id = id.split("::")[0] or id
         while not model_info:
             try:
-                model_info = self._fetch_model_info(repo_id, variant)
-            except UnknownMetadataException:
-                raise
-            except requests.HTTPError:
+                # Use the injected session when provided (supports testing with mock adapters).
+                # Otherwise use HfApi which uses httpx internally.
+                if self._has_custom_session:
+                    model_info = self._model_info_via_session(repo_id, variant)
+                else:
+                    model_info = HfApi().model_info(repo_id=repo_id, files_metadata=True, revision=variant)
+            except RepositoryNotFoundError as excp:
+                raise UnknownMetadataException(f"'{repo_id}' not found. See trace for details.") from excp
+            except RevisionNotFoundError:
                 if variant is None:
                     raise
                 else:
@@ -94,18 +112,15 @@ def from_id(self, id: str, variant: Optional[ModelRepoVariant] = None) -> AnyMod
 
         _, name = repo_id.split("/")
 
-        for s in model_info.get("siblings") or []:
-            rfilename = s.get("rfilename")
-            size = s.get("size")
-            assert rfilename is not None
-            assert size is not None
-            lfs = s.get("lfs")
+        for s in model_info.siblings or []:
+            assert s.rfilename is not None
+            assert s.size is not None
             files.append(
                 RemoteModelFile(
-                    url=hf_hub_url(repo_id, rfilename, revision=variant or "main"),
-                    path=Path(name, rfilename),
-                    size=size,
-                    sha256=lfs.get("sha256") if lfs else None,
+                    url=hf_hub_url(repo_id, s.rfilename, revision=variant or "main"),
+                    path=Path(name, s.rfilename),
+                    size=s.size,
+                    sha256=s.lfs.get("sha256") if s.lfs else None,
                 )
             )
 
@@ -132,10 +147,10 @@ def from_id(self, id: str, variant: Optional[ModelRepoVariant] = None) -> AnyMod
         )
 
         return HuggingFaceMetadata(
-            id=model_info["id"],
+            id=model_info.id,
             name=name,
             files=files,
-            api_response=json.dumps(model_info, default=str),
+            api_response=json.dumps(model_info.__dict__, default=str),
             is_diffusers=is_diffusers,
             ckpt_urls=ckpt_urls,
         )

invokeai/backend/quantization/scripts/quantize_t5_xxl_bnb_llm_int8.py

@@ -15,8 +15,11 @@ def load_state_dict_into_t5(model: T5EncoderModel, state_dict: dict):
     missing_keys, unexpected_keys = model.load_state_dict(state_dict, strict=False, assign=True)
     assert len(unexpected_keys) == 0
     assert set(missing_keys) == {"encoder.embed_tokens.weight"}
-    # Assert that the layers we expect to be shared are actually shared.
-    assert model.encoder.embed_tokens.weight is model.shared.weight
+    # Re-tie shared weights. In transformers 5.x, weight tying is implemented at the
+    # parameter level (via _tie_weights / tie_weights) rather than as a Python object
+    # alias.  load_state_dict(assign=True) replaces parameters in-place, which severs
+    # the parameter-level tie.  Calling tie_weights() re-establishes it.
+    model.tie_weights()
 
 
 def main():

pyproject.toml

@@ -35,7 +35,7 @@ dependencies = [
   # Core generation dependencies, pinned for reproducible builds.
   "accelerate",
   "bitsandbytes; sys_platform!='darwin'",
-  "compel==2.1.1",
+  "compel>=2.4.0,<3",
   "diffusers[torch]==0.37.0",
   "gguf",
   "mediapipe==0.10.14",                   # needed for "mediapipeface" controlnet model
@@ -53,7 +53,7 @@ dependencies = [
   "torch>=2.7.0,<2.8.0; sys_platform == 'darwin'",
   "torchsde",                             # diffusers needs this for SDE solvers, but it is not an explicit dep of diffusers
   "torchvision",
-  "transformers>=4.56.0",
+  "transformers>=5.5,<5.6",
 
   # Core application dependencies, pinned for reproducible builds.
   "fastapi-events",

tests/app/invocations/test_sd3_text_encoder.py

@@ -140,7 +140,6 @@ def test_sd3_t5_encode_uses_effective_device(monkeypatch):
 
     monkeypatch.setattr(f"{module_path}.T5EncoderModel", FakeSd3T5Encoder)
     monkeypatch.setattr(f"{module_path}.T5Tokenizer", FakeT5Tokenizer)
-    monkeypatch.setattr(f"{module_path}.T5TokenizerFast", FakeT5Tokenizer)
 
     invocation = Sd3TextEncoderInvocation.model_construct(
         clip_l=SimpleNamespace(text_encoder=SimpleNamespace(), tokenizer=SimpleNamespace(), loras=[]),