en_core_web_hftrf-3.8.1 fails to load with spacy-transformers 1.4.0 (safetensors weights)

[maripastor-mr]

Summary

en_core_web_hftrf-3.8.1 (release) ships with transformer weights in safetensors format and pins spacy-transformers>=1.4.0,<1.5.0. However, spacy-transformers 1.4.0 cannot load safetensors weights — HFShim.from_bytes calls torch.load() which only handles pickle/zip format.

The safetensors migration (explosion/spacy-transformers#423, commit 2cba136) landed on master 3 days after the 1.4.0 release, so there is currently no released version of spacy-transformers that can load this model.

Steps to reproduce

pip install spacy==3.8.7 spacy-transformers==1.4.0
pip install en_core_web_hftrf==3.8.1
python -c "import spacy; spacy.load('en_core_web_hftrf')"

Error

_pickle.UnpicklingError: Weights only load failed. [...]
WeightsUnpickler error: Unsupported operand 64

With weights_only=False forced, the error becomes:

_pickle.UnpicklingError: invalid load key, '@'

The first byte of the state is 0x40 (@), which is the start of a safetensors header (little-endian u64 header length), not a pickle stream.

Environment

• Python 3.12
• spacy 3.8.7
• spacy-transformers 1.4.0
• torch 2.8.0

Request

Could a new release of spacy-transformers be cut to include the safetensors migration from explosion/spacy-transformers#423? Currently there is no way to use en_core_web_hftrf-3.8.1 with a released version of spacy-transformers.