Unable to resume model training with a locally downloaded model and trust remote code

[conceptofmind]

System Info

requires-python = ">=3.12"
dependencies = [
"accelerate>=1.13.0",
"datasets>=4.8.5",
"sentence-transformers>=5.4.1",
"torch>=2.11.0",
"transformers>=5.8.0",
]

Who can help?

@tomaszcichy98

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

When downloading a model locally using:

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="voyageai/voyage-4-nano",
    local_dir="/e/data1/datasets/playground/mmlaion/shared/enrico/models/voyage-4-nano",
)

I am unable to resume training as there are missing files due to custom code not being saved on disk.

    model = SentenceTransformer(
        base_model_path, 
        trust_remote_code=trust_remote_code
    )

I get the error:

[rank1]: Traceback (most recent call last):
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 236, in <module>
[rank1]:     main()
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 217, in main
[rank1]:     trainer_stats = trainer.train(resume_from_checkpoint=resume)
[rank1]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 1415, in train
[rank1]:     self._load_from_checkpoint(resume_from_checkpoint)
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/trainer.py", line 966, in _load_from_checkpoint
[rank1]:     loaded_model = model_class(checkpoint_path, trust_remote_code=self.model.trust_remote_code)
[rank1]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 41, in wrapper
[rank1]:     return func(*args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/sentence_transformer/model.py", line 183, in __init__
[rank1]:     super().__init__(
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 198, in __init__
[rank1]:     modules, self.module_kwargs = self._load_modules(
[rank1]:                                   ^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 974, in _load_modules
[rank1]:     return self._load_config_modules(model_name_or_path, **load_kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 1165, in _load_config_modules
[rank1]:     module = module_class.load(
[rank1]:              ^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1708, in load
[rank1]:     return cls(model_name_or_path=model_name_or_path, **init_kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 87, in wrapper
[rank1]:     return func(*args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 642, in __init__
[rank1]:     self.model = self._load_model(
[rank1]:                  ^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1414, in _load_model
[rank1]:     return model_cls.from_pretrained(model_name_or_path, config=config, **model_kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 379, in from_pretrained
[rank1]:     model_class = get_class_from_dynamic_module(
[rank1]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 616, in get_class_from_dynamic_module
[rank1]:     final_module = get_cached_module_file(
[rank1]:                    ^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 425, in get_cached_module_file
[rank1]:     resolved_module_file = cached_file(
[rank1]:                            ^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 278, in cached_file
[rank1]:     file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 380, in cached_files
[rank1]:     raise OSError(
[rank1]: OSError: /e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205 does not appear to have a file named modeling_qwen3_bidirectional.py. Checkout 'https://huggingface.co//e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205/tree/main' for available files.
[rank3]: Traceback (most recent call last):
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 236, in <module>
[rank3]:     main()
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 217, in main
[rank3]:     trainer_stats = trainer.train(resume_from_checkpoint=resume)
[rank3]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 1415, in train
[rank3]:     self._load_from_checkpoint(resume_from_checkpoint)
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/trainer.py", line 966, in _load_from_checkpoint
[rank3]:     loaded_model = model_class(checkpoint_path, trust_remote_code=self.model.trust_remote_code)
[rank3]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 41, in wrapper
[rank3]:     return func(*args, **kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/sentence_transformer/model.py", line 183, in __init__
[rank3]:     super().__init__(
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 198, in __init__
[rank3]:     modules, self.module_kwargs = self._load_modules(
[rank3]:                                   ^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 974, in _load_modules
[rank3]:     return self._load_config_modules(model_name_or_path, **load_kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 1165, in _load_config_modules
[rank3]:     module = module_class.load(
[rank3]:              ^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1708, in load
[rank3]:     return cls(model_name_or_path=model_name_or_path, **init_kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 87, in wrapper
[rank3]:     return func(*args, **kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 642, in __init__
[rank3]:     self.model = self._load_model(
[rank3]:                  ^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1414, in _load_model
[rank3]:     return model_cls.from_pretrained(model_name_or_path, config=config, **model_kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 379, in from_pretrained
[rank3]:     model_class = get_class_from_dynamic_module(
[rank3]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 616, in get_class_from_dynamic_module
[rank3]:     final_module = get_cached_module_file(
[rank3]:                    ^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 425, in get_cached_module_file
[rank3]:     resolved_module_file = cached_file(
[rank3]:                            ^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 278, in cached_file
[rank3]:     file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 380, in cached_files
[rank3]:     raise OSError(
[rank3]: OSError: /e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205 does not appear to have a file named modeling_qwen3_bidirectional.py. Checkout 'https://huggingface.co//e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205/tree/main' for available files.
[rank2]: Traceback (most recent call last):
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 236, in <module>
[rank2]:     main()
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 217, in main
[rank2]:     trainer_stats = trainer.train(resume_from_checkpoint=resume)
[rank2]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 1415, in train
[rank2]:     self._load_from_checkpoint(resume_from_checkpoint)
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/trainer.py", line 966, in _load_from_checkpoint
[rank2]:     loaded_model = model_class(checkpoint_path, trust_remote_code=self.model.trust_remote_code)
[rank2]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 41, in wrapper
[rank2]:     return func(*args, **kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/sentence_transformer/model.py", line 183, in __init__
[rank2]:     super().__init__(
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 198, in __init__
[rank2]:     modules, self.module_kwargs = self._load_modules(
[rank2]:                                   ^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 974, in _load_modules
[rank2]:     return self._load_config_modules(model_name_or_path, **load_kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 1165, in _load_config_modules
[rank2]:     module = module_class.load(
[rank2]:              ^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1708, in load
[rank2]:     return cls(model_name_or_path=model_name_or_path, **init_kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 87, in wrapper
[rank2]:     return func(*args, **kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 642, in __init__
[rank2]:     self.model = self._load_model(
[rank2]:                  ^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1414, in _load_model
[rank2]:     return model_cls.from_pretrained(model_name_or_path, config=config, **model_kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 379, in from_pretrained
[rank2]:     model_class = get_class_from_dynamic_module(
[rank2]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 616, in get_class_from_dynamic_module
[rank2]:     final_module = get_cached_module_file(
[rank2]:                    ^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 425, in get_cached_module_file
[rank2]:     resolved_module_file = cached_file(
[rank2]:                            ^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 278, in cached_file
[rank2]:     file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 380, in cached_files
[rank2]:     raise OSError(
[rank2]: OSError: /e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205 does not appear to have a file named modeling_qwen3_bidirectional.py. Checkout 'https://huggingface.co//e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205/tree/main' for available files.
[rank0]: Traceback (most recent call last):
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 236, in <module>
[rank0]:     main()
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/train.py", line 217, in main
[rank0]:     trainer_stats = trainer.train(resume_from_checkpoint=resume)
[rank0]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 1415, in train
[rank0]:     self._load_from_checkpoint(resume_from_checkpoint)
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/trainer.py", line 966, in _load_from_checkpoint
[rank0]:     loaded_model = model_class(checkpoint_path, trust_remote_code=self.model.trust_remote_code)
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 41, in wrapper
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/sentence_transformer/model.py", line 183, in __init__
[rank0]:     super().__init__(
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 198, in __init__
[rank0]:     modules, self.module_kwargs = self._load_modules(
[rank0]:                                   ^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 974, in _load_modules
[rank0]:     return self._load_config_modules(model_name_or_path, **load_kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/model.py", line 1165, in _load_config_modules
[rank0]:     module = module_class.load(
[rank0]:              ^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1708, in load
[rank0]:     return cls(model_name_or_path=model_name_or_path, **init_kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/util/decorators.py", line 87, in wrapper
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 642, in __init__
[rank0]:     self.model = self._load_model(
[rank0]:                  ^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/sentence_transformers/base/modules/transformer.py", line 1414, in _load_model
[rank0]:     return model_cls.from_pretrained(model_name_or_path, config=config, **model_kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 379, in from_pretrained
[rank0]:     model_class = get_class_from_dynamic_module(
[rank0]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 616, in get_class_from_dynamic_module
[rank0]:     final_module = get_cached_module_file(
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/dynamic_module_utils.py", line 425, in get_cached_module_file
[rank0]:     resolved_module_file = cached_file(
[rank0]:                            ^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 278, in cached_file
[rank0]:     file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/e/project1/reformo/enrico/slurm-st/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 380, in cached_files
[rank0]:     raise OSError(
[rank0]: OSError: /e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205 does not appear to have a file named modeling_qwen3_bidirectional.py. Checkout 'https://huggingface.co//e/data1/datasets/playground/mmlaion/shared/enrico/saved_models/voyage-4-nano_lr3e-05_warmup0.1_bs8k/checkpoint-2205/tree/main' for available files.

This currently happens with the models:
https://huggingface.co/voyageai/voyage-4-nano
https://huggingface.co/jinaai/jina-embeddings-v5-text-small-retrieval

Expected behavior

I should be able to resume model training.

[ArjunSrivastava1]

the issue is that ur custom code cant be saved or gets overwritten via the downloads right? so cant you refactor those custom portions to ensure that the downloads dont overwrite it?

[conceptofmind]

Tom has noted that this might be a regression of this PR: #15379

[mustafahk27]

hi, i'd love to work on this issue.

From reading the traceback, it looks like the root cause is that _save_checkpoint in trainer.py doesn't copy over the custom remote code files (e.g. modeling_qwen3_bidirectional.py) when saving a checkpoint for a model loaded with trust_remote_code=True. So when resuming, the dynamic module loader can't find those files in the checkpoint directory.
My proposed fix would be to detect trust_remote_code usage at checkpoint save time and copy any custom .py files from the original model directory into the checkpoint folder. Does that approach sound right to you?