Commit cc6e911
authored
🐛 disallow downloads for fast tokenizer conversion (#63)
#### Motivation
Some of our models are failing internal integration tests where the
AutoTokenizer tries to download files even though they exist locally in
the cache.
#### Modifications
This adds the `local_files_only` kwarg to the `.from_pretrained()` call
in the fast tokenizer conversion in the launcher, to prevent hf from
attempting to download tokenizer files
#### Result
Local tests show this allows the failing models to successfully load up
tokenizers
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>1 parent a5ac374 commit cc6e911
1 file changed
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
828 | 828 | | |
829 | 829 | | |
830 | 830 | | |
831 | | - | |
| 831 | + | |
832 | 832 | | |
833 | 833 | | |
834 | 834 | | |
| |||
0 commit comments