feat(tokenizers): support loading a chat template from a file#4899
Open
omri-manolo wants to merge 2 commits into
Open
feat(tokenizers): support loading a chat template from a file#4899omri-manolo wants to merge 2 commits into
omri-manolo wants to merge 2 commits into
Conversation
Add an optional chat_template_path to the tokenizer config: a path to a jinja template file (local or fsspec-readable URI) applied to the built tokenizer. This lets CLI/Hydra callers swap chat templates per run without editing tokenizer_config.json inside the model checkpoint or embedding multi-line jinja strings in overrides. Signed-off-by: Omri Manor <omrim@ai21.com>
yaoyu-33
added
area:training
yaoyu-33
approved these changes
Jul 17, 2026
Contributor
|
/ok to test 678536c |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Adds an optional
chat_template_pathtoTokenizerConfig— a path to a jinja template file (local or fsspec-readable URI) applied to the built tokenizer — so CLI/Hydra callers can swap chat templates per run without editingtokenizer_config.jsoninside the model checkpoint or embedding multi-line jinja strings in overrides.Changelog
TokenizerConfig.chat_template_path: new optional field; mutually exclusive with the inlinechat_template(setting both raises a clear configuration error)._resolve_chat_template()intraining/tokenizers/tokenizer.py: resolves the inline template or reads the template file; supportsmsc://URIs when the Multi-Storage Client feature is enabled.build_hf_tokenizer()applies the resolved template to the built tokenizer.TestChatTemplatePathOverride): local-file resolution, inline passthrough, unset →None, inline+path mutual-exclusion error,msc://resolution, and end-to-end throughbuild_hf_tokenizer.GitHub Actions CI
See the CI section in the Contributing doc for how to trigger the CI. A Nvidia developer will need to approve and trigger the CI for external contributors.
Before your PR is "Ready for review"
Pre checks:
data-preparation.md) if reviewers want itmsc://path is behind the existing Multi-Storage Client feature flagIf you haven't finished some of the above items you can still open "Draft" PR.
Additional Information
TokenizerConfig.chat_template_path: load a chat template from a file #4833tests/unit_tests/trainingverified green locally against currentmain(82b47ab) + this change.