Add hi-IN , Ko-KR and pt-BR IPA tokenizer support#15567
Merged
XuesongYang merged 23 commits intoNVIDIA-NeMo:mainfrom Apr 24, 2026
Merged
Add hi-IN , Ko-KR and pt-BR IPA tokenizer support#15567XuesongYang merged 23 commits intoNVIDIA-NeMo:mainfrom
XuesongYang merged 23 commits intoNVIDIA-NeMo:mainfrom
Conversation
quapham
changed the title
Collaborator
|
Can you fix the linting and sign off issues? |
Contributor
There was a problem hiding this comment.
Pull request overview
Extends the NeMo TTS IPA G2P/tokenizer stack to better handle additional scripts/locales (Hindi with English code-switching, Korean, and Brazilian Portuguese) by expanding tokenization character coverage, dictionary parsing, and adding unit tests to validate expected IPA outputs.
Changes:
- Added unit tests for IPA tokenization in
pt-BR,hi-IN(Hindi/English code-switching), andko-KR. - Expanded “any-locale” tokenization character coverage to include Indic and Korean Unicode ranges.
- Updated
IpaG2pdictionary parsing and regex handling to accept Indic/Korean words and merge multiple dictionaries.
Reviewed changes
Copilot reviewed 4 out of 7 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| tests/collections/common/tokenizers/text_to_speech/test_tts_tokenizers.py | Adds unit tests and small in-test pronunciation dictionaries for pt-BR, hi-IN code-switching, and ko-KR. |
| nemo/collections/tts/g2p/models/i18n_ipa.py | Extends IpaG2p regex + dictionary parsing to support Indic/Korean and multi-dict merging for code-switching. |
| nemo/collections/common/tokenizers/text_to_speech/tokenizer_utils.py | Adds Indic and Korean Unicode ranges and expands any-locale word tokenization regex accordingly. |
| nemo/collections/common/tokenizers/text_to_speech/ipa_lexicon.py | Adds pt-BR and ko-KR to supported locales and extends punctuation sets for hi-IN and ko-KR. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
XuesongYang
force-pushed
the
hi_pt_BR_ipa
branch
from
065c5cb to
2151454
Compare
XuesongYang
reviewed
Apr 20, 2026
XuesongYang
reviewed
Apr 20, 2026
XuesongYang
reviewed
Apr 20, 2026
XuesongYang
reviewed
Apr 20, 2026
XuesongYang
reviewed
Apr 20, 2026
XuesongYang
reviewed
Apr 21, 2026
XuesongYang
reviewed
Apr 21, 2026
XuesongYang
reviewed
Apr 21, 2026
…kenizers.py Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: quanpham <youngkwan199@gmail.com>
Signed-off-by: quapham <quapham@users.noreply.github.com>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
XuesongYang
force-pushed
the
hi_pt_BR_ipa
branch
from
2cffea8 to
f5bd97d
Compare
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Signed-off-by: Xuesong Yang <1646669+XuesongYang@users.noreply.github.com>
Collaborator
|
@blisc all good now. pls have a look. |
8 tasks
blisc
approved these changes
Apr 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
The
Update branchbutton must only be pressed in very rare occassions.An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.
What does this PR do ?
Extends the IPAG2p tokenizer to support Hindi (hi-IN) with English code-switching , Korean (ko-KR) and Brazilian Portuguese (pt-BR) locale.
Collection: [Note which collection this PR will affect]
tts, common
Changelog
Usage
# Add a code snippet demonstrating how to use thisGitHub Actions CI
The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".
Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information