Add Korean TN support for cardinal numbers and postprocessing

[bbae0312]

What does this PR do ?

Classify and verbalize grammars
Unit tests with coverage up to 17-digit numbers
Support for spacing around units (억, 만, 조, 경)
Post-processing logic for formatting
Unit tests and Sparrowhawk tests

Before your PR is "Ready for review"

Pre checks:

• [x ] Have you signed your commits? Use git commit -s to sign.
• [x ] Do all unittests finish successfully before sending PR?
  1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
  2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
• [x ] If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here.
• [x ] Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files?
• [x ] Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ?
• [x ] Have you added the correct license header Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files?
• [x ] If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. See an example here.
• [x ] Remove import guards (try import: ... except: ...) if not already done.
• [x ] If you added a new language or a new feature please update the NeMo documentation (lives in different repo).
• [x ] Have you added your language support to tools/text_processing_deployment/pynini_export.py.

PR Type:

• [x ] New Feature
• Bugfix
• Documentation
• Test

If you haven't finished some of the above items you can still open "Draft" PR.

[mgrafu]

let's call this something like graph_digit_no_zero_one to make it more explicit in its 'alt'-ness

[bbae0312]

Renamed graph_digit_alt to graph_digit_no_zero_one for clarity as suggested.

[mgrafu]

are the rules for creating teens any different from the ones to create ties? does it make sense to have a separate file for them, or would it be possible to just add 1 십 to the ties file?

[bbae0312]

You're right! Since both follow the pattern of combining a base digit with “십”, it makes sense to consolidate them. I’ll go ahead and remove the teen.tsv file and just add 1 십 to the ty.tsv file.

[mgrafu]

this covers numbers from 1 to 99, right? can we rename the variable for more clarity?

[bbae0312]

I’ll rename it to graph_1_to_99 for clarity.

[mgrafu]

(NEMO_DIGIT**3 @ graph_hundred_component) == graph_hundred, let's try to use the same variable intead of redefining the rule (this applies to other _component graphs too)

[bbae0312]

Updated the code to use graph_hundred and graph_thousand instead of (NEMO_DIGIT3 @ graph_hundred_component) and (NEMO_DIGIT4 @ graph_thousand_component).

[mgrafu]

let's add a test case in test_cases_cardinal.txt for each of these rules

[bbae0312]

Sure! I’ll add one test case for each of these rules in test_cases_cardinal.txt

[mgrafu]

let's also add a few test cases in test_cases_cardinal.txt for negative numbers

[bbae0312]

I'll add some negative cases as well.

[mgrafu]

let's remove the commented out tests for now and add them as we develop each class

[bbae0312]

Deleted all the commented out tests

[mgrafu]

LGTM -- let's just get CI running properly