Skip to content

Commit c094e41

Browse files
authored
fix: move OOV test cases out of shared test data to fix C++ unit tests (#370)
The C++ runtime test loads test data from normalizer.txt but uses FSTs built without tag_oov=True. OOV test cases (Korean/Japanese characters) that require <oov> wrapping are moved to a separate file only loaded by the Python test (which uses tag_oov=True). Fixes #368
1 parent 9dd4a69 commit c094e41

3 files changed

Lines changed: 3 additions & 2 deletions

File tree

tn/chinese/test/data/normalizer.txt

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,8 +35,6 @@
3535
我儿子喜欢这地儿 => 我儿子喜欢这地
3636
O2O => O to O
3737
B2B => B to B
38-
我们안녕 => 我们<oov>안</oov><oov>녕</oov>
39-
雪の花 => 雪<oov>の</oov>花
4038
给12315打个电话 => 给幺二三幺五打个电话
4139
人均200以内 => 人均两百以内
4240
当场票数≥100万 => 当场票数大于等于一百万
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
我们안녕 => 我们<oov>안</oov><oov>녕</oov>
2+
雪の花 => 雪<oov>の</oov>花

tn/chinese/test/normalizer_test.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ class TestNormalizer:
3434
parse_test_case("data/time.txt"),
3535
parse_test_case("data/whitelist.txt"),
3636
parse_test_case("data/normalizer.txt"),
37+
parse_test_case("data/normalizer_tag_oov.txt"),
3738
)
3839

3940
@pytest.mark.parametrize("written, spoken", normalizer_cases)

0 commit comments

Comments
 (0)