Skip to content

Commit cb2fb03

Browse files
authored
fix: English TN no longer inserts spurious space after opening punctuation (#363)
The verbalizer's INSERT_SPACE was applied uniformly between all non-punct tokens, causing unwanted spaces after opening quotes and parens (e.g., `"hello"` → `" hello"`). Split the verbalizer pattern so classify tokens use INSERT_SPACE for inter-word spacing while punct tokens use DELETE_SPACE — punct values already carry surrounding spacing via the tagger's add_weight(accep(" "), -1.0).star. Add test cases for the reported issue and related patterns.
1 parent 9125e4a commit cb2fb03

2 files changed

Lines changed: 15 additions & 4 deletions

File tree

tn/english/normalizer.py

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,7 @@ def build_tagger_and_verbalizer(self):
7878
).optimize() + (add_weight(punctuation.tagger, 2.00).plus | self.DELETE_SPACE)
7979
self.tagger = (delete(" ").star + tagger.star) @ self.build_rule(delete(" "), r="[EOS]")
8080

81-
verbalizer = (
81+
classify = (
8282
cardinal.verbalizer
8383
| ordinal.verbalizer
8484
| word.verbalizer
@@ -92,7 +92,15 @@ def build_tagger_and_verbalizer(self):
9292
| electronic.verbalizer
9393
| serial.verbalizer
9494
| whitelist.verbalizer
95-
| punctuation.verbalizer
9695
| rang.verbalizer
97-
).optimize() + (punctuation.verbalizer.plus | self.INSERT_SPACE)
98-
self.verbalizer = verbalizer.star @ self.build_rule(delete(" "), r="[EOS]")
96+
).optimize()
97+
punct = punctuation.verbalizer.optimize()
98+
# Punct tokens carry surrounding spacing in their values (the tagger's
99+
# add_weight(accep(" "), -1.0).star absorbs spaces around punctuation).
100+
# So punct tokens handle their own spacing and don't need INSERT_SPACE.
101+
# Only classify tokens need INSERT_SPACE for inter-word spacing.
102+
verbalizer = (
103+
classify + (punct.plus | self.INSERT_SPACE)
104+
| punct + (punct.plus | self.DELETE_SPACE)
105+
).star
106+
self.verbalizer = verbalizer @ self.build_rule(delete(" "), r="[EOS]")

tn/english/test/data/normalizer.txt

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,3 +6,6 @@ Try searching for 'Toyota' or 'Investment' => Try searching for 'Toyota' or 'Inv
66
"" => ""
77
The HTML tag <p> defines a paragraph. => The HTML tag <p> defines a paragraph.
88
hello world => hello world
9+
"So, one hundred thousand merit shouldn't be a problem." => "So, one hundred thousand merit shouldn't be a problem."
10+
"hello" => "hello"
11+
(hello) => (hello)

0 commit comments

Comments
 (0)