Commit 0d44e35
committed
fix: English TN no longer inserts spurious space after opening punctuation
The verbalizer's INSERT_SPACE was applied uniformly between all
non-punct tokens, causing unwanted spaces after opening quotes and
parens (e.g., `"hello"` → `" hello"`). Split the verbalizer pattern
so classify tokens use INSERT_SPACE for inter-word spacing while
punct tokens use DELETE_SPACE — punct values already carry surrounding
spacing via the tagger's add_weight(accep(" "), -1.0).star.
Add test cases for the reported issue and related patterns.1 parent ef6824f commit 0d44e35
2 files changed
Lines changed: 15 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
78 | 78 | | |
79 | 79 | | |
80 | 80 | | |
81 | | - | |
| 81 | + | |
82 | 82 | | |
83 | 83 | | |
84 | 84 | | |
| |||
92 | 92 | | |
93 | 93 | | |
94 | 94 | | |
95 | | - | |
96 | 95 | | |
97 | | - | |
98 | | - | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
0 commit comments