Skip to content

Commit eabb5b9

Browse files
pengzhendongpengzhendong
authored
feat: English ITN with full rule coverage (#358)
* feat: integrate all English ITN rules with money, time and decimal fixes - Add Money rule: two dollars => $2, one cent => $0.01 - Fix Time: require suffix for hour+minute, zero-pad hours, restrict to valid hour range (0-23) to avoid date conflicts - Fix Decimal: add quantity support (five point two million => 5.2 million) - Fix Money cents: pad single-digit cents (1 => 01) - Extend _num_to_word to support 60-99 NeMo English ITN: 372/470 (79%) All 1442 unit tests pass. * feat: English ITN improvements - 412/470 NeMo coverage (88%) - decimal: add cardinal+quantity support (63/63 full pass) - time: add no-suffix hour+minute, quarter/half to, timezone (28/29) - money: add cents padding, quantity, decimal format (43/52) - measure: add compound units mph, sq ft, kgf/cm² (112/112 full pass) - word: support apostrophes and trailing punctuation (54/55) - cardinal: add 0-12 exception (consistent with NeMo) - Fix token_parser ITN_ORDERS for time zone and money quantity * feat: add electronic rule back and improve coverage to 93% (436/470) - Rewrite Electronic rule: require 'at' for email or dot-separated domain, preventing false matches on plain text - Add compound units to measurements.tsv (mph, sq ft, kgf/cm²) NeMo coverage: 436/470 (93%) Full pass: decimal(63), measure(112), ordinal(34) * feat: money quantity and decimal support, 442/470 (94%) * feat: telephone with double/IP/serial/country code, 446/470 (95%) * test: add 446 NeMo-based English ITN test cases * feat: fix money/telephone/IP, 450/470 (96%) - Money: add with_hundred pattern (one fifty five => $155), exclude thousand from quantity, fix fifteen thousand dollars => $15000 - Telephone: add double digit support in IP addresses - Update test cases to match improved coverage (450 cases) * feat: decades, serial weight fix, 451/470 (96%) - Date: add decades pattern (nineteen eighties => 1980s) - Telephone: increase serial weight to reduce false matches - Telephone: add double digit support in IP - Update test cases (451 cases) * feat: word uses NOT_SPACE, add decades, 451/470 (96%) * feat: NeMo-style tokenization, 455/470 (97%) Replace tagger.star with NeMo-style token + closure(delete_extra_space + token) pattern. This ensures explicit space consumption between tokens, resolving many segmentation ambiguities: - seven eleven stores => 7-eleven stores (whitelist now wins) - set alarm at ten to eleven pm => set alarm at 10:50 p.m. * feat: fix time min-to, IP twenty-three, serial, 456/470 (97%) - Time: fix minute_to composition (use raw digits without zero-padding) => time now 29/29 full pass - Telephone: fix IP to support single+two_digit combinations (one twenty three dot... => 123.123.0.40) - Cardinal: expose graph_two_digit for telephone serial * feat: 463/470 (98.5%) NeMo coverage - cardinal: fix zero in exception list - date: add Q2 quarter, 750BC, 3-digit year, decades => 36/36 full pass - time: fix date vs time priority => 29/29 full pass - whitelist: fixed via date priority => 12/12 full pass - telephone: fix serial two_digit weight, IP combinations - 7 full-pass rules: ordinal, decimal, measure, date, time, whitelist, money(51/52) * feat: 469/470 (99.8%) NeMo coverage - electronic: exclude "dot" as email username first token - money: reject singular "one" with plural currency ("one dollars") - telephone: add credit card 4-6-4/4-6-5 formats with optional country code - telephone: exclude "a" as serial first char to avoid "a thirty six" -> "a36" - punctuation: add Punctuation class, split punct from words ("twenty!" -> "20 !")
1 parent c7c9f75 commit eabb5b9

24 files changed

Lines changed: 949 additions & 182 deletions

‎itn/english/data/measurements.tsv‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -143,3 +143,7 @@ gy gray
143143
sv sievert
144144
cwt hundredweight
145145
cc c c
146+
mph miles per hour
147+
sq ft square feet
148+
kgf/cm² kilograms force per square centimeter
149+
kgf/cm² kilogram force per square centimeter

‎itn/english/inverse_normalizer.py‎

Lines changed: 47 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -13,12 +13,22 @@
1313
# limitations under the License.
1414

1515
from importlib_resources import files
16+
from pynini import closure
1617
from pynini.lib.pynutil import add_weight, delete
1718

1819
from itn.english.rules.cardinal import Cardinal
1920
from itn.english.rules.char import Char
21+
from itn.english.rules.date import Date
2022
from itn.english.rules.decimal import Decimal
23+
from itn.english.rules.electronic import Electronic
24+
from itn.english.rules.measure import Measure
25+
from itn.english.rules.money import Money
2126
from itn.english.rules.ordinal import Ordinal
27+
from itn.english.rules.punctuation import Punctuation
28+
from itn.english.rules.telephone import Telephone
29+
from itn.english.rules.time import Time
30+
from itn.english.rules.whitelist import Whitelist
31+
from itn.english.rules.word import Word
2232
from tn.processor import Processor
2333

2434

@@ -34,23 +44,53 @@ def build_tagger_and_verbalizer(self):
3444
cardinal = Cardinal()
3545
ordinal = Ordinal(cardinal=cardinal)
3646
decimal = Decimal(cardinal=cardinal)
47+
date = Date(cardinal=cardinal, ordinal=ordinal)
48+
time = Time(cardinal=cardinal)
49+
measure = Measure(cardinal=cardinal, decimal=decimal)
50+
money = Money(cardinal=cardinal, decimal=decimal)
51+
telephone = Telephone(cardinal=cardinal)
52+
electronic = Electronic()
53+
whitelist = Whitelist()
54+
word = Word()
3755
char = Char()
56+
punctuation = Punctuation()
3857

39-
tagger = (
40-
add_weight(ordinal.tagger, 1.0)
41-
| add_weight(decimal.tagger, 1.01)
42-
| add_weight(cardinal.tagger, 1.02)
58+
classify = (
59+
add_weight(date.tagger, 1.09)
60+
| add_weight(time.tagger, 1.1)
61+
| add_weight(measure.tagger, 1.1)
62+
| add_weight(money.tagger, 1.08)
63+
| add_weight(whitelist.tagger, 1.01)
64+
| add_weight(telephone.tagger, 1.1)
65+
| add_weight(electronic.tagger, 1.1)
66+
| add_weight(ordinal.tagger, 1.09)
67+
| add_weight(decimal.tagger, 1.1)
68+
| add_weight(cardinal.tagger, 1.1)
69+
| add_weight(word.tagger, 50)
4370
| add_weight(char.tagger, 100)
4471
).optimize()
4572

46-
tagger = tagger.star
47-
self.tagger = tagger @ self.build_rule(delete(" "), "", "[EOS]")
73+
punct = add_weight(punctuation.tagger, 1.1)
74+
token = closure(punct + delete(" ").ques) + classify + closure(delete(" ").ques + punct)
75+
graph = token + closure(self.DELETE_EXTRA_SPACE + token)
76+
self.tagger = delete(" ").star + graph + delete(" ").star
4877

4978
verbalizer = (
5079
cardinal.verbalizer
5180
| ordinal.verbalizer
5281
| decimal.verbalizer
82+
| date.verbalizer
83+
| time.verbalizer
84+
| measure.verbalizer
85+
| money.verbalizer
86+
| telephone.verbalizer
87+
| electronic.verbalizer
88+
| whitelist.verbalizer
89+
| word.verbalizer
5390
| char.verbalizer
91+
| punctuation.verbalizer
5492
).optimize()
5593

56-
self.verbalizer = verbalizer.star
94+
self.verbalizer = (verbalizer + self.INSERT_SPACE).star @ self.build_rule(
95+
self.DELETE_EXTRA_SPACE
96+
) @ self.build_rule(delete(" "), r="[EOS]")

‎itn/english/rules/cardinal.py‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,8 @@
1212
# See the License for the specific language governing permissions and
1313
# limitations under the License.
1414

15-
from pynini import closure, cross, string_file, union
16-
from pynini.lib.pynutil import delete, insert
15+
from pynini import closure, cross, difference, string_file, union
16+
from pynini.lib.pynutil import add_weight, delete, insert
1717

1818
from tn.processor import Processor
1919
from tn.utils import get_abs_path
@@ -35,7 +35,8 @@ def build_tagger(self):
3535

3636
# 1~9, 10~19, 20~99
3737
one_digit = digit
38-
two_digit = teen | (ties + (ds + digit | insert("0")))
38+
two_digit = teen | (ties + (ds + digit | add_weight(insert("0"), 0.1)))
39+
self.graph_two_digit = two_digit
3940
up_to_99 = one_digit | two_digit
4041

4142
# one hundred, one hundred twenty three, one hundred one
@@ -47,6 +48,7 @@ def build_tagger(self):
4748

4849
# 1~999
4950
up_to_999 = up_to_99 | hundreds
51+
self.up_to_999 = up_to_999
5052
# 1~999 with zero-padding to 3 digits
5153
up_to_999_padded = hundreds | insert("0") + two_digit | insert("00") + one_digit
5254

@@ -111,10 +113,17 @@ def _with_mag_padded(name):
111113
graph = (delete_and @ graph).optimize()
112114

113115
self.graph = graph
116+
self.graph_no_exception = graph
117+
118+
# exclude 0-12 from cardinal tagger (they stay as words)
119+
from itn.english.rules.time import _num_to_word
120+
exception_labels = ["zero"] + [_num_to_word(x) for x in range(1, 13)]
121+
exception = union(*exception_labels).optimize()
122+
graph_with_exception = (difference(self.VSIGMA, exception) @ graph).optimize()
114123

115124
minus = delete("minus") | delete("negative")
116125
optional_minus = closure(insert('negative: "-" ') + minus + ds, 0, 1)
117-
final_graph = optional_minus + insert('integer: "') + graph + insert('"')
126+
final_graph = optional_minus + insert('integer: "') + graph_with_exception + insert('"')
118127
self.tagger = self.add_tokens(final_graph)
119128

120129
def build_verbalizer(self):

‎itn/english/rules/date.py‎

Lines changed: 30 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,8 @@ def build_tagger(self):
5656

5757
# Year as two groups of two digits: "twenty twelve" => 2012
5858
year_two_parts = (teen | two_digit) + ds + (two_digit | oh_digit | teen)
59+
# 3-digit year: "seven fifty" => 750
60+
year_three_digit = digit + ds + (two_digit | oh_digit | teen)
5961

6062
# Year as "X thousand Y": "two thousand twelve" => 2012
6163
# Need zero-padded variants so "two thousand three" => 2003
@@ -116,9 +118,32 @@ def build_tagger(self):
116118
+ po
117119
)
118120
# Year only => "twenty twelve", "two thousand three"
119-
graph_y = add_weight(year, 0.01) + po
121+
graph_y = year + po
120122

121-
final_graph = graph_mdy | graph_md | graph_my | graph_dmy | graph_dm | graph_y
123+
# Decades: "nineteen eighties" => 1980s
124+
decade_suffix = closure(self.ALPHA, 1) + (cross("ies", "y") | delete("s"))
125+
decade_word = pynini.compose(decade_suffix, ties | cross("ten", "10"))
126+
graph_decade = (
127+
insert('year: "') + (teen | two_digit) + ds + decade_word + insert('0s"') + po
128+
)
129+
130+
# Quarter: "second quarter of twenty twenty two" => Q2 2022
131+
quarter_num = (
132+
cross("first", "1") | cross("second", "2")
133+
| cross("third", "3") | cross("fourth", "4")
134+
)
135+
graph_quarter = (
136+
insert('day: "Q') + quarter_num + insert('"')
137+
+ ds + delete("quarter") + ds + delete("of") + ds
138+
+ insert(' year: "') + year_graph + insert('"') + po
139+
)
140+
141+
# BC/AD suffix
142+
bc_ad = ds + (cross("b c", "BC") | cross("a d", "AD"))
143+
year_graph_with_3digit = year_graph | year_three_digit
144+
graph_y_bc = insert('year: "') + year_graph_with_3digit + bc_ad + insert('"') + po
145+
146+
final_graph = graph_mdy | graph_md | graph_my | graph_dmy | graph_dm | graph_y | graph_decade | graph_quarter | graph_y_bc
122147
self.tagger = self.add_tokens(final_graph)
123148

124149
def build_verbalizer(self):
@@ -160,6 +185,8 @@ def build_verbalizer(self):
160185
graph_dmy = day + self.DELETE_SPACE + insert(" ") + month + optional_year
161186
# year only
162187
graph_y = year
188+
# day + year (for quarter: Q2 2022)
189+
graph_dy = day + self.DELETE_SPACE + insert(" ") + year
163190

164-
graph = (graph_mdy | graph_dmy | graph_y) + self.DELETE_SPACE + delete_po
191+
graph = (graph_mdy | graph_dmy | graph_dy | graph_y) + self.DELETE_SPACE + delete_po
165192
self.verbalizer = self.delete_tokens(graph)

‎itn/english/rules/decimal.py‎

Lines changed: 27 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,12 +12,12 @@
1212
# See the License for the specific language governing permissions and
1313
# limitations under the License.
1414

15-
from pynini import closure, cross, string_file
15+
from pynini import closure, cross, string_file, union
1616
from pynini.lib.pynutil import delete, insert
1717

1818
from itn.english.rules.cardinal import Cardinal
1919
from tn.processor import Processor
20-
from tn.utils import get_abs_path
20+
from tn.utils import get_abs_path, load_labels
2121

2222

2323
class Decimal(Processor):
@@ -45,6 +45,25 @@ def build_tagger(self):
4545
point = delete("point")
4646

4747
graph = optional_negative + closure(integer_part + ds, 0, 1) + point + ds + frac_part
48+
49+
# quantity: "five point two million" => 5.2 million
50+
quantities = load_labels(get_abs_path("../itn/english/data/numbers/thousands.tsv"))
51+
quantity_all = union(*[x[0] for x in quantities])
52+
quantity_no_thousand = union(*[x[0] for x in quantities if x[0] != "thousand"])
53+
# decimal + quantity: five point two million, 164.58 thousand
54+
quantity_graph = (
55+
optional_negative + integer_part + ds + point + ds + frac_part
56+
+ ds + insert(' quantity: "') + quantity_all + insert('"')
57+
)
58+
# cardinal (up to 999) + quantity: four hundred million, five million
59+
# exclude thousand to let cardinal handle "ten thousand" => 10000
60+
cardinal_small = self.cardinal.up_to_999
61+
cardinal_quantity = (
62+
optional_negative + insert('integer_part: "') + cardinal_small + insert('"')
63+
+ ds + insert(' quantity: "') + quantity_no_thousand + insert('"')
64+
)
65+
graph |= quantity_graph | cardinal_quantity
66+
4867
self.tagger = self.add_tokens(graph)
4968

5069
def build_verbalizer(self):
@@ -56,6 +75,11 @@ def build_verbalizer(self):
5675
+ delete('"') + self.NOT_QUOTE.plus + delete('"')
5776
)
5877
optional_fractional = closure(fractional + self.DELETE_SPACE, 0, 1)
59-
graph = optional_sign + optional_integer + optional_fractional
78+
quantity = (
79+
insert(" ") + delete('quantity:') + self.DELETE_SPACE
80+
+ delete('"') + self.NOT_QUOTE.plus + delete('"')
81+
)
82+
optional_quantity = closure(quantity + self.DELETE_SPACE, 0, 1)
83+
graph = optional_sign + optional_integer + optional_fractional + optional_quantity
6084
self.numbers = graph
6185
self.verbalizer = self.delete_tokens(graph)

‎itn/english/rules/electronic.py‎

Lines changed: 28 additions & 69 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
# See the License for the specific language governing permissions and
1313
# limitations under the License.
1414

15-
from pynini import closure, cross, invert, string_file
15+
from pynini import accep, closure, cross, difference, invert, string_file
1616
from pynini.lib.pynutil import add_weight, delete, insert
1717

1818
from tn.processor import Processor
@@ -28,90 +28,49 @@ def __init__(self):
2828

2929
def build_tagger(self):
3030
ds = delete(" ")
31-
32-
# Single characters: digits and letters
3331
digit = string_file(get_abs_path("../itn/english/data/numbers/digit.tsv"))
3432
zero = string_file(get_abs_path("../itn/english/data/numbers/zero.tsv"))
35-
alpha_or_digit = self.ALPHA | digit | zero
36-
37-
# Symbols from TSV (symbol\tname): invert to get name -> symbol
38-
symbols = invert(
39-
string_file(get_abs_path("../itn/english/data/electronic/symbols.tsv"))
40-
)
33+
symbols = invert(string_file(get_abs_path("../itn/english/data/electronic/symbols.tsv")))
4134

42-
# A "token" is either a single char (letter/digit/symbol) or a
43-
# multi-letter word kept verbatim (e.g. "gmail", "nvidia").
44-
# Multi-letter words have lower priority so spelled-out letters are preferred.
45-
word = add_weight(closure(self.ALPHA, 2), 0.01)
46-
token = alpha_or_digit | symbols | word
35+
char = self.ALPHA | digit | zero
36+
word = add_weight(closure(self.ALPHA, 2), 0.1)
37+
token = char | symbols | word
38+
first_token = char | difference(word, accep("dot"))
39+
component = first_token + closure(ds + token)
4740

48-
# A component is one or more tokens separated by spaces
49-
component = token + closure(ds + token)
41+
dot = cross("dot", ".")
42+
domain = component + (ds + dot + ds + component).plus
5043

5144
username = insert('username: "') + component + insert('"')
45+
domain_field = insert('domain: "') + domain + insert('"')
5246

53-
# Domain: component(s) separated by "dot" => "."
54-
dot = cross("dot", ".")
55-
domain_content = component + closure(ds + dot + ds + component)
56-
domain = insert('domain: "') + domain_content + insert('"')
57-
58-
# Email: username at domain
59-
graph_email = (
60-
username
61-
+ ds
62-
+ delete("at")
63-
+ ds
64-
+ insert(" ")
65-
+ domain
66-
)
67-
68-
# URL protocol: "h t t p colon slash slash" or "h t t p s colon slash slash"
47+
# Email: X at Y dot Z (requires "at" keyword)
48+
graph_email = username + ds + delete("at") + ds + insert(" ") + domain_field
49+
50+
# URL: requires protocol or www prefix
6951
http = cross("h t t p", "http")
7052
https = cross("h t t p s", "https")
71-
colon_slash_slash = cross(" colon slash slash ", "://")
72-
protocol_start = (http | https) + colon_slash_slash
73-
74-
# www prefix
53+
protocol = (http | https) + cross(" colon slash slash ", "://")
7554
www = cross("w w w", "www")
7655

77-
# URL: [protocol] [www.] domain
78-
url_content = (
79-
closure(protocol_start, 0, 1)
80-
+ closure(www + ds + dot + ds, 0, 1)
81-
+ domain_content
82-
)
83-
graph_url = insert('protocol: "') + url_content + insert('"')
56+
# protocol + [www.] + domain
57+
url_with_protocol = protocol + closure(www + ds + dot + ds, 0, 1) + domain
58+
# www. + domain (no protocol)
59+
url_with_www = www + ds + dot + ds + domain
60+
# domain only (must have dot): nvidia dot com
61+
url_domain_only = domain
62+
63+
graph_url = insert('protocol: "') + (url_with_protocol | url_with_www | url_domain_only) + insert('"')
8464

8565
final_graph = graph_email | graph_url
8666
self.tagger = self.add_tokens(final_graph)
8767

8868
def build_verbalizer(self):
89-
username = (
90-
delete("username:")
91-
+ self.DELETE_SPACE
92-
+ delete('"')
93-
+ self.NOT_QUOTE.plus
94-
+ delete('"')
95-
)
96-
domain = (
97-
delete("domain:")
98-
+ self.DELETE_SPACE
99-
+ delete('"')
100-
+ self.NOT_QUOTE.plus
101-
+ delete('"')
102-
)
103-
protocol = (
104-
delete("protocol:")
105-
+ self.DELETE_SPACE
106-
+ delete('"')
107-
+ self.NOT_QUOTE.plus
108-
+ delete('"')
109-
)
110-
111-
# Email: username@domain
69+
username = delete("username:") + self.DELETE_SPACE + delete('"') + self.NOT_QUOTE.plus + delete('"')
70+
domain = delete("domain:") + self.DELETE_SPACE + delete('"') + self.NOT_QUOTE.plus + delete('"')
71+
protocol = delete("protocol:") + self.DELETE_SPACE + delete('"') + self.NOT_QUOTE.plus + delete('"')
72+
11273
graph_email = username + self.DELETE_SPACE + insert("@") + domain
113-
# URL: just output the protocol content directly
11474
graph_url = protocol
11575

116-
graph = graph_email | graph_url
117-
self.verbalizer = self.delete_tokens(graph)
76+
self.verbalizer = self.delete_tokens(graph_email | graph_url)

0 commit comments

Comments
 (0)