royin romanization raises IndexError: string index out of range on Thai words containing ฤ (U+0E24, THAI CHARACTER RU) — e.g. ฤก, ฤดู, ฤทธิ์ — which are common in real text.
Reproduction
from pythainlp.transliterate import romanize
romanize("ฤดู", engine="royin") # IndexError: string index out of range
Root cause
royin carries two inconsistent definitions of "consonant":
_CONSONANTS (the romanization table) has 45 keys: the 44 thai_consonants plus ฤ (_CONSONANTS['ฤ'] == ['rue', '']).
_RE_CONSONANT = re.compile(f"[{thai_consonants}]") matches only the 44 — it does not match ฤ.
In _romanize, the consonant list is built from the regex:
consonants = _RE_CONSONANT.findall(word) # ← excludes ฤ
...
word = _replace_consonants(word, "".join(consonants))
but _replace_consonants advances its index j into consonants for every character it treats as a consonant via word[i] not in _CONSONANTS — which includes ฤ. So when a word contains ฤ, the loop increments j more times than len(consonants), and consonants[j] indexes past the end → IndexError. (It only crashes when ฤ pushes j past the end — e.g. ฤ early with few following consonants, as in ฤดู; in longer words it silently mis-indexes instead, which is its own latent correctness bug.)
Fix
Build the consonant list with the same predicate the loop uses, so the two stay in sync (and ฤ is romanized through its existing _CONSONANTS entry):
def _romanize(word: str) -> str:
# Special case: single ห character should be empty (silent)
if word == "ห":
return ""
word = _replace_vowels(_normalize(word))
- consonants = _RE_CONSONANT.findall(word)
+ consonants = [c for c in word if c in _CONSONANTS]
# 2-character word, all consonants
if len(word) == 2 and len(consonants) == 2:
Validation
- 0 regressions across 15 common tokens (บริษัท, จำกัด, มหาชน, ไทย, กรุงเทพ, ธนาคาร, ประเทศไทย, พลังงาน, อุตสาหกรรม, เทคโนโลยี, การบินไทย, ปตท, ห, ก, สตริง) — byte-identical output.
- Previously-crashing ฤ words now romanize: ฤก→rueok, ฤดู→rueadu, ฤทธิ์→rueot.
- ฤ words that did not crash before (พฤษภาคม, อังกฤษ, พฤหัสบดี, นฤมล, ทฤษฎี) produce identical output.
Suggested test (tests/test_transliterate.py)
def test_romanize_royin_ru(self):
# ฤ (U+0E24) must not raise IndexError (regression test)
self.assertIsNotNone(romanize("ฤดู", engine="royin"))
self.assertIsNotNone(romanize("ฤทธิ์", engine="royin"))
self.assertIsNotNone(romanize("ฤก", engine="royin"))
royinromanization raisesIndexError: string index out of rangeon Thai words containing ฤ (U+0E24, THAI CHARACTER RU) — e.g. ฤก, ฤดู, ฤทธิ์ — which are common in real text.Reproduction
Root cause
royincarries two inconsistent definitions of "consonant":_CONSONANTS(the romanization table) has 45 keys: the 44thai_consonantsplus ฤ (_CONSONANTS['ฤ'] == ['rue', ''])._RE_CONSONANT = re.compile(f"[{thai_consonants}]")matches only the 44 — it does not match ฤ.In
_romanize, the consonant list is built from the regex:but
_replace_consonantsadvances its indexjintoconsonantsfor every character it treats as a consonant viaword[i] not in _CONSONANTS— which includes ฤ. So when a word contains ฤ, the loop incrementsjmore times thanlen(consonants), andconsonants[j]indexes past the end →IndexError. (It only crashes when ฤ pushesjpast the end — e.g. ฤ early with few following consonants, as in ฤดู; in longer words it silently mis-indexes instead, which is its own latent correctness bug.)Fix
Build the consonant list with the same predicate the loop uses, so the two stay in sync (and ฤ is romanized through its existing
_CONSONANTSentry):Validation
Suggested test (tests/test_transliterate.py)