Skip to content

IndexError on Thai words containing ฤ (U+0E24) #1444

Description

@nikola

royin romanization raises IndexError: string index out of range on Thai words containing ฤ (U+0E24, THAI CHARACTER RU) — e.g. ฤก, ฤดู, ฤทธิ์ — which are common in real text.

Reproduction

from pythainlp.transliterate import romanize
romanize("ฤดู", engine="royin")   # IndexError: string index out of range

Root cause

royin carries two inconsistent definitions of "consonant":

  • _CONSONANTS (the romanization table) has 45 keys: the 44 thai_consonants plus ฤ (_CONSONANTS['ฤ'] == ['rue', '']).
  • _RE_CONSONANT = re.compile(f"[{thai_consonants}]") matches only the 44 — it does not match ฤ.

In _romanize, the consonant list is built from the regex:

consonants = _RE_CONSONANT.findall(word)          # ← excludes ฤ
...
word = _replace_consonants(word, "".join(consonants))

but _replace_consonants advances its index j into consonants for every character it treats as a consonant via word[i] not in _CONSONANTS — which includes ฤ. So when a word contains ฤ, the loop increments j more times than len(consonants), and consonants[j] indexes past the end → IndexError. (It only crashes when ฤ pushes j past the end — e.g. ฤ early with few following consonants, as in ฤดู; in longer words it silently mis-indexes instead, which is its own latent correctness bug.)

Fix

Build the consonant list with the same predicate the loop uses, so the two stay in sync (and ฤ is romanized through its existing _CONSONANTS entry):

 def _romanize(word: str) -> str:
     # Special case: single ห character should be empty (silent)
     if word == "ห":
         return ""

     word = _replace_vowels(_normalize(word))
-    consonants = _RE_CONSONANT.findall(word)
+    consonants = [c for c in word if c in _CONSONANTS]

     # 2-character word, all consonants
     if len(word) == 2 and len(consonants) == 2:

Validation

  • 0 regressions across 15 common tokens (บริษัท, จำกัด, มหาชน, ไทย, กรุงเทพ, ธนาคาร, ประเทศไทย, พลังงาน, อุตสาหกรรม, เทคโนโลยี, การบินไทย, ปตท, ห, ก, สตริง) — byte-identical output.
  • Previously-crashing ฤ words now romanize: ฤก→rueok, ฤดู→rueadu, ฤทธิ์→rueot.
  • ฤ words that did not crash before (พฤษภาคม, อังกฤษ, พฤหัสบดี, นฤมล, ทฤษฎี) produce identical output.

Suggested test (tests/test_transliterate.py)

def test_romanize_royin_ru(self):
    # ฤ (U+0E24) must not raise IndexError (regression test)
    self.assertIsNotNone(romanize("ฤดู", engine="royin"))
    self.assertIsNotNone(romanize("ฤทธิ์", engine="royin"))
    self.assertIsNotNone(romanize("ฤก", engine="royin"))

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugbugs in the library

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions