Add comprehensive robustness tests for real-world edge cases

[Copilot]

What does this changes

Adds a comprehensive test suite (tests/core/test_robustness.py) that tests real-world edge cases important for production usage:

• 7 test methods covering edge cases across multiple categories
• Empty strings & whitespace: spaces, tabs, unicode spaces, ideographic spaces, mixed whitespace (10 test cases)
• Special characters: BOM, control chars, zero-width chars, smart quotes, various dashes (13 test cases)
• Truncated unicode: surrogate pairs, mid-encoding cuts, malformed sequences (4 test cases)
• Emoji: basic emoji, ZWJ sequences, skin tone modifiers, flags, mixed with Thai text (7 test cases)
• Control & hidden characters: zero-width space/joiner, control chars, invisible characters (9 test cases)
• Thai-specific edge cases: combining marks, mixed scripts, repetition marks (5 test cases)
• Very long strings: performance testing with repetitive patterns (3 test cases, addressing issue Report: newmm bug #893)

All tests validate behavior across 4 core tokenization engines (newmm, newmm-safe, longest, mm) to ensure consistent handling.

What was wrong

Text processing needs to handle various real-world edge cases that can cause crashes or incorrect output:

• Users copy/paste from terminals (special chars, BOM, encoding issues)
• Text from different encodings (truncated unicode, surrogate pairs)
• Modern messaging and social media (emoji, ZWJ sequences)
• Invisible characters from various sources (zero-width, control characters)
• Very long strings with repetitive patterns that could cause performance issues (issue Report: newmm bug #893)

Existing tests lacked comprehensive coverage of these production edge cases and multi-engine validation across different tokenization algorithms.

How this fixes it

Implements comprehensive testing for production edge cases with multi-engine validation:

Empty and whitespace handling:

EMPTY_AND_WHITESPACE = [
    "", " ", "\t", "\n", "\r\n",  # Basic
    "\u00a0", "\u2000\u2001",     # Unicode spaces
    "\u3000",                      # Ideographic space (CJK)
]

Special characters from encoding issues:

SPECIAL_CHARS = [
    "\ufeff",          # BOM (Byte Order Mark)
    "\u200c\u200d",    # Zero-width non-joiner, joiner
    "\u201c\u201d",    # Smart quotes (curly quotes)
    "—–-",             # Different dashes
]

Emoji and modern Unicode:

EMOJI_CASES = [
    "😀",              # Basic emoji
    "👨‍👩‍👧‍👦",         # Family (ZWJ sequence)
    "👍🏻",             # With skin tone modifier
    "🇹🇭",             # Flag (regional indicators)
    "สวัสดี😀ครับ",     # Thai with emoji
]

Very long strings (issue #893):

VERY_LONG_STRINGS = [
    "ชิ" * 50,                           # Repetitive single syllable
    "ด้านหน้า" * 20,                      # Repetitive compound word
    "ด้านหน้า" * 10 + "กกกกกก" * 10,    # Mixed patterns
]

Each test category is validated across all 4 tokenization engines to ensure consistent handling of edge cases. The test suite includes 51 test strings covering real-world scenarios from copy/paste operations, terminal input, different encodings, and modern text processing requirements.

Your checklist for this pull request

• Passed code styles and structures
• Passed code linting checks and unit test

Original prompt

please work on the issue #726

Pick categories from the Big List to test robustness, reliability, and security of functions in pythainlp package (the package hosted in this repo).

Put this as a new test suite in a new file.

Start small and gradually grow the categories from the Big List.

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[github-actions]

Hello @Copilot, thank you for submitting a PR! We will respond as soon as possible.

สวัสดี @Copilot ขอบคุณที่ส่ง PR เข้ามา เราจะตอบกลับให้เร็วที่สุดเท่าที่จะทำได้

[coveralls]

coverage: 53.699%. remained the same
when pulling cfb4007 on copilot/add-test-suite-for-pythainlp
into 4553c44 on dev.

[bact]

Is this really helpful? Is the code in this test really able to capture the execution? @copilot

[Copilot]

You're right - the test doesn't actually detect code execution. I've renamed it to test_functions_handle_injection_strings_safely and clarified in the docstring that it's a robustness test verifying functions handle injection patterns without crashing, not an execution detection test. Since PyThaiNLP functions are text processors that don't interpret HTML/JS/SQL, the value is in ensuring they can handle these patterns gracefully. (commit b9e02d0)

[bact]

@copilot word_tokenize() can have different engines, make the test cover it all

[Copilot]

@copilot word_tokenize() can have different engines, make the test cover it all

Done! Updated all word_tokenize tests to cover 4 core engines: newmm, newmm-safe, longest, and mm. Each test now loops through all engines to ensure robustness across different tokenization algorithms. (commit b9d7729)

[bact]

@copilot add test word_tokenize against very long string from #893

[Copilot]

@copilot add test word_tokenize against very long string from #893

Added test for very long strings addressing issue #893. The test includes repetitive patterns ("ชิ" * 50, "ด้านหน้า" * 20) and mixed patterns that can cause ambiguous breaking points. Uses newmm-safe engine which is designed to handle such cases without excessive processing time. (commit fb4a890)

[bact]

@copilot Recheck the worthiness of these new test cases.
If there are checks that already covered by the existing test cases, consider remove them.
If there are checks that probably cannot really cover what they claims (like the code injection, which looks irrelevant), remove them.

[Copilot]

@copilot Recheck the worthiness of these new test cases.
If there are checks that already covered by the existing test cases, consider remove them.
If there are checks that probably cannot really co...

Done! Streamlined the test suite from 18 tests (415 lines) to 2 focused tests (94 lines).

Removed:

• Tests for empty strings, reserved strings, numeric strings → already covered in test_tokenize.py
• Script injection tests → not relevant for text processors that don't interpret HTML/JS/SQL
• Util function tests → already covered in test_util.py

Kept:

• Multi-engine Thai edge cases (combining marks, zero-width spaces, mixed scripts) → unique value
• Very long strings (issue Report: newmm bug #893) → addresses specific performance issue

The remaining tests provide genuine value without duplicating existing coverage. (commit 4063489)

[bact]

@copilot I think these tests are worth to have, let's keep it or add it:

• empty strings and different whitespaces
• special characters that often seen in text in the wild, copy & paste from terminal, weird enconding, different BOM characters, are worth testing
• unicode characters that got truncated at the middle of encoding is worth testing
• emoji and control character dealings are worth testing
• hidden/invisible characters are worth testing

[Copilot]

@copilot I think these tests are worth to have, let's keep it or add it:

• empty strings and different whitespaces
  ...

Added all requested test categories! The test suite now includes:

• Empty strings & whitespace (10 cases): spaces, tabs, unicode spaces, ideographic spaces
• Special characters (13 cases): BOM, control chars, zero-width chars, smart quotes, various dashes
• Truncated unicode (4 cases): surrogate pairs, mid-encoding cuts
• Emoji (7 cases): basic emoji, ZWJ sequences, modifiers, flags, mixed with Thai
• Control & hidden chars (9 cases): zero-width space/joiner, control characters, invisible chars

All tested across 4 tokenization engines. Total: 7 test methods covering real-world edge cases. (commit e23bab7)

[bact]

@copilot there are some bidirectional, hidden, zero-width, or joiner characters in the test code now. This is not visible to human and can create issues. Please put those character as escaped one or as \u unicode point, so it is readable by both machine and human. Make the code more transparent.

[bact]

@copilot got it. We better revert to e23bab7

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud