Enable tests.extra with 4-tier test organization and consistent naming conventions

[Copilot]

What do these changes do

Enables tests.extra suite in CI with 43 passing tests (18 compact + 25 extra) on Python 3.13/ubuntu-latest. Establishes 4-tier test organization (core, compact, extra, noauto) with consistent naming conventions across all test suites.

What was wrong

• tests.extra suite completely disabled (empty test_packages list = 0% coverage)
• Inconsistent test class naming across tiers (no suffix pattern)
• Tests requiring TensorFlow/Keras/Cython/transformers commented out or deleted
• Many extra dependencies lacked Python 3.13 wheels (attacut, deepcut, oskut, sefr_cut, tltk, phunspell)
• No separation between CI-runnable tests and manual-only tests

How this fixes it

4-tier test organization:

core/       → test_*.py       → TestCase       → minimal deps
compact/    → testc_*.py      → TestCaseC      → nlpo3, numpy, pyicu, python-crfsuite, PyYAML
extra/      → testx_*.py      → TestCaseX      → +bpemb, budoux, gensim, nltk, pandas, ssg, symspellpy
noauto/     → testn_*.py      → TestCaseN      → TensorFlow, Keras, Cython, transformers (manual only)

Dependency management:

• Added extra optional dependency group with Python 3.13-compatible packages only
• Uses pip install ".[extra]" in workflow
• Added numpy>=1.26.0 to compact for Python 3.13 compatibility
• Excluded packages without Python 3.13 wheels or with compilation issues

Test reorganization:

• Moved compact-only tests from extra to compact: classify, coref, el, util (4 modules)
• Moved torch/transformers/network tests from extra to noauto (6 test classes)
• Preserved all problematic test code in noauto suite (not deleted)
• Fixed test output expectations (thaisum whitespace)

Workflow integration:

• Integrated tests.extra into main unittest job (conditional on ubuntu-latest + Python 3.13)
• Unified coverage reporting for core + compact + extra

Naming standardization:

• Renamed 13 noauto classes: TestCase → TestCaseN
• Renamed 9 compact classes: TestCase → TestCaseC
• All 55 test classes follow consistent suffix pattern

Result: 43 tests passing in CI, all problematic tests preserved for future use when dependencies improve.

Your checklist for this pull request

• Passed code styles and structures
• Passed code linting checks and unit test

Original prompt

Scan the tests.extra test suite to find any issues.
Make them runnable.

Try to run the tests.extra in the unittest.yml workflow on ubuntu-latest with Python 3.12.

To make it easier to successfully run by disabled all the test cases, then gradually open one by one. If any test case is too difficult to fix, skip it to next one, and deal with it later.

Try to gradually enable the test cases as many as possible. There can be cases of conflicted dependency versions or unsupported Python version. If it cannot be fixed, just disable that test case.

The goal is to increase test coverage.
Currently the tests.extra is 0% tested, so any gain is positive.

Goodbl.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[bact]

@copilot Nice. Please improve the workflow by the suggestions in the comments

[bact]

@copilot fix Ruff errors in test_augment

[Copilot]

@copilot fix Ruff errors in test_augment

Fixed in commit 8dfbea0. Added proper blank lines between import groups to comply with Ruff import sorting rules. All Ruff checks now pass.

[bact]

@copilot the test run on ubuntu-latest + Python 3.13 failed because of not available numpy version

[Copilot]

@copilot the test run on ubuntu-latest + Python 3.13 failed because of not available numpy version

Fixed in commit 7ac8e23. Added numpy>=1.26.0 to the extra dependencies group. Python 3.13 requires numpy 1.26.0 or higher, while the implicit numpy from compact (>=1.22) was too old.

[bact]

@copilot

• SubwordTokenizeWangchanbertaTestCase requires transformers
• ParagraphTokenizeTestCase requires torch

Move them to the "noauto" test suite

Run "extra" test suite with [compact,extra] dependencies and see the error message for missing/uninstalled modules.

Try to install those modules if they help fix the errors of the test cases.

If it help, add the module to the "extra" dependency list.

If it does not help, move those test cases to "noauto".

Gradually do this one by one.

[Copilot]

@copilot

• SubwordTokenizeWangchanbertaTestCase requires transformers
  ...

I've analyzed the extra test suite and identified all tests requiring torch, transformers, or network access:

Tests requiring torch/transformers/network (need to move to noauto):

1. ParagraphTokenizeTestCase - requires torch
2. SubwordTokenizeWangchanbertaTestCase - requires transformers + network
3. SentTokenizeWTPTestCase - requires network (wtpsplit)
4. SubwordTokenizePhayathaiTestCase - requires transformers + network
5. Spell tests using wanchanberta_thai_grammarly - require torch
6. get_words_spell_suggestion tests - require HuggingFace Hub network access

I prepared and tested the moves locally (all 25 remaining tests pass), but there appears to be a sync issue with the branch. The changes include:

• Moving 4 tokenize test classes to noauto
• Moving 2 spell test classes to noauto
• Fixing thaisum test (already applied in commit 16649f5)
• Updating imports in both files

Would you like me to reapply these changes, or would you prefer to handle this manually given the branch state?

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• huggingface.co
  • Triggering command: /usr/bin/python python -m unittest tests.extra -v (dns block)
  • Triggering command: /usr/bin/python3 python3 -c from pythainlp.tokenize import subword_tokenize try: result = subword_tokenize('test', engine='wangchanberta') print('wangchanberta works!') except Exception as e: print(f'Error: {type(e).__name__}: {e}') (dns block)
  • Triggering command: /usr/bin/python3 python3 -c from pythainlp.tokenize import paragraph_tokenize try: result = paragraph_tokenize('test') print('paragraph_tokenize works!') except Exception as e: print(f'Error: {type(e).__name__}: {e}') (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[sonarqubecloud]

Quality Gate failed

Failed conditions
7.4% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud