Merge pull request #1299 from PyThaiNLP/dev

Update dev to add-phupa-dataset branch

Author: bact

.github/ISSUE_TEMPLATE/feature_request.md

@@ -5,7 +5,7 @@ about: Propose a change or addition เสนอความสามารถ
 
 ## Detailed description
 
-<!--- Provide a detailed description of the change or addition you are proposing -->
+<!--- Provide a detailed description of the change or addition -->
 <!--- รายละเอียดการเปลี่ยนแปลงหรือสิ่งเพิ่มเติมที่คุณกำลังเสนอ -->
 
 ## Context
@@ -17,8 +17,8 @@ about: Propose a change or addition เสนอความสามารถ
 
 ## Possible implementation
 
-<!--- Not obligatory, but suggest an idea for implementing addition or change -->
-<!--- ไม่จำเป็นต้องใส่ แต่คุณสามารถแนะนำได้ว่าการเปลี่ยนแปลงหรือเพิ่มเติมดังกล่าวน่าจะทำได้ด้วยวิธีไหน -->
+<!--- Suggest an idea for implementing the change or addition (Optional) -->
+<!--- แนะนำว่าจะเปลี่ยนแปลงหรือเพิ่มเติมได้ด้วยวิธีไหน (ไม่จำเป็นต้องใส่) -->
 
 ## Your environment
 

.github/copilot-instructions.md

@@ -12,7 +12,7 @@
       configuration. Communicate this to the users during code review.
       See <https://www.nltk.org/py-modindex.html>.
 - [ ] The type information analyzer at
-      <https://github.com/PyThaiNLP/pythainlp/blob/dev/build_tools/analysis/type_hint_analyzer.py>
+      <https://github.com/PyThaiNLP/pythainlp/blob/dev/build_tools/analysis/type-analyzer.py>
       can generate information about annotation completeness of
       variables, functions, methods, type aliases, decorators, and
       classes in the PyThaiNLP repo.
@@ -23,6 +23,13 @@
       please refer to Python tyep specification when in doubt.
 - [ ] Complete type annotations for function, method, class, variable, etc.
       Maintain near-100% type annotation coverage.
+- [ ] Add tests for new functionality or behavior.
+      Keep the test coverage high.
+- [ ] Add test cases to capture edge cases.
+- [ ] Add test cases to cover code branches.
+- [ ] Major changes should be logged in the change log at
+      <https://github.com/PyThaiNLP/pythainlp/blob/dev/CHANGELOG.md>.
+      Provide issue number or PR number if available.
 
 ## Project contribution guidelines
 
@@ -38,9 +45,9 @@
 - [ ] For significant changes, update the CHANGELOG.md file
       to document the changes.
   - [ ] Follow "Keep a Changelog" principles
-        https://keepachangelog.com/en/1.0.0/
+        <https://keepachangelog.com/en/1.0.0/>
   - [ ] Use semantic versioning for version numbers
-        https://semver.org/
+        <https://semver.org/>
   - [ ] If it is a breaking change, indicate it clearly in the changelog.
     - [ ] Provide migration instructions if necessary.
 - [ ] Do not leave trailing whitespaces in the code or documentation files,
@@ -98,10 +105,10 @@
       in names.
 - [ ] For URLs/IRIs, use lowercase letters and hyphens to separate words
       (e.g., `my-api-endpoint`) and follow W3C Cool URIs for the Semantic Web:
-      https://www.w3.org/TR/cooluris/
+      <https://www.w3.org/TR/cooluris/>
 - [ ] Consult Schema.org vocabularies when deciding about names.
 - [ ] Consult "Style Guidelines for Naming and Labeling Ontologies in the
-      Multilingual Web" https://www.researchgate.net/publication/277224472
+      Multilingual Web" <https://www.researchgate.net/publication/277224472>
 
 ## Tidy code and documentation
 
@@ -121,7 +128,7 @@
 ## File header
 
 - [ ] When possible, put relevant SPDX File Tags at file header.
-      See https://spdx.github.io/spdx-spec/v2.3/file-information/
+      See <https://spdx.github.io/spdx-spec/v2.3/file-information/>
   - [ ] SPDX-FileContributor
   - [ ] SPDX-FileCopyrightText
   - [ ] Default SPDX-FileType for code is "SOURCE"
@@ -183,7 +190,7 @@
 ## API
 
 - [ ] The overall architecture, code, and API endpoints should follow the latest
-      version of OpenAPI specification at https://spec.openapis.org/oas/
+      version of OpenAPI specification at <https://spec.openapis.org/oas/>
 - [ ] API endpoints must use proper HTTP return codes.
 - [ ] Follow web best practices as recommended by OpenAPI, IETF, W3C, etc.
 
@@ -208,7 +215,7 @@
       as much as possible.
   - [ ] Follow best practices and standard Python type hint patterns.
   - [ ] Use mypy as an assistant.
-    - [ ] mypy is in "dev" optional dependency. 
+    - [ ] mypy is in "dev" optional dependency.
     - [ ] Sometimes mypy may report errors wrongly due to cache issues.
           Try to reset the cache if unexpected errors occured.
   - [ ] Use pyright, pyrefly, and pytype for second opinions.

.github/workflows/corpus.yml

@@ -0,0 +1,68 @@
+# SPDX-FileCopyrightText: 2026 PyThaiNLP Project
+# SPDX-FileType: SOURCE
+# SPDX-License-Identifier: Apache-2.0
+
+name: Corpus test
+
+on:
+  push:
+    branches:
+      - dev
+    paths:
+      - ".github/workflows/corpus.yml"
+      - "pythainlp/corpus/**"
+      - "tests/corpus/**"
+  pull_request:
+    branches:
+      - dev
+    paths:
+      - ".github/workflows/corpus.yml"
+      - "pythainlp/corpus/**"
+      - "tests/corpus/**"
+
+# Avoid duplicate runs for the same source branch and repository
+concurrency:
+  group: >-
+    ${{ github.workflow }}-${{
+      github.event.pull_request.head.repo.full_name || github.repository
+    }}-${{ github.head_ref || github.ref_name }}
+  cancel-in-progress: true
+
+jobs:
+  corpus:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v6
+
+      - name: Set up Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: "3.13"
+          cache: "pip"
+
+      - name: Install dependencies
+        run: |
+          pip install --upgrade pip
+          pip install .
+
+      - name: Test corpus catalog
+        env:
+          PYTHONIOENCODING: utf-8
+        run: |
+          python -m unittest discover -s tests/corpus -p "test_catalog*.py" -v
+
+      - name: Test built-in corpus files
+        env:
+          PYTHONIOENCODING: utf-8
+        run: |
+          python -m unittest discover -s tests/corpus -p "test_builtin_*.py" -v
+
+      - name: Test downloadable corpus files
+        env:
+          PYTHONIOENCODING: utf-8
+        run: |
+          python -m unittest discover -s tests/corpus -p "test_downloadable_*.py" -v

.github/workflows/markdownlint.yml

@@ -0,0 +1,36 @@
+# SPDX-FileCopyrightText: 2026 PyThaiNLP contributors
+# SPDX-FileType: SOURCE
+# SPDX-License-Identifier: Apache-2.0
+
+name: Markdown lint
+
+on:
+  push:
+    branches:
+      - dev
+    paths:
+      - ".github/workflows/markdownlint.yml"
+      - "**/*.md"
+  pull_request:
+    branches:
+      - dev
+    paths:
+      - ".github/workflows/markdownlint.yml"
+      - "**/*.md"
+
+jobs:
+  markdownlint:
+    name: Run Markdown lint
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+
+      - uses: DavidAnson/markdownlint-cli2-action@v22
+        with:
+          globs: |
+            **/*.md
+            !License.md
+            !LICENSE.md

.github/workflows/unittest.yml

@@ -1,10 +1,19 @@
 # SPDX-FileCopyrightText: 2016-2026 PyThaiNLP Project
 # SPDX-License-Identifier: Apache-2.0
 
+# To reduce test time and save resources,
+# unit tests that download large corpus files will not be covered
+# in this workflow.
+# A separated corpus test workflow is in
+# https://github.com/PyThaiNLP/pythainlp/blob/dev/.github/workflows/corpus.yml
+# which will be triggered only if there is a change in a corpus.
+
 name: Unit test
 
 on:
   push:
+    branches:
+    - dev
     paths:
       - ".github/workflows/unittest.yml"
       - "bin/**"

.gitignore

@@ -129,3 +129,7 @@ logs/
 # Temp files
 *.tmp
 *.temp
+
+# Type annotation analyzer
+build_tools/output/*.csv
+build_tools/output/*.json

.markdownlintrc .markdownlint.json.markdownlintrc renamed to .markdownlint.json

@@ -1,8 +1,9 @@
 {
   "default": true,
   "MD013": {
-    "line_length": 150,
-    "tables": false
+    "line_length": 80,
+    "tables": false,
+    "code_blocks": false
   },
   "MD033": false,
   "MD041": false,

CHANGELOG.md

@@ -11,9 +11,10 @@ Notable changes between versions.
 - For full release notes, see:
   <https://github.com/PyThaiNLP/pythainlp/releases>
 - For detailed commit changes, see:
-  <https://github.com/PyThaiNLP/pythainlp/compare/v5.2.0...dev> (select tags to compare)
+  <https://github.com/PyThaiNLP/pythainlp/compare/v5.2.0...dev>
+  (select tags to compare)
 
-# Version 5.2.0 -> dev
+## Version 5.2.0 -> dev
 
 Minimum required version is now 3.9.
 Support for Python 3.7 and 3.8 has been removed.
@@ -34,6 +35,8 @@ See PR for prompt and details.
 - Make package zip-safe #1212
 - Ensure thread-safety for tokenizers #1213
 - Add Thai-NNER integration with top-level entity filtering #1221
+- Reorganize noauto test suite by dependency groups
+  (torch, tensorflow, onnx, cython, network) #935
 - Improved documentation; code cleanup; more tests
 
 ## Version 5.1.2 -> 5.2.0
@@ -46,26 +49,31 @@ See PR for prompt and details.
 - Add budoux #1161
 - Remove conceptnet #1103
 - fix the connectivity of cli commands #1154
-- Fix Docker build failure, add docker compose file for convenience #1132
+- Fix Docker build failure, add docker compose file
+  for convenience #1132
 
 ## Version 5.1.1 -> 5.1.2
 
 - Update romanize docs and keep space #1110
 
 ## Version 5.1.0 -> 5.1.1
 
-- PR Description: Refactor thai_consonants_all to Use set in syllable.py #1087
+- PR Description: Refactor thai_consonants_all to
+  use set in syllable.py #1087
 - ThaiTransliterator: Select 1D CPU int64 tensor device #1089
 
 ## Version 5.0.5 -> 5.1.0
 
 - Add Thai Discourse Treebank postag #910
 - Add Thai Universal Dependency Treebank postag #916
 - Add Thai G2P v2 Grapheme-to-Phoneme model #923
-- Add support for list of strings as input to sent_tokenize() #927
-- Add pythainlp.tools.safe_print to handle UnicodeEncodeError on console #969
+- Add support for list of strings as input
+  to sent_tokenize() #927
+- Add pythainlp.tools.safe_print to handle UnicodeEncodeError
+  on console #969
 - Fix collate() to consider tonemark in ordering #926
-- Fix nlpo3.load_dict() that never print error msg when not success #979
+- Fix nlpo3.load_dict() that never print error msg when
+  not success #979
 - Add Thai Solar Date convert to Thai Lunar Date #998
 - Add Thai pangram text #1045
 - Remove clause_tokenize #1024
@@ -77,8 +85,8 @@ See PR for prompt and details.
 
 ## Version 5.0.3 -> 5.0.4
 
-- Fix: pythainlp.util.maiyamok does not duplicate words when more than one
-  Maiyamok is used #917
+- Fix: pythainlp.util.maiyamok does not duplicate words
+  when more than one Maiyamok is used #917
 
 ## Version 5.0.2 -> 5.0.3
 
@@ -87,8 +95,10 @@ See PR for prompt and details.
 
 ## Version 5.0.1 -> 5.0.2
 
-- Fix: crfcut: Ensure splitting of sentences using terminal punctuation #905
+- Fix: crfcut: Ensure splitting of sentences using terminal
+  punctuation #905
 
 ## Version 5.0.0 -> 5.0.1
 
-- Fix: delay calling syllable_tokenize to avoid pycrfsuite ImportError #901
+- Fix: delay calling syllable_tokenize to avoid
+  pycrfsuite ImportError #901