Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
2f4143f
Initial plan
Copilot Feb 6, 2026
712faf8
Mock large corpus file downloads in tests for speed and stability
Copilot Feb 6, 2026
6f00243
Fix linting issues in test_corpus.py
Copilot Feb 6, 2026
fa9f3a7
Add corpus integrity testing workflow and tests
Copilot Feb 6, 2026
fc1aaec
Add README for corpus integrity tests
Copilot Feb 6, 2026
0e28b15
Add explicit permissions to corpus-integrity workflow for security
Copilot Feb 6, 2026
b6f2bc1
Fix ruff issues and update tests/README.md with corpus_integrity refe…
Copilot Feb 6, 2026
1addde9
Fix unittest command syntax in tests/README.md
Copilot Feb 6, 2026
8457587
Update README.md
bact Feb 6, 2026
bb11971
Suppress CLI test output and mock network calls to avoid corpus downl…
Copilot Feb 6, 2026
01879e3
Add corpus catalog tests and rename workflow to "Corpus Data Test"
Copilot Feb 6, 2026
1b03213
Address PR review feedback: remove trailing whitespace and unnecessar…
Copilot Feb 6, 2026
8ffba33
Update README.md
bact Feb 6, 2026
aa1657e
Restore assertion with clarifying comment to verify command success
Copilot Feb 6, 2026
6a4d46a
Simplify test assertion and make README more concise
Copilot Feb 6, 2026
464ca6b
Rename corpus_integrity to corpus_data to reflect expanded test scope
Copilot Feb 6, 2026
1bcfbe3
Simplify to "Corpus test" with tests.corpus package name
Copilot Feb 6, 2026
dde237b
Improve clarity of corpus test suite description
bact Feb 6, 2026
88a9b3c
Fix formatting and capitalization in README.md
bact Feb 6, 2026
7053488
Update README with SPDX metadata
bact Feb 6, 2026
03e53bc
Update README with SPDX metadata
bact Feb 6, 2026
99cfe6a
Fix SPDX headers formatting in README.md
bact Feb 6, 2026
d2f6d6b
Update corpus.yml
bact Feb 6, 2026
2a817fd
Correct README formatting for corpus test instructions
bact Feb 6, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions .github/workflows/corpus.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# SPDX-FileCopyrightText: 2026 PyThaiNLP Project
# SPDX-FileType: SOURCE
# SPDX-License-Identifier: Apache-2.0

name: Corpus test

on:
push:
branches:
- dev
paths:
- ".github/workflows/corpus.yml"
- "pythainlp/corpus/**"
- "tests/corpus/**"
pull_request:
branches:
- dev
paths:
- ".github/workflows/corpus.yml"
- "pythainlp/corpus/**"
- "tests/corpus/**"

# Avoid duplicate runs for the same source branch and repository
concurrency:
group: >-
${{ github.workflow }}-${{
github.event.pull_request.head.repo.full_name || github.repository
}}-${{ github.head_ref || github.ref_name }}
cancel-in-progress: true

jobs:
corpus:
runs-on: ubuntu-latest
permissions:
contents: read

steps:
- name: Checkout
uses: actions/checkout@v6

- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: "3.13"
cache: "pip"

- name: Install dependencies
run: |
pip install --upgrade pip
pip install .

- name: Test corpus catalog
env:
PYTHONIOENCODING: utf-8
run: |
python -m unittest discover -s tests/corpus -p "test_catalog*.py" -v

- name: Test built-in corpus files
env:
PYTHONIOENCODING: utf-8
run: |
python -m unittest discover -s tests/corpus -p "test_builtin_*.py" -v

- name: Test downloadable corpus files
env:
PYTHONIOENCODING: utf-8
run: |
python -m unittest discover -s tests/corpus -p "test_downloadable_*.py" -v
27 changes: 26 additions & 1 deletion tests/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
SPDX-FileCopyrightText: 2026 PyThaiNLP Project
SPDX-FileType: DOCUMENTATION
SPDX-License-Identifier: Apache-2.0
---

# Test suites and execution

To run a test suite, run:
Expand Down Expand Up @@ -155,7 +160,6 @@ By separating tests by dependency group, we can:
- Requires: Internet connection, may involve large downloads
- Test case class suffix: `TestCaseN`


## Robustness tests (test_robustness.py)

A comprehensive test suite within core tests that tests edge cases important
Expand All @@ -169,3 +173,24 @@ for real-world usage:
- Thai-specific edge cases with combining characters and mixed scripts
- Multi-engine robustness testing across all core tokenization engines
- Very long strings that can cause performance issues (issue #893)

## Corpus test (corpus/)

A separate test suite that verifies the integrity, format, parseability,
and catalog functionality of corpus in PyThaiNLP.

These tests are separate from regular unit tests because they test actual
file loading and parsing (not mocked), require network access, and
can be resource intensive.

For detailed information about corpus test, see:
[tests/corpus/README.md](corpus/README.md)

The corpus test is triggered automatically via GitHub Actions
when changes are made to `pythainlp/corpus/**` or `tests/corpus/**`.

Run corpus test:

```shell
python -m unittest tests.corpus
```
Loading
Loading