diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md index 4f3b414eb..4acbb6ace 100644 --- a/.github/ISSUE_TEMPLATE/feature_request.md +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -5,7 +5,7 @@ about: Propose a change or addition เสนอความสามารถ ## Detailed description - + ## Context @@ -17,8 +17,8 @@ about: Propose a change or addition เสนอความสามารถ ## Possible implementation - - + + ## Your environment diff --git a/README.md b/README.md index 54ee858c7..d2c191c66 100644 --- a/README.md +++ b/README.md @@ -13,12 +13,12 @@ [![Facebook](https://img.shields.io/badge/Facebook-0866FF?style=flat&logo=facebook&logoColor=white)](https://www.facebook.com/pythainlp/) [![Chat on Matrix](https://matrix.to/img/matrix-badge.svg)](https://matrix.to/#/#thainlp:matrix.org) -[pythainlp.org](https://pythainlp.org/) | -[Tutorials](https://pythainlp.org/tutorials) | -[License info](https://pythainlp.org/dev-docs/notes/license.html) | -[Model cards](https://github.com/PyThaiNLP/pythainlp/wiki/Model-Cards) | -[Adopters](https://github.com/PyThaiNLP/pythainlp/blob/dev/INTHEWILD.md) | -*[เอกสารภาษาไทย](https://github.com/PyThaiNLP/pythainlp/blob/dev/README_TH.md)* +[pythainlp.org](https://pythainlp.org/) +| [Tutorials](https://pythainlp.org/tutorials) +| [License info](https://pythainlp.org/dev-docs/notes/license.html) +| [Model cards](https://github.com/PyThaiNLP/pythainlp/wiki/Model-Cards) +| [Adopters](https://github.com/PyThaiNLP/pythainlp/blob/dev/INTHEWILD.md) +| *[เอกสารภาษาไทย](https://github.com/PyThaiNLP/pythainlp/blob/dev/README_TH.md)* Designed to be a Thai-focused counterpart to [NLTK](https://www.nltk.org/), **PyThaiNLP** provides standard tools for linguistic analysis under @@ -186,9 +186,12 @@ with this BibTeX entry: ------
- We have only one official repository at https://github.com/PyThaiNLP/pythainlp and another mirror at https://gitlab.com/pythainlp/pythainlp + We have only one official repository at + https://github.com/PyThaiNLP/pythainlp and another mirror at + https://gitlab.com/pythainlp/pythainlp
- Beware of malware if you use code from mirrors other than the official two on GitHub and GitLab. + Beware of malware if you use code from mirrors other than the + official two on GitHub and GitLab.
diff --git a/README_TH.md b/README_TH.md index ee1696616..58f92ea9b 100644 --- a/README_TH.md +++ b/README_TH.md @@ -13,12 +13,12 @@ [![Facebook](https://img.shields.io/badge/Facebook-0866FF?style=flat&logo=facebook&logoColor=white)](https://www.facebook.com/pythainlp/) [![Chat on Matrix](https://matrix.to/img/matrix-badge.svg)](https://matrix.to/#/#thainlp:matrix.org) -[pythainlp.org](https://pythainlp.org/) | -[วิธีใช้งาน](https://pythainlp.org/tutorials) | -[ข้อมูลสัญญาอนุญาต](https://pythainlp.org/dev-docs/notes/license.html) | -[ใบข้อมูลโมเดล](https://github.com/PyThaiNLP/pythainlp/wiki/Model-Cards) | -[ใครใช้ PyThaiNLP บ้าง](https://github.com/PyThaiNLP/pythainlp/blob/dev/INTHEWILD.md) | -*[English](https://github.com/PyThaiNLP/pythainlp/blob/dev/README.md)* +[pythainlp.org](https://pythainlp.org/) +| [วิธีใช้งาน](https://pythainlp.org/tutorials) +| [ข้อมูลสัญญาอนุญาต](https://pythainlp.org/dev-docs/notes/license.html) +| [ใบข้อมูลโมเดล](https://github.com/PyThaiNLP/pythainlp/wiki/Model-Cards) +| [ใครใช้ PyThaiNLP บ้าง](https://github.com/PyThaiNLP/pythainlp/blob/dev/INTHEWILD.md) +| *[English](https://github.com/PyThaiNLP/pythainlp/blob/dev/README.md)* **PyThaiNLP** ถูกออกแบบให้เป็นเครื่องมือมาตรฐานสำหรับการวิเคราะห์ภาษาศาสตร์ภาษาไทย ภายใต้สัญญาอนุญาต Apache-2.0 โดยข้อมูลและโมเดลอยู่ภายใต้ CC0-1.0 และ CC-BY-4.0 @@ -100,7 +100,8 @@ pip install "pythainlp[extra1,extra2,...]" -สำหรับรายละเอียด dependencies สามารถดูได้ที่ส่วน `[project.optional-dependencies]` ใน +สำหรับรายละเอียด dependencies +สามารถดูได้ที่ส่วน `[project.optional-dependencies]` ใน [`pyproject.toml`](https://github.com/PyThaiNLP/pythainlp/blob/dev/pyproject.toml) ## ไดเรกทอรีข้อมูล @@ -225,9 +226,12 @@ PyThaiNLP ดาวน์โหลดข้อมูล (ดูแค็ตต ------
- เรามีที่เก็บข้อมูลอย่างเป็นทางการที่เดียวที่ https://github.com/PyThaiNLP/pythainlp และมีที่เก็บสำเนาอีกแห่งที่ https://gitlab.com/pythainlp/pythainlp + เรามีที่เก็บข้อมูลอย่างเป็นทางการที่เดียวที่ + https://github.com/PyThaiNLP/pythainlp และมีที่เก็บสำเนาอีกแห่งที่ + https://gitlab.com/pythainlp/pythainlp
- โปรดระมัดระวังซอฟต์แวร์ประสงค์ร้ายหรือมัลแวร์ ถ้าคุณใช้โค้ดจากที่เก็บข้อมูลอื่นนอกเหนือจากที่ GitHub และ GitLab ข้างต้น + โปรดระมัดระวังซอฟต์แวร์ประสงค์ร้ายหรือมัลแวร์ + ถ้าคุณใช้โค้ดจากที่เก็บข้อมูลอื่นนอกเหนือจากที่ GitHub และ GitLab ข้างต้น
diff --git a/SECURITY.md b/SECURITY.md index 169e2f5a0..d2961ac84 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -20,7 +20,11 @@ The following security improvements are planned for future releases: -- Migrate from pickle to a safer serialization format such as JSON or MessagePack. -- Upgrade the hashing algorithm for integrity verification from MD5 to SHA-256 or SHA-3. +- Migrate from pickle to a safer serialization format such as JSON or + [MessagePack][]. +- Upgrade the hashing algorithm for integrity verification from MD5 to SHA-256 + or SHA-3. - Implement digital signatures for corpus files to ensure authenticity. - Add version tracking to the corpus to prevent rollback attacks. + +[MessagePack]: https://msgpack.org/ diff --git a/build_tools/analysis/README.md b/build_tools/analysis/README.md index 64d8d541e..9f4582f1d 100644 --- a/build_tools/analysis/README.md +++ b/build_tools/analysis/README.md @@ -90,10 +90,12 @@ easy analysis in spreadsheet applications or data analysis tools. - `output/functions_no_hints.csv` - Functions without any type hints - `output/functions_incomplete_hints.csv` - Functions with partial hints - `output/class_variables_no_hints.csv` - Class variables without type hints -- `output/instance_variables_no_hints.csv` - Instance variables without type hints +- `output/instance_variables_no_hints.csv` - + Instance variables without type hints - `output/module_variables_no_hints.csv` - Module variables without type hints - `output/type_aliases.csv` - All type aliases defined in the codebase -- `output/submodule_summary.csv` - Summary statistics by submodule with mypy errors +- `output/submodule_summary.csv` - + Summary statistics by submodule with mypy errors **CSV Schema:** @@ -155,9 +157,9 @@ cat output/submodule_summary.csv **Example Output:** ```text -================================================================================ +============================================================================== TYPE ANNOTATION COVERAGE ANALYSIS FOR PYTHAINLP -================================================================================ +============================================================================== Repository root: /path/to/pythainlp Output directory: ./output @@ -181,17 +183,17 @@ Analyzed 426 instance variables Analyzed 508 module variables Analyzed 0 type aliases -================================================================================ +============================================================================== OVERALL STATISTICS - FUNCTIONS/METHODS -================================================================================ +============================================================================== Total functions/methods: 720 Complete type hints: 592 (82.22%) Incomplete type hints: 56 ( 7.78%) No type hints: 72 (10.00%) -================================================================================ +============================================================================== OVERALL STATISTICS - VARIABLES -================================================================================ +============================================================================== Total variables: 959 Class variables: 25 Instance variables: 426 @@ -202,7 +204,8 @@ No type hints: 909 (94.79%) ### Automated Analysis -The repository includes a GitHub Actions workflow that automatically runs the type hint analyzer on every push to the `dev` branch: +The repository includes a GitHub Actions workflow that automatically runs +the type hint analyzer on every push to the `dev` branch: - **Workflow**: `.github/workflows/type-hint-analysis.yml` - **Trigger**: Push to `dev` branch @@ -210,11 +213,13 @@ The repository includes a GitHub Actions workflow that automatically runs the ty - **Artifacts**: JSON and CSV files (30-day retention) - **Summary**: Displayed in GitHub Actions UI -The workflow provides continuous monitoring of type hint coverage as the codebase evolves. +The workflow provides continuous monitoring of type hint coverage +as the codebase evolves. ### Type Completeness Standards -This analyzer follows the type completeness guidelines from the Python typing documentation: +This analyzer follows the type completeness guidelines from +the Python typing documentation: The analysis covers: @@ -241,13 +246,16 @@ a library is considered to have complete type hints when: **Type hint status:** -- **Complete:** All parameters and return value have type hints (for functions), or variable has type annotation (for variables) -- **Incomplete:** Some parameters or return value missing type hints (for functions only) +- **Complete:** All parameters and return value have type hints + (for functions), or variable has type annotation (for variables) +- **Incomplete:** Some parameters or return value missing type hints + (for functions only) - **None:** No type hints at all **Analyzed Elements:** -- **Functions/Methods:** Function signatures including parameters and return types +- **Functions/Methods:** Function signatures including parameters + and return types - Excludes `self` and `cls` parameters from parameter counts - Considers both parameters and return type for completeness - Tracks decorator usage (e.g., `@staticmethod`, `@lru_cache`) @@ -361,7 +369,8 @@ The analyzer codebase maintains high documentation standards: **Key Design decisions:** -1. **AST-based Analysis**: Uses Python's `ast` module rather than runtime inspection +1. **AST-based Analysis**: Uses Python's `ast` module rather than runtime + inspection - Pros: No need to import/execute code, faster, safer - Cons: Cannot detect dynamically generated code @@ -396,7 +405,8 @@ The analyzer codebase maintains high documentation standards: 2. **More Accurate Reference Counting** - Use AST-based import analysis instead of text search - Track actual usage vs. string mentions - - Distinguish between different types of references (call, attribute access, etc.) + - Distinguish between different types of references + (call, attribute access, etc.) 3. **Additional Metrics** - Generic type parameterization completeness @@ -447,9 +457,12 @@ type annotations should be added to: ### What should not be annotated -1. **Reassignments** - Adding type annotations to reassignments causes `no-redef` errors -2. **Dictionary subscript operations** - Cannot annotate `dict[key] = value` operations -3. **Variables with obvious literal types** - Optional, but generally omitted for simple cases +1. **Reassignments** - + Adding type annotations to reassignments causes `no-redef` errors +2. **Dictionary subscript operations** - + Cannot annotate `dict[key] = value` operations +3. **Variables with obvious literal types** - + Optional, but generally omitted for simple cases ## Future Tools diff --git a/tokenization-benchmark.md b/tokenization-benchmark.md index c8f973e45..163d9effe 100644 --- a/tokenization-benchmark.md +++ b/tokenization-benchmark.md @@ -3,7 +3,8 @@ **Note: This benchmark framework is obsolete and no longer actively maintained.** A framework for benchmarking tokenization algorithms for Thai. -It provides a command-line interface that allows users to conveniently execute the benchmarks, +It provides a command-line interface that allows users to +conveniently execute the benchmarks, as well as a module interface for use in development pipelines. ## Metrics @@ -107,7 +108,8 @@ pip install "pythainlp[benchmarks]" ## Related Work -- [Thai Tokenizers Docker][docker]: collection of Docker containers of pre-built Thai tokenizers. +- [Thai Tokenizers Docker][docker]: collection of Docker containers of + pre-built Thai tokenizers. ## Development