Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ about: Propose a change or addition เสนอความสามารถ

## Detailed description

<!--- Provide a detailed description of the change or addition you are proposing -->
<!--- Provide a detailed description of the change or addition -->
<!--- รายละเอียดการเปลี่ยนแปลงหรือสิ่งเพิ่มเติมที่คุณกำลังเสนอ -->

## Context
Expand All @@ -17,8 +17,8 @@ about: Propose a change or addition เสนอความสามารถ

## Possible implementation

<!--- Not obligatory, but suggest an idea for implementing addition or change -->
<!--- ไม่จำเป็นต้องใส่ แต่คุณสามารถแนะนำได้ว่าการเปลี่ยนแปลงหรือเพิ่มเติมดังกล่าวน่าจะทำได้ด้วยวิธีไหน -->
<!--- Suggest an idea for implementing the change or addition (Optional) -->
<!--- แนะนำว่าจะเปลี่ยนแปลงหรือเพิ่มเติมได้ด้วยวิธีไหน (ไม่จำเป็นต้องใส่) -->

## Your environment

Expand Down
19 changes: 11 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,12 @@
[![Facebook](https://img.shields.io/badge/Facebook-0866FF?style=flat&logo=facebook&logoColor=white)](https://www.facebook.com/pythainlp/)
[![Chat on Matrix](https://matrix.to/img/matrix-badge.svg)](https://matrix.to/#/#thainlp:matrix.org)

[pythainlp.org](https://pythainlp.org/) |
[Tutorials](https://pythainlp.org/tutorials) |
[License info](https://pythainlp.org/dev-docs/notes/license.html) |
[Model cards](https://github.com/PyThaiNLP/pythainlp/wiki/Model-Cards) |
[Adopters](https://github.com/PyThaiNLP/pythainlp/blob/dev/INTHEWILD.md) |
*[เอกสารภาษาไทย](https://github.com/PyThaiNLP/pythainlp/blob/dev/README_TH.md)*
[pythainlp.org](https://pythainlp.org/)
| [Tutorials](https://pythainlp.org/tutorials)
| [License info](https://pythainlp.org/dev-docs/notes/license.html)
| [Model cards](https://github.com/PyThaiNLP/pythainlp/wiki/Model-Cards)
| [Adopters](https://github.com/PyThaiNLP/pythainlp/blob/dev/INTHEWILD.md)
| *[เอกสารภาษาไทย](https://github.com/PyThaiNLP/pythainlp/blob/dev/README_TH.md)*

Designed to be a Thai-focused counterpart to [NLTK](https://www.nltk.org/),
**PyThaiNLP** provides standard tools for linguistic analysis under
Expand Down Expand Up @@ -186,9 +186,12 @@ with this BibTeX entry:
------

<div align="center">
<strong>We have only one official repository at https://github.com/PyThaiNLP/pythainlp and another mirror at https://gitlab.com/pythainlp/pythainlp</strong>
<strong>We have only one official repository at
https://github.com/PyThaiNLP/pythainlp and another mirror at
https://gitlab.com/pythainlp/pythainlp</strong>
</div>

<div align="center">
<strong>Beware of malware if you use code from mirrors other than the official two on GitHub and GitLab.</strong>
<strong>Beware of malware if you use code from mirrors other than the
official two on GitHub and GitLab.</strong>
</div>
22 changes: 13 additions & 9 deletions README_TH.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,12 @@
[![Facebook](https://img.shields.io/badge/Facebook-0866FF?style=flat&logo=facebook&logoColor=white)](https://www.facebook.com/pythainlp/)
[![Chat on Matrix](https://matrix.to/img/matrix-badge.svg)](https://matrix.to/#/#thainlp:matrix.org)

[pythainlp.org](https://pythainlp.org/) |
[วิธีใช้งาน](https://pythainlp.org/tutorials) |
[ข้อมูลสัญญาอนุญาต](https://pythainlp.org/dev-docs/notes/license.html) |
[ใบข้อมูลโมเดล](https://github.com/PyThaiNLP/pythainlp/wiki/Model-Cards) |
[ใครใช้ PyThaiNLP บ้าง](https://github.com/PyThaiNLP/pythainlp/blob/dev/INTHEWILD.md) |
*[English](https://github.com/PyThaiNLP/pythainlp/blob/dev/README.md)*
[pythainlp.org](https://pythainlp.org/)
| [วิธีใช้งาน](https://pythainlp.org/tutorials)
| [ข้อมูลสัญญาอนุญาต](https://pythainlp.org/dev-docs/notes/license.html)
| [ใบข้อมูลโมเดล](https://github.com/PyThaiNLP/pythainlp/wiki/Model-Cards)
| [ใครใช้ PyThaiNLP บ้าง](https://github.com/PyThaiNLP/pythainlp/blob/dev/INTHEWILD.md)
| *[English](https://github.com/PyThaiNLP/pythainlp/blob/dev/README.md)*

**PyThaiNLP** ถูกออกแบบให้เป็นเครื่องมือมาตรฐานสำหรับการวิเคราะห์ภาษาศาสตร์ภาษาไทย
ภายใต้สัญญาอนุญาต Apache-2.0 โดยข้อมูลและโมเดลอยู่ภายใต้ CC0-1.0 และ CC-BY-4.0
Expand Down Expand Up @@ -100,7 +100,8 @@ pip install "pythainlp[extra1,extra2,...]"

</details>

สำหรับรายละเอียด dependencies สามารถดูได้ที่ส่วน `[project.optional-dependencies]` ใน
สำหรับรายละเอียด dependencies
สามารถดูได้ที่ส่วน `[project.optional-dependencies]` ใน
[`pyproject.toml`](https://github.com/PyThaiNLP/pythainlp/blob/dev/pyproject.toml)

## ไดเรกทอรีข้อมูล
Expand Down Expand Up @@ -225,9 +226,12 @@ PyThaiNLP ดาวน์โหลดข้อมูล (ดูแค็ตต
------

<div align="center">
<strong>เรามีที่เก็บข้อมูลอย่างเป็นทางการที่เดียวที่ https://github.com/PyThaiNLP/pythainlp และมีที่เก็บสำเนาอีกแห่งที่ https://gitlab.com/pythainlp/pythainlp</strong>
<strong>เรามีที่เก็บข้อมูลอย่างเป็นทางการที่เดียวที่
https://github.com/PyThaiNLP/pythainlp และมีที่เก็บสำเนาอีกแห่งที่
https://gitlab.com/pythainlp/pythainlp</strong>
</div>

<div align="center">
<strong>โปรดระมัดระวังซอฟต์แวร์ประสงค์ร้ายหรือมัลแวร์ ถ้าคุณใช้โค้ดจากที่เก็บข้อมูลอื่นนอกเหนือจากที่ GitHub และ GitLab ข้างต้น</strong>
<strong>โปรดระมัดระวังซอฟต์แวร์ประสงค์ร้ายหรือมัลแวร์
ถ้าคุณใช้โค้ดจากที่เก็บข้อมูลอื่นนอกเหนือจากที่ GitHub และ GitLab ข้างต้น</strong>
</div>
8 changes: 6 additions & 2 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,11 @@

The following security improvements are planned for future releases:

- Migrate from pickle to a safer serialization format such as JSON or MessagePack.
- Upgrade the hashing algorithm for integrity verification from MD5 to SHA-256 or SHA-3.
- Migrate from pickle to a safer serialization format such as JSON or
[MessagePack][].
- Upgrade the hashing algorithm for integrity verification from MD5 to SHA-256
or SHA-3.
- Implement digital signatures for corpus files to ensure authenticity.
- Add version tracking to the corpus to prevent rollback attacks.

[MessagePack]: https://msgpack.org/
51 changes: 32 additions & 19 deletions build_tools/analysis/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,10 +90,12 @@ easy analysis in spreadsheet applications or data analysis tools.
- `output/functions_no_hints.csv` - Functions without any type hints
- `output/functions_incomplete_hints.csv` - Functions with partial hints
- `output/class_variables_no_hints.csv` - Class variables without type hints
- `output/instance_variables_no_hints.csv` - Instance variables without type hints
- `output/instance_variables_no_hints.csv` -
Instance variables without type hints
- `output/module_variables_no_hints.csv` - Module variables without type hints
- `output/type_aliases.csv` - All type aliases defined in the codebase
- `output/submodule_summary.csv` - Summary statistics by submodule with mypy errors
- `output/submodule_summary.csv` -
Summary statistics by submodule with mypy errors

**CSV Schema:**

Expand Down Expand Up @@ -155,9 +157,9 @@ cat output/submodule_summary.csv
**Example Output:**

```text
================================================================================
==============================================================================
TYPE ANNOTATION COVERAGE ANALYSIS FOR PYTHAINLP
================================================================================
==============================================================================

Repository root: /path/to/pythainlp
Output directory: ./output
Expand All @@ -181,17 +183,17 @@ Analyzed 426 instance variables
Analyzed 508 module variables
Analyzed 0 type aliases

================================================================================
==============================================================================
OVERALL STATISTICS - FUNCTIONS/METHODS
================================================================================
==============================================================================
Total functions/methods: 720
Complete type hints: 592 (82.22%)
Incomplete type hints: 56 ( 7.78%)
No type hints: 72 (10.00%)

================================================================================
==============================================================================
OVERALL STATISTICS - VARIABLES
================================================================================
==============================================================================
Total variables: 959
Class variables: 25
Instance variables: 426
Expand All @@ -202,19 +204,22 @@ No type hints: 909 (94.79%)

### Automated Analysis

The repository includes a GitHub Actions workflow that automatically runs the type hint analyzer on every push to the `dev` branch:
The repository includes a GitHub Actions workflow that automatically runs
the type hint analyzer on every push to the `dev` branch:

- **Workflow**: `.github/workflows/type-hint-analysis.yml`
- **Trigger**: Push to `dev` branch
- **Environment**: ubuntu-latest, Python 3.9
- **Artifacts**: JSON and CSV files (30-day retention)
- **Summary**: Displayed in GitHub Actions UI

The workflow provides continuous monitoring of type hint coverage as the codebase evolves.
The workflow provides continuous monitoring of type hint coverage
as the codebase evolves.

### Type Completeness Standards

This analyzer follows the type completeness guidelines from the Python typing documentation:
This analyzer follows the type completeness guidelines from
the Python typing documentation:
<https://typing.python.org/en/latest/guides/libraries.html#type-completeness>

The analysis covers:
Expand All @@ -241,13 +246,16 @@ a library is considered to have complete type hints when:

**Type hint status:**

- **Complete:** All parameters and return value have type hints (for functions), or variable has type annotation (for variables)
- **Incomplete:** Some parameters or return value missing type hints (for functions only)
- **Complete:** All parameters and return value have type hints
(for functions), or variable has type annotation (for variables)
- **Incomplete:** Some parameters or return value missing type hints
(for functions only)
- **None:** No type hints at all

**Analyzed Elements:**

- **Functions/Methods:** Function signatures including parameters and return types
- **Functions/Methods:** Function signatures including parameters
and return types
- Excludes `self` and `cls` parameters from parameter counts
- Considers both parameters and return type for completeness
- Tracks decorator usage (e.g., `@staticmethod`, `@lru_cache`)
Expand Down Expand Up @@ -361,7 +369,8 @@ The analyzer codebase maintains high documentation standards:

**Key Design decisions:**

1. **AST-based Analysis**: Uses Python's `ast` module rather than runtime inspection
1. **AST-based Analysis**: Uses Python's `ast` module rather than runtime
inspection
- Pros: No need to import/execute code, faster, safer
- Cons: Cannot detect dynamically generated code

Expand Down Expand Up @@ -396,7 +405,8 @@ The analyzer codebase maintains high documentation standards:
2. **More Accurate Reference Counting**
- Use AST-based import analysis instead of text search
- Track actual usage vs. string mentions
- Distinguish between different types of references (call, attribute access, etc.)
- Distinguish between different types of references
(call, attribute access, etc.)

3. **Additional Metrics**
- Generic type parameterization completeness
Expand Down Expand Up @@ -447,9 +457,12 @@ type annotations should be added to:

### What should not be annotated

1. **Reassignments** - Adding type annotations to reassignments causes `no-redef` errors
2. **Dictionary subscript operations** - Cannot annotate `dict[key] = value` operations
3. **Variables with obvious literal types** - Optional, but generally omitted for simple cases
1. **Reassignments** -
Adding type annotations to reassignments causes `no-redef` errors
2. **Dictionary subscript operations** -
Cannot annotate `dict[key] = value` operations
3. **Variables with obvious literal types** -
Optional, but generally omitted for simple cases

## Future Tools

Expand Down
6 changes: 4 additions & 2 deletions tokenization-benchmark.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@
**Note: This benchmark framework is obsolete and no longer actively maintained.**

A framework for benchmarking tokenization algorithms for Thai.
It provides a command-line interface that allows users to conveniently execute the benchmarks,
It provides a command-line interface that allows users to
conveniently execute the benchmarks,
as well as a module interface for use in development pipelines.

## Metrics
Expand Down Expand Up @@ -107,7 +108,8 @@ pip install "pythainlp[benchmarks]"

## Related Work

- [Thai Tokenizers Docker][docker]: collection of Docker containers of pre-built Thai tokenizers.
- [Thai Tokenizers Docker][docker]: collection of Docker containers of
pre-built Thai tokenizers.

## Development

Expand Down
Loading