diff --git a/TYPE_HINT_ANALYSIS.md b/TYPE_HINT_ANALYSIS.md deleted file mode 100644 index 1726f08db..000000000 --- a/TYPE_HINT_ANALYSIS.md +++ /dev/null @@ -1,219 +0,0 @@ -# Type Hint Coverage Analysis for PyThaiNLP - -**Analysis Date:** 2026-02-03 - -This document provides a comprehensive analysis of type hint coverage -across the PyThaiNLP codebase. - -## Executive Summary - -The PyThaiNLP codebase demonstrates **strong type hint coverage** with -82.22% of functions and methods having complete type hints. - -- **Total functions/methods analyzed:** 720 -- **Complete type hints:** 592 (82.22%) -- **Incomplete type hints:** 56 (7.78%) -- **No type hints:** 72 (10.00%) - -## Overall Statistics - -| Category | Count | Percentage | -| -------- | ----- | ---------- | -| Complete type hints | 592 | 82.22% | -| Incomplete type hints | 56 | 7.78% | -| No type hints | 72 | 10.00% | -| **Total** | **720** | **100.00%** | - -## Breakdown by Submodule - -### Excellent Coverage (95%+) - -| Submodule | Total | Complete | Incomplete | None | % Complete | Mypy | -| --------- | ----- | -------- | ---------- | ---- | ---------- | ---- | -| **cli** | 21 | 21 | 0 | 0 | 100.00% | 6 | -| **lm** | 2 | 2 | 0 | 0 | 100.00% | 6 | -| **morpheme** | 2 | 2 | 0 | 0 | 100.00% | 6 | -| **parse** | 9 | 9 | 0 | 0 | 100.00% | 6 | -| **tokenizeicu** | 3 | 3 | 0 | 0 | 100.00% | - | -| **tools** | 9 | 9 | 0 | 0 | 100.00% | 6 | -| **transliterateicu** | 1 | 1 | 0 | 0 | 100.00% | - | -| **word_vector** | 7 | 7 | 0 | 0 | 100.00% | 8 | -| **ancient** | 2 | 2 | 0 | 0 | 100.00% | 6 | -| **corpus** | 70 | 68 | 1 | 1 | 97.14% | 6 | -| **tag** | 68 | 66 | 2 | 0 | 97.06% | 7 | -| **soundex** | 27 | 26 | 0 | 1 | 96.30% | 6 | -| **spell** | 43 | 41 | 2 | 0 | 95.35% | 6 | - -### Good Coverage (80-94%) - -| Submodule | Total | Complete | Incomplete | None | % Complete | Mypy | -| --------- | ----- | -------- | ---------- | ---- | ---------- | ---- | -| **util** | 109 | 103 | 4 | 2 | 94.50% | 6 | -| **phayathaibert** | 19 | 17 | 0 | 2 | 89.47% | 6 | -| **benchmarks** | 8 | 7 | 1 | 0 | 87.50% | 6 | -| **tokenize** | 73 | 62 | 3 | 8 | 84.93% | 6 | -| **translate** | 44 | 37 | 5 | 2 | 84.09% | 6 | - -### Moderate Coverage (50-79%) - -| Submodule | Total | Complete | Incomplete | None | % Complete | Mypy | -| --------- | ----- | -------- | ---------- | ---- | ---------- | ---- | -| **khavee** | 9 | 7 | 0 | 2 | 77.78% | 6 | -| **summarize** | 17 | 12 | 5 | 0 | 70.59% | 9 | -| **ulmfit** | 25 | 17 | 4 | 4 | 68.00% | 6 | -| **augment** | 29 | 18 | 4 | 7 | 62.07% | 6 | -| **generate** | 15 | 8 | 6 | 1 | 53.33% | 6 | -| **wsd** | 4 | 2 | 2 | 0 | 50.00% | 6 | - -### Needs Improvement (<50%) - -| Submodule | Total | Complete | Incomplete | None | % Complete | Mypy | -| --------- | ----- | -------- | ---------- | ---- | ---------- | ---- | -| **transliterate** | 75 | 36 | 9 | 30 | 48.00% | 6 | -| **wangchanberta** | 9 | 4 | 1 | 4 | 44.44% | 6 | -| **coref** | 5 | 2 | 2 | 1 | 40.00% | 6 | -| **chat** | 4 | 1 | 1 | 2 | 25.00% | 6 | -| **classify** | 5 | 1 | 3 | 1 | 20.00% | 6 | -| **el** | 5 | 1 | 1 | 3 | 20.00% | 6 | -| **main** | 1 | 0 | 0 | 1 | 0.00% | 0 | - -## Priority Analysis - -Functions and methods are categorized by priority based on: - -- **Scope:** Public vs. private -- **Internal references:** How frequently they're used within the package -- **Test coverage:** Which test suite they belong to (core, compact, extra, - noauto) - -### High Priority Items - -**Criteria:** Public functions with >10 internal references in core/compact -test suites - -**Finding:** No functions without type hints fall into this category, -indicating that the most critical, widely-used public APIs are well-covered. - -### Medium Priority Items - -**Criteria:** Public functions with 3-10 references - -**Functions without type hints (51 total):** - -Top items include: - -- `pythainlp.corpus.util.tokenize` (991 references) -- `pythainlp.classify.param_free.GzipModel.train` (164 references) -- Various `__init__` methods across multiple classes (113 references each) - -**Functions with incomplete type hints (43 total):** - -Top items include: - -- `pythainlp.tokenize.nercut.segment` (2/3 params hinted) -- `pythainlp.classify.param_free.GzipModel.load` (missing return type) -- Various `__init__` methods missing return type hints - -### Low Priority Items - -**Criteria:** Private functions or rarely referenced functions - -- **21 functions** without type hints -- **13 functions** with incomplete type hints - -## Test Suite Coverage - -Functions are mapped to test suites based on the test categorization: - -- **core:** Core functionality tests (no external dependencies) -- **compact:** Tests with stable, small dependencies -- **extra:** Tests with larger dependencies -- **noauto:** Tests not run in CI/CD (e.g., TensorFlow dependencies) -- **unknown:** Functions not clearly mapped to a test suite - -Most functions requiring type hint improvements are either in "unknown" -test suite category or are tested indirectly. - -## Recommendations - -### Immediate Actions (High Priority) - -1. **Complete coverage for widely-used public APIs** - - ✅ Already achieved - no high-priority items without type hints - -### Short-term Improvements (Medium Priority) - -1. **transliterate submodule** (48.00% coverage) - - Focus on 30 functions without type hints - - Address 9 functions with incomplete hints - -2. **classify submodule** (20.00% coverage) - - Add type hints to 3 incomplete functions - - Add type hints to 1 function without hints - -3. **el (entity linking) submodule** (20.00% coverage) - - Add type hints to 3 functions without hints - - Complete 1 function with incomplete hints - -4. **chat submodule** (25.00% coverage) - - Add type hints to 2 functions without hints - - Complete 1 function with incomplete hints - -5. **wangchanberta submodule** (44.44% coverage) - - Add type hints to 4 functions without hints - - Complete 1 function with incomplete hints - -6. **Add return type hints to `__init__` methods** - - Many class `__init__` methods are missing `-> None` return type - - This is a quick win that significantly improves completeness - -### Long-term Goals - -1. **Achieve 95%+ coverage across all submodules** -2. **Maintain type hint standards for new code** -3. **Gradually improve coverage in legacy code during maintenance** - -## Detailed Results - -Full analysis results with file locations and line numbers are available in: - -- **JSON format:** `/tmp/type_hint_analysis.json` -- **Text report:** Contains full listings of all functions by priority - -## Analysis Methodology - -This analysis was performed using a custom AST-based analyzer that: - -1. Scans all Python files in the repository -2. Parses function and method definitions using Python's `ast` module -3. Checks for type hints on parameters and return values -4. Categorizes functions by: - - Completeness (complete/incomplete/none) - - Scope (public/private) - - Internal reference count - - Test suite mapping -5. Assigns priority based on the above factors -6. Runs mypy on each submodule to count type-related errors - -**Notes:** - -- Classes themselves are not scored (only their methods) -- `self` and `cls` parameters are excluded from type hint requirements -- Functions with no parameters and no return type hint are marked as "none" -- Functions with all parameters typed and return type are marked as - "complete" -- Everything else is marked as "incomplete" - -## Conclusion - -PyThaiNLP demonstrates **strong type hint adoption** with 82.22% complete -coverage. The codebase has excellent coverage in core modules (cli, corpus, -tag, spell) and good coverage in most other areas. - -The main areas for improvement are specialized modules like `transliterate`, -`classify`, `el`, and `chat`, which represent opportunities for enhancing -type safety and developer experience. - ---- - -This report was automatically generated on 2026-02-03. diff --git a/TYPE_HINT_COMPLETE_REPORT.md b/TYPE_HINT_COMPLETE_REPORT.md deleted file mode 100644 index a206f35a9..000000000 --- a/TYPE_HINT_COMPLETE_REPORT.md +++ /dev/null @@ -1,303 +0,0 @@ - - -# Type Hint Coverage Analysis - Complete Report - -**Repository:** PyThaiNLP/pythainlp -**Analysis Date:** 2026-02-03 -**Commit:** 1b86a05 - ---- - -## 📊 Executive Summary - -This comprehensive analysis scanned **720 functions and methods** across **30 submodules** in the PyThaiNLP repository to assess type hint coverage. - -### Overall Results - -| Metric | Count | Percentage | -|--------|-------|------------| -| ✅ **Complete type hints** | 592 | **82.22%** | -| ⚠️ **Incomplete type hints** | 56 | 7.78% | -| ❌ **No type hints** | 72 | 10.00% | -| **Total analyzed** | **720** | **100.00%** | - -**Conclusion:** PyThaiNLP demonstrates **strong type hint adoption** with over 82% complete coverage. - ---- - -## 📁 Analysis Artifacts - -All analysis results have been committed to the repository: - -### Documentation - -- 📄 **[TYPE_HINT_ANALYSIS.md](./TYPE_HINT_ANALYSIS.md)** - Comprehensive analysis report (7.2 KB) -- 📄 **[TYPE_HINT_QUICKSTART.md](./TYPE_HINT_QUICKSTART.md)** - Quick reference guide (3.7 KB) - -### Tools - -- 🔧 **[build_tools/analysis/type_hint_analyzer.py](./build_tools/analysis/type_hint_analyzer.py)** - Main analyzer (18 KB) -- 🔧 **[build_tools/analysis/generate_csv.py](./build_tools/analysis/generate_csv.py)** - CSV generator (2.6 KB) -- 📖 **[build_tools/analysis/README.md](./build_tools/analysis/README.md)** - Tool documentation - -### Data Files - -- 📊 **[docs/type_hint_analysis/submodule_summary.csv](./docs/type_hint_analysis/submodule_summary.csv)** - Statistics (843 B) -- 📊 **[docs/type_hint_analysis/functions_no_hints.csv](./docs/type_hint_analysis/functions_no_hints.csv)** - 72 functions (12 KB) -- 📊 **[docs/type_hint_analysis/functions_incomplete_hints.csv](./docs/type_hint_analysis/functions_incomplete_hints.csv)** - 56 functions (9.3 KB) -- 📖 **[docs/type_hint_analysis/README.md](./docs/type_hint_analysis/README.md)** - Data documentation - ---- - -## 🎯 Key Findings - -### Excellent Coverage (100%) - -Nine submodules have **perfect type hint coverage**: - -- cli, lm, morpheme, parse, tokenizeicu, tools, transliterateicu, word_vector, ancient - -### Good Coverage (95%+) - -Four submodules with excellent coverage: - -- corpus (97.14%), tag (97.06%), soundex (96.30%), spell (95.35%) - -### Areas Needing Improvement - -Six submodules below 50% coverage: - -| Submodule | Coverage | Functions Missing Hints | Priority | -|-----------|----------|------------------------|----------| -| **transliterate** | 48% | 30 functions, 9 incomplete | 🔴 High | -| **wangchanberta** | 44% | 4 functions, 1 incomplete | 🔴 High | -| **coref** | 40% | 1 function, 2 incomplete | 🟡 Medium | -| **chat** | 25% | 2 functions, 1 incomplete | 🟡 Medium | -| **classify** | 20% | 1 function, 3 incomplete | 🟡 Medium | -| **el** | 20% | 3 functions, 1 incomplete | 🟡 Medium | - ---- - -## 🎖️ Priority Classification - -### High Priority (0 items) ✅ - -**Definition:** Public functions with >10 internal references in core/compact test suites - -**Result:** All critical, widely-used public APIs have complete type hints! - -### Medium Priority (94 items) ⚠️ - -**Definition:** Public functions with 3-10 references - -- **51 functions** without type hints -- **43 functions** with incomplete type hints - -Top items by reference count: - -1. `pythainlp.corpus.util.tokenize` - 991 references, no hints -2. `pythainlp.classify.param_free.GzipModel.train` - 164 references, no hints -3. Multiple `__init__` methods - 113 references each, no hints - -### Low Priority (34 items) - -**Definition:** Private functions or rarely referenced - -- **21 functions** without type hints -- **13 functions** with incomplete type hints - ---- - -## 📈 Coverage by Category - -### By Scope - -- **Public functions:** Most are well-covered, but some `__init__` methods need `-> None` -- **Private functions:** Lower priority, but should still be improved - -### By Test Suite - -Analysis mapped functions to test suites: - -- **core:** Core functionality (no external dependencies) -- **compact:** Stable, small dependencies -- **extra:** Larger dependencies -- **noauto:** Not in CI/CD (e.g., TensorFlow) -- **unknown:** No clear test mapping - ---- - -## 🛠️ How to Use This Analysis - -### Quick Look - -```bash -# View the quick start guide -cat TYPE_HINT_QUICKSTART.md - -# Check submodule summary -cat docs/type_hint_analysis/submodule_summary.csv | column -t -s, -``` - -### Re-run Analysis - -```bash -# From repository root -python3 build_tools/analysis/type_hint_analyzer.py -python3 build_tools/analysis/generate_csv.py -cp /tmp/*.csv docs/type_hint_analysis/ -``` - -### Filter Data - -```bash -# Functions in transliterate submodule -csvgrep -c Submodule -m "transliterate" docs/type_hint_analysis/functions_no_hints.csv - -# Sort by references -csvsort -c References -r docs/type_hint_analysis/functions_no_hints.csv | head -20 -``` - ---- - -## 💡 Recommendations - -### Immediate Actions - -1. ✅ **No critical issues** - All high-priority public APIs have type hints - -### Short-term Improvements (Quick Wins) - -1. **Add `-> None` to `__init__` methods** - Many incomplete hints are just missing this -2. **Focus on transliterate submodule** - 30 functions need hints (48% coverage) -3. **Improve classify and el submodules** - Both at 20% coverage - -### Long-term Goals - -1. **Achieve 95%+ coverage** across all submodules -2. **Maintain standards** for new code -3. **Gradually improve** legacy code during maintenance - -### Example Fixes - -#### Missing Type Hints - -```python -# Before -def tokenize(text): - return newmm.segment(text, custom_dict=trie) - -# After -def tokenize(text: str) -> List[str]: - return newmm.segment(text, custom_dict=trie) -``` - -#### Incomplete Type Hints - -```python -# Before -def train(self): - temp_list = [] - return temp_list - -# After -def train(self) -> List[int]: - temp_list = [] - return temp_list -``` - -#### Missing Return Type - -```python -# Before -def __init__(self, model_path: str): - self.model = load(model_path) - -# After -def __init__(self, model_path: str) -> None: - self.model = load(model_path) -``` - ---- - -## 🔍 Methodology - -The analysis uses an AST-based approach: - -1. **Scan** all Python files in `pythainlp/` and `tests/` -2. **Parse** using Python's `ast` module -3. **Analyze** function signatures for type hints -4. **Categorize** by completeness (complete/incomplete/none) -5. **Count** internal references to determine importance -6. **Map** to test suites (core/compact/extra/noauto) -7. **Assign** priority based on scope, references, and test coverage -8. **Generate** reports in multiple formats (markdown, CSV, JSON) - -**Notes:** - -- `self` and `cls` parameters are excluded from requirements -- Functions with no parameters and no return hint are marked "none" -- Functions with all parameters typed and return type are marked "complete" -- Everything else is marked "incomplete" - ---- - -## 📊 Detailed Statistics by Submodule - -| Submodule | Total | Complete | Incomplete | None | % Complete | -|-----------|-------|----------|------------|------|------------| -| cli | 21 | 21 | 0 | 0 | 100.00% | -| lm | 2 | 2 | 0 | 0 | 100.00% | -| morpheme | 2 | 2 | 0 | 0 | 100.00% | -| parse | 9 | 9 | 0 | 0 | 100.00% | -| tokenizeicu | 3 | 3 | 0 | 0 | 100.00% | -| tools | 9 | 9 | 0 | 0 | 100.00% | -| transliterateicu | 1 | 1 | 0 | 0 | 100.00% | -| word_vector | 7 | 7 | 0 | 0 | 100.00% | -| ancient | 2 | 2 | 0 | 0 | 100.00% | -| corpus | 70 | 68 | 1 | 1 | 97.14% | -| tag | 68 | 66 | 2 | 0 | 97.06% | -| soundex | 27 | 26 | 0 | 1 | 96.30% | -| spell | 43 | 41 | 2 | 0 | 95.35% | -| util | 109 | 103 | 4 | 2 | 94.50% | -| phayathaibert | 19 | 17 | 0 | 2 | 89.47% | -| benchmarks | 8 | 7 | 1 | 0 | 87.50% | -| tokenize | 73 | 62 | 3 | 8 | 84.93% | -| translate | 44 | 37 | 5 | 2 | 84.09% | -| khavee | 9 | 7 | 0 | 2 | 77.78% | -| summarize | 17 | 12 | 5 | 0 | 70.59% | -| ulmfit | 25 | 17 | 4 | 4 | 68.00% | -| augment | 29 | 18 | 4 | 7 | 62.07% | -| generate | 15 | 8 | 6 | 1 | 53.33% | -| wsd | 4 | 2 | 2 | 0 | 50.00% | -| transliterate | 75 | 36 | 9 | 30 | 48.00% | -| wangchanberta | 9 | 4 | 1 | 4 | 44.44% | -| coref | 5 | 2 | 2 | 1 | 40.00% | -| chat | 4 | 1 | 1 | 2 | 25.00% | -| classify | 5 | 1 | 3 | 1 | 20.00% | -| el | 5 | 1 | 1 | 3 | 20.00% | -| **main** | 1 | 0 | 0 | 1 | 0.00% | - ---- - -## ✅ Conclusion - -PyThaiNLP demonstrates **strong type hint adoption** with 82.22% complete coverage. The codebase has: - -- ✅ **Excellent coverage** in core modules (cli, corpus, tag, spell) -- ✅ **Good coverage** in most utility and processing modules -- ⚠️ **Improvement needed** in specialized modules (transliterate, classify, el, chat) -- ✅ **No critical gaps** - all widely-used public APIs have type hints - -The main opportunities for enhancement are in specialized modules that -represent about 18% of the codebase. These improvements will further -strengthen type safety and developer experience across the entire project. - ---- - -**Analysis Generated:** 2026-02-03 -**Tools Version:** 1.0 -**Python Version:** 3.9+ diff --git a/TYPE_HINT_QUICKSTART.md b/TYPE_HINT_QUICKSTART.md deleted file mode 100644 index ab4fce55e..000000000 --- a/TYPE_HINT_QUICKSTART.md +++ /dev/null @@ -1,137 +0,0 @@ -# Type Hint Coverage Analysis - Quick Start - -This is a quick reference guide for the type hint coverage analysis tools. - -## TL;DR - -**Overall Status:** 82.22% of functions have complete type hints ✅ - -**What to do:** - -1. Check `TYPE_HINT_ANALYSIS.md` for the full report -2. Use CSV files in `docs/type_hint_analysis/` for detailed analysis -3. Focus on improving coverage in: `transliterate`, `classify`, `el`, `chat`, `wangchanberta` - -## Running the Analysis - -```bash -# From repository root -python3 build_tools/analysis/type_hint_analyzer.py - -# Generate CSV files -python3 build_tools/analysis/generate_csv.py - -# Copy CSV files to docs -cp /tmp/*.csv docs/type_hint_analysis/ -``` - -## Quick Stats by Submodule - -| Coverage | Submodules | -|----------|------------| -| 💚 100% | cli, lm, morpheme, parse, tokenizeicu, tools, transliterateicu, word_vector, ancient | -| ✅ 95%+ | corpus (97%), tag (97%), soundex (96%), spell (95%) | -| 👍 80-94% | util (94%), phayathaibert (89%), benchmarks (87%), tokenize (84%), translate (84%) | -| ⚠️ 50-79% | khavee (77%), summarize (70%), ulmfit (68%), augment (62%), generate (53%), wsd (50%) | -| ❌ <50% | **transliterate (48%)**, **wangchanberta (44%)**, **coref (40%)**, **chat (25%)**, **classify (20%)**, **el (20%)** | - -## Priority Items to Fix - -### High Priority - -✅ None! All critical public APIs have type hints. - -### Medium Priority (51 functions without hints, 43 incomplete) - -Key items: - -- `pythainlp.corpus.util.tokenize` (991 references) -- `pythainlp.classify.param_free.GzipModel.train` (164 references) -- Many `__init__` methods missing `-> None` return type - -### Low Priority (21 without, 13 incomplete) - -Private functions or rarely used functions. - -## Examples of Issues - -### Missing Type Hints - -```python -# Before -def tokenize(text): - return newmm.segment(text, custom_dict=trie) - -# After -def tokenize(text: str) -> List[str]: - return newmm.segment(text, custom_dict=trie) -``` - -### Incomplete Type Hints - -```python -# Before -def train(self): - temp_list = [] - ... - return temp_list - -# After -def train(self) -> List[int]: - temp_list = [] - ... - return temp_list -``` - -### Missing Return Type on **init** - -```python -# Before -def __init__(self, model_path: str): - self.model = load(model_path) - -# After -def __init__(self, model_path: str) -> None: - self.model = load(model_path) -``` - -## Data Files - -All detailed data is available in: - -- **Main Report:** `TYPE_HINT_ANALYSIS.md` -- **CSV Data:** `docs/type_hint_analysis/` - - `submodule_summary.csv` - Statistics by submodule - - `functions_no_hints.csv` - Functions without type hints - - `functions_incomplete_hints.csv` - Functions with incomplete hints -- **JSON Data:** `/tmp/type_hint_analysis.json` (generated during analysis) - -## Using CSV Files - -```bash -# View functions in transliterate submodule -csvgrep -c Submodule -m "transliterate" docs/type_hint_analysis/functions_no_hints.csv | csvlook - -# Sort by references (most used first) -csvsort -c References -r docs/type_hint_analysis/functions_no_hints.csv | head -20 | csvlook - -# Count by priority -csvcut -c Priority docs/type_hint_analysis/functions_no_hints.csv | tail -n +2 | sort | uniq -c -``` - -## Recommendations - -1. **Quick wins:** Add `-> None` to all `__init__` methods (many incomplete hints are this) -2. **Focus on:** transliterate, classify, el, chat submodules -3. **Maintain:** Keep 100% coverage in new code -4. **Gradual:** Improve legacy code during maintenance - -## Tools - -- `build_tools/analysis/type_hint_analyzer.py` - Main analyzer -- `build_tools/analysis/generate_csv.py` - CSV generator -- `build_tools/analysis/README.md` - Detailed tool documentation - ---- - -**Last Updated:** 2026-02-03 diff --git a/pythainlp/py.typed b/pythainlp/py.typed new file mode 100644 index 000000000..6943d75e7 --- /dev/null +++ b/pythainlp/py.typed @@ -0,0 +1,2 @@ +# Do not delete. +# This file tells type checkers that this package provides type information.