Skip to content

Commit 1f7959c

Browse files
committed
Sync with latest dev branch updates
2 parents 80fa60e + 8c880d9 commit 1f7959c

27 files changed

Lines changed: 705 additions & 375 deletions

CONTRIBUTING.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -175,13 +175,13 @@ This will create distribution packages in the `dist/` directory.
175175
#current_version = 2.3.3
176176

177177
bump-my-version bump patch
178-
#current_version = 2.3.6-dev0
178+
#current_version = 2.3.4-dev0
179179

180180
bump-my-version bump minor
181-
#current_version = 2.3.1-dev0
181+
#current_version = 2.4.0-dev0
182182

183183
bump-my-version bump build
184-
#current_version = 2.3.1-dev1
184+
#current_version = 2.4.0-dev1
185185

186186
bump-my-version bump major
187187
#current_version = 3.0.0-dev0

README.md

Lines changed: 48 additions & 117 deletions
Original file line numberDiff line numberDiff line change
@@ -6,76 +6,54 @@
66
[![Python 3.9](https://img.shields.io/badge/python-3.9-blue.svg)](https://www.python.org/downloads/)
77
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
88
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.3519354.svg)](https://doi.org/10.5281/zenodo.3519354)
9-
109
[![Project Status: Active](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)
1110
[![Codacy Grade](https://app.codacy.com/project/badge/Grade/5821a0de122041c79999bbb280230ffb)](https://www.codacy.com/gh/PyThaiNLP/pythainlp/dashboard?utm_source=github.com&utm_medium=referral&utm_content=PyThaiNLP/pythainlp&utm_campaign=Badge_Grade)
1211
[![Coverage Status](https://coveralls.io/repos/github/PyThaiNLP/pythainlp/badge.svg?branch=dev)](https://coveralls.io/github/PyThaiNLP/pythainlp?branch=dev)
13-
1412
[![Google Colab Badge](https://badgen.net/badge/Launch%20Quick%20Start%20Guide/on%20Google%20Colab/blue?icon=terminal)](https://colab.research.google.com/github/PyThaiNLP/tutorials/blob/master/source/notebooks/pythainlp_get_started.ipynb)
13+
[![Facebook](https://img.shields.io/badge/Facebook-0866FF?style=flat&logo=facebook&logoColor=white)](https://www.facebook.com/pythainlp/)
1514
[![Chat on Matrix](https://matrix.to/img/matrix-badge.svg)](https://matrix.to/#/#thainlp:matrix.org)
1615

17-
PyThaiNLP is a Python package for text processing and linguistic analysis,
18-
similar to [NLTK](https://www.nltk.org/) with a focus on Thai language.
16+
[pythainlp.org](https://pythainlp.org/) |
17+
[Tutorials](https://pythainlp.org/tutorials) |
18+
[License info](https://pythainlp.org/dev-docs/notes/license.html) |
19+
[Model cards](https://github.com/PyThaiNLP/pythainlp/wiki/Model-Cards) |
20+
[Adopters](https://github.com/PyThaiNLP/pythainlp/blob/dev/INTHEWILD.md) |
21+
*[เอกสารภาษาไทย](https://github.com/PyThaiNLP/pythainlp/blob/dev/README_TH.md)*
1922

20-
PyThaiNLP เป็นไลบารีภาษาไพทอนสำหรับประมวลผลภาษาธรรมชาติ คล้ายกับ NLTK โดยเน้นภาษาไทย
21-
[ดูรายละเอียดภาษาไทยได้ที่ README_TH.MD](https://github.com/PyThaiNLP/pythainlp/blob/dev/README_TH.md)
22-
23-
## Quick install
23+
Designed to be a Thai-focused counterpart to [NLTK](https://www.nltk.org/),
24+
**PyThaiNLP** provides standard tools for linguistic analysis under
25+
an Apache-2.0 source license, with its data and models covered by CC0-1.0
26+
and CC-BY-4.0.
2427

2528
```sh
2629
pip install pythainlp
2730
```
2831

29-
| Version | Description | Status |
30-
|:------:|:--:|:------:|
31-
| [5.2.0](https://github.com/PyThaiNLP/pythainlp/releases) | Stable | [Change Log](https://github.com/PyThaiNLP/pythainlp/issues/1080) |
32-
| [`dev`](https://github.com/PyThaiNLP/pythainlp/tree/dev) | Release Candidate for 5.3 | [Change Log](https://github.com/PyThaiNLP/pythainlp/issues/1169) |
33-
34-
## Getting Started
35-
36-
- PyThaiNLP requires Python 3.9+.
37-
- Python 2.7 users can use PyThaiNLP 1.6. See [2.0 change log](https://github.com/PyThaiNLP/pythainlp/issues/118) | [Upgrading from 1.7](https://pythainlp.org/docs/2.0/notes/pythainlp-1_7-2_0.html) | [Upgrading ThaiNER from 1.7](https://github.com/PyThaiNLP/pythainlp/wiki/Upgrade-ThaiNER-from-PyThaiNLP-1.7-to-PyThaiNLP-2.0)
38-
- [PyThaiNLP Get Started notebook](https://pythainlp.org/tutorials/notebooks/pythainlp_get_started.html) | [API document](https://pythainlp.org/docs) | [Tutorials](https://pythainlp.org/tutorials)
39-
- [Official website](https://pythainlp.org/) | [PyPI](https://pypi.org/project/pythainlp/) | [Facebook page](https://www.facebook.com/pythainlp/)
40-
- [Who uses PyThaiNLP?](https://github.com/PyThaiNLP/pythainlp/blob/dev/INTHEWILD.md)
41-
- [Model cards](https://github.com/PyThaiNLP/pythainlp/wiki/Model-Cards) - for technical details, caveats, and ethical considerations of the models developed and used in PyThaiNLP
42-
43-
## Capabilities
44-
45-
PyThaiNLP provides standard linguistic analysis for Thai language and standard Thai locale utility functions.
46-
Some of these functions are also available via the command-line interface (run `thainlp` in your shell).
47-
48-
Partial list of features:
32+
| Version | Python version | Changes | Documentation |
33+
|:-------:|:--------------:|:-------:|:-------------:|
34+
| [5.2.0](https://github.com/PyThaiNLP/pythainlp/releases) | 3.7+ | [Log](https://github.com/PyThaiNLP/pythainlp/issues/1080) | [pythainlp.org/docs](https://pythainlp.org/docs) |
35+
| [`dev`](https://github.com/PyThaiNLP/pythainlp/tree/dev) | 3.9+ | [Log](https://github.com/PyThaiNLP/pythainlp/issues/1169) | [pythainlp.org/dev-docs](https://pythainlp.org/dev-docs/) |
4936

50-
- Convenient character and word classes, like Thai consonants (`pythainlp.thai_consonants`), vowels (`pythainlp.thai_vowels`), digits (`pythainlp.thai_digits`), and stop words (`pythainlp.corpus.thai_stopwords`) -- comparable to constants like `string.letters`, `string.digits`, and `string.punctuation`
51-
- Linguistic unit segmentation at different levels: sentence (`sent_tokenize`), word (`word_tokenize`), and subword (`subword_tokenize`)
52-
- Part-of-speech tagging (`pos_tag`)
53-
- Spelling suggestion and correction (`spell` and `correct`)
54-
- Phonetic algorithm and transliteration (`soundex` and `transliterate`)
55-
- Collation (sorted by dictionary order) (`collate`)
56-
- Number read out (`num_to_thaiword` and `bahttext`)
57-
- Datetime formatting (`thai_strftime`)
58-
- Thai-English keyboard misswitched fix (`eng_to_thai`, `thai_to_eng`)
37+
## Features
5938

60-
## Installation
39+
- **Linguistic units:** Sentence, word, and subword segmentation
40+
(`sent_tokenize`, `word_tokenize`, `subword_tokenize`).
41+
- **Tagging:** Part-of-speech tagging (`pos_tag`).
42+
- **Transliteration:** Romanization (`transliterate`) and IPA conversion.
43+
- **Correction:** Spelling suggestion and correction (`spell`, `correct`).
44+
- **Utilities:** Soundex, collation, number-to-text (`bahttext`), datetime
45+
formatting (`thai_strftime`), and keyboard layout correction.
46+
- **Data:** Built-in Thai character sets, word lists, and stop words.
47+
- **CLI:** Command-line interface via `thainlp`.
6148

62-
```sh
63-
pip install --upgrade pythainlp
64-
```
65-
66-
This will install the latest stable release of PyThaiNLP.
67-
68-
Install different releases:
49+
```sh
50+
thainlp data catalog # List datasets
51+
thainlp help # Show usage
52+
```
6953

70-
- Stable release: `pip install --upgrade pythainlp`
71-
- Pre-release (nearly ready): `pip install --upgrade --pre pythainlp`
72-
- Development (likely to break things): `pip install https://github.com/PyThaiNLP/pythainlp/archive/dev.zip`
54+
## Installation options
7355

74-
### Installation Options
75-
76-
Some functionalities, like Thai WordNet, may require extra packages.
77-
To install those requirements,
78-
specify a set of `[name]` immediately after `pythainlp`:
56+
To install with specific extras (e.g., `translate`, `wordnet`, `full`):
7957

8058
```sh
8159
pip install "pythainlp[extra1,extra2,...]"
@@ -86,94 +64,47 @@ Possible `extras` included:
8664
- `compact` — install a stable and small subset of dependencies (recommended)
8765
- `translate` — machine translation support
8866
- `wordnet` — WordNet support
89-
- `full` — install all optional dependencies
90-
(may introduce large dependencies and conflicts)
67+
- `full` — install all optional dependencies (may introduce conflicts)
9168

9269
The documentation website maintains
9370
[full list of extras](https://pythainlp.org/dev-docs/notes/installation.html).
9471

95-
For dependency details,
96-
look at the `[project.optional-dependencies]` section in
72+
For details, see `[project.optional-dependencies]` section in
9773
[`pyproject.toml`](https://github.com/PyThaiNLP/pythainlp/blob/dev/pyproject.toml).
9874

99-
## Data Directory
100-
101-
- Some additional data, like word lists and language models,
102-
may be automatically downloaded during runtime.
103-
- PyThaiNLP caches these data under the directory `~/pythainlp-data`
104-
by default.
105-
- The data directory can be changed by specifying the environment variable
106-
`PYTHAINLP_DATA_DIR`.
107-
- See the data catalog (`db.json`) at
108-
<https://github.com/PyThaiNLP/pythainlp-corpus>
75+
## Data directory
10976

110-
### Using PyThaiNLP in Distributed Environments
77+
PyThaiNLP downloads data (see the data catalog `db.json` at
78+
[pythainlp-corpus](https://github.com/PyThaiNLP/pythainlp-corpus))
79+
to `~/pythainlp-data` by default.
80+
Set the `PYTHAINLP_DATA_DIR` environment variable to override this location.
11181

11282
When using PyThaiNLP in distributed computing environments
11383
(e.g., Apache Spark), set the `PYTHAINLP_DATA_DIR` environment variable
114-
inside the function that will be distributed to worker nodes:
115-
116-
```python
117-
def tokenize_thai(text):
118-
import os
119-
os.environ['PYTHAINLP_DATA_DIR'] = './pythainlp-data'
120-
from pythainlp.tokenize import word_tokenize
121-
return word_tokenize(text)
122-
123-
rdd.map(tokenize_thai)
124-
```
125-
126-
This ensures each worker uses a local writable directory.
127-
See `examples/distributed_pyspark.py` for more examples.
128-
129-
## Command-Line Interface
130-
131-
Some of PyThaiNLP functionalities can be used via command line
132-
with the `thainlp` command.
133-
134-
For example, to display a catalog of datasets:
135-
136-
```sh
137-
thainlp data catalog
138-
```
84+
inside the function that will be distributed to worker nodes.
85+
See details in
86+
[the documentation](https://pythainlp.org/dev-docs/notes/installation.html).
13987

140-
To show how to use:
141-
142-
```sh
143-
thainlp help
144-
```
145-
146-
## Testing and test suites
88+
## Testing
14789

14890
We test core functionalities on all officially supported Python versions.
14991

150-
Some functionality requiring extra dependencies may be tested less frequently
151-
due to potential version conflicts or incompatibilities between packages.
152-
153-
Test cases are categorized into three groups: core, compact, and extra.
154-
You can find these tests in the [tests/](/tests/) directory.
155-
156-
For more detailed information on testing, please refer to the tests README:
157-
[tests/README.md](./tests/README.md)
92+
See [tests/README.md](./tests/README.md) for test matrix and other details.
15893

15994
## Licenses
16095

161-
| | License |
162-
| :-- | :-- |
96+
| Content | License |
97+
| :------ | :------ |
16398
| PyThaiNLP source codes and notebooks | [Apache Software License 2.0](https://github.com/PyThaiNLP/pythainlp/blob/dev/LICENSE) |
16499
| Corpora, datasets, and documentations created by PyThaiNLP | [Creative Commons Zero 1.0 Universal Public Domain Dedication License (CC0)](https://creativecommons.org/publicdomain/zero/1.0/)|
165100
| Language models created by PyThaiNLP | [Creative Commons Attribution 4.0 International Public License (CC-by)](https://creativecommons.org/licenses/by/4.0/) |
166101
| Other corpora and models that may be included in PyThaiNLP | See [Corpus License](https://github.com/PyThaiNLP/pythainlp/blob/dev/pythainlp/corpus/corpus_license.md) |
167102

168103
## Contribute to PyThaiNLP
169104

170-
- Please fork and create a pull request :)
171-
- For style guides and other information, including references to algorithms we use,
172-
please refer to our [contributing](https://github.com/PyThaiNLP/pythainlp/blob/dev/CONTRIBUTING.md) page.
173-
174-
## Who uses PyThaiNLP?
175-
176-
You can read [INTHEWILD.md](https://github.com/PyThaiNLP/pythainlp/blob/dev/INTHEWILD.md).
105+
Please fork and create a pull request.
106+
See [CONTRIBUTING.md](https://github.com/PyThaiNLP/pythainlp/blob/dev/CONTRIBUTING.md)
107+
for guidelines and algorithm references.
177108

178109
## Citations
179110

docs/notes/start.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,5 +19,5 @@ Thai NLP faces several challenges. A brief list includes:
1919

2020
Tutorial notebooks
2121
==================
22-
- `PyThaiNLP Get Started <https://pythainlp.org/tutorials/notebooks/pythainlp-get-started.html>`_
22+
- `PyThaiNLP Get Started <https://pythainlp.org/tutorials/notebooks/pythainlp_get_started.html>`_
2323
- `Other tutorials <https://pythainlp.org/tutorials/>`_

pyproject.toml

Lines changed: 27 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@ classifiers = [
5050
"Programming Language :: Python :: 3.11",
5151
"Programming Language :: Python :: 3.12",
5252
"Programming Language :: Python :: 3.13",
53+
"Programming Language :: Python :: 3.14",
5354
"Intended Audience :: Developers",
5455
"Natural Language :: Thai",
5556
"Topic :: Scientific/Engineering :: Artificial Intelligence",
@@ -70,7 +71,8 @@ dependencies = [
7071
# Development and testing dependencies
7172
dev = [
7273
"black>=25.11.0",
73-
"bump-my-version>=1.2.6 ",
74+
"build>=1.0.0",
75+
"bump-my-version>=1.2.6",
7476
"coverage>=7.10.7",
7577
"flake8>=7.3.0",
7678
"ruff>=0.14.14",
@@ -185,6 +187,7 @@ extra = [
185187
"pandas>=0.24",
186188
"ssg>=0.0.8",
187189
"symspellpy>=6.7.6",
190+
"tltk>=1.10",
188191
]
189192

190193
# Full dependencies - pinned where available
@@ -223,7 +226,7 @@ full = [
223226
"sefr_cut>=1.1",
224227
"symspellpy==6.9.0",
225228
"thai-nner==0.3",
226-
"tltk>=1.6.8,<2",
229+
"tltk>=1.10,<2",
227230
"torch>=1.13.1,<3",
228231
"transformers==4.57.6",
229232
"ufal.chu-liu-edmonds==1.0.3",
@@ -275,6 +278,28 @@ filename = "pythainlp/__init__.py"
275278
search = '__version__ = "{current_version}"'
276279
replace = '__version__ = "{new_version}"'
277280

281+
[[tool.bumpversion.files]]
282+
filename = "CITATION.cff"
283+
search = 'version: {current_version}'
284+
replace = 'version: {new_version}'
285+
286+
[[tool.bumpversion.files]]
287+
filename = "CITATION.cff"
288+
regex = true
289+
search = 'date-released: \d{{4}}-\d{{2}}-\d{{2}}'
290+
replace = 'date-released: {now:%Y-%m-%d}'
291+
292+
[[tool.bumpversion.files]]
293+
filename = "codemeta.json"
294+
search = '"version": "{current_version}"'
295+
replace = '"version": "{new_version}"'
296+
297+
[[tool.bumpversion.files]]
298+
filename = "codemeta.json"
299+
regex = true
300+
search = '"dateModified": "\d{{4}}-\d{{2}}-\d{{2}}"'
301+
replace = '"dateModified": "{now:%Y-%m-%d}"'
302+
278303
[tool.bumpversion.parts.release]
279304
optional_value = "prod"
280305
first_value = "dev"

pythainlp/tag/named_entity.py

Lines changed: 26 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -16,12 +16,15 @@ class NER:
1616
:param str corpus: corpus
1717
1818
**Options for engine**
19-
* *thainer-v2* - Thai NER engine v2.0 for Thai NER 2.0 (default)
19+
* *phayathaibert* - PhayaThaiBERT-based Thai NER engine
2020
* *thainer* - Thai NER engine
21+
* *thainer-v2* - Thai NER engine v2.0 for Thai NER 2.0 (default)
2122
* *tltk* - wrapper for `TLTK <https://pypi.org/project/tltk/>`_.
23+
* *wangchanberta* - WangchanBERTa-based Thai NER engine
2224
2325
**Options for corpus**
2426
* *thainer* - Thai NER corpus (default)
27+
* *thainer-v2* - Thai NER v2 corpus
2528
2629
**Note**: The tltk engine supports NER models from tltk only.
2730
"""
@@ -34,29 +37,33 @@ def __init__(
3437
def load_engine(self, engine: str, corpus: str) -> None:
3538
self.name_engine = engine
3639
self.engine: Any = None
37-
if engine == "thainer" and corpus == "thainer":
38-
from pythainlp.tag.thainer import ThaiNameTagger
40+
if corpus == "thainer":
41+
if engine == "thainer":
42+
from pythainlp.tag.thainer import ThaiNameTagger
3943

40-
self.engine = ThaiNameTagger()
41-
elif engine == "thainer-v2" and corpus == "thainer":
42-
from pythainlp.wangchanberta import NamedEntityRecognition
44+
self.engine = ThaiNameTagger()
45+
elif engine == "thainer-v2":
46+
from pythainlp.wangchanberta import NamedEntityRecognition
4347

44-
self.engine = NamedEntityRecognition(
45-
model="pythainlp/thainer-corpus-v2-base-model"
46-
)
47-
elif engine == "tltk":
48-
from pythainlp.tag import tltk
48+
self.engine = NamedEntityRecognition(
49+
model="pythainlp/thainer-corpus-v2-base-model"
50+
)
51+
elif engine == "wangchanberta":
52+
from pythainlp.wangchanberta import ThaiNameTagger as WangchanbertaThaiNameTagger # type: ignore[assignment] # noqa: I001,E501
53+
54+
self.engine = WangchanbertaThaiNameTagger(dataset_name=corpus) # type: ignore[call-arg]
55+
elif corpus == "thainer-v2":
56+
if engine == "phayathaibert":
57+
from pythainlp.phayathaibert.core import NamedEntityTagger
4958

50-
self.engine = tltk
51-
elif engine == "wangchanberta" and corpus == "thainer":
52-
from pythainlp.wangchanberta import ThaiNameTagger # type: ignore[assignment] # noqa: I001
59+
self.engine = NamedEntityTagger()
60+
else: # No corpus matched
61+
if engine == "tltk":
62+
from pythainlp.tag import tltk
5363

54-
self.engine = ThaiNameTagger(dataset_name=corpus) # type: ignore[call-arg]
55-
elif engine == "phayathaibert" and corpus == "thainer-v2":
56-
from pythainlp.phayathaibert.core import NamedEntityTagger
64+
self.engine = tltk
5765

58-
self.engine = NamedEntityTagger()
59-
else:
66+
if self.engine is None:
6067
raise ValueError(
6168
f"NER class not support {engine} engine or {corpus} corpus."
6269
)

pythainlp/util/__init__.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,8 @@
4545
"reorder_vowels",
4646
"rhyme",
4747
"sound_syllable",
48+
"spell_syllable",
49+
"spell_word",
4850
"spelling",
4951
"spell_words",
5052
"syllable_length",
@@ -121,6 +123,7 @@
121123
from pythainlp.util.remove_trailing_repeat_consonants import (
122124
remove_trailing_repeat_consonants,
123125
)
126+
from pythainlp.util.spell_words import spell_syllable, spell_word
124127
from pythainlp.util.strftime import thai_strftime
125128

126129
from pythainlp.util.thai import (

0 commit comments

Comments
 (0)