Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: Feature request
about: Propose a change or an addition เสนอความสามารถใหม่
about: Propose a change or addition เสนอความสามารถใหม่
---

## Detailed description
Expand Down
18 changes: 9 additions & 9 deletions .github/copilot-instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,14 +113,14 @@
- [ ] Be defensive on variable expansion.
- [ ] Use quotes or other constructs to encapsulate paths, make it compatible
with different kinds of shells.
- [ ] Be mindful about semantic of different types of quotation marks.
- [ ] Be mindful about the semantics of different types of quotation marks.

## Library imports and dependencies

- [ ] Recheck the correctness of library/module/package name.
- [ ] Check the correctness of library/module/package names.
Be very careful of slopsquatting and typosquatting attacks.
- [ ] Use the most updated version of the library that is supported
by the OS/compiler/framework currently being in used.
by the OS/compiler/framework currently being used.
- [ ] In source code, group and sort imports by the programming language
convention (e.g., in Python, typically by standard library first,
then by third-party libraries)
Expand All @@ -130,8 +130,8 @@
- [ ] Remove unused imports.
- [ ] In build metadata (like pyproject.toml in Python) or
dependency list (like requirements.txt in Python), sort dependencies.
- [ ] Warn users about abandoned dependency with no maintenance
for long time and suggest equivalent drop-in replacement.
- [ ] Warn users about abandoned dependencies with no maintenance
for a long time and suggest equivalent drop-in replacements.

## Security

Expand All @@ -153,15 +153,15 @@
- [ ] Avoid the deserialization of untrusted data (CWE-502).
- [ ] In Python, avoid using `pickle` module for
serialization/deserialization.
- [ ] When handling file and path, be careful of path traversal vulnerabilities
- [ ] When handling files and paths, be careful of path traversal vulnerabilities
like CWE-22.

## API

- [ ] The overall architecture, code, API endpoints to follow the latest
- [ ] The overall architecture, code, and API endpoints should follow the latest
version of OpenAPI specification at https://spec.openapis.org/oas/
- [ ] API endpoints must use proper HTTP return code
- [ ] Follows web best practices as recommended by OpenAPI, IETF, W3C, etc.
- [ ] API endpoints must use proper HTTP return codes.
- [ ] Follow web best practices as recommended by OpenAPI, IETF, W3C, etc.

## Python

Expand Down
2 changes: 1 addition & 1 deletion .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
### What does this changes
### What do these changes do

<!--- Brief summary of the changes -->

Expand Down
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Notable changes between versions.

## Version 5.2.0 -> dev

Some features and fixes in this version are AI assisted.
Some features and fixes in this version are AI-assisted.
See PR for prompt and details.

- Lazy load dictionaries to reduce memory usage #1186
Expand Down
2 changes: 1 addition & 1 deletion CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
## Our Pledge

In the interest of fostering an open and welcoming environment,
we as contributors and maintainers pledge to making participation
we as contributors and maintainers pledge to make participation
in our project and our community a harassment-free experience for everyone,
regardless of age, body size, disability, ethnicity, sex characteristics,
gender identity and expression, level of experience, education,
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Please refer to our [Contributor Covenant Code of Conduct](https://github.com/Py
(see "Testing" section below);
- Run all tests before pushing (just execute `tox`) so you will know if your
changes broke something;
- Commented out codes are [dead codes][dead-codes];
- Commented-out code is [dead code][dead-codes];
- All `#TODO` comments should be turned into [issues][issues] in GitHub;
- When appropriate, use [f-string][pep0498]
(use `f"{a} = {b}"`,
Expand Down
2 changes: 1 addition & 1 deletion INTHEWILD.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Who uses PyThaiNLP?

We would like to keep track of who are using the package.
We would like to keep track of who is using the package.
Please send a PR with your company name or `@githubhandle`
or both company name and `@githubhandle`.

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -273,5 +273,5 @@ and its BibTeX entry:
</div>

<div align="center">
<strong>Beware of malware if you use codes from mirrors other than the official two on GitHub and GitLab.</strong>
<strong>Beware of malware if you use code from mirrors other than the official two on GitHub and GitLab.</strong>
</div>
4 changes: 3 additions & 1 deletion SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,9 @@

## Future Security Recommendations

- Migrate from pickle to a safer serialization format like JSON or MessagePack.
The following security improvements are planned for future releases:

- Migrate from pickle to a safer serialization format such as JSON or MessagePack.
- Upgrade the hashing algorithm for integrity verification from MD5 to SHA-256 or SHA-3.
- Implement digital signatures for corpus files to ensure authenticity.
- Add version tracking to the corpus to prevent rollback attacks.
2 changes: 1 addition & 1 deletion docs/notes/installation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ The extras can include:
- ``abbreviation`` — abbreviation expansion utilities
- ``full`` — install all optional dependencies

For dependency details, see the `extras` variable in `setup.py <https://github.com/PyThaiNLP/pythainlp/blob/dev/setup.py>`_.
For dependency details, see the `project.optional-dependencies` section in `pyproject.toml <https://github.com/PyThaiNLP/pythainlp/blob/dev/pyproject.toml>`_.

Notes for Windows installation
-----------------------------
Expand Down
2 changes: 1 addition & 1 deletion docs/notes/start.rst
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ PyThaiNLP is a Python library for Thai natural language processing (NLP). With t

Thai NLP faces several challenges. A brief list includes:

#. **Sentence boundary detection** — This is one of the biggest challenges in Thai NLP. The lack of explicit end-of-sentence markers makes it difficult to create training sets for many tasks. The issue is twofold: in the writing system, Thai punctuation and spacing do not always indicate sentence endings; in language use, sentences often begin with conjunctions such as 'because' or 'but', which can make sentence boundaries ambiguous even for native speakers.
#. **Sentence boundary detection** — This is one of the biggest challenges in Thai NLP. The lack of explicit end-of-sentence markers makes it difficult to create training sets for many tasks. The issue is twofold: (1) in the writing system, Thai punctuation and spacing do not always indicate sentence endings; (2) in language use, sentences often begin with conjunctions such as 'because' or 'but', which can make sentence boundaries ambiguous even for native speakers.

#. **Word segmentation** — Thai does not use spaces to separate words, so segmentation is challenging. Solving it often requires understanding context to rule out unlikely word breaks. This is similar to issues in other Asian languages such as Japanese and Chinese. Recently, techniques that represent words, subwords, and characters as vectors (embeddings) have improved performance and help address this problem.

Expand Down
8 changes: 4 additions & 4 deletions pythainlp/corpus/corpus_license.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Corpus License

- Corpora, datasets, and documentation created by PyThaiNLP project are released under [Creative Commons Zero 1.0 Universal Public Domain Dedication License](https://creativecommons.org/publicdomain/zero/1.0/) (CC0).
- Language models created by PyThaiNLP project are released under [Creative Commons Attribution 4.0 International Public License](https://creativecommons.org/licenses/by/4.0/) (CC-by).
- For more information about corpora that PyThaiNLP use, see [https://github.com/PyThaiNLP/pythainlp-corpus/](https://github.com/PyThaiNLP/pythainlp-corpus/).
- Corpora, datasets, and documentation created by the PyThaiNLP project are released under [Creative Commons Zero 1.0 Universal Public Domain Dedication License](https://creativecommons.org/publicdomain/zero/1.0/) (CC0).
- Language models created by the PyThaiNLP project are released under [Creative Commons Attribution 4.0 International Public License](https://creativecommons.org/licenses/by/4.0/) (CC-by).
- For more information about corpora that PyThaiNLP uses, see [https://github.com/PyThaiNLP/pythainlp-corpus/](https://github.com/PyThaiNLP/pythainlp-corpus/).

## Dictionaries and Word Lists

Expand All @@ -13,7 +13,7 @@ The following word lists are created by the PyThaiNLP project and released under
| Filename | Description |
| ---------------------------- | ------------------------------------------------------ |
| countries_th.txt | List of countries in Thai |
| etcc.txt List of | Enhanced Thai Character Clusters |
| etcc.txt | List of Enhanced Thai Character Clusters |
| negations_th.txt | Negation word list |
| stopwords_th.txt | Stop word list |
| syllables_th.txt | List of Thai syllables |
Expand Down
2 changes: 1 addition & 1 deletion pythainlp/soundex/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
def soundex(
text: str, engine: str = DEFAULT_SOUNDEX_ENGINE, length: int = 4
) -> str:
"""This function converts Thai text into phonetic code.
"""Converts Thai text into phonetic code.

:param str text: word
:param str engine: soundex engine
Expand Down
2 changes: 1 addition & 1 deletion pythainlp/soundex/lk82.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@


def lk82(text: str) -> str:
"""This function converts Thai text into phonetic code with the
"""Converts Thai text into phonetic code with the
Thai soundex algorithm named **LK82** [#lk82]_.

:param str text: Thai word
Expand Down
2 changes: 1 addition & 1 deletion pythainlp/soundex/metasound.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@


def metasound(text: str, length: int = 4) -> str:
"""This function converts Thai text into phonetic code with the
"""Converts Thai text into phonetic code with the
matching technique called **MetaSound**
[#metasound]_ (combination between Soundex and Metaphone algorithms).
MetaSound algorithm was developed specifically for the Thai language.
Expand Down
2 changes: 1 addition & 1 deletion pythainlp/soundex/prayut_and_somchaip.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@


def prayut_and_somchaip(text: str, length: int = 4) -> str:
"""This function converts English-Thai Cross-Language Transliterated Word into
"""Converts English-Thai Cross-Language Transliterated Words into
phonetic code with the matching technique called **Soundex** [#prayut_and_somchaip]_.

:param str text: English-Thai Cross-Language Transliterated Word
Expand Down
2 changes: 1 addition & 1 deletion pythainlp/soundex/udom83.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@


def udom83(text: str) -> str:
"""This function converts Thai text into phonetic code with the
"""Converts Thai text into phonetic code with the
Thai soundex algorithm named **Udom83** [#udom83]_.

:param str text: Thai word
Expand Down
4 changes: 2 additions & 2 deletions pythainlp/spell/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@ def default_spell_checker() -> "NorvigSpellChecker":

def spell(word: str, engine: str = "pn") -> list[str]:
"""Provides a list of possible correct spellings of the given word.
The list of words are from the words in the dictionary
that incurs an edit distance value of 1 or 2.
The list of words is from words in the dictionary
that have an edit distance value of 1 or 2.
The result is a list of words sorted by their occurrences
in the spelling dictionary in descending order.

Expand Down
6 changes: 3 additions & 3 deletions pythainlp/summarize/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ def summarize(
engine: str = DEFAULT_SUMMARIZE_ENGINE,
tokenizer: str = "newmm",
) -> list[str]:
"""This function summarizes text based on frequency of words.
"""Summarizes text based on frequency of words.

Under the hood, this function first tokenizes sentences from the given
text with :func:`pythainlp.tokenize.sent_tokenize`.
Expand Down Expand Up @@ -126,12 +126,12 @@ def extract_keywords(
tokenizer: str = "newmm",
stop_words: Optional[Iterable[str]] = None,
) -> list[str]:
"""This function returns most-relevant keywords (and/or keyphrases) from the input document.
"""Returns most-relevant keywords (and/or keyphrases) from the input document.
Each algorithm may produce completely different keywords from each other,
so please be careful when choosing the algorithm.

*Note*: Calling :func: `extract_keywords()` is expensive. For repetitive use of KeyBERT (the default engine),
creating KeyBERT object is highly recommended.
creating a KeyBERT object is highly recommended.

:param str text: text to be summarized
:param Tuple[int, int] keyphrase_ngram_range: Number of token units to be defined as keyword.
Expand Down
2 changes: 1 addition & 1 deletion pythainlp/tokenize/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ def word_detokenize(
) -> Union[list[str], str]:
"""Word detokenizer.

This function will detokenize the list of words in each sentence into text.
Detokenizes the list of words in each sentence into text.

:param str segments: List of sentences, each with a list of words.
:param str output: the output type (str or list)
Expand Down
2 changes: 1 addition & 1 deletion pythainlp/tokenize/nlpo3.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ def load_dict(file_path: str, dict_name: str) -> bool:
"""Load a dictionary file into an in-memory dictionary collection.

The loaded dictionary will be accessible through the assigned dict_name.
*** This function will not override an existing dict name. ***
**Note: This function will not override an existing dict name.**

:param file_path: Path to a dictionary file
:type file_path: str
Expand Down
6 changes: 3 additions & 3 deletions pythainlp/transliterate/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ def romanize(
engine: str = DEFAULT_ROMANIZE_ENGINE,
fallback_engine: str = DEFAULT_ROMANIZE_ENGINE,
) -> str:
"""This function renders Thai word in the Latin alphabet or "romanization",
"""Renders Thai words in the Latin alphabet or "romanization",
using the Royal Thai General System of Transcription (RTGS)
[#rtgs_transcription]_. RTGS is the official system published
by the Royal Institute of Thailand. (Thai: ถอดเสียงภาษาไทยเป็นอักษรละติน)
Expand Down Expand Up @@ -104,7 +104,7 @@ def select_romanize_engine(engine: str) -> Callable[[str], str]:
def transliterate(
text: str, engine: str = DEFAULT_TRANSLITERATE_ENGINE
) -> str:
"""This function transliterates Thai text.
"""Transliterates Thai text.

:param str text: Thai text to be transliterated
:param str engine: 'icu', 'ipa', or 'thaig2p' (default)
Expand Down Expand Up @@ -187,7 +187,7 @@ def transliterate(


def pronunciate(word: str, engine: str = DEFAULT_PRONUNCIATE_ENGINE) -> str:
"""This function pronunciates Thai word.
"""Pronunciates Thai words.

:param str word: Thai text to be pronunciated
:param str engine: 'w2p' (default)
Expand Down
2 changes: 1 addition & 1 deletion pythainlp/transliterate/spoonerism.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
def puan(word: str, show_pronunciation: bool = True) -> str:
"""Thai Spoonerism

This function converts Thai word to spoonerism word.
Converts a Thai word to a spoonerism word.

:param str word: Thai word to be spoonerized
:param bool show_pronunciation: True (default) or False
Expand Down
6 changes: 3 additions & 3 deletions pythainlp/util/abbreviation.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@
def abbreviation_to_full_text(
text: str, top_k: int = 2
) -> list[tuple[str, Optional[float]]]:
"""This function converts Thai text (with abbreviation) to full text.
"""Converts Thai text (with abbreviations) to full text.

This function uses KhamYo for handles abbreviations.
See more `KhamYo <https://github.com/wannaphong/KhamYo>`_.
Uses KhamYo to handle abbreviations.
See more: `KhamYo <https://github.com/wannaphong/KhamYo>`_.

:param str text: Thai text
:param int top_k: Top K
Expand Down
2 changes: 1 addition & 1 deletion pythainlp/util/collate.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ def _thkey(word: str) -> str:


def collate(data: Iterable, reverse: bool = False) -> list[str]:
"""This function sorts strings (almost) according to Thai dictionary.
"""Sorts strings (almost) according to Thai dictionary.

Important notes: this implementation ignores tone marks and symbols

Expand Down
8 changes: 4 additions & 4 deletions pythainlp/util/digitconv.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@


def thai_digit_to_arabic_digit(text: str) -> str:
"""This function converts Thai digits (i.e. ๑, ๓, ๑๐) to Arabic digits
"""Converts Thai digits (i.e. ๑, ๓, ๑๐) to Arabic digits
(i.e. 1, 3, 10).

:param str text: Text with Thai digits such as '๑', '๒', '๓'
Expand All @@ -89,7 +89,7 @@ def thai_digit_to_arabic_digit(text: str) -> str:


def arabic_digit_to_thai_digit(text: str) -> str:
"""This function converts Arabic digits (i.e. 1, 3, 10) to Thai digits
"""Converts Arabic digits (i.e. 1, 3, 10) to Thai digits
(i.e. ๑, ๓, ๑๐).

:param str text: Text with Arabic digits such as '1', '2', '3'
Expand Down Expand Up @@ -143,7 +143,7 @@ def digit_to_text(text: str) -> str:


def text_to_arabic_digit(text: str) -> str:
"""This function converts spelled out digits in Thai to Arabic digits.
"""Converts spelled out digits in Thai to Arabic digits.

:param text: A digit spelled out in Thai
:return: An Arabic digit such as '1', '2', '3' if the text is
Expand Down Expand Up @@ -180,7 +180,7 @@ def text_to_arabic_digit(text: str) -> str:


def text_to_thai_digit(text: str) -> str:
"""This function converts spelled out digits in Thai to Thai digits.
"""Converts spelled out digits in Thai to Thai digits.

:param text: A digit spelled out in Thai
:return: A Thai digit such as '๑', '๒', '๓' if the text is digit
Expand Down
2 changes: 1 addition & 1 deletion pythainlp/util/emojiconv.py
Original file line number Diff line number Diff line change
Expand Up @@ -1834,7 +1834,7 @@


def emoji_to_thai(text: str, delimiters=(_delimiter, _delimiter)) -> str:
"""This function converts emojis to their Thai meanings
"""Converts emojis to their Thai meanings.

:param str text: Text with emojis
:return: Text with emojis converted to their Thai meanings
Expand Down
4 changes: 2 additions & 2 deletions pythainlp/util/keywords.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,8 +73,8 @@ def rank(words: list[str], exclude_stopwords: bool = False) -> Optional[Counter]


def find_keyword(word_list: list[str], min_len: int = 3) -> dict[str, int]:
"""This function counts the frequencies of words in the list
where stopword is excluded and returns a frequency dictionary.
"""Counts the frequencies of words in the list
where stopwords are excluded and returns a frequency dictionary.

:param list word_list: a list of words
:param int min_len: the minimum frequency for words to be retained
Expand Down
2 changes: 1 addition & 1 deletion pythainlp/util/normalize.py
Original file line number Diff line number Diff line change
Expand Up @@ -246,7 +246,7 @@ def reorder_vowels(text: str) -> str:
def remove_repeat_vowels(text: str) -> str:
"""Remove repeating vowels, tone marks, and signs.

This function will call reorder_vowels() first, to make sure that
Calls reorder_vowels() first to ensure that
double Sara E will be converted to Sara Ae and not be removed.

:param str text: input text
Expand Down
Loading
Loading