diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md index e985f859c..4f3b414eb 100644 --- a/.github/ISSUE_TEMPLATE/feature_request.md +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -1,6 +1,6 @@ --- name: Feature request -about: Propose a change or an addition เสนอความสามารถใหม่ +about: Propose a change or addition เสนอความสามารถใหม่ --- ## Detailed description diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 42bcfbbfa..81701d8cf 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -113,14 +113,14 @@ - [ ] Be defensive on variable expansion. - [ ] Use quotes or other constructs to encapsulate paths, make it compatible with different kinds of shells. -- [ ] Be mindful about semantic of different types of quotation marks. +- [ ] Be mindful about the semantics of different types of quotation marks. ## Library imports and dependencies -- [ ] Recheck the correctness of library/module/package name. +- [ ] Check the correctness of library/module/package names. Be very careful of slopsquatting and typosquatting attacks. - [ ] Use the most updated version of the library that is supported - by the OS/compiler/framework currently being in used. + by the OS/compiler/framework currently being used. - [ ] In source code, group and sort imports by the programming language convention (e.g., in Python, typically by standard library first, then by third-party libraries) @@ -130,8 +130,8 @@ - [ ] Remove unused imports. - [ ] In build metadata (like pyproject.toml in Python) or dependency list (like requirements.txt in Python), sort dependencies. -- [ ] Warn users about abandoned dependency with no maintenance - for long time and suggest equivalent drop-in replacement. +- [ ] Warn users about abandoned dependencies with no maintenance + for a long time and suggest equivalent drop-in replacements. ## Security @@ -153,15 +153,15 @@ - [ ] Avoid the deserialization of untrusted data (CWE-502). - [ ] In Python, avoid using `pickle` module for serialization/deserialization. -- [ ] When handling file and path, be careful of path traversal vulnerabilities +- [ ] When handling files and paths, be careful of path traversal vulnerabilities like CWE-22. ## API -- [ ] The overall architecture, code, API endpoints to follow the latest +- [ ] The overall architecture, code, and API endpoints should follow the latest version of OpenAPI specification at https://spec.openapis.org/oas/ -- [ ] API endpoints must use proper HTTP return code -- [ ] Follows web best practices as recommended by OpenAPI, IETF, W3C, etc. +- [ ] API endpoints must use proper HTTP return codes. +- [ ] Follow web best practices as recommended by OpenAPI, IETF, W3C, etc. ## Python diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 4838d666a..4d6ce8c15 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,4 +1,4 @@ -### What does this changes +### What do these changes do diff --git a/CHANGELOG.md b/CHANGELOG.md index bca5fc44d..458b31c50 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,7 +15,7 @@ Notable changes between versions. ## Version 5.2.0 -> dev -Some features and fixes in this version are AI assisted. +Some features and fixes in this version are AI-assisted. See PR for prompt and details. - Lazy load dictionaries to reduce memory usage #1186 diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index 4da1fcde4..4239f5e50 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -3,7 +3,7 @@ ## Our Pledge In the interest of fostering an open and welcoming environment, -we as contributors and maintainers pledge to making participation +we as contributors and maintainers pledge to make participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2ec505a53..3f3a07ef0 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -32,7 +32,7 @@ Please refer to our [Contributor Covenant Code of Conduct](https://github.com/Py (see "Testing" section below); - Run all tests before pushing (just execute `tox`) so you will know if your changes broke something; -- Commented out codes are [dead codes][dead-codes]; +- Commented-out code is [dead code][dead-codes]; - All `#TODO` comments should be turned into [issues][issues] in GitHub; - When appropriate, use [f-string][pep0498] (use `f"{a} = {b}"`, diff --git a/INTHEWILD.md b/INTHEWILD.md index b8e49a813..ee4dad16f 100644 --- a/INTHEWILD.md +++ b/INTHEWILD.md @@ -1,6 +1,6 @@ # Who uses PyThaiNLP? -We would like to keep track of who are using the package. +We would like to keep track of who is using the package. Please send a PR with your company name or `@githubhandle` or both company name and `@githubhandle`. diff --git a/README.md b/README.md index 0e855765c..066a32062 100644 --- a/README.md +++ b/README.md @@ -273,5 +273,5 @@ and its BibTeX entry:
- Beware of malware if you use codes from mirrors other than the official two on GitHub and GitLab. + Beware of malware if you use code from mirrors other than the official two on GitHub and GitLab.
diff --git a/SECURITY.md b/SECURITY.md index 6054330dd..169e2f5a0 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -18,7 +18,9 @@ ## Future Security Recommendations -- Migrate from pickle to a safer serialization format like JSON or MessagePack. +The following security improvements are planned for future releases: + +- Migrate from pickle to a safer serialization format such as JSON or MessagePack. - Upgrade the hashing algorithm for integrity verification from MD5 to SHA-256 or SHA-3. - Implement digital signatures for corpus files to ensure authenticity. - Add version tracking to the corpus to prevent rollback attacks. diff --git a/docs/notes/installation.rst b/docs/notes/installation.rst index cb8426b48..44a8d81c5 100644 --- a/docs/notes/installation.rst +++ b/docs/notes/installation.rst @@ -42,7 +42,7 @@ The extras can include: - ``abbreviation`` — abbreviation expansion utilities - ``full`` — install all optional dependencies -For dependency details, see the `extras` variable in `setup.py `_. +For dependency details, see the `project.optional-dependencies` section in `pyproject.toml `_. Notes for Windows installation ----------------------------- diff --git a/docs/notes/start.rst b/docs/notes/start.rst index 3da48c700..59512de22 100644 --- a/docs/notes/start.rst +++ b/docs/notes/start.rst @@ -13,7 +13,7 @@ PyThaiNLP is a Python library for Thai natural language processing (NLP). With t Thai NLP faces several challenges. A brief list includes: -#. **Sentence boundary detection** — This is one of the biggest challenges in Thai NLP. The lack of explicit end-of-sentence markers makes it difficult to create training sets for many tasks. The issue is twofold: in the writing system, Thai punctuation and spacing do not always indicate sentence endings; in language use, sentences often begin with conjunctions such as 'because' or 'but', which can make sentence boundaries ambiguous even for native speakers. +#. **Sentence boundary detection** — This is one of the biggest challenges in Thai NLP. The lack of explicit end-of-sentence markers makes it difficult to create training sets for many tasks. The issue is twofold: (1) in the writing system, Thai punctuation and spacing do not always indicate sentence endings; (2) in language use, sentences often begin with conjunctions such as 'because' or 'but', which can make sentence boundaries ambiguous even for native speakers. #. **Word segmentation** — Thai does not use spaces to separate words, so segmentation is challenging. Solving it often requires understanding context to rule out unlikely word breaks. This is similar to issues in other Asian languages such as Japanese and Chinese. Recently, techniques that represent words, subwords, and characters as vectors (embeddings) have improved performance and help address this problem. diff --git a/pythainlp/corpus/corpus_license.md b/pythainlp/corpus/corpus_license.md index 70b72c335..f4e0efba0 100644 --- a/pythainlp/corpus/corpus_license.md +++ b/pythainlp/corpus/corpus_license.md @@ -1,8 +1,8 @@ # Corpus License -- Corpora, datasets, and documentation created by PyThaiNLP project are released under [Creative Commons Zero 1.0 Universal Public Domain Dedication License](https://creativecommons.org/publicdomain/zero/1.0/) (CC0). -- Language models created by PyThaiNLP project are released under [Creative Commons Attribution 4.0 International Public License](https://creativecommons.org/licenses/by/4.0/) (CC-by). -- For more information about corpora that PyThaiNLP use, see [https://github.com/PyThaiNLP/pythainlp-corpus/](https://github.com/PyThaiNLP/pythainlp-corpus/). +- Corpora, datasets, and documentation created by the PyThaiNLP project are released under [Creative Commons Zero 1.0 Universal Public Domain Dedication License](https://creativecommons.org/publicdomain/zero/1.0/) (CC0). +- Language models created by the PyThaiNLP project are released under [Creative Commons Attribution 4.0 International Public License](https://creativecommons.org/licenses/by/4.0/) (CC-by). +- For more information about corpora that PyThaiNLP uses, see [https://github.com/PyThaiNLP/pythainlp-corpus/](https://github.com/PyThaiNLP/pythainlp-corpus/). ## Dictionaries and Word Lists @@ -13,7 +13,7 @@ The following word lists are created by the PyThaiNLP project and released under | Filename | Description | | ---------------------------- | ------------------------------------------------------ | | countries_th.txt | List of countries in Thai | -| etcc.txt List of | Enhanced Thai Character Clusters | +| etcc.txt | List of Enhanced Thai Character Clusters | | negations_th.txt | Negation word list | | stopwords_th.txt | Stop word list | | syllables_th.txt | List of Thai syllables | diff --git a/pythainlp/soundex/core.py b/pythainlp/soundex/core.py index 3c85ee887..504aca6ef 100644 --- a/pythainlp/soundex/core.py +++ b/pythainlp/soundex/core.py @@ -23,7 +23,7 @@ def soundex( text: str, engine: str = DEFAULT_SOUNDEX_ENGINE, length: int = 4 ) -> str: - """This function converts Thai text into phonetic code. + """Converts Thai text into phonetic code. :param str text: word :param str engine: soundex engine diff --git a/pythainlp/soundex/lk82.py b/pythainlp/soundex/lk82.py index 47f8aaddc..8d4d9561c 100644 --- a/pythainlp/soundex/lk82.py +++ b/pythainlp/soundex/lk82.py @@ -38,7 +38,7 @@ def lk82(text: str) -> str: - """This function converts Thai text into phonetic code with the + """Converts Thai text into phonetic code with the Thai soundex algorithm named **LK82** [#lk82]_. :param str text: Thai word diff --git a/pythainlp/soundex/metasound.py b/pythainlp/soundex/metasound.py index ef764eb52..e5876ef27 100644 --- a/pythainlp/soundex/metasound.py +++ b/pythainlp/soundex/metasound.py @@ -26,7 +26,7 @@ def metasound(text: str, length: int = 4) -> str: - """This function converts Thai text into phonetic code with the + """Converts Thai text into phonetic code with the matching technique called **MetaSound** [#metasound]_ (combination between Soundex and Metaphone algorithms). MetaSound algorithm was developed specifically for the Thai language. diff --git a/pythainlp/soundex/prayut_and_somchaip.py b/pythainlp/soundex/prayut_and_somchaip.py index 0480e1f73..c71507043 100644 --- a/pythainlp/soundex/prayut_and_somchaip.py +++ b/pythainlp/soundex/prayut_and_somchaip.py @@ -31,7 +31,7 @@ def prayut_and_somchaip(text: str, length: int = 4) -> str: - """This function converts English-Thai Cross-Language Transliterated Word into + """Converts English-Thai Cross-Language Transliterated Words into phonetic code with the matching technique called **Soundex** [#prayut_and_somchaip]_. :param str text: English-Thai Cross-Language Transliterated Word diff --git a/pythainlp/soundex/udom83.py b/pythainlp/soundex/udom83.py index cfb6a72f8..e5736c6a5 100644 --- a/pythainlp/soundex/udom83.py +++ b/pythainlp/soundex/udom83.py @@ -49,7 +49,7 @@ def udom83(text: str) -> str: - """This function converts Thai text into phonetic code with the + """Converts Thai text into phonetic code with the Thai soundex algorithm named **Udom83** [#udom83]_. :param str text: Thai word diff --git a/pythainlp/spell/core.py b/pythainlp/spell/core.py index 978dddd36..c4bbeaec5 100644 --- a/pythainlp/spell/core.py +++ b/pythainlp/spell/core.py @@ -24,8 +24,8 @@ def default_spell_checker() -> "NorvigSpellChecker": def spell(word: str, engine: str = "pn") -> list[str]: """Provides a list of possible correct spellings of the given word. - The list of words are from the words in the dictionary - that incurs an edit distance value of 1 or 2. + The list of words is from words in the dictionary + that have an edit distance value of 1 or 2. The result is a list of words sorted by their occurrences in the spelling dictionary in descending order. diff --git a/pythainlp/summarize/core.py b/pythainlp/summarize/core.py index 690efa03a..f1c4f975d 100644 --- a/pythainlp/summarize/core.py +++ b/pythainlp/summarize/core.py @@ -24,7 +24,7 @@ def summarize( engine: str = DEFAULT_SUMMARIZE_ENGINE, tokenizer: str = "newmm", ) -> list[str]: - """This function summarizes text based on frequency of words. + """Summarizes text based on frequency of words. Under the hood, this function first tokenizes sentences from the given text with :func:`pythainlp.tokenize.sent_tokenize`. @@ -126,12 +126,12 @@ def extract_keywords( tokenizer: str = "newmm", stop_words: Optional[Iterable[str]] = None, ) -> list[str]: - """This function returns most-relevant keywords (and/or keyphrases) from the input document. + """Returns most-relevant keywords (and/or keyphrases) from the input document. Each algorithm may produce completely different keywords from each other, so please be careful when choosing the algorithm. *Note*: Calling :func: `extract_keywords()` is expensive. For repetitive use of KeyBERT (the default engine), - creating KeyBERT object is highly recommended. + creating a KeyBERT object is highly recommended. :param str text: text to be summarized :param Tuple[int, int] keyphrase_ngram_range: Number of token units to be defined as keyword. diff --git a/pythainlp/tokenize/core.py b/pythainlp/tokenize/core.py index 0d8e98129..bcb76aa72 100644 --- a/pythainlp/tokenize/core.py +++ b/pythainlp/tokenize/core.py @@ -35,7 +35,7 @@ def word_detokenize( ) -> Union[list[str], str]: """Word detokenizer. - This function will detokenize the list of words in each sentence into text. + Detokenizes the list of words in each sentence into text. :param str segments: List of sentences, each with a list of words. :param str output: the output type (str or list) diff --git a/pythainlp/tokenize/nlpo3.py b/pythainlp/tokenize/nlpo3.py index 381f93ec2..55fa7c877 100644 --- a/pythainlp/tokenize/nlpo3.py +++ b/pythainlp/tokenize/nlpo3.py @@ -44,7 +44,7 @@ def load_dict(file_path: str, dict_name: str) -> bool: """Load a dictionary file into an in-memory dictionary collection. The loaded dictionary will be accessible through the assigned dict_name. - *** This function will not override an existing dict name. *** + **Note: This function will not override an existing dict name.** :param file_path: Path to a dictionary file :type file_path: str diff --git a/pythainlp/transliterate/core.py b/pythainlp/transliterate/core.py index 861b1c282..b54cd41d1 100644 --- a/pythainlp/transliterate/core.py +++ b/pythainlp/transliterate/core.py @@ -15,7 +15,7 @@ def romanize( engine: str = DEFAULT_ROMANIZE_ENGINE, fallback_engine: str = DEFAULT_ROMANIZE_ENGINE, ) -> str: - """This function renders Thai word in the Latin alphabet or "romanization", + """Renders Thai words in the Latin alphabet or "romanization", using the Royal Thai General System of Transcription (RTGS) [#rtgs_transcription]_. RTGS is the official system published by the Royal Institute of Thailand. (Thai: ถอดเสียงภาษาไทยเป็นอักษรละติน) @@ -104,7 +104,7 @@ def select_romanize_engine(engine: str) -> Callable[[str], str]: def transliterate( text: str, engine: str = DEFAULT_TRANSLITERATE_ENGINE ) -> str: - """This function transliterates Thai text. + """Transliterates Thai text. :param str text: Thai text to be transliterated :param str engine: 'icu', 'ipa', or 'thaig2p' (default) @@ -187,7 +187,7 @@ def transliterate( def pronunciate(word: str, engine: str = DEFAULT_PRONUNCIATE_ENGINE) -> str: - """This function pronunciates Thai word. + """Pronunciates Thai words. :param str word: Thai text to be pronunciated :param str engine: 'w2p' (default) diff --git a/pythainlp/transliterate/spoonerism.py b/pythainlp/transliterate/spoonerism.py index 9a19deb07..7e0b3c9aa 100644 --- a/pythainlp/transliterate/spoonerism.py +++ b/pythainlp/transliterate/spoonerism.py @@ -12,7 +12,7 @@ def puan(word: str, show_pronunciation: bool = True) -> str: """Thai Spoonerism - This function converts Thai word to spoonerism word. + Converts a Thai word to a spoonerism word. :param str word: Thai word to be spoonerized :param bool show_pronunciation: True (default) or False diff --git a/pythainlp/util/abbreviation.py b/pythainlp/util/abbreviation.py index 0916cad39..542ca872f 100644 --- a/pythainlp/util/abbreviation.py +++ b/pythainlp/util/abbreviation.py @@ -12,10 +12,10 @@ def abbreviation_to_full_text( text: str, top_k: int = 2 ) -> list[tuple[str, Optional[float]]]: - """This function converts Thai text (with abbreviation) to full text. + """Converts Thai text (with abbreviations) to full text. - This function uses KhamYo for handles abbreviations. - See more `KhamYo `_. + Uses KhamYo to handle abbreviations. + See more: `KhamYo `_. :param str text: Thai text :param int top_k: Top K diff --git a/pythainlp/util/collate.py b/pythainlp/util/collate.py index 6f6feb5e9..dc7db672f 100644 --- a/pythainlp/util/collate.py +++ b/pythainlp/util/collate.py @@ -24,7 +24,7 @@ def _thkey(word: str) -> str: def collate(data: Iterable, reverse: bool = False) -> list[str]: - """This function sorts strings (almost) according to Thai dictionary. + """Sorts strings (almost) according to Thai dictionary. Important notes: this implementation ignores tone marks and symbols diff --git a/pythainlp/util/digitconv.py b/pythainlp/util/digitconv.py index 389a721c7..b80164b04 100644 --- a/pythainlp/util/digitconv.py +++ b/pythainlp/util/digitconv.py @@ -64,7 +64,7 @@ def thai_digit_to_arabic_digit(text: str) -> str: - """This function converts Thai digits (i.e. ๑, ๓, ๑๐) to Arabic digits + """Converts Thai digits (i.e. ๑, ๓, ๑๐) to Arabic digits (i.e. 1, 3, 10). :param str text: Text with Thai digits such as '๑', '๒', '๓' @@ -89,7 +89,7 @@ def thai_digit_to_arabic_digit(text: str) -> str: def arabic_digit_to_thai_digit(text: str) -> str: - """This function converts Arabic digits (i.e. 1, 3, 10) to Thai digits + """Converts Arabic digits (i.e. 1, 3, 10) to Thai digits (i.e. ๑, ๓, ๑๐). :param str text: Text with Arabic digits such as '1', '2', '3' @@ -143,7 +143,7 @@ def digit_to_text(text: str) -> str: def text_to_arabic_digit(text: str) -> str: - """This function converts spelled out digits in Thai to Arabic digits. + """Converts spelled out digits in Thai to Arabic digits. :param text: A digit spelled out in Thai :return: An Arabic digit such as '1', '2', '3' if the text is @@ -180,7 +180,7 @@ def text_to_arabic_digit(text: str) -> str: def text_to_thai_digit(text: str) -> str: - """This function converts spelled out digits in Thai to Thai digits. + """Converts spelled out digits in Thai to Thai digits. :param text: A digit spelled out in Thai :return: A Thai digit such as '๑', '๒', '๓' if the text is digit diff --git a/pythainlp/util/emojiconv.py b/pythainlp/util/emojiconv.py index 985a965da..6aa9265b4 100644 --- a/pythainlp/util/emojiconv.py +++ b/pythainlp/util/emojiconv.py @@ -1834,7 +1834,7 @@ def emoji_to_thai(text: str, delimiters=(_delimiter, _delimiter)) -> str: - """This function converts emojis to their Thai meanings + """Converts emojis to their Thai meanings. :param str text: Text with emojis :return: Text with emojis converted to their Thai meanings diff --git a/pythainlp/util/keywords.py b/pythainlp/util/keywords.py index cd5328063..751bca013 100644 --- a/pythainlp/util/keywords.py +++ b/pythainlp/util/keywords.py @@ -73,8 +73,8 @@ def rank(words: list[str], exclude_stopwords: bool = False) -> Optional[Counter] def find_keyword(word_list: list[str], min_len: int = 3) -> dict[str, int]: - """This function counts the frequencies of words in the list - where stopword is excluded and returns a frequency dictionary. + """Counts the frequencies of words in the list + where stopwords are excluded and returns a frequency dictionary. :param list word_list: a list of words :param int min_len: the minimum frequency for words to be retained diff --git a/pythainlp/util/normalize.py b/pythainlp/util/normalize.py index 2f4d6f3d5..beb3f4eac 100644 --- a/pythainlp/util/normalize.py +++ b/pythainlp/util/normalize.py @@ -246,7 +246,7 @@ def reorder_vowels(text: str) -> str: def remove_repeat_vowels(text: str) -> str: """Remove repeating vowels, tone marks, and signs. - This function will call reorder_vowels() first, to make sure that + Calls reorder_vowels() first to ensure that double Sara E will be converted to Sara Ae and not be removed. :param str text: input text diff --git a/pythainlp/util/numtoword.py b/pythainlp/util/numtoword.py index 7548f6943..81c437d13 100644 --- a/pythainlp/util/numtoword.py +++ b/pythainlp/util/numtoword.py @@ -29,11 +29,11 @@ def bahttext(number: float) -> str: - """This function converts a number to Thai text and adds + """Converts a number to Thai text and adds a suffix "บาท" (Baht). The precision will be fixed at two decimal places (0.00) - to fits "สตางค์" (Satang) unit. - This function works similar to `BAHTTEXT` function in Microsoft Excel. + to fit "สตางค์" (Satang) unit. + This function works similarly to the `BAHTTEXT` function in Microsoft Excel. :param float number: number to be converted into Thai Baht currency format :return: text representing the amount of money in the format @@ -78,7 +78,7 @@ def bahttext(number: float) -> str: def num_to_thaiword(number: int) -> str: - """This function converts number to Thai text + """Converts a number to Thai text. :param int number: an integer number to be converted to Thai text :return: text representing the number in Thai diff --git a/pythainlp/util/remove_trailing_repeat_consonants.py b/pythainlp/util/remove_trailing_repeat_consonants.py index 5ec01b3cb..af1deef5f 100644 --- a/pythainlp/util/remove_trailing_repeat_consonants.py +++ b/pythainlp/util/remove_trailing_repeat_consonants.py @@ -27,15 +27,15 @@ def remove_trailing_repeat_consonants( ) -> str: """Remove repeating consonants at the last of the sentence. - This function will remove the repeating consonants - before a whitespace, new line or at the last + Removes the repeating consonants + before a whitespace, new line, or at the end so that the last word matches a word in the given dictionary. If there is no match, the repeating consonants will be reduced to one. - If there are several match, the longest word will be used. - Since this function uses a dictionary, the result may differs + If there are several matches, the longest word will be used. + Since this function uses a dictionary, the result may differ depending on the dictionary used. - Plus, it is recommended to use normalize() to have a better result. + It is recommended to use normalize() for better results. :param str text: input text :param Trie dictionary: Trie dictionary to check the last word. @@ -102,8 +102,8 @@ def remove_trailing_repeat_consonants( def _remove_repeat_trailing_consonants_from_segment(segment: str) -> str: """Remove repeating consonants at the last of the segment. - This function process only at the last of the given text. - Details is same as remove_repeat_consonants(). + Processes only at the end of the given text. + Details are the same as remove_repeat_consonants(). :param str segment: segment of text :return: segment without repeating Thai consonants @@ -115,7 +115,7 @@ def _remove_repeat_trailing_consonants_from_segment(segment: str) -> str: (len(segment) > 1) # last is Thai consonant and (segment[-1] in consonants) - # has repiitition + # has repetition and (segment[-1] == segment[-2]) ): # no need to process @@ -153,7 +153,7 @@ def _remove_repeat_trailing_consonants_from_segment(segment: str) -> str: def _remove_all_last_consonants(text: str, dup: str) -> str: """Reduce repeating characters at the end of the text. - This function will remove the repeating characters at the last. + Removes the repeating characters at the end. The text just before the repeating characters will be returned. :param str text: input text @@ -194,8 +194,8 @@ def _update_consonant_repeaters(custom_dict: Iterable[str]) -> None: def _is_last_consonant_repeater(word: str) -> bool: """Check if the word has repeating consonants at the end. - This function checks if the word has - more than 1 repeating consonants at the end. + Checks if the word has + more than 1 repeating consonant at the end. :param str word: word to be checked :return: True if the word has repeating consonants at the end. diff --git a/pythainlp/word_vector/core.py b/pythainlp/word_vector/core.py index 6ad1e9242..1c4c58e0d 100644 --- a/pythainlp/word_vector/core.py +++ b/pythainlp/word_vector/core.py @@ -77,7 +77,7 @@ def doesnt_match(self, words: list[str]) -> str: :param list words: a list of words :raises KeyError: if there is any word in `positive` or `negative` that is not in the vocabulary of the model. - :return: the word is that mostly unrelated + :return: the word that is mostly unrelated :rtype: str :Note: @@ -248,11 +248,11 @@ def similarity(self, word1: str, word2: str) -> float: return self.model.similarity(word1, word2) def sentence_vectorizer(self, text: str, use_mean: bool = True) -> ndarray: - """This function converts a Thai sentence into vector. - Specifically, it first tokenizes that text and map each tokenized word + """Converts a Thai sentence into a vector. + Specifically, it first tokenizes that text and maps each tokenized word with the word vectors from the model. - Then, word vectors are aggregated into one vector of 300 dimension - by calculating either mean or summation of all word vectors. + Then, word vectors are aggregated into one vector of 300 dimensions + by calculating either the mean or summation of all word vectors. :param str text: text input :param bool use_mean: if `True` aggregate word vectors with mean of all diff --git a/pythainlp/wsd/core.py b/pythainlp/wsd/core.py index a53b39355..68d132790 100644 --- a/pythainlp/wsd/core.py +++ b/pythainlp/wsd/core.py @@ -56,7 +56,7 @@ def get_sense( custom_tokenizer: Tokenizer = _word_cut, ) -> list[tuple[str, float]]: """Get word sense from the sentence. - This function will get definition and distance from context in sentence. + Gets definition and distance from context in sentence. :param str sentence: Thai sentence :param str word: Thai word diff --git a/release.md b/release.md index a17693527..b1682a410 100644 --- a/release.md +++ b/release.md @@ -11,8 +11,8 @@ SPDX-License-Identifier: CC0-1.0 1. Update `CHANGELOG.md` with a short summary of important changes since the previous stable release. For example, deprecation or termination of support. -3. This project follows [semantic versioning][semver]. - Ensure the version and release date fields (if any) in these files +2. This project follows [semantic versioning][semver]. + Ensure the version and release date fields in these files have been updated to the version of the new planned release: - `codemeta.json` - `pyproject.toml` @@ -29,14 +29,14 @@ SPDX-License-Identifier: CC0-1.0 For instance, the title could be `v5.0.1`. 7. Add a short summary of important changes since the previous stable release. - _This should be similar to what have been logged in `CHANGELOG.md`._ + _This should be similar to what has been logged in `CHANGELOG.md`._ Then click the "Generate release notes" button. -9. You can optionally include any particular thank-you's to contributors or +8. You can optionally include any particular thank-yous to contributors or reviewers in a note at the bottom of the release. -10. You can then click "Publish release" button. -11. If [the CI][ci] run is [successful][actions], +9. You can then click "Publish release" button. +10. If [the CI][ci] run is [successful][actions], then the release will be published on both - the GitHub release page and also the [Python Package Index][pypi]. + the GitHub release page and the [Python Package Index][pypi]. [semver]: https://semver.org/ [releases]: https://github.com/PyThaiNLP/pythainlp/releases diff --git a/tests/README.md b/tests/README.md index 5ea8dc72b..1f4d0a97b 100644 --- a/tests/README.md +++ b/tests/README.md @@ -38,7 +38,7 @@ for real-world usage: - Run `unittest tests.extra` - Explore functionalities that rely on optional dependencies specified in the - `extras` section of `setup.py`. + `project.optional-dependencies` section of `pyproject.toml`. - These dependencies might include libraries like `gensim`, `tltk`, or `torch`. - Due to dependency complexities, these functionalities are not part of the automated test suite and will not be tested in the CI/CD pipeline. diff --git a/tests/compact/__init__.py b/tests/compact/__init__.py index b5ba439d0..9e884d2c4 100644 --- a/tests/compact/__init__.py +++ b/tests/compact/__init__.py @@ -3,7 +3,7 @@ # SPDX-License-Identifier: Apache-2.0 """Unit test. Compact version. -Test functions that require "compact" dependencies (see setup.py). +Test functions that require "compact" dependencies (see pyproject.toml). """ from unittest import TestLoader, TestSuite diff --git a/tests/extra/__init__.py b/tests/extra/__init__.py index 51e5bf006..f9f519fce 100644 --- a/tests/extra/__init__.py +++ b/tests/extra/__init__.py @@ -3,7 +3,7 @@ # SPDX-License-Identifier: Apache-2.0 """Unit test. Extra version. -Test functions that require dependencies beyond "compact" (see setup.py). +Test functions that require dependencies beyond "compact" (see pyproject.toml). """ from unittest import TestLoader, TestSuite diff --git a/tokenization-benchmark.md b/tokenization-benchmark.md index 4c7d79320..9ee486da2 100644 --- a/tokenization-benchmark.md +++ b/tokenization-benchmark.md @@ -1,8 +1,10 @@ -# Word Tokenization Benchmark for Thai (obsolete) +# Word Tokenization Benchmark for Thai + +**Note: This benchmark framework is obsolete and no longer actively maintained.** A framework for benchmarking tokenization algorithms for Thai. -It has a command-line interface that allows users to conveniently execute the benchmarks -as well as a module interface for later use in their development pipelines. +It provides a command-line interface that allows users to conveniently execute the benchmarks, +as well as a module interface for use in development pipelines. ## Metrics @@ -46,13 +48,17 @@ as well as a module interface for later use in their development pipelines. [res-BEST-val-Sertis-BiGRU]: https://pythainlp.org/tokenization-benchmark-visualization/?experiment-name=BEST-val-Sertis-BiGRU [res-WiseSight160-Sertis-BiGRU]: https://pythainlp.org/tokenization-benchmark-visualization/?experiment-name=WiseSight160-Sertis-BiGRU -## Installation (WIP) +## Installation + +**Note: Installation instructions are a work in progress.** ```shell -pip ... +pip install "pythainlp[benchmarks]" ``` -## Usages (to be updated) +## Usage + +**Note: Usage instructions will be updated.** 1. Command-line Interface