You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/api/soundex.rst
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -90,7 +90,7 @@ References
90
90
91
91
.. [#metasound] Snae & Brückner. (2009). `Novel Phonetic Name Matching Algorithm with a Statistical Ontology for Analyzing Names Given in Accordance with Thai Astrology <https://pdfs.semanticscholar.org/3983/963e87ddc6dfdbb291099aa3927a0e3e4ea6.pdf>`_.
Copy file name to clipboardExpand all lines: docs/api/tokenize.rst
+39-40Lines changed: 39 additions & 40 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,40 +8,39 @@ The :mod:`pythainlp.tokenize` module contains a comprehensive set of functions a
8
8
Modules
9
9
-------
10
10
11
-
12
11
.. autofunction:: sent_tokenize
13
12
:noindex:
14
-
13
+
15
14
Splits Thai text into sentences. This function identifies sentence boundaries, which is essential for text segmentation and analysis.
16
15
17
16
.. autofunction:: paragraph_tokenize
18
17
:noindex:
19
-
18
+
20
19
Segments text into paragraphs, which can be valuable for document-level analysis or summarization.
21
20
22
21
.. autofunction:: subword_tokenize
23
22
:noindex:
24
-
23
+
25
24
Tokenizes text into subwords, which can be helpful for various NLP tasks, including subword embeddings.
26
25
27
26
.. autofunction:: syllable_tokenize
28
27
:noindex:
29
-
28
+
30
29
Divides text into syllables, allowing you to work with individual Thai language phonetic units.
31
30
32
31
.. autofunction:: word_tokenize
33
32
:noindex:
34
-
33
+
35
34
Splits text into words. This function is a fundamental tool for Thai language text analysis.
36
35
37
36
.. autofunction:: word_detokenize
38
37
:noindex:
39
-
38
+
40
39
Reverses the tokenization process, reconstructing text from tokenized units. Useful for text generation tasks.
41
40
42
41
.. autoclass:: Tokenizer
43
42
:members:
44
-
43
+
45
44
The `Tokenizer` class is a versatile tool for customizing tokenization processes and managing tokenization models. It provides various methods and attributes to fine-tune tokenization according to your specific needs.
46
45
47
46
.. autoclass:: display_cell_tokenize
@@ -55,119 +54,119 @@ Sentence level
55
54
--------------
56
55
57
56
**crfcut**
58
-
57
+
59
58
.. automodule:: pythainlp.tokenize.crfcut
60
59
:members:
61
-
60
+
62
61
A tokenizer that operates at the sentence level using Conditional Random Fields (CRF). It is suitable for segmenting text into sentences accurately.
63
62
64
63
**thaisumcut**
65
-
64
+
66
65
.. automodule:: pythainlp.tokenize.thaisumcut
67
66
:members:
68
-
67
+
69
68
A sentence tokenizer based on a maximum entropy model. It's a great choice for sentence boundary detection in Thai text.
70
69
71
70
Word level
72
71
----------
73
72
74
73
**attacut**
75
-
74
+
76
75
.. automodule:: pythainlp.tokenize.attacut
77
76
:members:
78
-
77
+
79
78
A tokenizer designed for word-level segmentation. It provides accurate word boundary detection in Thai text.
80
79
81
80
**deepcut**
82
-
81
+
83
82
.. automodule:: pythainlp.tokenize.deepcut
84
83
:members:
85
-
84
+
86
85
Utilizes deep learning techniques for word segmentation, achieving high accuracy and performance.
87
86
88
87
**multi_cut**
89
-
88
+
90
89
.. automodule:: pythainlp.tokenize.multi_cut
91
90
:members:
92
-
91
+
93
92
An ensemble tokenizer that combines multiple tokenization strategies for improved word segmentation.
94
93
95
94
**nlpo3**
96
-
95
+
97
96
.. automodule:: pythainlp.tokenize.nlpo3
98
97
:members:
99
-
98
+
100
99
A word tokenizer based on the NLPO3 model. It offers advanced word boundary detection and is suitable for various NLP tasks.
101
100
102
101
**longest**
103
-
102
+
104
103
.. automodule:: pythainlp.tokenize.longest
105
104
:members:
106
-
105
+
107
106
A tokenizer that identifies word boundaries by selecting the longest possible words in a text.
108
107
109
108
**pyicu**
110
-
109
+
111
110
.. automodule:: pythainlp.tokenize.pyicu
112
111
:members:
113
-
112
+
114
113
An ICU-based word tokenizer offering robust support for Thai text segmentation.
115
114
116
115
**nercut**
117
-
116
+
118
117
.. automodule:: pythainlp.tokenize.nercut
119
118
:members:
120
-
119
+
121
120
A tokenizer optimized for Named Entity Recognition (NER) tasks, ensuring accurate tokenization for entity recognition.
122
121
123
122
**sefr_cut**
124
-
123
+
125
124
.. automodule:: pythainlp.tokenize.sefr_cut
126
125
:members:
127
-
126
+
128
127
An advanced word tokenizer for segmenting Thai text, with a focus on precision.
129
128
130
129
**oskut**
131
-
130
+
132
131
.. automodule:: pythainlp.tokenize.oskut
133
132
:members:
134
-
133
+
135
134
A tokenizer that uses a pre-trained model for word segmentation. It's a reliable choice for general-purpose text analysis.
136
135
137
136
**newmm (Default)**
138
-
137
+
139
138
.. automodule:: pythainlp.tokenize.newmm
140
139
:members:
141
-
140
+
142
141
The default word tokenization engine that provides a balance between accuracy and efficiency for most use cases.
143
142
144
143
Subword level
145
144
-------------
146
145
147
146
**tcc**
148
-
147
+
149
148
.. automodule:: pythainlp.tokenize.tcc
150
149
:members:
151
-
150
+
152
151
Tokenizes text into Thai Character Clusters (TCCs), a subword level representation.
153
152
154
153
**tcc+**
155
-
154
+
156
155
.. automodule:: pythainlp.tokenize.tcc_p
157
156
:members:
158
-
157
+
159
158
A subword tokenizer that includes additional rules for more precise subword segmentation.
160
159
161
160
**etcc**
162
-
161
+
163
162
.. automodule:: pythainlp.tokenize.etcc
164
163
:members:
165
-
164
+
166
165
Enhanced Thai Character Clusters (eTCC) tokenizer for subword-level analysis.
167
166
168
167
**han_solo**
169
-
168
+
170
169
.. automodule:: pythainlp.tokenize.han_solo
171
170
:members:
172
-
171
+
173
172
A subword tokenizer specialized for Han characters and mixed scripts, suitable for various text processing scenarios.
Copy file name to clipboardExpand all lines: docs/api/tools.rst
+4-4Lines changed: 4 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,25 +9,25 @@ Modules
9
9
10
10
.. autofunction:: get_full_data_path
11
11
:noindex:
12
-
12
+
13
13
Retrieves the full path to the PyThaiNLP data directory. This function is essential for internal data management, enabling PyThaiNLP to locate resources efficiently.
14
14
15
15
.. autofunction:: get_pythainlp_data_path
16
16
:noindex:
17
-
17
+
18
18
Obtains the path to the PyThaiNLP data directory. This function is useful for accessing the library's data resources for internal processes.
19
19
20
20
.. autofunction:: get_pythainlp_path
21
21
:noindex:
22
-
22
+
23
23
Returns the path to the PyThaiNLP library directory. This function is vital for PyThaiNLP's internal operations and library management.
This module appears to be related to handling misspellings within PyThaiNLP. While not explicitly documented here, it likely provides functionality for identifying and correcting misspelled words, which can be crucial for text preprocessing and language processing tasks.
32
32
33
33
The `pythainlp.tools` module contains these functions, which are mainly intended for PyThaiNLP's internal workings. While they may not be directly utilized by external users, they play a pivotal role in ensuring the smooth operation of the library. Understanding the purpose of these functions can be valuable for contributors and developers working on PyThaiNLP, as it sheds light on the internal mechanisms and data management within the library.
Copy file name to clipboardExpand all lines: docs/api/translate.rst
+7-7Lines changed: 7 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,40 +9,40 @@ Modules
9
9
10
10
.. autoclass:: Translate
11
11
:members:
12
-
12
+
13
13
The `Translate` class is the central component of the module, offering a unified interface for various translation tasks. It acts as a coordinator, directing translation requests to specific language pairs and models.
This function facilitates the download of all available English to Thai translation models. It ensures that the required models are accessible for translation tasks, enhancing the usability of the module.
The `EnThTranslator` class specializes in translating text from English to Thai. It offers a range of methods for translating sentences and text, enabling accurate and meaningful translations between these languages.
Conversely, the `ThEnTranslator` class focuses on translating text from Thai to English. It provides functionality for translating Thai text into English, contributing to effective language understanding and communication.
The `ThZhTranslator` class specializes in translating text from Thai to Chinese (Simplified). This class is valuable for bridging language gaps between these two languages, promoting cross-cultural communication.
The `ZhThTranslator` class is designed for translating text from Chinese (Simplified) to Thai. It assists in making content accessible to Thai-speaking audiences by converting Chinese text into Thai.
Lastly, the `ThFrTranslator` class specializes in translating text from Thai to French. It serves as a tool for expanding language accessibility and promoting content sharing in French-speaking communities.
47
47
48
48
.. The `pythainlp.translate` module extends the language processing capabilities of PyThaiNLP, offering machine translation functionality for various language pairs. Whether you need to translate text between English and Thai, Thai and Chinese, or Thai and French, this module provides the necessary tools and classes to facilitate seamless language conversion. The `Translate` class acts as the central coordinator, while language-specific classes ensure accurate and meaningful translations for diverse linguistic scenarios.
The `WunsenTransliterate` class represents a transliteration engine known as "Wunsen." It offers specific transliteration methods for rendering Thai text into a phonetic English format.
The `thai2rom` engine specializes in transliterating Thai text into romanized form. It's particularly useful for rendering Thai words accurately in an English phonetic format.
The `royin` engine focuses on transliterating Thai text into English characters. It provides an alternative approach to transliteration, ensuring accurate representation of Thai words.
Copy file name to clipboardExpand all lines: docs/api/ulmfit.rst
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,7 +9,7 @@ Modules
9
9
10
10
.. autoclass:: ThaiTokenizer
11
11
:members:
12
-
12
+
13
13
The `ThaiTokenizer` class is a critical component of ULMFiT, designed for tokenizing Thai text effectively. Tokenization is the process of breaking down text into individual tokens, and this class allows you to do so with precision and accuracy.
0 commit comments