|
| 1 | +# Word Tokenization Benchmark for Thai |
| 2 | + |
| 3 | +**Note: This benchmark framework is obsolete and no longer actively maintained.** |
| 4 | + |
| 5 | +A framework for benchmarking tokenization algorithms for Thai. |
| 6 | +It provides a command-line interface that allows users to conveniently execute the benchmarks, |
| 7 | +as well as a module interface for use in development pipelines. |
| 8 | + |
| 9 | +## Metrics |
| 10 | + |
| 11 | +<div align="center"> |
| 12 | + <img src="https://i.imgur.com/jVBOLa2.png"/> |
| 13 | +</div> |
| 14 | + |
| 15 | +### Character-Level (CL) |
| 16 | + |
| 17 | +- True Positive (TP): no. of starting characters that are correctly predicted. |
| 18 | +- True Negative (TN): no. of non-starting characters that are correctly predicted. |
| 19 | +- False Positive (FP): no. of non-starting characters that are wrongly predicted as starting characters. |
| 20 | +- False Negative (FN): no. of starting characters that are wrongly predicted as non-starting characters. |
| 21 | +- Precision: TP / (TP + FP) |
| 22 | +- Recall: TP / (TP+FN) |
| 23 | +- f1: ... |
| 24 | + |
| 25 | +### Word-Level (WL) |
| 26 | + |
| 27 | +- Correctly Tokenized Words (CTW): no. of words in reference that are correctly tokenized. |
| 28 | +- Precision: CTW / no. words in reference solution |
| 29 | +- Recall: CTW / no. words in sample |
| 30 | +- f1: ... |
| 31 | + |
| 32 | +## Benchmark Results |
| 33 | + |
| 34 | +| Vendor | Approach | Datasets | |
| 35 | +|---|---|---| |
| 36 | +| DeepCut | CNN | [-yellow.svg)][res-BEST-val-DeepCut] [-yellow.svg)][res-THNC-DeepCut] [-yellow.svg)][res-Orchid-DeepCut] [-yellow.svg)][res-WiseSight160-DeepCut] | |
| 37 | +| PyThaiNLP-newmm | dictionary-based | [-yellow.svg)][res-BEST-val-PyThaiNLP-newmm] [-yellow.svg)][res-THNC-PyThaiNLP-newmm] [-yellow.svg)][res-Orchid-PyThaiNLP-newmm] [-yellow.svg)][res-WiseSight160-PyThaiNLP-newmm] | |
| 38 | +| Sertis-BiGRU | Bi-directional RNN | [-yellow.svg)][res-BEST-val-Sertis-BiGRU] [-yellow.svg)][res-WiseSight160-Sertis-BiGRU] | |
| 39 | + |
| 40 | +[res-BEST-val-DeepCut]: https://pythainlp.org/tokenization-benchmark-visualization/?experiment-name=BEST-val-DeepCut |
| 41 | +[res-THNC-DeepCut]: https://pythainlp.org/tokenization-benchmark-visualization/?experiment-name=THNC-DeepCut |
| 42 | +[res-Orchid-DeepCut]: https://pythainlp.org/tokenization-benchmark-visualization/?experiment-name=Orchid-DeepCut |
| 43 | +[res-WiseSight160-DeepCut]: https://pythainlp.org/tokenization-benchmark-visualization/?experiment-name=WiseSight160-DeepCut |
| 44 | +[res-BEST-val-PyThaiNLP-newmm]: https://pythainlp.org/tokenization-benchmark-visualization/?experiment-name=BEST-val-PyThaiNLP-newmm |
| 45 | +[res-THNC-PyThaiNLP-newmm]: https://pythainlp.org/tokenization-benchmark-visualization/?experiment-name=THNC-PyThaiNLP-newmm |
| 46 | +[res-Orchid-PyThaiNLP-newmm]: https://pythainlp.org/tokenization-benchmark-visualization/?experiment-name=Orchid-PyThaiNLP-newmm |
| 47 | +[res-WiseSight160-PyThaiNLP-newmm]: https://pythainlp.org/tokenization-benchmark-visualization/?experiment-name=WiseSight160-PyThaiNLP-newmm |
| 48 | +[res-BEST-val-Sertis-BiGRU]: https://pythainlp.org/tokenization-benchmark-visualization/?experiment-name=BEST-val-Sertis-BiGRU |
| 49 | +[res-WiseSight160-Sertis-BiGRU]: https://pythainlp.org/tokenization-benchmark-visualization/?experiment-name=WiseSight160-Sertis-BiGRU |
| 50 | + |
| 51 | +## Installation |
| 52 | + |
| 53 | +**Note: Installation instructions are a work in progress.** |
| 54 | + |
| 55 | +```shell |
| 56 | +pip install "pythainlp[benchmarks]" |
| 57 | +``` |
| 58 | + |
| 59 | +## Usage |
| 60 | + |
| 61 | +**Note: Usage instructions will be updated.** |
| 62 | + |
| 63 | +1. Command-line Interface |
| 64 | + |
| 65 | + ```shell |
| 66 | + PYTHONPATH=`pwd` python scripts/thai-tokenisation-benchmark.py \ |
| 67 | + --test-file ./data/best-2010/TEST_100K_ANS.txt \ |
| 68 | + --input ./data/best-2010-syllable.txt |
| 69 | + ``` |
| 70 | + |
| 71 | + Sample output: |
| 72 | + |
| 73 | + ```text |
| 74 | + Benchmarking ./data/best-2010-deepcut.txt against ./data/best-2010/TEST_100K_ANS.txt with 2252 samples in total |
| 75 | + ============== Benchmark Result ============== |
| 76 | + metric mean±std min max |
| 77 | + char_level:tp 47.82±47.22 1.000000 354.0 |
| 78 | + char_level:tn 144.19±145.97 1.000000 887.0 |
| 79 | + char_level:fp 1.34±2.02 0.000000 23.0 |
| 80 | + char_level:fn 0.70±1.19 0.000000 14.0 |
| 81 | + char_level:precision 0.96±0.08 0.250000 1.0 |
| 82 | + char_level:recall 0.98±0.04 0.500000 1.0 |
| 83 | + char_level:f1 0.97±0.06 0.333333 1.0 |
| 84 | + word_level:precision 0.92±0.14 0.000000 1.0 |
| 85 | + word_level:recall 0.93±0.12 0.000000 1.0 |
| 86 | + word_level:f1 0.93±0.13 0.000000 1.0 |
| 87 | + ``` |
| 88 | + |
| 89 | +2. Module Interface |
| 90 | + |
| 91 | + ```python |
| 92 | + from pythainlp.benchmarks import word_tokenisation as bwt |
| 93 | +
|
| 94 | + ref_samples = array of reference tokenised samples |
| 95 | + tokenised_samples = array of tokenised samples, aka. from your algorithm |
| 96 | +
|
| 97 | + # dataframe contains metrics for each sample |
| 98 | + df = bwt.benchmark(ref_samples, tokenised_samples) |
| 99 | + ``` |
| 100 | + |
| 101 | +## Related Work |
| 102 | + |
| 103 | +- [Thai Tokenizers Docker][docker]: collection of Docker containers of pre-built Thai tokenizers. |
| 104 | + |
| 105 | +## Development |
| 106 | + |
| 107 | +Unit tests |
| 108 | + |
| 109 | +```shell |
| 110 | +TEST_VERBOSE=1 PYTHONPATH=. python tests/__init__.py |
| 111 | +``` |
| 112 | + |
| 113 | +## Acknowledgement |
| 114 | + |
| 115 | +This project was initially started by [Pattarawat Chormai][pat], while he was interning at [Dr. Attapol Thamrongrattanarit][ate]'s lab. |
| 116 | +
|
| 117 | +[docker]: https://github.com/PyThaiNLP/docker-thai-tokenizers |
| 118 | +[ate]: https://attapol.github.io |
| 119 | +[pat]: https://pat.chormai.org |
0 commit comments