|
1 | | -# Botok – Python Tibetan Tokenizer |
| 1 | +<h1 align="center"> |
| 2 | + <br> |
| 3 | + <a href="https://buddhistai.tools/"><img src="https://raw.githubusercontent.com/WeBuddhist/visual-assets/refs/heads/main/logo/WB-logo-purple.png" alt="OpenPecha" width="150"></a> |
| 4 | + <br> |
| 5 | +</h1> |
2 | 6 |
|
3 | | -[](https://github.com/OpenPecha/botok/releases) |
4 | | -[](https://botok.readthedocs.io/en/latest/?badge=latest) |
5 | | -[](https://coveralls.io/github/OpenPecha/botok?branch=master) |
6 | | -[](https://black.readthedocs.io/en/stable/) |
7 | | -[](https://github.com/OpenPecha/botok/blob/master/LICENSE) |
8 | | -[](https://github.com/OpenPecha/Botok) |
| 7 | +<h1 align="center">Botok</h1> |
9 | 8 |
|
10 | | -## Project Description |
| 9 | +<p align="center"> |
| 10 | + |Apache-2.0| |Python| |NLP| |
| 11 | +</p> |
11 | 12 |
|
12 | | -Botok is a powerful Python library for tokenizing Tibetan text. It segments text into words with high accuracy and provides optional attributes such as lemma, part-of-speech (POS) tags, and clean forms. The library supports various text formats, custom dialects, and multiple tokenization modes, making it a versatile tool for Tibetan Natural Language Processing (NLP). |
| 13 | +🏷 བོད་ཏོག [pʰøtɔk̚] Tibetan word tokenizer in Python |
13 | 14 |
|
14 | | -## Who This Project Is For |
| 15 | +## Table of Contents |
15 | 16 |
|
16 | | -This project is intended for developers and researchers working with Tibetan language text processing who need accurate word segmentation and tokenization. |
| 17 | +- [Features](#features) |
| 18 | +- [Prerequisites](#prerequisites) |
| 19 | +- [Installation](#installation) |
| 20 | +- [Configuration](#configuration) |
| 21 | +- [Usage](#usage) |
| 22 | +- [Development](#development) |
| 23 | +- [Testing](#testing) |
| 24 | +- [Contributing](#contributing) |
| 25 | +- [How to get help](#how-to-get-help) |
| 26 | +- [Terms of use](#terms-of-use) |
17 | 27 |
|
18 | | -## Project Dependencies |
| 28 | +## Features |
19 | 29 |
|
20 | | -Before using Botok, ensure you have: |
| 30 | +- Tibetan word tokenization using statistical and rule-based methods |
| 31 | +- Support for both classical and modern Tibetan |
| 32 | +- Integration with PyBo (Python Buddhist) |
| 33 | +- Customizable segmentation rules |
| 34 | +-字典-based word lookup |
21 | 35 |
|
22 | | -* Python 3.7+ |
23 | | -* pip (Python package installer) |
| 36 | +## Prerequisites |
24 | 37 |
|
25 | | -## Instructions for Installing Botok |
| 38 | +- Python 3.8+ |
| 39 | +- pip |
26 | 40 |
|
27 | | -### Install Botok |
| 41 | +## Installation |
28 | 42 |
|
29 | | -1. **Install via pip:** |
| 43 | +```bash |
| 44 | +# Clone the repository |
| 45 | +git clone https://github.com/OpenPecha/Botok.git |
| 46 | +cd Botok |
30 | 47 |
|
31 | | - ```bash |
32 | | - pip install botok |
33 | | - ``` |
| 48 | +# Install dependencies |
| 49 | +pip install -r requirements.txt |
34 | 50 |
|
35 | | -2. **Install from source:** |
| 51 | +# Install the package |
| 52 | +pip install -e . |
| 53 | +``` |
| 54 | + |
| 55 | +## Configuration |
36 | 56 |
|
37 | | - ```bash |
38 | | - git clone https://github.com/OpenPecha/Botok.git |
39 | | - cd Botok |
40 | | - pip install -e . |
41 | | - ``` |
| 57 | +Botok can be configured via: |
| 58 | +- Environment variables |
| 59 | +- YAML configuration files in `config/` |
| 60 | +- Python API |
42 | 61 |
|
43 | | -### Run Botok |
| 62 | +## Usage |
44 | 63 |
|
45 | 64 | ```python |
46 | | -from botok import Botok |
| 65 | +import botok |
47 | 66 |
|
48 | | -botok = Botok() |
49 | | -text = "བོད་ཡིག" |
50 | | -tokens = botok.tokenize(text) |
| 67 | +# Create a tokenizer instance |
| 68 | +tokenizer = botok.Tok() |
| 69 | + |
| 70 | +# Tokenize Tibetan text |
| 71 | +text = "བོད་ཡིག་གི་དཔེ་ཆ་" |
| 72 | +tokens = tokenizer.tokenize(text) |
51 | 73 | print(tokens) |
52 | 74 | ``` |
53 | 75 |
|
54 | | -## Directory Structure |
| 76 | +## Development |
55 | 77 |
|
56 | | -``` |
57 | | -Botok/ |
58 | | -├── .github/ # GitHub configuration |
59 | | -├── botok/ # Main package source code |
60 | | -├── docs/ # Documentation |
61 | | -├── tests/ # Test suite |
62 | | -├── requirements.txt # Development dependencies |
63 | | -├── setup.py # Package setup configuration |
64 | | -├── setup.cfg # Package metadata |
65 | | -├── README.md # Project documentation |
66 | | -├── CHANGELOG.md # Release history |
67 | | -├── LICENSE # License file |
68 | | -└── usage.py # Usage examples |
69 | | -``` |
| 78 | +```bash |
| 79 | +# Install dev dependencies |
| 80 | +pip install -e .[dev] |
| 81 | + |
| 82 | +# Run tests |
| 83 | +pytest |
70 | 84 |
|
71 | | -## Contributing Guidelines |
| 85 | +# Lint |
| 86 | +flake8 botok/ |
| 87 | +``` |
72 | 88 |
|
73 | | -Contributions are welcome! Please read the [contributing guidelines](https://github.com/OpenPecha/Botok/blob/master/README.md#contributing) for details on how to submit pull requests. |
| 89 | +## Testing |
74 | 90 |
|
75 | | -## Additional Documentation |
| 91 | +```bash |
| 92 | +pytest tests/ |
| 93 | +``` |
76 | 94 |
|
77 | | -* [Full Documentation](https://botok.readthedocs.io/) |
78 | | -* [API Reference](https://botok.readthedocs.io/en/latest/) |
| 95 | +## Contributing |
79 | 96 |
|
80 | | -## How to Get Help |
| 97 | +1. Fork the repository |
| 98 | +2. Create a feature branch |
| 99 | +3. Commit your changes |
| 100 | +4. Push to the branch |
| 101 | +5. Open a Pull Request |
81 | 102 |
|
82 | | -* [GitHub Issues](https://github.com/OpenPecha/Botok/issues) |
83 | | -* [OpenPecha Community](https://openpecha.org) |
| 103 | +Please read [CONTRIBUTING.md](https://github.com/OpenPecha/.github/blob/main/CONTRIBUTING.md) for details. |
84 | 104 |
|
85 | | -## Terms of Use |
| 105 | +## How to get help |
| 106 | +* File an issue. |
| 107 | +* Join our [discord](https://discord.com/invite/7GFpPFSTeA). |
86 | 108 |
|
87 | | -Botok is licensed under the MIT License. |
| 109 | +## Terms of use |
| 110 | +Botok is licensed under the [Apache-2.0 License](/LICENSE). |
0 commit comments