|
| 1 | +# Contributing to TinyBPE |
| 2 | + |
| 3 | +Thank you for your interest in contributing! TinyBPE is a CPython extension implementing a fast, lightweight BPE tokenizer and trainer. |
| 4 | + |
| 5 | +## Development Setup |
| 6 | + |
| 7 | +### Prerequisites |
| 8 | + |
| 9 | +- Python 3.9+ |
| 10 | +- A C compiler (GCC on Linux, Clang on macOS, MSVC on Windows) |
| 11 | +- Git |
| 12 | + |
| 13 | +### Setting up the development environment |
| 14 | + |
| 15 | +```bash |
| 16 | +# Clone the repository |
| 17 | +git clone https://github.com/neluca/tinybpe.git |
| 18 | +cd tinybpe |
| 19 | + |
| 20 | +# Create a virtual environment |
| 21 | +python -m venv .venv |
| 22 | +source .venv/bin/activate # Linux/macOS |
| 23 | +# .venv\Scripts\activate # Windows |
| 24 | + |
| 25 | +# Install development dependencies |
| 26 | +pip install -r requirements_dev.txt |
| 27 | + |
| 28 | +# Build the C extension in-place |
| 29 | +pip install -e . |
| 30 | + |
| 31 | +# Run tests |
| 32 | +python -m pytest |
| 33 | +``` |
| 34 | + |
| 35 | +### Building the C extension manually |
| 36 | + |
| 37 | +```bash |
| 38 | +# Build the extension module (places .pyd/.so in the source tree) |
| 39 | +python setup.py build_ext --inplace |
| 40 | + |
| 41 | +# Or use pip for an editable install |
| 42 | +pip install -e . |
| 43 | +``` |
| 44 | + |
| 45 | +## Coding Conventions |
| 46 | + |
| 47 | +### C code (`src/`) |
| 48 | +- **Standard**: C99 |
| 49 | +- **Style**: Consistent with existing code — 4-space indentation, snake_case for function and variable names |
| 50 | +- **Naming**: Use descriptive names. Struct fields use `snake_case`. Functions are prefixed with `bpe_`. |
| 51 | +- **Memory**: Use `bpe_malloc` / `bpe_free` wrappers which integrate with Python's memory manager. Always check for NULL returns. |
| 52 | +- **Error handling**: Set Python exceptions via `PyErr_SetString` / `PyErr_NoMemory` and return NULL or -1. |
| 53 | + |
| 54 | +### Python code (`tinybpe/`) |
| 55 | +- **Standard**: Python 3.9+ compatible |
| 56 | +- **Style**: Follow [PEP 8](https://peps.python.org/pep-0008/) — enforced by ruff |
| 57 | +- **Type hints**: All public methods must have type annotations |
| 58 | +- **Docstrings**: NumPy-style docstrings required for all public classes and methods |
| 59 | + |
| 60 | +### Pre-commit hooks |
| 61 | + |
| 62 | +```bash |
| 63 | +# Install pre-commit hooks (run once) |
| 64 | +pre-commit install |
| 65 | + |
| 66 | +# Run all hooks manually |
| 67 | +pre-commit run --all-files |
| 68 | +``` |
| 69 | + |
| 70 | +## Running Tests |
| 71 | + |
| 72 | +```bash |
| 73 | +# Run all tests |
| 74 | +python -m pytest |
| 75 | + |
| 76 | +# Run with coverage |
| 77 | +python -m pytest --cov=tinybpe --cov-report=term |
| 78 | + |
| 79 | +# Run a specific test file |
| 80 | +python -m pytest tests/test_tinybpe.py |
| 81 | +``` |
| 82 | + |
| 83 | +## Pull Request Process |
| 84 | + |
| 85 | +1. Fork the repository and create a feature branch from `main`. |
| 86 | +2. Make your changes, following the coding conventions above. |
| 87 | +3. Add or update tests as needed. |
| 88 | +4. Run `pre-commit run --all-files` to ensure style compliance. |
| 89 | +5. Run `python -m pytest` to verify all tests pass. |
| 90 | +6. Update `CHANGELOG.md` under the `[Unreleased]` section. |
| 91 | +7. Submit a pull request with a clear description of the changes. |
| 92 | + |
| 93 | +## Project Structure |
| 94 | + |
| 95 | +``` |
| 96 | +tinybpe/ |
| 97 | +├── src/ # C extension source |
| 98 | +│ ├── _tree_core.h/c # AVL tree (self-balancing binary search tree) |
| 99 | +│ ├── bpe_common.h/c # Common types (pairs, pieces) and utilities |
| 100 | +│ ├── bpe_tokenizer.h/c # Encoder/decoder implementation |
| 101 | +│ ├── bpe_trainer.h/c # BPE training algorithm |
| 102 | +│ └── bpe_module.c # CPython type definitions and method wrappers |
| 103 | +├── tinybpe/ # Python package |
| 104 | +│ ├── __init__.py # Public API exports |
| 105 | +│ ├── _version.py # Single source of truth for version |
| 106 | +│ ├── _model_io.py # Model file save/load |
| 107 | +│ ├── _tiktoken.py # tiktoken model conversion |
| 108 | +│ ├── core.py # CommonTokenizer and Tokenizer wrappers |
| 109 | +│ ├── simple.py # SimpleTrainer wrapper |
| 110 | +│ └── bpe.pyi # Type stubs for the C extension |
| 111 | +├── tests/ # Test suite |
| 112 | +├── examples/ # Example scripts |
| 113 | +├── benchmarks/ # Performance benchmarks |
| 114 | +└── .github/workflows/ # CI/CD pipelines |
| 115 | +``` |
| 116 | + |
| 117 | +## License |
| 118 | + |
| 119 | +By contributing, you agree that your contributions will be licensed under the MIT License. |
0 commit comments