Commit 130df1e
Add TextSpitter 2.0 Rust/PyO3 core with full test suite
Introduces a Maturin-based Rust extension (_core.pyd) that accelerates
encoding detection, text normalization, token counting, and token-aware
chunking, while keeping a pure-Python fallback path (_fallback.py) for
environments where the compiled extension is unavailable.
Rust crates: pyo3 0.21 (Bound API), chardetng, tiktoken-rs 0.5, rayon,
unicode-normalization, regex. All batch methods release the GIL via
py.allow_threads so CPython threads stay unblocked during heavy work.
Python layer changes:
- __init__.py: try-import _core, fall back to _fallback, export
_RUST_AVAILABLE flag alongside Chunk, TextChunker, TextNormalizer,
TokenCounter, detect_encoding
- core.py: uses detect_encoding() instead of the old four-attempt loop
- _fallback.py: tiktoken-backed fallback with model validation at
construction time; word-based middle-truncation when tiktoken absent
- pyproject.toml: switched build-backend to maturin; added tiktoken to
dev deps; added [tool.maturin] config; expanded .gitignore for Rust
Tests added (239 total, all green):
- test_detect_encoding.py -- both Rust and fallback paths
- test_normalizer.py -- Unicode forms, whitespace, OCR, headers
- test_token_counter.py -- count, batch, truncate, parallel stress
- test_chunker.py -- field correctness, max_tokens, tables,
section titles, batch, parallel stress
- test_rust_integration.py -- end-to-end pipeline and interface parity
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>1 parent e8e6db1 commit 130df1e
20 files changed
Lines changed: 2671 additions & 47 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | | - | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
0 commit comments