Skip to content

Commit 130df1e

Browse files
fsecada01claude
andcommitted
Add TextSpitter 2.0 Rust/PyO3 core with full test suite
Introduces a Maturin-based Rust extension (_core.pyd) that accelerates encoding detection, text normalization, token counting, and token-aware chunking, while keeping a pure-Python fallback path (_fallback.py) for environments where the compiled extension is unavailable. Rust crates: pyo3 0.21 (Bound API), chardetng, tiktoken-rs 0.5, rayon, unicode-normalization, regex. All batch methods release the GIL via py.allow_threads so CPython threads stay unblocked during heavy work. Python layer changes: - __init__.py: try-import _core, fall back to _fallback, export _RUST_AVAILABLE flag alongside Chunk, TextChunker, TextNormalizer, TokenCounter, detect_encoding - core.py: uses detect_encoding() instead of the old four-attempt loop - _fallback.py: tiktoken-backed fallback with model validation at construction time; word-based middle-truncation when tiktoken absent - pyproject.toml: switched build-backend to maturin; added tiktoken to dev deps; added [tool.maturin] config; expanded .gitignore for Rust Tests added (239 total, all green): - test_detect_encoding.py -- both Rust and fallback paths - test_normalizer.py -- Unicode forms, whitespace, OCR, headers - test_token_counter.py -- count, batch, truncate, parallel stress - test_chunker.py -- field correctness, max_tokens, tables, section titles, batch, parallel stress - test_rust_integration.py -- end-to-end pipeline and interface parity Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent e8e6db1 commit 130df1e

20 files changed

Lines changed: 2671 additions & 47 deletions

.gitignore

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,4 +7,15 @@ labs
77
.venv
88
*.7z
99
*.env
10-
*.egg*/
10+
*.egg*/
11+
12+
# Rust / Maturin build artifacts
13+
target/
14+
*.pyd
15+
*.pdb
16+
*.so
17+
18+
# IDE / tool state
19+
.idea/
20+
.serena/
21+
claudedocs/

0 commit comments

Comments
 (0)