Remove Phi-2 tokenizer

- Remove models/phi2.tbm
- Remove Phi-2 from registry, tests, and READMEs
- 7 built-in models remaining

Author: neluca

README.md

@@ -8,15 +8,15 @@
 
 **An ultra-fast, lightweight BPE tokenizer and trainer with a pure-C core.**
 
-Ever wished you could load a GPT-4 compatible tokenizer in **one line** without network calls? TinyBPE ships 8 pre-built ByteLevel BPE models directly in the package. The CPython C core runs BPE encoding/decoding at native speed — typically **10-50× faster** than pure-Python implementations while depending only on `regex`.
+Ever wished you could load a GPT-4 compatible tokenizer in **one line** without network calls? TinyBPE ships 7 pre-built ByteLevel BPE models directly in the package. The CPython C core runs BPE encoding/decoding at native speed — typically **10-50× faster** than pure-Python implementations while depending only on `regex`.
 
 ## Why TinyBPE?
 
 | Feature | TinyBPE | tiktoken | HuggingFace tokenizers |
 |---|---|---|---|
 | **Core engine** | Pure C (CPython) | Pure Rust (PyO3) | Pure Rust (PyO3) |
 | **Dependencies** | `regex` only | `tiktoken` + Rust toolchain | `tokenizers` + Rust toolchain |
-| **Built-in models** | 8 models ship in package | Downloads on first use | Downloads on first use |
+| **Built-in models** | 7 models ship in package | Downloads on first use | Downloads on first use |
 | **Offline ready** | ✅ Fully offline | ❌ Requires download | ❌ Requires download |
 | **Model format** | Human-readable `.tbm` text | Binary blob | JSON / binary |
 | **One-liner load** | `Tokenizer.from_pretrained("cl100k_base")` | `tiktoken.get_encoding("cl100k_base")` | `AutoTokenizer.from_pretrained(...)` |
@@ -60,8 +60,8 @@ tok.decode(ids)  # → 'hello world'
 import tinybpe
 
 tinybpe.list_models()
-# ['cl100k_base', 'deepseek-v4', 'minicpm', 'o200k_base',
-#  'p50k_base', 'phi2', 'qwen35', 'r50k_base']
+# ['cl100k_base', 'deepseek-v4', 'minicpm5', 'o200k_base',
+#  'p50k_base', 'qwen35', 'r50k_base']
 ```
 
 ### Built-in Model Catalog
@@ -73,7 +73,6 @@ tinybpe.list_models()
 | `p50k_base` | GPT-3 (davinci, curie, babbage, ada) | 50,280 |
 | `r50k_base` | GPT-2 | 50,256 |
 | `qwen35` | Qwen3.5 (0.8B-35B) | 247,843 |
-| `phi2` | Microsoft Phi-2 | 50,257 |
 | `deepseek-v4` | DeepSeek-V4 Flash | 127,997 |
 | `minicpm5` | MiniCPM5-1B (ByteLevel BPE) | 130,050 |
 

README_zh.md

@@ -8,15 +8,15 @@
 
 **纯 C 内核的超轻量高性能 BPE 分词器与训练器。**
 
-只需**一行代码**即可加载 GPT-4 兼容分词器，无需联网。TinyBPE 内置 8 个 ByteLevel BPE 预训练模型，开箱即用。CPython C 扩展以原生速度执行 BPE 编码/解码——通常比纯 Python 实现**快 10–50 倍**，且仅依赖 `regex` 一个第三方库。
+只需**一行代码**即可加载 GPT-4 兼容分词器，无需联网。TinyBPE 内置 7 个 ByteLevel BPE 预训练模型，开箱即用。CPython C 扩展以原生速度执行 BPE 编码/解码——通常比纯 Python 实现**快 10–50 倍**，且仅依赖 `regex` 一个第三方库。
 
 ## 为什么选择 TinyBPE？
 
 | 特性 | TinyBPE | tiktoken | HuggingFace tokenizers |
 |---|---|---|---|
 | **核心引擎** | 纯 C（CPython 扩展） | 纯 Rust（PyO3） | 纯 Rust（PyO3） |
 | **依赖** | 仅 `regex` | `tiktoken` + Rust 工具链 | `tokenizers` + Rust 工具链 |
-| **内置模型** | 8 个模型随包分发 | 首次使用时下载 | 首次使用时下载 |
+| **内置模型** | 7 个模型随包分发 | 首次使用时下载 | 首次使用时下载 |
 | **离线可用** | ✅ 完全离线 | ❌ 需要下载 | ❌ 需要下载 |
 | **模型格式** | 可读文本 `.tbm` 文件 | 二进制 blob | JSON / 二进制 |
 | **一行加载** | `Tokenizer.from_pretrained("cl100k_base")` | `tiktoken.get_encoding("cl100k_base")` | `AutoTokenizer.from_pretrained(...)` |
@@ -60,8 +60,8 @@ tok.decode(ids)  # → 'hello world'
 import tinybpe
 
 tinybpe.list_models()
-# ['cl100k_base', 'deepseek-v4', 'minicpm', 'o200k_base',
-#  'p50k_base', 'phi2', 'qwen35', 'r50k_base']
+# ['cl100k_base', 'deepseek-v4', 'minicpm5', 'o200k_base',
+#  'p50k_base', 'qwen35', 'r50k_base']
 ```
 
 ### 内置模型目录
@@ -73,7 +73,6 @@ tinybpe.list_models()
 | `p50k_base` | GPT-3（davinci、curie、babbage、ada） | 50,280 |
 | `r50k_base` | GPT-2 | 50,256 |
 | `qwen35` | Qwen3.5（0.8B-35B） | 247,843 |
-| `phi2` | Microsoft Phi-2 | 50,257 |
 | `deepseek-v4` | DeepSeek-V4 Flash | 127,997 |
 | `minicpm5` | MiniCPM5-1B（ByteLevel BPE） | 130,050 |
 

models/README.md

@@ -31,7 +31,6 @@ ids = tok.encode("hello world")
 | Model | LLM Compatibility | Vocab | Pre-tokenization | Source |
 |---|---|---|---|---|
 | `qwen35` | Qwen3.5 (0.8B-35B) | 247,843 | GPT-2 regex | `Qwen/Qwen3.5-0.8B` |
-| `phi2` | Microsoft Phi-2 | 50,257 | GPT-2 regex | `microsoft/phi-2` |
 | `deepseek-v4` | DeepSeek-V4 Flash | 127,997 | None (raw) | `deepseek-ai/DeepSeek-V4-Flash` |
 
 ## Model Format
@@ -70,7 +69,6 @@ After conversion, add the model to the registry in `tinybpe/_registry.py` so it
 | GPT-3 | `p50k_base` BPE | Full support |
 | GPT-2 | `r50k_base` BPE | Full support |
 | Qwen3.5 | ByteLevel BPE | Full support |
-| Phi-2 | ByteLevel BPE | Full support |
 | Llama 3 / 4 | ByteLevel BPE | Via `convert_hf_tokenizer.py` |
 | DeepSeek V2 / V3 | ByteLevel BPE | Full support |
 | MiniCPM5-1B | ByteLevel BPE | Full support |