Skip to content

Commit a13eede

Browse files
nelucaneluca
committed
Add MiniCPM5-1B tokenizer support
- Convert MiniCPM5-1B ByteLevel BPE tokenizer to .tbm format - Handle list-format merges in convert_hf_tokenizer.py - Add minicpm5 to model registry (vocab_size=130050) - Add tests for minicpm5 from_pretrained
1 parent e143f30 commit a13eede

4 files changed

Lines changed: 129866 additions & 16 deletions

File tree

0 commit comments

Comments
 (0)