tokenizer: add tokenize_with_range
#134
| Job | Run time |
|---|---|
| 47s | |
| 18s | |
| 55s | |
| 22s | |
| 34s | |
| 3m 38s | |
| 3m 37s | |
| 31s | |
| 3m 34s | |
| 14m 16s |
tokenize_with_range
#134
| Job | Run time |
|---|---|
| 47s | |
| 18s | |
| 55s | |
| 22s | |
| 34s | |
| 3m 38s | |
| 3m 37s | |
| 31s | |
| 3m 34s | |
| 14m 16s |