Commit 82c8ab7
nemo
Implement
Not every token seems to be valid unicode but every token is interpreted
as such in `pw.whisper_token_to_str`. While this can be caught with an
exception handler it might be worthwhile to have a way of getting the
token bytes instead and parsing them using `.decode`, e.g.:
```python
str(pw.whisper_token_to_bytes(ctx, tid), 'utf8', 'ignore')
```whisper_token_to_bytes
1 parent 4ab9616 commit 82c8ab7
1 file changed
Lines changed: 7 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
133 | 133 | | |
134 | 134 | | |
135 | 135 | | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
136 | 142 | | |
137 | 143 | | |
138 | 144 | | |
| |||
488 | 494 | | |
489 | 495 | | |
490 | 496 | | |
| 497 | + | |
491 | 498 | | |
492 | 499 | | |
493 | 500 | | |
| |||
0 commit comments