Skip to content

Commit a6a3611

Browse files
committed
fix bug where 0 was sent to else path
1 parent dd8f595 commit a6a3611

1 file changed

Lines changed: 2 additions & 1 deletion

File tree

model2vec/tokenizer/tokenizer.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,8 @@ def turn_tokens_into_ids(tokens: list[str], model: TokenizerModel) -> list[list[
8989

9090
token_ids: list[list[int]] = []
9191
for token in tokens:
92-
if token_id := vocabulary.get(token):
92+
token_id = vocabulary.get(token)
93+
if token_id is not None:
9394
token_ids.append([*prefix, token_id, *suffix])
9495
else:
9596
token_ids.append(tokenizer.encode(token).ids)

0 commit comments

Comments
 (0)