Skip to content

Commit 19d96eb

Browse files
committed
Fix eod masking + strategy refactoring
1 parent 9e786c8 commit 19d96eb

11 files changed

Lines changed: 94552 additions & 41 deletions

File tree

out.txt

Lines changed: 753 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
{
2+
"bos_token": {
3+
"content": "<s>",
4+
"lstrip": false,
5+
"normalized": false,
6+
"rstrip": false,
7+
"single_word": false
8+
},
9+
"eos_token": {
10+
"content": "</s>",
11+
"lstrip": false,
12+
"normalized": false,
13+
"rstrip": false,
14+
"single_word": false
15+
},
16+
"unk_token": {
17+
"content": "<unk>",
18+
"lstrip": false,
19+
"normalized": false,
20+
"rstrip": false,
21+
"single_word": false
22+
}
23+
}

0 commit comments

Comments
 (0)