add quality score script#69

Merged

AstroVasseL merged 9 commits into

mainfrom

feature/add-quality-score

Jun 17, 2026

Commits on Jun 11, 2026

add quality score script
Lasse Vulto
committed
format script
Lasse Vulto
committed
remove min and max from normalize because std python buildin
Lasse Vulto
committed

Commits on Jun 16, 2026

lookup special tokens from tokenizer.json rather than hardcoding them
Lasse Vulto
committed
update formatting
Lasse Vulto
committed
load tokenizer outside map function
Lasse Vulto
committed
add more cpu to map
Lasse Vulto
committed
change sep to padding
Lasse Vulto
committed
output path from argv
Lasse Vulto
committed