Potential bug in ch02/05_bpe-from-scratch/bpe-from-scratch-simple.ipynb

[h1yuol]

The line words = text.replace("\n", " \n ").split() in the encode() function in ch02/05_bpe-from-scratch/bpe-from-scratch-simple.ipynb is a bit confusing. Adding .replace("\n", " \n ") or not would not change the split result. In the split result, '\n' is not treated as a separate token but instead not included. This seems to be a bug? Can I get clarification on this please?

[d-kleine]

@rasbt gentle ping in case this bug report got overlooked 🙂

[rasbt]

Thanks for the ping and no worries, I didn't forget. I am out of office though and can't work on it at the moment. Will tackle it in a few weeks.