fix: binary file check

[ix-56h]

Closes #375

Topic

tiktoken crash sometimes with binary files, after diging i've found that we did some check to ignore binary files but not strong enough.

[ix-56h]

@NicolasIRAGNE i've fix encoding issue on windows, we now use utf-16 le instead of utf-16.

I let you decide what we need to do from now.

[NicolasIRAGNE]

I'll take a look whenever I have a bit of time, I'm not sure what this does but it seems better than initial check

But I am curious as to what the actual error is. Do we read too many characters? What happens if the file is just a huge numbers of ascii chars?

Also, #375 did this fix the problem as well?

[ix-56h]

But I am curious as to what the actual error is. Do we read too many characters? What happens if the file is just a huge numbers of ascii chars?

The current problem:

1. we read a chunk of a file
2. we wrongly determine binaries using a too much simple method
3. this leads to include bytes within the final output context
4. the tiktoken package, which uses rust, panic when unwrapping parsing result

Also, #375 did this fix the problem as well?

Yes. This is the main purpose of this with another one : The context should now ignore a lot more (if it's not all) binaries, so the context should be a lot more usable for LLMs on various repositories.

[github-actions]

Hi there! We haven’t seen activity on this pull request for 45 days, so I’m marking it as stale.
If you’d like to keep it open, please leave a comment within 10 days. Thanks!

[github-actions]

Hi there! We haven’t heard anything for 10 days, so I’m closing this pull request. Feel free to reopen if you’d like to continue working on it. Thanks!