Fix beam search#2157
Open
parkan wants to merge 1 commit into
Open
Conversation
|
Please sign our contributor license agreement at https://cla.futo.org |
|
Please sign our contributor license agreement at https://cla.futo.org |
1 similar comment
|
Please sign our contributor license agreement at https://cla.futo.org |
Author
|
oops sorry pushed the license-required modification notice to this branch instead of my main, fixing |
Author
|
ok the deadlines have a 50ms gap where we could drop the prediction/learning on the floor or double dispatch; extending both to ~600ms total is too slow for faster typing and doesn't solve that problem rethinking how to optimize this but beam fix is correct, I will trim this branch to only include the beam fix or you can pick it separately |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fixes
a few issuesbeam search, because I noticed the keyboard performing very poorly. After installing the custom APK my typing experience has improved dramatically.- increased LLM suggestion timeout; this appears to have no negative effect on latency- transformer context was too narrow, increased the params and validated empiricallyI also noticed that the finetuning path was fully disabled in 44c8fcd even though USE_TRANSFORMER_FINETUNING is still consumed and I am seeing the tuning setting in the UI. Not touched in this PR but I suggest either completely removing if you think it's not viable on device or re-enabling it since it appears fully viable (and valuable) to me.
I am also trying out training retroactively on my own corpus/on my local workstation from https://huggingface.co/breadlicker45/futo-keyboard-lm/blob/main/ml4_1_f16_meta_fixed.gguf since apparently no training occurred since ~last October and I have my own idiosyncratic speech patterns that the default model doesn't understand. So far even training on TinyStories gives better/equal performance to the stock model (though it talks about playgrounds and sleepy bears way too much) -- the "explore transformer models" link in the app still leads to an "under construction" page so even replacing that with a brief "how to locally train on your own writing" snippet would help, it trained in under 3 minutes at 2000 steps on a W7800 with no issues and imports fine.