spec : save the dynamic/static ngram cache file#22055
Open
petersid2022 wants to merge 1 commit intoggml-org:masterfrom
Open
spec : save the dynamic/static ngram cache file#22055petersid2022 wants to merge 1 commit intoggml-org:masterfrom
petersid2022 wants to merge 1 commit intoggml-org:masterfrom
Conversation
petersid2022
force-pushed
the
self-speculation-save-cache
branch
from
2e1c956 to
430c0ca
Compare
petersid2022
force-pushed
the
self-speculation-save-cache
branch
2 times, most recently
from
d5448ea to
ba99720
Compare
petersid2022
requested review from
a team,
CISC,
IMbackK,
ggerganov and
pwilkin
as code owners
petersid2022
force-pushed
the
self-speculation-save-cache
branch
4 times, most recently
from
cf7a308 to
8ae6c04
Compare
petersid2022
force-pushed
the
self-speculation-save-cache
branch
2 times, most recently
from
afc3295 to
dc2ab62
Compare
petersid2022
force-pushed
the
self-speculation-save-cache
branch
5 times, most recently
from
c402b3d to
9da23a4
Compare
petersid2022
changed the title
petersid2022
force-pushed
the
self-speculation-save-cache
branch
9 times, most recently
from
89b10b8 to
5c5bea4
Compare
petersid2022
commented
Apr 29, 2026
| }; | ||
|
|
||
| struct common_params_speculative_ngram_cache { | ||
| struct common_params_speculative_ngram_cache : common_params_speculative_ngram_map { |
Contributor
Author
There was a problem hiding this comment.
This is probably the wrong way of going about this, but I am curious if the same concept of m-gram speculative tokens can be applied in the ngram-cache implemetantion
petersid2022
force-pushed
the
self-speculation-save-cache
branch
from
5c5bea4 to
cee5400
Compare
* fix todo on providing n_draft, save_static and save_dynamic from common/common.h * implement the functionality by saving the cache at the common_speculative_state_ngram_cache destruction
petersid2022
force-pushed
the
self-speculation-save-cache
branch
from
cee5400 to
4d256ae
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
When we select the
COMMON_SPECULATIVE_TYPE_NGRAM_CACHEspeculative implementation we create a newcommon_speculative_state_ngram_cachestate usingcreate_state_ngram_cache, where we instantiate the new state by specifying various parameters (e.g,n_draft,save_staticandsave_dynamic) by hardcoding them.Instead we extend
common_params_speculativeto include those options as well.An attempt was also made to implement the
save_static/save_dynamicbehavior by callingcommon_ngram_cache_saveon object destruction.Additional information
Add self‑speculative decoding (no draft model required)#18471
Requirements