Remove the use of pickle throughout codebase#15629
Conversation
…iles elsewhere Signed-off-by: Jason <jasoli@nvidia.com>
Signed-off-by: blisc <blisc@users.noreply.github.com>
Signed-off-by: Jason <jasoli@nvidia.com>
Signed-off-by: blisc <blisc@users.noreply.github.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Signed-off-by: Jason <jasoli@nvidia.com>
Signed-off-by: Jason <jasoli@nvidia.com>
Signed-off-by: blisc <blisc@users.noreply.github.com>
Signed-off-by: blisc <blisc@users.noreply.github.com>
| logging.info(f"Loading the cached pickle file of hypotheses from '{cfg.hyps_cache_file}' ...") | ||
| with open(cfg.hyps_cache_file, 'rb') as probs_file: | ||
| all_hyps = pickle.load(probs_file) | ||
| all_hyps = msgpack.load(probs_file) |
There was a problem hiding this comment.
not sure - did you check if cfg.hyps_cache_file default extension is .pkl? if it is we should update, similar with other changes
There was a problem hiding this comment.
TBH, I don't know where this config file is supposed to exist, so leaving it as is.
| timestamp = json_mtime.strftime('%Y%m%d_%H%M') | ||
| pickle_filename += '_' + timestamp + '.pkl' | ||
| if os.path.exists(pickle_filename): | ||
| with open(pickle_filename, 'rb') as f: |
There was a problem hiding this comment.
update pickle_filename and the extension
There was a problem hiding this comment.
It's been changed throughout the codebase as best as I can find
There was a problem hiding this comment.
It looks like there is an update to SDE in https://github.com/NVIDIA-NeMo/NeMo/pull/15500/changes. I will unstage my changes from this PR
Signed-off-by: Jason <jasoli@nvidia.com>
Signed-off-by: Jason <jasoli@nvidia.com>
Signed-off-by: Jason <jasoli@nvidia.com>
|
/ok to test 878336f |
|
[🤖]: Hi @blisc 👋, We wanted to let you know that a CICD pipeline for this PR just finished successfully. So it might be time to merge this PR or get some approvals. |
4 participants
What does this PR do ?
Collection: core
Changelog
PR Type: