Add --import-json to compare against previously saved benchmark runs#873
Open
ChrisJr404 wants to merge 1 commit into
Open
Add --import-json to compare against previously saved benchmark runs#873ChrisJr404 wants to merge 1 commit into
ChrisJr404 wants to merge 1 commit into
Conversation
The new --import-json <file> flag reads a JSON file previously written by --export-json and folds the benchmarks inside it into the current run. The imported entries are not re-executed: they appear in the relative speed comparison and in every export format alongside any commands given on the command line. This makes it possible to keep a long-running baseline on disk (or share one across machines) and compare new commands against it without paying the cost of the original measurements again. The flag may be specified more than once to combine several saved files. Closes sharkdp#607
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #607.
This adds an
--import-json <file>flag that loads benchmarks from a JSON file previously written by--export-jsonand folds them into the current run. The imported entries are not re-executed: they appear in the relative speed comparison and in every export format alongside any commands given on the command line.--import-jsoncan be specified more than once to combine several files. It can also be used on its own (without any commands) to convert a saved JSON run into another export format, which is the use case @dkaszews mentioned on the issue.Implementation notes
BenchmarkResultnow derivesDeserializein addition toSerialize. The fields that--export-jsonskips serialize asdefaultso the existing on-disk schema parses without loss.src/import/module reads the JSON and reconstructsBenchmarkResultvalues.command_with_unused_parametersis not part of the on-disk schema, so it falls back tocommandwhen missing.add_imported_resultsentry point. Imported results are pushed onto its results vector before any live benchmark runs, so they participate inprint_relative_speed_comparisonand in all configured exporters.display_numberso theBenchmark N:headers continue counting from the imported tail rather than restarting at 1. The originalnumberfield still indexes the per-command option arrays (command_output_policies, prepare, conclude), keeping that bookkeeping unchanged.--commandis no longer required when--import-jsonis supplied, so a saved file can be replayed (or re-exported) without inventing a dummy command.Tests
src/import/mod.rs: a JSON round-trip (re-emit aBenchmarkResultvia serde and reload it), tolerance for missing optional fields, and a friendlier error for a missing path.tests/integration_tests.rscovering: import + a live command, import-only, multiple--import-jsonflags in the same invocation, and the missing-file error message.Output looks like: