Skip to content

Add --import-json to compare against previously saved benchmark runs#873

Open
ChrisJr404 wants to merge 1 commit into
sharkdp:masterfrom
ChrisJr404:feat/import-json-607
Open

Add --import-json to compare against previously saved benchmark runs#873
ChrisJr404 wants to merge 1 commit into
sharkdp:masterfrom
ChrisJr404:feat/import-json-607

Conversation

@ChrisJr404
Copy link
Copy Markdown

Closes #607.

This adds an --import-json <file> flag that loads benchmarks from a JSON file previously written by --export-json and folds them into the current run. The imported entries are not re-executed: they appear in the relative speed comparison and in every export format alongside any commands given on the command line.

hyperfine 'old-binary args' --export-json baseline.json
hyperfine --import-json baseline.json 'new-binary args'

--import-json can be specified more than once to combine several files. It can also be used on its own (without any commands) to convert a saved JSON run into another export format, which is the use case @dkaszews mentioned on the issue.

Implementation notes

  • BenchmarkResult now derives Deserialize in addition to Serialize. The fields that --export-json skips serialize as default so the existing on-disk schema parses without loss.
  • A new src/import/ module reads the JSON and reconstructs BenchmarkResult values. command_with_unused_parameters is not part of the on-disk schema, so it falls back to command when missing.
  • The scheduler grew a small add_imported_results entry point. Imported results are pushed onto its results vector before any live benchmark runs, so they participate in print_relative_speed_comparison and in all configured exporters.
  • Live benchmarks now carry a separate display_number so the Benchmark N: headers continue counting from the imported tail rather than restarting at 1. The original number field still indexes the per-command option arrays (command_output_policies, prepare, conclude), keeping that bookkeeping unchanged.
  • --command is no longer required when --import-json is supplied, so a saved file can be replayed (or re-exported) without inventing a dummy command.

Tests

  • Unit tests under src/import/mod.rs: a JSON round-trip (re-emit a BenchmarkResult via serde and reload it), tolerance for missing optional fields, and a friendlier error for a missing path.
  • Integration tests in tests/integration_tests.rs covering: import + a live command, import-only, multiple --import-json flags in the same invocation, and the missing-file error message.

Output looks like:

Benchmark 1: sleep 1.0 (imported)
Benchmark 2: sleep 2.0 (imported)
Benchmark 3: sleep 1.5
  Time (mean ± σ):      1.500 s ±  0.000 s    [User: 0.000 s, System: 0.000 s]
  Range (min … max):    1.500 s …  1.500 s    10 runs

Summary
  sleep 1.0 ran
    1.50 ± 0.00 times faster than sleep 1.5
    2.00 ± 0.00 times faster than sleep 2.0

The new --import-json <file> flag reads a JSON file previously written by
--export-json and folds the benchmarks inside it into the current run.
The imported entries are not re-executed: they appear in the relative
speed comparison and in every export format alongside any commands given
on the command line.

This makes it possible to keep a long-running baseline on disk (or share
one across machines) and compare new commands against it without paying
the cost of the original measurements again. The flag may be specified
more than once to combine several saved files.

Closes sharkdp#607
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add an option to save a benchmark run for later comparisons

1 participant