feat: run benchmarks in non-bench targets#85
Conversation
|
Thinking about it further, an alternative design could be to make It would look like this: Maybe make the subcommand name optional and default to |
|
Another limitation of this pull request is that it often rebuilds the whole target even if I just ran |
|
hey sorry was just away for a couple weeks. I am a bit confused by this. The behaviour I would like is that you use |
|
@cmyr cargo criterion cannot run successfully when using cargo instruments currently |
As discussed in and closes #84. I also made it use the
benchprofile by default when--benchmarksis supplied, and add the--benchargument to the target args likecargo benchdoes (otherwise it runs unit tests instead).For reference, you can (optionally) specify which benchmarks to run by adding them as target args:
Otherwise it'll run all of the benchmarks in the target.
Limitations:
cargo instruments -t time --benchmarksto fail because it'll build multiple targets and it expects a single one (like if--benchmarkshadn't been supplied). AFAIK,cargo instrumentscan only run one target, so it has no analogous operation tocargo benchrunning benchmarks in multiple targets by default, so maybe this is OK. An alternative would be to run benchmarks in the main target by default, but this might be unexpected for users ofcargo bench. Workaround is to specify a single target likecargo instruments -t time --benchmarks --bin <crate_name>. Might be nice to point users in this direction in the error message.--benchmarkssince everywhere else incargo"bench" is used, but overloading the--benchoption seems more confusing. I could see an argument for--benchmarkas it looks like an action, instead of using two plurals in a row ("instruments", "benchmarks"). But--benchmarksmakes it clear that it'll run all of them in the chosen target.cargo instruments --benchwithout a value, the error message doesn't ask them to use--benchmarksinstead. But I did add a note about it in the--helpmessage for--bench.--benchto the target args likecargo benchdoes, but I haven't checked thecargo benchsource code to see if there are exceptions to this behavior (say, when the default test harness is disabled).cargo instruments --benchshould imply--benchmarks? We'd probably want to use thebenchprofile by default, for example. I'm not using non-default test harnesses or separate bench targets though, so I haven't tested what the best behavior here should be.