Current Testworks command line:
$ _build/bin/testworks-run --help
Usage: testworks-run [options]
Run test suites.
--debug WHAT Enter the debugger on failure: NO|crashes|failures
-p, --progress TYPE Show output as the test run progresses: none|DEFAULT|verbose
--report TYPE Final report to generate: failures|full|json|log|none|summary|surefire|xml
--report-file FILE File in which to store the report.
--load FILE Load the given shared library file before searching for test suites. May be repeated.
--suite SUITE Run (or list) only these named suites. May be repeated.
--test TEST Run (or list) only these named tests. May be repeated.
--skip-suite SUITE Skip these named suites. May be repeated.
--skip-test TEST Skip these named tests. May be repeated.
-l, --list WHAT List components: all|suites|tests|benchmarks
-t, --tag TAG Only run tests matching this tag. If tag is prefixed with '-', the test will only run if it does NOT have the tag. May be repeated. Ex: --tag=-slow,-benchmark means don't run benchmarks or tests tagged as slow.
-h, --help Display this message.
Off the top of my head, I propose these changes:
- Always output the result summary. If someone truly doesn't want to see this (e.g., automation) then
>/dev/null is available.
- Get rid of
--debug=no. This is equivalent to not specifying the --debug flag.
--report should be only about the report format. Therefore the options are json, xml, surefire, and log. (But we should delete log, and maybe xml.)
- Replace
--progress with --verbose boolean option to display what you currently get from --report full. --report full goes away.)
- Replace
--suite and --test with --match, which accepts a regular expression and may be repeated. Suites and tests occupy the same namespace so there is no ambiguity.
- Replace
--skip-test and --skip-suite with --skip, which accepts a regular expression and may be repeated. Ditto.
- Make
--list a boolean flag (i.e., and list the tests that would be run based on --match and --skip.
- Add a few more short options (see below).
- Provide a default location for
--report-file. Probably ./testworks-report.json etc. Tell user where the report was written.
The end result looks something like this:
Run tests.
-d, --debug WHAT Enter the debugger on failure: crashes|failures
-v, --verbose Show output as the test run progresses.
--report TYPE Final report to generate: json|log|surefire|xml
--report-file FILE File in which to store the report.
-l, --load FILE Load the given shared library file before searching for tests. May be repeated.
--match REGEX Run only tests and suites matching this regular expression. May be repeated.
--skip REGEX Skip tests and suites matching this regular expression. May be repeated.
-l, --list Only list the tests that would have been run.
-t, --tag TAG Only run tests matching this tag. If tag is prefixed with '-', the test will only run if it does NOT have the tag. May be repeated. Ex: --tag=-slow,-benchmark means don't run benchmarks or tests tagged as slow.
-h, --help Display this message.
Current Testworks command line:
Off the top of my head, I propose these changes:
>/dev/nullis available.--debug=no. This is equivalent to not specifying the--debugflag.--reportshould be only about the report format. Therefore the options are json, xml, surefire, and log. (But we should delete log, and maybe xml.)--progresswith--verboseboolean option to display what you currently get from--report full.--report fullgoes away.)--suiteand--testwith--match, which accepts a regular expression and may be repeated. Suites and tests occupy the same namespace so there is no ambiguity.--skip-testand--skip-suitewith--skip, which accepts a regular expression and may be repeated. Ditto.--lista boolean flag (i.e., and list the tests that would be run based on--matchand--skip.--report-file. Probably./testworks-report.jsonetc. Tell user where the report was written.The end result looks something like this: