Skip to content

Command-line improvements #121

@cgay

Description

@cgay

Current Testworks command line:

$ _build/bin/testworks-run --help
Usage: testworks-run [options]

Run test suites.
      --debug WHAT        Enter the debugger on failure: NO|crashes|failures
  -p, --progress TYPE     Show output as the test run progresses: none|DEFAULT|verbose
      --report TYPE       Final report to generate: failures|full|json|log|none|summary|surefire|xml
      --report-file FILE  File in which to store the report.
      --load FILE         Load the given shared library file before searching for test suites. May be repeated.
      --suite SUITE       Run (or list) only these named suites. May be repeated.
      --test TEST         Run (or list) only these named tests. May be repeated.
      --skip-suite SUITE  Skip these named suites. May be repeated.
      --skip-test TEST    Skip these named tests. May be repeated.
  -l, --list WHAT         List components: all|suites|tests|benchmarks
  -t, --tag TAG           Only run tests matching this tag. If tag is prefixed with '-', the test will only run if it does NOT have the tag. May be repeated. Ex: --tag=-slow,-benchmark means don't run benchmarks or tests tagged as slow.
  -h, --help              Display this message.

Off the top of my head, I propose these changes:

  1. Always output the result summary. If someone truly doesn't want to see this (e.g., automation) then >/dev/null is available.
  2. Get rid of --debug=no. This is equivalent to not specifying the --debug flag.
  3. --report should be only about the report format. Therefore the options are json, xml, surefire, and log. (But we should delete log, and maybe xml.)
  4. Replace --progress with --verbose boolean option to display what you currently get from --report full. --report full goes away.)
  5. Replace --suite and --test with --match, which accepts a regular expression and may be repeated. Suites and tests occupy the same namespace so there is no ambiguity.
  6. Replace --skip-test and --skip-suite with --skip, which accepts a regular expression and may be repeated. Ditto.
  7. Make --list a boolean flag (i.e., and list the tests that would be run based on --match and --skip.
  8. Add a few more short options (see below).
  9. Provide a default location for --report-file. Probably ./testworks-report.json etc. Tell user where the report was written.

The end result looks something like this:

Run tests.
  -d, --debug WHAT        Enter the debugger on failure: crashes|failures
  -v, --verbose           Show output as the test run progresses.
      --report TYPE       Final report to generate: json|log|surefire|xml
      --report-file FILE  File in which to store the report.
  -l, --load FILE         Load the given shared library file before searching for tests. May be repeated.
      --match REGEX       Run only tests and suites matching this regular expression. May be repeated.
      --skip REGEX        Skip tests and suites matching this regular expression. May be repeated.
  -l, --list              Only list the tests that would have been run.
  -t, --tag TAG           Only run tests matching this tag. If tag is prefixed with '-', the test will only run if it does NOT have the tag. May be repeated. Ex: --tag=-slow,-benchmark means don't run benchmarks or tests tagged as slow.
  -h, --help              Display this message.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels
    No fields configured for Feature.

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions