Skip to content

nk3 test: Add --deny-skipped option#375

Draft
robin-nitrokey wants to merge 1 commit into
masterfrom
nk3-test-deny-skipped
Draft

nk3 test: Add --deny-skipped option#375
robin-nitrokey wants to merge 1 commit into
masterfrom
nk3-test-deny-skipped

Conversation

@robin-nitrokey
Copy link
Copy Markdown
Member

This patch adds an option to the nk3 test subcommand that causes one or more skipped tests to fail the test run. This can be used in production to make sure that all selected tests are actually executed.

Open questions:

  • Should --all automatically activate this flag?

Changes

  • Add --deny-skipped flag to nk3 test.

Checklist

  • tested with Python3.9
  • signed commits
  • updated documentation (e.g. parameter description, inline doc, docs.nitrokey)
  • added labels

Test Environment and Execution

  • OS: Debian 11
  • device's model: NK3AM
  • device's firmware version: v1.3.1

This patch adds an option to the nk3 test subcommand that causes one or
more skipped tests to fail the test run.  This can be used in production
to make sure that all selected tests are actually executed.
@jans23
Copy link
Copy Markdown
Member

jans23 commented Apr 13, 2023

Should --all automatically activate this flag?

IMO yes, this is what I expect from it.

Why to introduce --deny-skipped additionally?

@robin-nitrokey
Copy link
Copy Markdown
Member Author

--all decides which tests are executed, and --deny-skipped how the results are interpreted. The reasoning behind the skipped state is that we don’t want to indicate failures to end users that are not really a failure.

@robin-nitrokey robin-nitrokey marked this pull request as draft April 23, 2023 17:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants