Our tests generally check if their requirements are met (relevant lib installed etc), and will skip themselves if there is something missing.
It is a nice behavior when one wants to just run some tests locally, but it is a footgun for CI: tests may be silently skipped becuse we, for example. forgot to install something.
This is currently the case with any libev-based unit test: they are not running in any configuration!
Worse, when inspecting CI output its hard to notice this, because the default pytest output just prints dots, instead of printing each test, reason for skip etc.
Fix for pytest output is just adding -v flag to all its invocations.
How to fix silent skipping?
Imo the solution is to disallow skipping in CI, and list precise test filters in command line.
So instead of skipping libev due to no library on windows, we could add relevant -k invocation.
It will be a bit more verbose, but imo a lot safer.
Our tests generally check if their requirements are met (relevant lib installed etc), and will skip themselves if there is something missing.
It is a nice behavior when one wants to just run some tests locally, but it is a footgun for CI: tests may be silently skipped becuse we, for example. forgot to install something.
This is currently the case with any libev-based unit test: they are not running in any configuration!
Worse, when inspecting CI output its hard to notice this, because the default pytest output just prints dots, instead of printing each test, reason for skip etc.
Fix for pytest output is just adding
-vflag to all its invocations.How to fix silent skipping?
Imo the solution is to disallow skipping in CI, and list precise test filters in command line.
So instead of skipping libev due to no library on windows, we could add relevant
-kinvocation.It will be a bit more verbose, but imo a lot safer.