Tests are marked or not marked as skipped, depending which filtering option was used

[gladykov]

Before opening issues, I wanted to discuss what do you think about such behavior of Vitest.

We use different filtering options together, to run different sets of tests:

1. vitest tests/feature1 --testNamePattern="Create scheduled notifications"
2. vitest tests/feature1 --exclude=**/*SEQ.test.ts"
   3 .vitest tests/feature1/*SEQ.test.ts

Those options provide same functionality in my mind.

But:

1. when I will use testNamePattern all other tests which does not match the pattern are marked as skipped
2. when I will use exclude, tests in excluded files are not marked as skipped - they are absent from reports
3. when I use path + glob, tests in files which are not matched, are not marked as skipped - they are absent from reports

So in case of 1, I have 6 tests which run, and 200 tests which are marked as skipped. I didn't skip them. I just didn't want them to run :D
I skip the test, when I do test.skip

This pollutes our results and skews stats.

I know workaround is to divide our tests into more subfolders or use file name filtering. But wouldn't it be nice, if testNamePattern would not skip all other tests and just omit them? Especially if we use --testNamePattern=Smoke as a way to designate smoke tests or other groups of tests.

[sheremet-va]

--testNamePattern is the least performant out of the three. Vitest treats test files as "test specifications". To collect tests, Vitest needs to run code inside a test specification (test file). Using a different file name glob or an exclude is the most performant way to filter your tests since Vitest won't even need to run the test file. If test file didn't run, there is nothing to report so it's not reported and not included in the statistic.

On the other hand, --testNamePattern has to run every test file to collect tests first and then start filtering (one by one). This happens dynamically (we report as we collect and run tests) and we can't just avoid reporting it because the whole point of Vitest is to report test files. Some tools rely on this behaviour. For example, the vscode extension sets the name pattern to an impossible to reach regexp and just collects tests. If you don't want to report tests, you can write a custom reporter.

[gladykov]

I see. While I understand this is core behavior and unlikely to change, just want to leave my (purely subjective) opinion that it strips skipped status from semantic meaning. skipped should be on purpose, because of reason (ex. known regression, feature flag disabled) or called explicitly by skip() and skipIf() . skipped because of test filtering does not fulfill this criteria IMHO. So skipped status as piece of information about executed test run becomes useless if we mix both: 1. intentional skip of test by developer (sometimes in some condition) + 2. filtering, which happens to use skipped as means to filter tests.