Create multiple coverage files and merge them all together

[swissspidy]

See #241 and specifically #241 (comment)

In a nutshell:

• Do not collect coverage if the command is executed as part of the tests setup, e.g. when downloading core or getting wp cli info in FeatureContext.
• Update coverage file name so that a new file is created for each individual step.
• Save coverage files as serialized classes so that they can be merged together later
• In FeatureContext, after finishing the test suite, merge all coverage files together into a single XML file

This requires passing the --xdebug flag to Behat so that it's available there. Otherwise we can't instantiate a new CodeCoverage class. See wp-cli/.github#124 for adding this change in our CI.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 25 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/Context/FeatureContext.php	0.00%	25 Missing ⚠️

📢 Thoughts on this report? Let us know!

[swissspidy]

aand everything's passing now 🎉

[mrsdizzie]

Do you have an example you used to test this? Similar to your helpful comment here #241 (comment) is there a small change I can make in another project to see that this fixes the previous issues (since it doesn't seem possible to test it here directly) ?

[swissspidy]

@mrsdizzie sure yeah. I tested this with the rewrite-command repo. When you use this branch of wp-cli-tests there and run tests with WP_CLI_TEST_COVERAGE=1 composer behat -- --xdebug, you will see many .cov files created in build/logs during the test. They will have file names such as manage-wordpress-rewrites-change-site-permastructs-32-mysql.cov, where 32 is the line number in the feature file. After the tests finish, they will be deleted and there is only a single behat-coverage.xml file left. In there, the count="" values will be higher than 1 in most places, which means these code paths were hit more than once during the whole test suite run (previously in each individual xml file the value was usually only 0 or 1)

[mrsdizzie]

@swissspidy great! Followed those steps and agree it looks better and I see higher counts now