Iterate further on the integration tests via Travis

[jrfnl]

This ticket is meant as the next iteration, following up on #91

In this iteration, the following additional variants should be explored:

• Having some builds with a global install of PHPCS + global install of the external standard used for testing.
• Having some builds with a mix of global/local installs of external standards(s).
• Having some builds using Windows as OS instead of Linux.

[Potherca]

A couple of years ago I create some code for phpcodesniffer-composer-installer issue-24 that might be usable for some of this.

[Potherca]

Not sure how/where to best run build steps in Windows. Maybe @mjrider has some ideas?

[jrfnl]

@Potherca I know you can set the OS for specific builds in both Travis as well as actions, so either could work.

[jrfnl]

Especially with an eye on preventing regressions, while still broadening support to newer versions, like PHPCS 4.x, PHP 8.x and Composer 2.x, I've made an inventory of test cases which would need to be included, by going through every single issue and PR in this repo from the very beginning to the time of writing.

My focus has been on integration tests and I have made "shorthand" notes.
I don't have time at this moment to get this setup completely, nor have I a clear idea on how we can set up these type of CLI integration tests easily.

All the same, I figured it may be useful to make these notes public for whomever starts work on getting tests set up when they have time.

---

OS matrix

• Windows
• Unix

Travis matrix:

• PHPCS 2 low/high - PHP 5.3 - PHP 8
• PHPCS 3 low/high - PHP 5.4 - PHP 8
• PHPCS 4 (allowed to fail) - PHP 7.2 - PHP 8
• Composer 1
• Composer 2

Possibly use:
--prefer-lowest (works with update/require)
--prefer-stable (works with update/require)
--prefer-source (works with update/require/install)

Basic test scenarios:

• Local install
• Global install

More detailed test scenarios:

• With another dependency which has a file autoload (Symfony polyfill)
• In combination with other Composer plugins
• In combination with post-install/update-cmd's (see Bugfix: Composer scripts/commands broken in 0.6.0 update #105)

Note: When error output is expected, test with both --verbose and without and verify the output.

Caveats:
As the tests would include tests which change the global composer install, running tests locally should be done in a VM/Docker container.

On top of that, it would be a good idea to make a backup of the Composer global composer.json, composer.lock and phpcodesniffer.conf files before running the tests and put those back afterwards (and running a composer install to resync the global Composer setup to what it was before running the tests)

Rough idea about basic test setup:

• create-project test
• fill in initial setup for test situation
• if different results are expected for PHPCS 2/3/4, use phpcs --version to retrieve the version number and set the $expected results based on that.
• run test
• remove the test project

=> figure out how to use the "current" branch of the plugin in the tests as it may not (yet) be public

Things to test:

• catch the output of the composer command and check the bit coming from this plugin is displayed
• run phpcs -i
• run phpcs --config-show

composer install with composer.json with

• no external standards
• one external standard
• three different external standards

composer require & require-dev

• adding one external standard
• adding two external standards
• removing one external standard
• removing two external standards

composer require/remove this plugin

• without PHPCS installed
• with PHPCS installed, no external standards
• with PHPCS installed, multiple external standards, paths set already relative paths
• with PHPCS installed, multiple external standards, paths set already absolute paths

other

• confirm it doesn't run when using --no-scripts
• confirm it doesn't run when using --no-plugins
• confirm running "Dealerdirect\Composer\Plugin\Installers\PHPCodeSniffer\Plugin::run" directly then works

test with installing something with --ignore-platform-reqs where one of the other dependencies of the project has a higher PHP requirement

test situation where the root project is a PHPCS standard which requires the plugin

• make sure it only sets this path if the root package contains phpcodesniffer-standard as type & contains a ruleset.xml file in the root ?

test with different search depths set via extra

{
    "extra": {
        "phpcodesniffer-search-depth": 5
    }
}

test with different search depths set via composer config extra.phpcodesniffer-search-depth 4

-- see if we can set up a test which triggers other Composer command steps to verify this plugin does not interfere
See: #48, #49, #53

test with the plugin installed with different versions locally and globally

• See: https://github.com/Dealerdirect/phpcodesniffer-composer-installer#caveats and https://getcomposer.org/doc/articles/plugins.md#using-plugins
• Issue Plugin not working correctly when sniffs install depth is equal to "1" #13
• local old / global new
• global old / local new
• what about when PHPCS is installed both globally and locally ?

test with other Composer PHPCS plugin installed as well

• do they play nice together ?
  -> or should a new feature be added to detect other plugins and alert on them ?

Candidates:

• https://packagist.org/explore/?query=php_codesniffer&type=composer-plugin
• higidi/composer-phpcodesniffer-standards-plugin (abandoned in favour of this plugin)
• simplyadmire/composer-plugins (also abandoned, no alternative mentioned)

Issue #13

• install older PHPCS version
• run composer update
• see if it all worked & the path is set for the new PHPCS version

Issue #32

• mix of absolute and relative paths in installed_paths including duplicates (once as absolute, once as relative)
• run the plugin
  -> verify the output is correct and contains all paths only once

Issue #33

• PHPCS 2.x: Make sure that path is always set to the directory above the directory containing ruleset.xml and potentially the /Sniffs/ directory as otherwise the path won't be compatible with PHPCS 2
• PHPCS 3.x: Verify the path is set to the standards directory.

Issue #63

• Test with standard installed in non-vendor directory, but somewhere under the root of the project anyway.

Issue #103

• do a full install with PHPCS, the plugin + external standards
• --no-dev install (uninstall the PHPCS + plugin packages)
• make sure that nothing breaks (no exit code 1)
  plugin will still run even when uninstalled as it uses the pre-uninstall state and has been loaded in memory already

test without PHPCS installed
i.e. "manually" removed it from the vendor directory and verify that the plugin gives an appropriate error message

External standards types to test with:

• standards with "default" directory layout, i.e. root/StandardName/Sniffs
• standard with additional PHPCS dependency (PHPCSExtra -> PHPCSUtils)
• standard with external dependency (Slevomat -> PHPParser)
• standard with sniffs directory at top level of the package, i.e. root/Sniffs (old PHPCompatibility could be used - make sure path is set to dir above for compatibility with PHPCS 2)
• standard with deeper directory structure, i.e. root/Sub/StandardName/Sniffs (DrupalCoder/ObjectCalisthenics - make sure path is set to "Sub" for compatibility with PHPCS 2)
• standard with multiple standards (WPCS/DrupalCoder/PHPCSExtra)
• standard without sniffs, just ruleset.xml (Doctrine = also in /lib/ subdirectory or PHPCSUtils)
• standard where this plugin is a dependency of the standard (PHPCSExtra)
• standard which doesn't have the phpcodesniffer-standard type set (mamis - shouldn't be installed)

wp-coding-standards/wpcs
slevomat/coding-standard
phpcompatibility/php-compatibility
object-calisthenics/phpcs-calisthenics-rules
pheromone/phpcs-security-audit
sirbrillig/phpcs-variable-analysis
drupal/coder
doctrine/coding-standard
phpcsstandards/phpcsutils
phpcsstandards/phpcsextra
mamis/phpcs-coding-standard

[Potherca]

That's a really nice overview to have. Very worthwhile!

I've found the test code I wrote for #103 and the test code for #105 .

We might want to make separate tickets from the shorthand notes so that each topics/sections can be worked on individually.

I think one of the first things to do is to set up "something" generic to make it easier to write other tests on top of.
At the moment I have no idea when I'll have time to create something...

[Potherca]

By the way, I concur with the suggested setup.

The testcode originally written for #24 has a similar setup.

Specifically

• Local install vs. Global install

• In combination with other Composer plugins

• In combination with post-install/update-cmd's (see Bugfix: Composer scripts/commands broken in 0.6.0 update #105)

Although written in/for PHPUnit, the test script calls Composer on the commandline (instead of calling the PHP code directly, as the unit-tests do).

The test code written in BASH was good to get things up and running quickly but the PHP code is better for the long-term as it is much easier to read, re-use, maintain and extend.

Running the PHP-code test with --verbose would be trivial, probably just extend the original class and extend the executeComposerCommand function to accept an extra CLI flag.

make a backup of the Composer global composer.json, composer.lock and phpcodesniffer.conf files before running the tests and put those back afterwards

The first two files are taken care of, the last could be added. They are backed-up once (with setUpBeforeClass) and restored after each test (with tearDown).

figure out how to use the "current" branch of the plugin in the tests as it may not (yet) be public

Assuming we want the tests live in this repo (maybe in tests/integration/ ?), the tests should run on the checked out branch. I can't remember exactly how I did that but I know I've got something for that somewhere.

The output is caught by running commands through exec: exec($command, $output, $exitCode);

Several (I think most, except three of four) of the composer commands to run would be quite easy to implement. The phpcs commands would be better served by another/separate test-file, i think.

Definitely interesting stuff I would like to work on. Earliest available time currently looks like September after the holidays, when my work with PDS Interop is further along, and when there is less Corona craziness.

[jrfnl]

Just came across this Composer plugin which may be useful when setting up the integration tests: https://packagist.org/packages/g1a/composer-test-scenarios