Enabling Surefire to run test classes and test methods in any order specified by a runOrder

[winglam]

The changes in this pull request enable Surefire to run test classes and test methods in any order specified by a newly added <runOrder> (testorder). Namely, the changes include

• A new <runOrder> called testorder that would make Surefire run test classes and test methods in the order specified by <test>. Regex is supported as is the case for other runOrders
• A new feature so that Surefire will attempt to parse the content of a file path for a list of tests to include or exclude if the argument specified by <test> is a file path (each line is assumed to be a test to include or exclude)
• Tests to TestListResolverTest and RunOrderCalculatorTest for the newly added features

The newly added testorder <runOrder> is supported for all JUnit 3 and JUnit 4 tests. I will plan on supporting JUnit 5 tests after these changes are accepted.

Simple (no regex) example using Dubbo:

mvn test -Dsurefire.runOrder=testorder \
-Dtest=org.apache.dubbo.rpc.protocol.dubbo.DubboLazyConnectTest#testSticky2,\
org.apache.dubbo.rpc.protocol.dubbo.DubboLazyConnectTest#testSticky1,\
org.apache.dubbo.rpc.protocol.dubbo.DubboLazyConnectTest#testSticky3,\
org.apache.dubbo.rpc.protocol.dubbo.DubboProtocolTest#testNonSerializedParameter\
 -pl dubbo-rpc/dubbo-rpc-dubbo

Output from Maven should say that the test class DubboLazyConnectTest ran before DubboProtocolTest

The file DubboLazyConnectTest.xml (dubbo-rpc/dubbo-rpc-dubbo/target/surefire-reports/TEST-org.apache.dubbo.rpc.protocol.dubbo.DubboLazyConnectTest.xml) should say

<testcase name="testSticky2"...>
<testcase name="testSticky1"...>
<testcase name="testSticky3...>

Regex example using Dubbo:

mvn test -Dsurefire.runOrder=testorder \
-Dtest=org.apache.dubbo.rpc.protocol.dubbo.DubboLazyConnectTest*,\
org.apache.dubbo.rpc.protocol.dubbo.DubboProtocolTest#testDubboProtocol*\
 -pl dubbo-rpc/dubbo-rpc-dubbo

Output from Maven should say

Tests run: 4 … in org.apache.dubbo.rpc.protocol.dubbo.DubboLazyConnectTest
Tests run: 3 … in org.apache.dubbo.rpc.protocol.dubbo.DubboProtocolTest

The file DubboProtocolTest.xml (dubbo-rpc/dubbo-rpc-dubbo/target/surefire-reports/TEST-org.apache.dubbo.rpc.protocol.dubbo.DubboProtocolTest.xml) should say

<testcase name=”testDubboProtocol”...>
<testcase name="testDubboProtocolWithMina"...>
<testcase name="testDubboProtocolMultiService”...>

Include/Exclude example using Dubbo:

mvn test -Dsurefire.runOrder=testorder -Dtest=org.apache.dubbo.rpc.protocol.dubbo.DubboLazyConnectTest#test*,\!org.apache.dubbo.rpc.protocol.dubbo.DubboLazyConnectTest#testSticky2  -pl dubbo-rpc/dubbo-rpc-dubbo

Output from Maven should say
Tests run: 3 … in org.apache.dubbo.rpc.protocol.dubbo.DubboLazyConnectTest

The file DubboLazyConnectTest.xml (dubbo-rpc/dubbo-rpc-dubbo/target/surefire-reports/TEST-org.apache.dubbo.rpc.protocol.dubbo.DubboLazyConnectTest.xml) should say

<testcase name="testSticky1"...>
<testcase name="testSticky3"...>
<testcase name="testSticky4...>

[waterlooyh]

Look Good