Improve reports

[berezovskyi]

I have finally managed to:

1. Run the suite against the RefImpl 2020 CM server.
2. Generate a report from the command line without Eclipse (but with Ant, potentially we may be able to replace it with a standalone XSLT processor like Saxon).

The idea behind this is to run the test suite against the RefImpl server containers via GH Action (+taking advantage of their support for the "service" containers that run along the Action container) and publish the reports via GH Pages.

The output looks like this:

Here are a few things that are wrong with it:

1. It hardcodes the number of expected MUSTs and corresponding tests that have to pass in order for the OSLC server to be considered conformant.
2. It uses the word "Provider". We agreed long time ago to call it "OSLC Server" AFAIK.
3. It defines levels arbitrarily just based on the number of MUSTs, SHOULDs, and MAYs satisfied. As discussed in the OSLC OP call on 2021-07-29, we might want to define a "bare minimum" level (just MUSTs) with new levels on top but focused on business-level features offered by conformant OSLC Servers (e.g. error reporting if the TRS supported well or graphical resource creation and selection if all DUI capabilities are in places and CORS/CSP are set correctly).
4. It says on top " OSLC Unit Test Report". It should be "OSLC Conformance Test Report (Eclipse Lyo TestSuite)" or similar. Better yet, "'OSLC RefImpl 2020 CM' Server OSLC Conformance Test Report". And on the next line, "Generated by Eclipse Lyo TestSuite v4.1.0.alpha2 (2020-08-01)".
5. It says in the image "OSLC TestSuite Execution Summary". I am confused a bit by the font size, took a little while to understand it's not a page heading but a title of the image. Maybe we can make it smaller.
6. "Error (MUSTS)" in the picture is misleading. As you see at the bottom of the report, 55 tests were actually run and 0 failed. This is because I tweaked the test suite config for the RefImpl 2020 CM server to turn off checking various (optional) OSLC capabilities. "Error (MUSTS)" makes the server look completely non-conformant. Cf. point 1 on hardcoding.
7. The "OSLC TestSuite Execution Summary" table only lists the server MUST conformance and does not mention SHOULDs and MAYs. Why?
8. Some tests fail on conditions that do not have a corresponding CC in the spec. See AbstractCreationAndUpdateRdfTests requires at least one instanceShape property value to update a resource #30, for example.
9. The "Unit Test Summary" table is useless. We should go over a mapping between unit test methods and CM RFC clauses and add CC IDs and maybe the definitions of levels as discussed in the meeting such that we could print them all in a table and show which CCs are violated and group them by "levels" such as QC, DUI etc.

[berezovskyi]

@jamsden @frejfrej I got inspired by the "OSLC levels" discussion (a revival of the old OSLC Profiles discussion) on the last call and tried to get the test suite running against the OSLC RefImpl. I think the test suite is actually a good first step, it already caught some errors in the RefImpl code.

cc @jadelkhoury @axelreichwein

[berezovskyi]

For the reference, a full test conformance output for the CM server:

<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogRdfXmlTests" name="baseUrlIsValid[0]" time="0.064" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogRdfXmlTests" name="catalogRootIsRdfNamespaceDeclaration[0]" time="0.05" assessment="passedMay"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogRdfXmlTests" name="catalogRootAboutElementPointsToSelf[0]" time="0.051" assessment="passedMay"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogRdfXmlTests" name="serviceProviderCatalogsHaveAtMostOneTitle[0]" time="0.049" assessment="passedMay"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogRdfXmlTests" name="serviceProviderCatalogHaveAtMostOneTitle[0]" time="0.047" assessment="passedMay"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogRdfXmlTests" name="serviceProvidersHaveAtMostOneTitle[0]" time="0.047" assessment="passedMay"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogRdfXmlTests" name="serviceProviderCatalogsHaveAtMostOnePublisher[0]" time="0.047" assessment="passedMay"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogRdfXmlTests" name="serviceProvidersHaveAtMostOnePublisher[0]" time="0.047" assessment="passedMay"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogRdfXmlTests" name="publisherElementsAreValid[0]" time="0.045" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogRdfXmlTests" name="misplacedParametersDoNotEffectResponse[0]" time="0.103" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogRdfXmlTests" name="invalidContentTypeGivesNotSupportedOPTIONAL[0]" time="0.103" assessment="passedMay" />

<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogRdfXmlTests" name="contentTypeIsSuggestedType[0]" time="0.085" assessment="passedMay"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderRdfXmlTests" name="misplacedParametersDoNotEffectResponse[0]" time="0.093" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderRdfXmlTests" name="invalidContentTypeGivesNotSupportedOPTIONAL[0]" time="0.093" assessment="passedMay" />
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderRdfXmlTests" name="currentUrlIsValid[0]" time="0.043" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderRdfXmlTests" name="rootAboutElementPointsToSelf[0]" time="0.042" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderRdfXmlTests" name="typeIsServiceProvider[0]" time="0.042" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderRdfXmlTests" name="responseContentTypeIsXML[0]" time="0.079" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderRdfXmlTests" name="serviceProviderHasAtMostOneTitle[0]" time="0.042" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderRdfXmlTests" name="eachServiceHasOneDomain[0]" time="0.042" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderRdfXmlTests" name="serviceProviderHasService[0]" time="0.042" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderRdfXmlTests" name="serviceProviderHasAtMostOnePublisher[0]" time="0.042" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogXmlTests" name="baseUrlIsValid[0]" time="0.044" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogXmlTests" name="catalogRootIsRdfNamespaceDeclaration[0]" time="0.044" assessment="passedMay"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogXmlTests" name="catalogRootAboutElementPointsToSelf[0]" time="0.043" assessment="passedMay"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogXmlTests" name="serviceProviderCatalogsHaveAtMostOneTitle[0]" time="0.048" assessment="passedMay"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogXmlTests" name="serviceProvidersHaveAtMostOneTitle[0]" time="0.048" assessment="passedMay"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogXmlTests" name="serviceProviderCatalogsHaveAtMostOnePublisher[0]" time="0.046" assessment="passedMay"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogXmlTests" name="serviceProvidersHaveAtMostOnePublisher[0]" time="0.047" assessment="passedMay"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogXmlTests" name="publisherElementsAreValid[0]" time="0.045" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogXmlTests" name="misplacedParametersDoNotEffectResponse[0]" time="0.147" assessment="passedMay"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogXmlTests" name="serviceProviderCatalogsHaveAtMostOneOAuthElement[0]" time="0.046" assessment="passedMay"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogXmlTests" name="serviceProvidersHaveAtMostOneOAuthElement[0]" time="0.051" assessment="passedMay"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogXmlTests" name="oAuthElementsAreValid[0]" time="0.045" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogXmlTests" name="serviceProviderCatalogsHaveValidResourceUrl[0]" time="0.086" assessment="passedMay"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogXmlTests" name="servicesProvidersHaveValidResourceUrl[0]" time="0.134" assessment="passedMay"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogXmlTests" name="detailsElementsHaveValidResourceAttribute[0]" time="0.045" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogXmlTests" name="invalidContentTypeGivesNotSupportedOPTIONAL[0]" time="0.045" assessment="passedMay" />
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderCatalogXmlTests" name="contentTypeIsSuggestedType[0]" time="0.083" assessment="passedMay"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderXmlTests" name="baseUrlIsValid[0]" time="0.038" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderXmlTests" name="publisherElementsAreValid[0]" time="0.04" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderXmlTests" name="misplacedParametersDoNotEffectResponse[0]" time="0.123" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderXmlTests" name="invalidContentTypeGivesNotSupportedOPTIONAL[0]" time="0.123" assessment="passedMay" />
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderXmlTests" name="responseContentTypeIsXML[0]" time="0.074" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderXmlTests" name="serviceProviderHasAtMostOneTitle[0]" time="0.039" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderXmlTests" name="eachServiceHasOneDomain[0]" time="0.04" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderXmlTests" name="serviceProviderHasService[0]" time="0.039" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderXmlTests" name="serviceProviderHasAtMostOnePublisher[0]" time="0.04" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderXmlTests" name="oAuthElementsAreValid[0]" time="0.039" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderXmlTests" name="serviceProviderHasValidDetails[0]" time="0.077" assessment="passedMay"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderXmlTests" name="prefixDefinitionsAreValid[0]" time="0.044" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderXmlTests" name="serviceProviderHasAtMostOneOAuthElement[0]" time="0.038" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderXmlTests" name="creationFactoriesAreValid[0]" time="0.039" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderXmlTests" name="queryCapabilityBlocksAreValid[0]" time="0.04" assessment="passedMust"/>
<testcase classname="org.eclipse.lyo.testsuite.oslcv2.core.ServiceProviderXmlTests" name="dialogsAreValid[0]" time="0.039" assessment="passedMust"/>

The assessment is done by a hardcoded association between a unit test and a corresponding clause severity. We can add levels to that mapping. A thornier issue is a MUST clause that is hidden behind a SHOULD or MAY. CORE-60 is a good example.