JUnit test reports don't show tests in execution order

[trancexpress]

See e.g. this report:

https://download.eclipse.org/eclipse/downloads/drops4/I20260318-1800/testresults/html/org.eclipse.jdt.ui.tests_ep440I-unit-macosx-x86_64-java21_macosx.cocoa.x86_64_21.html

The HTML starts with:

        <h3>TestCase AutomatedSuite</h3>
        <table class="details" border="0" cellpadding="5" cellspacing="2" width="95%">
            <tr valign="top">
                <th>Class</th><th>Name</th><th>Status</th><th width="80%">Type</th><th nowrap>Time(s)</th>
            </tr>
            <tr valign="top" class="">
                <td>[{0}]</td><td title="[{0}]">testBug213248_1[{0}]</td><td>Success</td><td></td><td>0.342</td>
            </tr>
            <tr valign="top" class="">
                <td>[{0}]</td><td title="[{0}]">testBug213248_1[{0}]</td><td>Success</td><td></td><td>0.307</td>
            </tr>

Screenshot:

The workspace log shows the actual execution order, the first few tests are:

!ENTRY org.eclipse.jdt.ui.tests 1 0 2026-03-18 22:08:47.590
!MESSAGE AutomatedSuite STARTING

!ENTRY org.eclipse.jdt.ui.tests 1 0 2026-03-18 22:08:47.616
!MESSAGE JUnit Jupiter STARTING

!ENTRY org.eclipse.jdt.ui.tests 1 0 2026-03-18 22:08:47.624
!MESSAGE VisitorTest STARTING

!ENTRY org.eclipse.jdt.ui.tests 1 0 2026-03-18 22:08:49.161
!MESSAGE simpleTest() STARTING
coll.add(var)
System.out.println(var)
System.err.println(var)
System.out.println(arr)

!ENTRY org.eclipse.jdt.ui.tests 1 0 2026-03-18 22:08:49.188
!MESSAGE simpleTest() DONE: SUCCESSFUL

!ENTRY org.eclipse.jdt.ui.tests 1 0 2026-03-18 22:08:49.189
!MESSAGE simpleTest2() STARTING
coll.add(var)

!ENTRY org.eclipse.jdt.ui.tests 1 0 2026-03-18 22:08:49.195
!MESSAGE simpleTest2() DONE: SUCCESSFUL

!ENTRY org.eclipse.jdt.ui.tests 1 0 2026-03-18 22:08:49.197
!MESSAGE simpleTest3() STARTING

Not all test bundles show log test start/end, so in some cases all we have is the JUnit report. When multiple failures seem connected, we want to know what the test order is, but we can't for such test bundles.

[trancexpress]

See also:

eclipse-jdt/eclipse.jdt.ui#79 (comment)
eclipse-jdt/eclipse.jdt.ui#79 (comment)

[HannesWell]

Not all test bundles show log test start/end, so in some cases all we have is the JUnit report. When multiple failures seem connected, we want to know what the test order is, but we can't for such test bundles.

Makes sense to me. My suggestion is to record the tests results in execution order in the generated html file and add a JS code so that users can sort the results in alphabetical order on demand.
If preferred, the client-side alphabetical sorting could be applied by default.

The workspace log shows the actual execution order, the first few tests are:

How you checked if the XML files, e.g. at https://download.eclipse.org/eclipse/downloads/drops4/I20260328-0750/testresults/xml/, are in execution order?
If yes, it would just have to be made sure that during the XSLT to generate the HTML files the order is maintained in
https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/master/products/eclipse-junit-tests/src/main/scripts/JUNIT.XSL

Maybe it's even explicitly sorted there?

[trancexpress]

Here is one of the recent reports:

https://download.eclipse.org/eclipse/downloads/drops4/I20260328-0750/testresults/html/org.eclipse.jdt.ui.tests_ep440I-unit-linux-x86_64-java21_linux.gtk.x86_64_21.html
https://download.eclipse.org/eclipse/downloads/drops4/I20260328-0750/testresults/ep440I-unit-linux-x86_64-java21_linux.gtk.x86_64_21/org.eclipse.jdt.ui.tests.AutomatedSuite.xml
https://download.eclipse.org/eclipse/downloads/drops4/I20260328-0750/testresults/ep440I-unit-linux-x86_64-java21_linux.gtk.x86_64_21/org.eclipse.jdt.ui.tests.AutomatedSuite.txt

First test in the workspace log (matches the runtime order, as we log with an execution listener):

!ENTRY org.eclipse.jdt.ui.tests 1 0 2026-03-28 15:01:22.816
!MESSAGE VisitorTest STARTING

!ENTRY org.eclipse.jdt.ui.tests 1 0 2026-03-28 15:01:23.420
!MESSAGE simpleTest() STARTING

First test in the html report:

[{0}]	testBug213248_1[{0}]	Success	

First test in the xml file:

<testcase classname="org.eclipse.jdt.ui.tests.quickfix.CleanUpTest" name="testDoNotUsePrimitiveBooleanRatherThanWrapper" time="0.1"/>

Looking at more content from the xml file, the order seems to be random there. Maybe a hash map is used for it:

<testcase classname="org.eclipse.jdt.ui.tests.quickfix.CleanUpTest" name="testDoNotUsePrimitiveBooleanRatherThanWrapper" time="0.1"/>
<testcase classname="org.eclipse.jdt.ui.tests.core.ImportOrganizeTest" name="testCommentAfterImport" time="0.054"/>
<testcase classname="org.eclipse.jdt.ui.tests.quickfix.LocalCorrectionsQuickFixTest" name="testIndirectStaticAccess_bug307407" time="0.037"/>
<testcase classname="org.eclipse.jdt.ui.tests.quickfix.PropertiesFileQuickAssistTest" name="testCreateFieldInAccessor1" time="0.413"/>
<testcase classname="org.eclipse.jdt.ui.tests.core.ImportOrganizeTest" name="testImportToStarWithExplicit3" time="0.06"/>

[HannesWell]

Looking at more content from the xml file, the order seems to be random there. Maybe a hash map is used for it:

That's of course a point that has to be fixed first. But I assume that these XML files are generated by JUnit directly. Within Eclipse in the JUnit view, one can sort the results in execution order, so I assume that this information is somehow retained.
Bug I'll have to dig into the details and investigate what is exactly generating the XML files and if there are any intermediate steps that maybe randomize the test element order.

In general I think the test details should be reworked in order to give us more flexibility for displaying them:

• Inline detailed I/Y-build test results into test overview at build reports page #3747

[trancexpress]

I can try to find the XML reporter that should be configured, its probably one that comes from ant.

Should be the usual one:

[echoproperties] formatter=org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter

products/eclipse-junit-tests/src/main/scripts/library.xml:            value="org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter" />

XMLJUnitResultFormatter should be writing in execution order...

There is the following task that maybe messes with ordering, but that I assume is for the final report... Each .xml file should not be affected by it.

    <target name="collect">
        <!--
            This target can be used to aggragate test runs from multiple test suites into a single report.

            Parameters to this target:
            includes        - the names of the files to include
            output-file        - the name of the output file to produce
        -->
        <junitreport
            todir="."
            tofile="${output-file}">
            <fileset dir=".">
                <include name="${includes}" />
            </fileset>
        </junitreport>
    </target>

[trancexpress]

I'm not sure if I had trouble with the XML formatter when migrating from JUnit 4 to JUnit 5, but I do see we implement: org.apache.tools.ant.taskdefs.optional.junitlauncher.TestResultFormatter

Which in turn implements the JUnit 5 listener: org.junit.platform.launcher.TestExecutionListener

Whereas org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter implements the JUnit 4 one: junit.framework.TestListener

I wonder if we can try using org.apache.tools.ant.taskdefs.optional.junitlauncher.LegacyXmlResultFormatter... Which implements TestResultFormatter...