Skip to content

Commit 6e334c2

Browse files
committed
fix: stop tagging "test exception" testcases as synthetic skip
"test exception" is not a synthetic placeholder emitted by any test framework — JUnit 4, JUnit Platform, Gradle's adapters, Spock and Kotest all reserve "initializationError"/"executionError" for that purpose, not generic English names. In dd-trace-java the entries observed in CI came from a real Spock feature method (`def "test exception"()` in HttpServerTest.groovy), not a framework synthetic. Tagging that name unconditionally silently masks real pass/fail outcomes for any Spock/JUnit 5 @DisplayName/Kotest test that happens to use it. Drop "test exception" from JUnitReport.tagSyntheticFailures and document the criteria for future synthetic entries (must be framework-emitted, must link to the source pinned to a release tag). Annotate the two remaining entries — initializationError (JUnit 4 + Gradle) and executionError (Gradle) — with the canonical source locations.
1 parent 5181a21 commit 6e334c2

1 file changed

Lines changed: 20 additions & 1 deletion

File tree

.gitlab/collect-result/JUnitReport.java

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,15 +66,34 @@ boolean normalizeStableTestNames() {
6666
return changed;
6767
}
6868

69+
// Tags framework-emitted synthetic testcases so Test Optimization does not treat them as real
70+
// failures. Any name added here must be a placeholder that the test framework/runner produces
71+
// itself — never a name that a user-authored test could legitimately have. New entries must
72+
// include a link to the framework/library source that emits the literal, pinned to a release
73+
// tag (not a commit hash). Frameworks/libraries to audit when adding a synthetic: JUnit 4
74+
// (org.junit.internal.runners.ErrorReportingRunner), JUnit Platform / Vintage / Jupiter
75+
// engines, Gradle's JUnit/JUnitPlatform test event adapters, TestNG, Spock, Kotest, Maven
76+
// Surefire/Failsafe. Generic English names (e.g. "test exception") must not be added — any
77+
// Spock feature method (`def "..."()`), JUnit 5 `@DisplayName`, or Kotest spec can produce
78+
// them, which would silently mask real pass/fail outcomes.
6979
void tagSyntheticFailures() {
7080
Map<String, List<Element>> initializationErrorsByClassname = new LinkedHashMap<>();
7181
for (var testcase : testcases()) {
7282
var name = testcase.getAttribute("name");
83+
// JUnit 4 emits "initializationError" when the test class cannot be loaded, has no runnable
84+
// methods, or its @BeforeClass fails — see
85+
// https://github.com/junit-team/junit4/blob/r4.13.2/src/main/java/org/junit/internal/runners/ErrorReportingRunner.java#L83
86+
// Gradle's JUnit Platform listener reuses the same name for synthetic container failures —
87+
// see
88+
// https://github.com/gradle/gradle/blob/v9.5.0/platforms/jvm/testing-jvm-infrastructure/src/main/java/org/gradle/api/internal/tasks/testing/junitplatform/JUnitPlatformTestExecutionListener.java#L426
7389
if ("initializationError".equals(name)) {
7490
initializationErrorsByClassname
7591
.computeIfAbsent(testcase.getAttribute("classname"), ignored -> new ArrayList<>())
7692
.add(testcase);
77-
} else if ("executionError".equals(name) || "test exception".equals(name)) {
93+
} else if ("executionError".equals(name)) {
94+
// Gradle's JUnit Platform listener emits "executionError" for container nodes whose
95+
// descendants already started before the failure — see
96+
// https://github.com/gradle/gradle/blob/v9.5.0/platforms/jvm/testing-jvm-infrastructure/src/main/java/org/gradle/api/internal/tasks/testing/junitplatform/JUnitPlatformTestExecutionListener.java#L426
7897
addFinalStatusProperty(testcase, "skip", MissingPropertiesPlacement.APPEND_TO_TESTCASE);
7998
}
8099
}

0 commit comments

Comments
 (0)