Skip to content

Commit f60311a

Browse files
committed
fix: stop tagging "test exception" testcases as synthetic skip
"test exception" is not a synthetic placeholder emitted by any test framework — JUnit 4, JUnit Platform, Gradle's adapters, Spock and Kotest all reserve "initializationError"/"executionError" for that purpose, not generic English names. In dd-trace-java the entries observed in CI came from a real Spock feature method (`def "test exception"()` in HttpServerTest.groovy), not a framework synthetic. Tagging that name unconditionally silently masks real pass/fail outcomes for any Spock/JUnit 5 @DisplayName/Kotest test that happens to use it. Drop "test exception" from JUnitReport.tagSyntheticFailures and document the criteria for future synthetic entries (must be framework-emitted, must link to the source pinned to a release tag). Annotate the two remaining entries — initializationError (JUnit 4 + Gradle) and executionError (Gradle) — with the canonical source locations.
1 parent 5181a21 commit f60311a

1 file changed

Lines changed: 34 additions & 7 deletions

File tree

.gitlab/collect-result/JUnitReport.java

Lines changed: 34 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -66,16 +66,43 @@ boolean normalizeStableTestNames() {
6666
return changed;
6767
}
6868

69+
/// Tags framework-emitted synthetic testcases so Test Optimization does not treat them as
70+
/// real failures.
71+
///
72+
/// **Criteria for new entries:**
73+
/// - Must be a name the framework/runner emits itself — never a name a user-authored test
74+
/// could legitimately have.
75+
/// - Must link to the source that emits the literal, pinned to a release tag (not a commit
76+
/// hash).
77+
///
78+
/// **Frameworks/libraries to audit before adding a name:** JUnit 4, JUnit Platform engines
79+
/// (Vintage / Jupiter), Gradle's JUnit and JUnit Platform adapters, TestNG, Spock, Kotest,
80+
/// Maven Surefire/Failsafe. JUnit 5's main sources do not define these literals — Vintage
81+
/// forwards `"initializationError"` from JUnit 4 as-is, and `"executionError"` is
82+
/// Gradle-specific.
83+
///
84+
/// **Do not add** generic English names such as `"test exception"`. Spock feature methods
85+
/// (`def "..."()`), JUnit 5 `@DisplayName`, and Kotest specs can all produce them, so the
86+
/// tagger would silently mask real pass/fail outcomes.
6987
void tagSyntheticFailures() {
7088
Map<String, List<Element>> initializationErrorsByClassname = new LinkedHashMap<>();
7189
for (var testcase : testcases()) {
72-
var name = testcase.getAttribute("name");
73-
if ("initializationError".equals(name)) {
74-
initializationErrorsByClassname
75-
.computeIfAbsent(testcase.getAttribute("classname"), ignored -> new ArrayList<>())
76-
.add(testcase);
77-
} else if ("executionError".equals(name) || "test exception".equals(name)) {
78-
addFinalStatusProperty(testcase, "skip", MissingPropertiesPlacement.APPEND_TO_TESTCASE);
90+
switch (testcase.getAttribute("name")) {
91+
// JUnit 4 ErrorReportingRunner — initializationError
92+
// https://github.com/junit-team/junit4/blob/r4.13.2/src/main/java/org/junit/internal/runners/ErrorReportingRunner.java#L83
93+
// Gradle JUnit Platform listener — reuses the same name for synthetic container failures
94+
// https://github.com/gradle/gradle/blob/v8.14.5/platforms/jvm/testing-junit-platform/src/main/java/org/gradle/api/internal/tasks/testing/junitplatform/JUnitPlatformTestExecutionListener.java#L248
95+
// https://github.com/gradle/gradle/blob/v9.5.0/platforms/jvm/testing-jvm-infrastructure/src/main/java/org/gradle/api/internal/tasks/testing/junitplatform/JUnitPlatformTestExecutionListener.java#L426
96+
case "initializationError" ->
97+
initializationErrorsByClassname
98+
.computeIfAbsent(testcase.getAttribute("classname"), ignored -> new ArrayList<>())
99+
.add(testcase);
100+
// Gradle JUnit Platform listener — executionError, used when descendants already started
101+
// https://github.com/gradle/gradle/blob/v8.14.5/platforms/jvm/testing-junit-platform/src/main/java/org/gradle/api/internal/tasks/testing/junitplatform/JUnitPlatformTestExecutionListener.java#L248
102+
// https://github.com/gradle/gradle/blob/v9.5.0/platforms/jvm/testing-jvm-infrastructure/src/main/java/org/gradle/api/internal/tasks/testing/junitplatform/JUnitPlatformTestExecutionListener.java#L426
103+
case "executionError" ->
104+
addFinalStatusProperty(testcase, "skip", MissingPropertiesPlacement.APPEND_TO_TESTCASE);
105+
default -> {}
79106
}
80107
}
81108

0 commit comments

Comments
 (0)