Diagnose log injection smoke test flakiness instead of masking it (#11075)

# What Does This Do

Adds diagnostic instrumentation to the `check raw file injection` smoke test so the next CI failure tells us the root cause instead of a bare "Condition not satisfied after 30s" with `traceCount=0`.

Changes to `LogInjectionSmokeTest`:
1. **`waitForTraceCountAlive`** — checks process liveness on every poll iteration; if the process dies, fails immediately with exit code + last 20 lines of process output
2. **Enriched timeout errors** — on timeout, dumps: process alive?, traceCount, RC polls received, last 30 lines of process output
3. **Reorder `waitForTraceCount(4)` before `waitFor`** + assert `waitFor` return value

# Motivation

CI Visibility data for the last 30 days on master shows 10 failures of `check raw file injection`:

| Failure mode | Count | Line | Duration | Root cause |
|---|---|---|---|---|
| `traceCount=0` at `waitForTraceCount(2)` | 9/10 | 368 | 30.3s | Unknown — no diagnostics |
| `logLines.size()=3` at `assertRawLogLinesWithInjection` | 1/10 | 229 | 8.3s | Incomplete log file |

The failure distribution is **bimodal** — successful runs complete in 3.5-8.7s (80 data points, zero above 9s), while failures sit at exactly 30.3s. There is nothing in between. This means the process either works or is totally broken — a timeout increase would just delay the same failure.

```
<9s:  ████████████████████████████████████████  80/80 passes
9-30s:                                           0 runs
30s:  █████████                                  9/10 failures (at timeout)
```

The current test is blind during the wait — it just polls `traceCount` in a loop. We don't know if the process crashed, hung during agent init, failed to connect to the test server, or something else entirely. This PR makes the next failure self-diagnosing.

**Example output when process crashes:**
```
Process exited with code 1 while waiting for 2 traces (received 0, RC polls: 3).
Last process output:
[dd.trace ...] ERROR ... NullPointerException during instrumentation
...
```

**Example output on timeout (process alive but not sending traces):**
```
Timed out waiting for 2 traces after 30s. traceCount=0, process.alive=true, RC polls received: 142.
Last process output:
[dd.trace ...] DEBUG ... Still loading instrumentations...
...
```

# Additional Notes

- Only `LogInjectionSmokeTest.groovy` is changed
- No timeout increase — the 30s `defaultPoll` is kept as-is
- All 11 historically flaky backends pass locally
- `rcClientMessages.size()` tells us whether the agent connected to the test server at all (RC polls hit `/v0.7/config` every 200ms)

# Contributor Checklist

- [x] Format the title according to [the contribution guidelines](https://github.com/DataDog/dd-trace-java/blob/master/CONTRIBUTING.md#title-format)
- [x] Assign the `type:` and (`comp:` or `inst:`) labels
- [x] Avoid using `close`, `fix`, or [any linking keywords](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword) when referencing an issue
- [x] Update the [CODEOWNERS](https://github.com/DataDog/dd-trace-java/blob/master/.github/CODEOWNERS) file on source file addition, migration, or deletion — N/A (no file additions)
- [x] Update [public documentation](https://docs.datadoghq.com/tracing/trace_collection/library_config/java/) with any new configuration flags or behaviors — N/A (test-only change)

tag: no release notes
tag: ai generated

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: devflow.devflow-routing-intake <devflow.devflow-routing-intake@kubernetes.us1.ddbuild.io>

Author: bm1549

dd-smoke-tests/log-injection/src/test/groovy/datadog/smoketest/LogInjectionSmokeTest.groovy

@@ -347,6 +347,60 @@ abstract class LogInjectionSmokeTest extends AbstractSmokeTest {
     return logEvent[key]
   }
 
+  /**
+   * Like {@link AbstractSmokeTest#waitForTraceCount} but checks process liveness on every poll
+   * iteration and dumps diagnostic state on failure, so CI failures produce actionable output
+   * instead of a bare "Condition not satisfied" after a 30s timeout.
+   */
+  int waitForTraceCountAlive(int count) {
+    try {
+      defaultPoll.eventually {
+        if (traceDecodingFailure != null) {
+          throw traceDecodingFailure
+        }
+        // Check the count BEFORE liveness — the process may have exited normally
+        // after delivering all traces, and we don't want to treat that as a failure.
+        if (traceCount.get() >= count) {
+          return
+        }
+        if (testedProcess != null && !testedProcess.isAlive()) {
+          def lastLines = tailProcessLog(20)
+          // RuntimeException (not AssertionError) so PollingConditions propagates
+          // immediately instead of retrying for the full timeout.
+          throw new RuntimeException(
+          "Process exited with code ${testedProcess.exitValue()} while waiting for ${count} traces " +
+          "(received ${traceCount.get()}, RC polls: ${rcClientMessages.size()}).\n" +
+          "Last process output:\n${lastLines}")
+        }
+        assert traceCount.get() >= count
+      }
+    } catch (AssertionError e) {
+      // The default error ("Condition not satisfied after 30s") is useless — enrich with diagnostic state
+      def alive = testedProcess?.isAlive()
+      def lastLines = tailProcessLog(30)
+      throw new AssertionError(
+      "Timed out waiting for ${count} traces after ${defaultPoll.timeout}s. " +
+      "traceCount=${traceCount.get()}, process.alive=${alive}, " +
+      "RC polls received: ${rcClientMessages.size()}.\n" +
+      "Last process output:\n${lastLines}", e)
+    }
+    traceCount.get()
+  }
+
+  private String tailProcessLog(int lines) {
+    try {
+      def logFile = new File(logFilePath)
+      if (!logFile.exists()) {
+        return "(log file does not exist: ${logFilePath})"
+      }
+      def allLines = logFile.readLines()
+      def tail = allLines.size() > lines ? allLines[-lines..-1] : allLines
+      return tail.join("\n")
+    } catch (Exception e) {
+      return "(failed to read log: ${e.message})"
+    }
+  }
+
   def parseTraceFromStdOut( String line ) {
     if (line == null) {
       throw new IllegalArgumentException("Line is null")
@@ -365,7 +419,7 @@ abstract class LogInjectionSmokeTest extends AbstractSmokeTest {
   @Flaky(condition = () -> JavaVirtualMachine.isIbm8() || JavaVirtualMachine.isOracleJDK8())
   def "check raw file injection"() {
     when:
-    def count = waitForTraceCount(2)
+    def count = waitForTraceCountAlive(2)
 
     def newConfig = """
         {"lib_config":
@@ -374,14 +428,15 @@ abstract class LogInjectionSmokeTest extends AbstractSmokeTest {
      """.toString()
     setRemoteConfig("datadog/2/APM_TRACING/config_overrides/config", newConfig)
 
-    count = waitForTraceCount(3)
+    count = waitForTraceCountAlive(3)
 
     setRemoteConfig("datadog/2/APM_TRACING/config_overrides/config", """{"lib_config":{}}""".toString())
 
-    testedProcess.waitFor(TIMEOUT_SECS, SECONDS)
-    def exitValue = testedProcess.exitValue()
+    // Wait for all 4 traces before waiting for process exit to ensure trace delivery is confirmed
+    count = waitForTraceCountAlive(4)
 
-    count = waitForTraceCount(4)
+    assert testedProcess.waitFor(TIMEOUT_SECS, SECONDS) : "Process did not exit within ${TIMEOUT_SECS}s"
+    def exitValue = testedProcess.exitValue()
 
     def logLines = outputLogFile.readLines()
     println "log lines: " + logLines