When a test job launched from alloy.py fails, the build log only contains a high-level summary (the STABILITY REPORT) followed by a diff of fail_db.txt listing the newly failing tests. For example:
_________________________STABILITY REPORT_________________________
Running native - AVX2
Using test compiler: Intel(r) Implicit SPMD Program Compiler (Intel(r) ISPC), 1.31.0dev (build commit 2857de55446851ee @ 20260622, LLVM 21.1.8)
Using C/C++ compiler: Microsoft (R) C/C++ Optimizing Compiler Version 19.44.35228 for x64
x86-64 generic-i32x4 unspec Windows LLVM 21.1 cl O2 obj. Fails: 12, New fails: 12. 00h09m44sec.
NO NEW RUNFAILS
NEW COMPFAILS: 12
NO NEW PASSES RUNFAILS
NO NEW PASSES COMPFAILS
The summary tells you which tests failed, but not why. The actual diagnostics from run_tests.py (compiler errors, mismatched output, stderr, etc.) are not included in the log, so there is no straightforward way to understand the root cause from the failed job alone.
Problem:
Finding the real diagnostics is non-obvious and time-consuming. To investigate a failure you need to dig through artifacts manually, even though run_tests.py already produced the detailed output during the run.
Suggested improvement:
Make the per-test diagnostics from run_tests.py easily discoverable from the failed alloy.py job — for example, by including the failing tests' compile/run output in the log (or as a saved artifact) alongside the stability report.
When a test job launched from
alloy.pyfails, the build log only contains a high-level summary (theSTABILITY REPORT) followed by a diff offail_db.txtlisting the newly failing tests. For example:The summary tells you which tests failed, but not why. The actual diagnostics from
run_tests.py(compiler errors, mismatched output, stderr, etc.) are not included in the log, so there is no straightforward way to understand the root cause from the failed job alone.Problem:
Finding the real diagnostics is non-obvious and time-consuming. To investigate a failure you need to dig through artifacts manually, even though
run_tests.pyalready produced the detailed output during the run.Suggested improvement:
Make the per-test diagnostics from
run_tests.pyeasily discoverable from the failedalloy.pyjob — for example, by including the failing tests' compile/run output in the log (or as a saved artifact) alongside the stability report.