Skip to content

Surface run_tests.py diagnostics in alloy.py build logs when tests fail #3868

Description

@aneshlya

When a test job launched from alloy.py fails, the build log only contains a high-level summary (the STABILITY REPORT) followed by a diff of fail_db.txt listing the newly failing tests. For example:

  _________________________STABILITY REPORT_________________________
  Running native - AVX2

  Using test compiler: Intel(r) Implicit SPMD Program Compiler (Intel(r) ISPC), 1.31.0dev (build commit 2857de55446851ee @ 20260622, LLVM 21.1.8)
  Using C/C++ compiler: Microsoft (R) C/C++ Optimizing Compiler Version 19.44.35228 for x64

  x86-64  generic-i32x4 unspec Windows LLVM 21.1         cl O2 obj. Fails: 12, New fails: 12. 00h09m44sec.
  NO NEW RUNFAILS
  NEW COMPFAILS: 12
  NO NEW PASSES RUNFAILS
  NO NEW PASSES COMPFAILS

The summary tells you which tests failed, but not why. The actual diagnostics from run_tests.py (compiler errors, mismatched output, stderr, etc.) are not included in the log, so there is no straightforward way to understand the root cause from the failed job alone.

Problem:

Finding the real diagnostics is non-obvious and time-consuming. To investigate a failure you need to dig through artifacts manually, even though run_tests.py already produced the detailed output during the run.

Suggested improvement:

Make the per-test diagnostics from run_tests.py easily discoverable from the failed alloy.py job — for example, by including the failing tests' compile/run output in the log (or as a saved artifact) alongside the stability report.

Metadata

Metadata

Assignees

No one assigned

    Labels

    CI / GitHub ActionsAll issues involving build automation and GitHub actions.

    Type

    Fields

    No fields configured for Task.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions