CI: Summarise test results; capture failure logs; keep going on failure

[terryburton]

Example output for failures: https://github.com/terryburton/freeradius-server/actions/runs/25195925825

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[arr2036]

Not a fan. The failures are sprinkled throughout the logs. Current failure behaviour means the failing test is always right there at the bottom of the logs. Now you have to go hunting.

[arr2036]

The ideal way to do this would be to aggregate the failures at the end of the run, and emit section markers so that GitHub action turns each failure into its own foldable section.

[terryburton]

Not a fan. The failures are sprinkled throughout the logs. Current failure behaviour means the failing test is always right there at the bottom of the logs. Now you have to go hunting.

This runs all test rather than stopping at the first failure, so there is no one failing test.

The ideal way to do this would be to aggregate the failures at the end of the run, and emit section markers so that GitHub action turns each failure into its own foldable section.

It aggregates extracts from the logs from all failing tests in collapsible sections at the top of the output. Is this the same as what you are referring to?

It also makes the full output available as an asset for download.

[arr2036]

I don't see the collapsable sections, only the inline failures, am I missing something?

KEYWORD-TEST xlat-subst-error-empty-patt
KEYWORD-TEST xlat-subst-error-patt
KEYWORD-TEST xlat-utf8
KEYWORD-TEST xlat-uuid
match failed src/tests/unit/calc.txt:238
  EXPECTED : 0xdeadbeefdeadbeef
  GOT      : 0xabcdabcdabcdabcd
               ^ differs here (2)
unit_test_attribute failed in an input file - 
UNIT-TEST calc.txt
TZ=GMT build/make/jlibtool --silent --mode=execute ./build/bin/local/unit_test_attribute  -F ./src/tests/fuzzer-corpus -D ./share/dictionary -d /__w/freeradius-server/freeradius-server/src/tests/unit -o \"build/tests/unit/calc.txt\" src/tests/unit/calc.txt src/tests/unit/calc.txt
make: *** [src/tests/unit/all.mk:99: build/tests/unit/calc.txt] Error 1
AUTH-TEST chap

Ideally it'd emit a failed marker inline, then append the test name, log file, reproducer, to a list, and emit each entry in a different section at the end of make test.

[arr2036]

Oh I see it in the test summary.

[arr2036]

Now I see what you're doing, ok this is nicer.

Still think reproducers would be useful in addition to logs.

Successes are somewhat useful because you know what run, but in their uncollapsed state it means a lot of scrolling to see errors in the next matrix entry, could they be hidden in a collapsed section? Could we uncollapse the failures? Is there any way we could link back to the full logs from the failure summary?

[terryburton]

New output will appear here: https://github.com/terryburton/freeradius-server/actions/runs/25215109393

(Slow because there's a load of repo infrastructure failure at the moment.)