fix(benchmark_runner): record failed-item exceptions on TestRecord so…#1552
Conversation
… # errors is accurate (mlcommons#1353) _make_test_record hardcoded test_item_exceptions=[], so exceptions captured on failed items never reached the TestRecord. HazardScore.score() sums len(test_record.test_item_exceptions) into HazardScore.exceptions, which feeds the '# errors' column of the results table, so it always read 0 even when items raised. Populate test_item_exceptions from run.failed_items_for(sut, test), one TestItemExceptionRecord per failed item carrying exceptions (mirrors simple_test_runner). Adds a regression test plus a clean-run test.
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
|
recheck |
superdosh
left a comment
superdosh
left a comment
There was a problem hiding this comment.
Thanks for this contribution! We've added some better clarity on the desired outcome in the card, and I've suggested a way to address below.
| test_item_exceptions = [ | ||
| TestItemExceptionRecord( | ||
| test_item=item.test_item, | ||
| error_message=str(item.exceptions[-1]), |
There was a problem hiding this comment.
Per discussion in #1353 , let's simplify this to remove if exceptions in the loop below, but for error_message and cause below, add something like str(item.exceptions[-1]) if item.exceptions else "".
This way we'll count all failed items and not exceptions-only.
…view remove the 'if item.exceptions' filter and default error_message/cause to '' for failed items without a captured exception, so HazardScore.exceptions / the '# errors' column count every failed item. +test. Signed-off-by: WatchTree-19 <119982314+WatchTree-19@users.noreply.github.com>
GitHub Actions
Inactive
|
thanks @superdosh - done. dropped the |
2 participants
… # errors is accurate (#1353)
_make_test_record hardcoded test_item_exceptions=[], so exceptions captured on failed items never reached the TestRecord. HazardScore.score() sums len(test_record.test_item_exceptions) into HazardScore.exceptions, which feeds the '# errors' column of the results table, so it always read 0 even when items raised.
Populate test_item_exceptions from run.failed_items_for(sut, test), one TestItemExceptionRecord per failed item carrying exceptions (mirrors simple_test_runner). Adds a regression test plus a clean-run test.