Ur test summary python#22091
Draft
kekaczma wants to merge 85 commits into
Draft
Conversation
Add comprehensive test result logging to Unified Runtime tests similar to SYCL E2E tests: - Enhanced LIT_OPTS with --show-unsupported, --show-pass, --show-xfail to display test status details - Added --no-progress-bar for cleaner CI output - Redirect test output to log files (adapter_tests.log, conformance_tests.log) - Added steps to display full logs in collapsible groups - Added steps to extract and report test summaries (Failed Tests, Unexpectedly Passed Tests, Unresolved Tests, Timed Out Tests, Testing Time) - Report summaries both in step output and GitHub Step Summary This improves visibility into which tests pass, fail, or are skipped, making it easier to diagnose CI issues.
GoogleTest format used by UR doesn't print full test lists like ShTest format in E2E. Parse LIT output to extract and display: - Passed tests list (collapsible) - Failed tests list (expanded if present) - Unsupported tests list (collapsible) - Expected failures list (collapsible) Includes statistics with emoji indicators for better readability.
- Parse LIT test output in single-pass awk for efficiency (O(n) vs O(n²)) - Generate GitHub step summary with collapsible test lists - Show TIMEOUT/FAIL/XPASS tests expanded by default for visibility - Collapse PASS/XFAIL/UNSUPPORTED/SKIPPED for long test lists - Simplify LIT_OPTS using `-a` flag for all test statuses - Add security hardening: printf instead of echo, input validation - Add error handling and graceful degradation for missing logs - Memory optimization: use arrays instead of string concatenation Tested with all LIT statuses: PASS, FAIL, XFAIL, XPASS, TIMEOUT, UNSUPPORTED, UNRESOLVED, SKIPPED
- Format test lists in code blocks instead of bullets for better readability - Remove misleading raw log steps that showed truncated output (535 vs 5536) - Add check for empty log files with warning message - Add debug output showing current directory and log file status - Add error handling to show warnings if summary generation fails - Explicitly call bash interpreter for script execution This fixes issue where test lists appeared with asterisks and numbers didn't match (showed 535 tests when actually 5536 were run due to GitHub Actions output truncation).
…lidation test suite
…gories, run CI validation
- Move statistics and test lists to composite actions - Cut full log at asterisks (remove lists/stats duplication) - Report failures shows only error details - Show statistics and collapsed sections in composite actions - Fix clang-format issues in ci_validation_tests.cpp This eliminates information duplication across workflow and composite actions while preserving all test details.
- Pipe tee directly to awk to avoid showing output twice - Change Report failures pattern to stop at test list headers instead of 'Testing Time' (which doesn't exist in GoogleTest) - Now shows error details + statistics but not test lists (lists are in collapsed sections)
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
… root Root cause: LIT runs in BUILD_DIR/test/conformance/ and writes XML there, but Python script was looking for it in workspace root. Solution: Use full path BUILD_DIR/test/conformance/conformance_results.xml when calling Python script. Add check if XML exists before passing to script. This fixes L0_V2+UR_L0 config showing 0/3886 skipped test names.
Adds --time-tests, --show-flakypass, --show-skipped flags to improve test reporting and debugging capabilities for Unified Runtime tests.
Required by ur_test_summary.py for secure XML parsing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.