Add test filter standardisation for hip-tests#5524
Draft
dileepr1 wants to merge 3 commits into
Draft
Conversation
Mirrors the test-filter standardisation pattern from TheRock#4694 (rccl) for hip-tests, paired with the rocm-systems branch users/dravindr/tf_hiptests (ROCm/rocm-systems#6315), which wires tier (quick/standard/comprehensive/full) and arch-exclude tags into hip_test_config.hh at build time and exposes them as CTest labels via catch_discover_tests(ADD_TAGS_AS_LABELS ... DISCOVERY_MODE PRE_TEST). - fetch_test_configurations.py: hip-tests test_script -> test_runner.py so the generic runner drives `ctest -L <tier>` from the installed Catch2 directory using the labels baked into the binaries. - test_runner.py: add hip-tests COMPONENT_OVERRIDES with test_dir = share/hip/catch_tests (not bin/<component>/) LD_LIBRARY_PATH = ROCM_PATH/lib (so the Catch2 binaries dlopen HIP). artifact-core-hiptests.toml already includes `share/hip/**`, so the install-tree `CTestTestfile.cmake` and Catch2 binaries are already bundled - no toml change needed. Follow-ups (not blocking): - The legacy `test_hiptests.py` is kept as a fallback per the test-filter standardisation skill; remove it once the new runner has soaked. - The rocm-systems YAML uses `<arch>_<os>_exclude` labels (e.g. `gfx950_linux_exclude`) instead of the runner's `ex_gpu_<arch>` convention, so on gfx94X / gfx950 Linux those known-broken tests will not yet be auto-excluded. Either rename the YAML labels to `ex_gpu_<arch>` (with the usual inclusive semantics) or extend `test_runner.py` to recognise the new convention - tracked separately. Co-authored-by: Cursor <cursoragent@cursor.com>
Generalise the exclude-label handling so per-arch / per-OS known-broken
tests can be expressed purely in test_categories.yaml without baking the
convention into individual component runners.
Recognised label shapes:
gfx950_exclude -> OS-agnostic, exact arch
gfx94X_linux_exclude -> Linux-only, wildcard arch
gfx110X_windows_exclude -> Windows-only, wildcard arch
A new helper find_matching_arch_exclude_labels() parses each discovered
*_exclude label with a strict regex anchored to `gfx<...>`, skips labels
whose OS suffix does not match the current host, and uses the existing
find_matching_gpu_arch() wildcard semantics so `gfx94X` correctly
matches gfx940/941/942 while `gfx950` does not.
build_ctest_command() OR's matching labels into the same -LE regex used
for {category}_exclude, preserving "any-match excludes" semantics.
Motivation: hip-tests (rocm-systems#6315) injects tier and arch-exclude
tags into hip_test_config.hh at compile time via
catch_discover_tests(ADD_TAGS_AS_LABELS ... PRE_TEST). The tier labels
worked out of the box, but `gfx950_linux_exclude` /
`gfx110X_windows_exclude` were collected then dropped on the floor,
which would have regressed gfx950 / gfx94X Linux runs vs the legacy
test_hiptests.py TEST_TO_IGNORE table.
The regex is strictly anchored to a `gfx` prefix and a known OS suffix,
so plain `<tier>_exclude` labels and any other component's existing
`*_exclude` labels are unaffected. Verified against eight scenarios
covering exact-arch, wildcard-arch, OS-mismatched, plain tier-exclude
and empty-gpu-arch shapes.
Co-authored-by: Cursor <cursoragent@cursor.com>
The hip-tests Windows CI job sets TEST_COMPONENT="hip-tests (PAL)" (and
optionally "hip-tests (ROCR)") from fetch_test_configurations.py, where
"(PAL)" / "(ROCR)" are matrix-variant tags rather than distinct test
components. The runner was passing those literal strings into both
COMPONENT_DIR_MAPPING and COMPONENT_OVERRIDES lookups, neither of which
keys on the variant suffix, so the "hip-tests" override silently did
not apply and TEST_DIR fell back to the non-existent
`build/bin/hip-tests (PAL)`:
Error: Test directory does not exist: build\bin\hip-tests (PAL)
https://github.com/ROCm/rocm-systems/actions/runs/26480955488/job/78042511831?pr=6315
Normalise the lookup by stripping a trailing parenthesised suffix
(`\s*\([^)]*\)\s*$`) before COMPONENT_DIR_MAPPING / COMPONENT_OVERRIDES
lookups. Both `hip-tests (PAL)` and `hip-tests (ROCR)` now resolve via
the single `hip-tests` entry to `share/hip/catch_tests`; component
names without a parenthesised suffix (miopen, rocprofiler-compute,
rocprofiler-systems, ...) keep their existing behaviour.
Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add test-filter standardisation for hip-tests, mirroring the pattern
established by TheRock#4694 (rccl).
Paired with the rocm-systems branch
users/dravindr/tf_hiptests(rocm-systems#6315),which wires tier (
quick/standard/comprehensive/full) andarch-exclude tags into
hip_test_config.hhat build time and exposesthem as CTest labels via
catch_discover_tests(ADD_TAGS_AS_LABELS ... DISCOVERY_MODE PRE_TEST).Three commits:
Add test filter standardisation for hip-tests
fetch_test_configurations.py: route hip-teststest_scriptto thegeneric
test_runner.py, which now drivesctest -L <tier>fromthe installed Catch2 directory using the labels baked into the
binaries.
test_runner.py: addhip-testsCOMPONENT_OVERRIDES:test_dir = share/hip/catch_tests(notbin/<component>/).LD_LIBRARY_PATH = ROCM_PATH/libso the Catch2 binaries candlopenHIP.artifact-core-hiptests.tomlalready includesshare/hip/**, sothe install-tree
CTestTestfile.cmakeand Catch2 binaries arealready bundled — no toml change needed.
`test_runner`: honour `<gfx_pattern>[_]_exclude` labels
known-broken tests can be expressed purely in
test_categories.yamlwithout baking the convention intoindividual component runners.
`*_exclude` label with a strict regex anchored to `gfx<...>`,
skips labels whose OS suffix does not match the current host, and
uses the existing `find_matching_gpu_arch()` wildcard semantics.
regex used for `{category}_exclude`, preserving
"any-match excludes" semantics.
`*_exclude` labels are unaffected.
`test_runner`: strip `(variant)` suffix from `TEST_COMPONENT`
(and optionally `"hip-tests (ROCR)"`) from
`fetch_test_configurations.py`, where `(PAL)` / `(ROCR)` are
matrix-variant tags rather than distinct test components. The
runner passed those literal strings into both
`COMPONENT_DIR_MAPPING` and `COMPONENT_OVERRIDES` lookups,
neither of which keys on the variant suffix, so the override
silently did not apply and `TEST_DIR` fell back to the
non-existent `build\bin\hip-tests (PAL)`.
(`\s*\([^)]\)\s$`) before the mapping / overrides lookups.
Companion PR
Test plan
the new `__exclude` label handling).
jobs (exercises the variant-suffix strip).
still parse the same way.
Made with Cursor