Fix Cloud Hypervisor live migration nextest filtering by vyadavmsft · Pull Request #4563 · microsoft/lisa

vyadavmsft · 2026-06-26T20:35:45Z

Run Cloud Hypervisor live migration tests through the CLI path and construct the nextest filter as a shell-safe expression. Preserve the base test filter while excluding the serial-only live migration test from the parallel run.

Description

Related Issue

Type of Change

Checklist

Description is filled in above
No credentials, secrets, or internal details are included
Peer review requested (if not, add required peer reviewers after raising PR)
Tests executed and results posted below

Test Validation

Key Test Cases:

Impacted LISA Features:

Tested Azure Marketplace Images:

Test Results

Image	VM Size	Result
		PASSED / FAILED / SKIPPED

github-actions · 2026-06-26T20:37:26Z

🤖 AI Test Selection

No test cases were selected for this PR.

Copilot

Pull request overview

This PR updates the Cloud Hypervisor test harness to run live migration tests through the CLI path and to build a shell-safe nextest filter expression while preserving the base filter behavior.

Changes:

Route live migration execution through the integration CLI test type and apply a nextest --filterset-based filter.
Add helper logic to construct a combined nextest filterset (base substring + include + exclude).
Adjust subtest listing and run orchestration to support cli_test_type / cli_test_filter overrides.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
lisa/microsoft/testsuites/cloud_hypervisor/ch_tests.py	Switch live migration run to the CLI integration path and pass `only/skip` plus CLI filtering parameters.
lisa/microsoft/testsuites/cloud_hypervisor/ch_tests_tool.py	Add nextest filterset construction and CLI-path execution for filtered runs, plus supporting subtest prep parameters.

Run Cloud Hypervisor live migration tests through the CLI path and construct the nextest filter as a shell-safe expression. Preserve the base test filter while excluding the serial-only live migration test from the parallel run.

github-actions · 2026-06-26T20:43:40Z

🤖 AI Test Selection

No test cases were selected for this PR.

Replace the custom bash argument quoting helper with shlex.quote and quote the generated bash script as a whole before passing it to bash -lc. Remove the redundant cli_test_type override by running live migration through the integration test type and selecting live migration cases with the nextest filter.

github-actions · 2026-06-29T19:45:19Z

🤖 AI Test Selection

No test cases were selected for this PR.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Derive the expected subtest set when running Cloud Hypervisor tests with a nextest CLI filter instead of passing an empty set into result processing. Fail fast if the filter and include/exclude lists select no subtests so a zero-test run cannot pass silently.

github-actions · 2026-06-29T20:51:43Z

🤖 AI Test Selection

No test cases were selected for this PR.

anirudhrb · 2026-07-01T04:45:23Z

        ref: str = "",
        only: Optional[List[str]] = None,
        skip: Optional[List[str]] = None,
+        only_test_prefixes: Optional[List[str]] = None,


Is this unused? I don't see any caller passing this.

Good catch. This was leftover from an earlier live-migration filtering approach. The current [cli_test_filter] path now derives the filtered subtest set directly, and no caller passes only_test_prefixes, so I removed it.

Drop the only_test_prefixes parameter from the Cloud Hypervisor test runner because no caller supplies it. Remove the corresponding private prefix-filter branches so the run_tests signature only exposes supported include, exclude, and CLI substring filters.

github-actions · 2026-07-01T19:04:08Z

🤖 AI Test Selection

No test cases were selected for this PR.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

+            test_script_args = f"--test-filter {shlex.quote(nextest_filter)}"
+            cmd_args = (
+                f"tests --hypervisor {hypervisor} --{test_type} -- "
+                f"{test_script_args}"
+            )


CH integration tests run under cargo-nextest, which rejects the libtest '-- -- --list' flag ('failed to parse test binary arguments --list: arguments are unsupported', exit 96). _list_subtests therefore returned an empty list and _prepare_filtered_subtests failed with 'No Cloud Hypervisor integration subtests matched filter', even though the tests exist and the nextest run path works. List subtests with 'cargo nextest list --message-format json' instead, run in the dev container via 'dev_cli.sh shell' and mirroring the integration build env (mshv feature, devcli_testenv cfg). The return value stays the same List[str] of '<module>::<test>' names, so the runbook-driven include/exclude (only/skip/cli_test_filter), the no-match validation, and per-subtest result tracking are unchanged.

github-actions · 2026-07-09T00:38:25Z

🤖 AI Test Selection

No test cases were selected for this PR.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

+    @staticmethod
+    def _parse_nextest_subtest_list(stdout: str) -> List[str]:
+        # "cargo nextest list --message-format json" prints a JSON document with
+        # a "rust-suites" map; each suite has a "testcases" map keyed by the test
+        # name (e.g. "live_migration::test_live_migration_local"). Build progress
+        # is written to stderr, so stdout starts at the first '{'. Return the
+        # test names to preserve the previous "<module>::<test>" list contract.
+        brace = stdout.find("{")
+        if brace == -1:
+            return []
+        try:
+            data = json.loads(stdout[brace:])
+        except json.JSONDecodeError:
+            return []
+        subtests: List[str] = []
+        for suite in data.get("rust-suites", {}).values():
+            subtests.extend(suite.get("testcases", {}).keys())
+        return subtests


The previous fix listed subtests via 'dev_cli.sh shell -- RUSTFLAGS=..(--cfg devcli_testenv).. cargo nextest list'. dev_cli.sh's shell command runs its arguments as 'bash -c \$*', which re-joins them on spaces and drops quoting, so the space inside the RUSTFLAGS value was split: bash set RUSTFLAGS=--cfg and then tried to run 'devcli_testenv' as a command (exit 127, 'devcli_testenv: command not found'). cargo never ran, the subtest list came back empty, and _prepare_filtered_subtests still failed with 'No subtests matched filter'. Use the space-free 'RUSTFLAGS=--cfg=devcli_testenv' form so it survives the bash -c hand-off, and make the JSON parser scan each '{' for the nextest object with a 'rust-suites' map, since dev_cli.sh interleaves docker pull and cargo build output on the same stream.

github-actions · 2026-07-09T01:15:38Z

🤖 AI Test Selection

No test cases were selected for this PR.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

    def _list_subtests(self, hypervisor: str, test_type: str) -> List[str]:
-        cmd_args = f"tests --hypervisor {hypervisor} --{test_type} -- -- --list"
+        # Cloud Hypervisor's integration tests run under cargo-nextest, which
+        # rejects the libtest "-- -- --list" flag ("failed to parse test binary
+        # arguments `--list`: arguments are unsupported"). Enumerate the subtests
+        # with "cargo nextest list" instead. The return value is still the same
+        # List[str] of "<module>::<test>" names, so the runbook-driven
+        # include/exclude (only/skip/cli_test_filter) and per-subtest result


… of failing Cloud Hypervisor gates every live migration integration test behind cfg(not(feature = "mshv")). The mshv integration build (cargo nextest ... --features mshv) compiles them out, so _list_subtests correctly returns no live_migration tests and the "live_migration" substring filter matches nothing. _prepare_filtered_subtests previously called fail() in that case, turning a legitimately-empty selection into a hard AssertionError. This made verify_cloud_hypervisor_live_migration_tests fail on mshv even though the tests do not exist in that build (they had been silently passing before the subtest-list assertion was added). Raise SkippedException instead so an unavailable test family (live migration on mshv, or an include/exclude filter that removes all matches) is reported as a clean skip per LISA conventions. KVM builds (no mshv feature) still compile these tests, so the filter matches and they run normally.

github-actions · 2026-07-09T06:17:26Z

🤖 AI Test Selection

No test cases were selected for this PR.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

+        filterset = f"test(~{self._escape_nextest_filter_matcher(base_filter)})"
+
+        if only:
+            included_tests = "|".join(
+                f"test(={self._escape_nextest_filter_matcher(test_name)})"
+                for test_name in only
+            )
+            filterset = f"({filterset}&({included_tests}))"
+
+        if skip:
+            for skipped_test in skip:
+                skipped_filter = self._escape_nextest_filter_matcher(skipped_test)
+                filterset = f"{filterset}-test(={skipped_filter})"
+


+        filtered_subtests = [
+            subtest for subtest in subtests if cli_test_filter in subtest
+        ]
+        if only is not None:
+            filtered_subtests = [
+                subtest for subtest in filtered_subtests if subtest in only
+            ]
+        if skip is not None:
+            filtered_subtests = [
+                subtest for subtest in filtered_subtests if subtest not in skip
+            ]


@@ -1380,7 +1480,27 @@ def _cache_vmm_version(self) -> None:
            self._log.debug(f"Could not cache VMM version: {e}")

    def _list_subtests(self, hypervisor: str, test_type: str) -> List[str]:


Copilot AI review requested due to automatic review settings June 26, 2026 20:35

vyadavmsft requested review from anirudhrb and pupacha as code owners June 26, 2026 20:35

Copilot started reviewing on behalf of vyadavmsft June 26, 2026 20:36 View session

Copilot AI reviewed Jun 26, 2026

View reviewed changes

Comment thread lisa/microsoft/testsuites/cloud_hypervisor/ch_tests_tool.py Outdated

Fix Cloud Hypervisor live migration nextest filtering

24a16f0

Run Cloud Hypervisor live migration tests through the CLI path and construct the nextest filter as a shell-safe expression. Preserve the base test filter while excluding the serial-only live migration test from the parallel run.

vyadavmsft force-pushed the ch_52 branch from db66144 to 24a16f0 Compare June 26, 2026 20:42

LiliDeng previously approved these changes Jun 29, 2026

View reviewed changes

anirudhrb requested changes Jun 29, 2026

View reviewed changes

Comment thread lisa/microsoft/testsuites/cloud_hypervisor/ch_tests_tool.py Outdated

Comment thread lisa/microsoft/testsuites/cloud_hypervisor/ch_tests_tool.py Outdated

Copilot AI review requested due to automatic review settings June 29, 2026 19:44

vyadavmsft dismissed LiliDeng’s stale review via deefd9a June 29, 2026 19:44

Copilot started reviewing on behalf of vyadavmsft June 29, 2026 19:45 View session

Copilot AI reviewed Jun 29, 2026

View reviewed changes

Comment thread lisa/microsoft/testsuites/cloud_hypervisor/ch_tests_tool.py

anirudhrb requested changes Jul 1, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 1, 2026 19:03

Copilot started reviewing on behalf of vyadavmsft July 1, 2026 19:03 View session

Copilot AI reviewed Jul 1, 2026

View reviewed changes

Comment thread lisa/microsoft/testsuites/cloud_hypervisor/ch_tests_tool.py

Comment on lines +501 to +505

test_script_args = f"--test-filter {shlex.quote(nextest_filter)}"

cmd_args = (

f"tests --hypervisor {hypervisor} --{test_type} -- "

f"{test_script_args}"

)

vyadavmsft requested a review from anirudhrb July 6, 2026 00:19

Copilot AI review requested due to automatic review settings July 9, 2026 00:37

Copilot started reviewing on behalf of vyadavmsft July 9, 2026 00:38 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 9, 2026 01:15

Copilot started reviewing on behalf of vyadavmsft July 9, 2026 01:15 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 9, 2026 06:16

Copilot started reviewing on behalf of vyadavmsft July 9, 2026 06:17 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

		@@ -1380,7 +1480,27 @@ def _cache_vmm_version(self) -> None:
		self._log.debug(f"Could not cache VMM version: {e}")

		def _list_subtests(self, hypervisor: str, test_type: str) -> List[str]:

Uh oh!

Conversation

vyadavmsft commented Jun 26, 2026

Description

Related Issue

Type of Change

Checklist

Test Validation

Test Results

Uh oh!

github-actions Bot commented Jun 26, 2026

🤖 AI Test Selection

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

github-actions Bot commented Jun 26, 2026

🤖 AI Test Selection

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 29, 2026

🤖 AI Test Selection

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

github-actions Bot commented Jun 29, 2026

🤖 AI Test Selection

Uh oh!

anirudhrb Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

vyadavmsft Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 1, 2026

🤖 AI Test Selection

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

github-actions Bot commented Jul 9, 2026

🤖 AI Test Selection

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

github-actions Bot commented Jul 9, 2026

🤖 AI Test Selection

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

github-actions Bot commented Jul 9, 2026

🤖 AI Test Selection

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants