Add --test_parallel to reduce CTest concurrency for ASan builds (#28675)

## Description

The `windows_x64_asan / build_x64` CI pipeline has been failing with OOM
(out-of-memory) because ASan-instrumented test binaries consume
significantly more memory than normal builds, and CTest was running them
at full CPU-count parallelism.

This PR adds a `--test_parallel` argument to `build.py` that allows
CTest concurrency to be configured independently from the build
parallelism (`--parallel`). It then uses `--test_parallel 4` in the
Windows x64 ASan workflow to cap test execution to 4 parallel jobs,
preventing OOM while keeping build parallelism at full speed.

## Motivation and Context

- ASan instrumentation inflates per-process memory usage by ~2-3x.
- The existing `--parallel` flag controls both MSBuild and CTest
concurrency together; there was no way to keep fast parallel builds
while limiting test concurrency.
- The CI runner has limited memory, and running all tests in parallel
under ASan exceeded available RAM.

## Changes

| File | Change |
|------|--------|
| `tools/ci_build/build_args.py` | Add `--test_parallel` argument
(default: `None`, falls back to `--parallel`) |
| `tools/ci_build/build.py` | Add `number_of_test_parallel_jobs()`
helper; use it for CTest `--parallel`; validate negative values |
| `.github/workflows/windows_build_x64_asan.yml` | Pass `--test_parallel
4` to cap ASan test concurrency |

## Testing

- `python -m py_compile tools/ci_build/build.py
tools/ci_build/build_args.py` — passes
- `python tools/ci_build/build.py --help` shows the new
`--test_parallel` option
- `git diff --check` — no whitespace issues
- When `--test_parallel` is omitted, behavior is unchanged (falls back
to `--parallel` value)

Author: tianleiwu

.github/workflows/windows_build_x64_asan.yml

@@ -44,4 +44,4 @@ jobs:
           @echo off
           echo %PATH%
           python -m pip install -r "%GITHUB_WORKSPACE%\tools\ci_build/github/windows\python\requirements.txt"
-          python "%GITHUB_WORKSPACE%\tools\ci_build\build.py" --config Debug --build_dir "%RUNNER_TEMP%\build" --skip_submodule_sync --parallel --use_vcpkg --use_vcpkg_ms_internal_asset_cache --cmake_generator "Visual Studio 17 2022" --disable_memleak_checker --enable_address_sanitizer
+          python "%GITHUB_WORKSPACE%\tools\ci_build\build.py" --config Debug --build_dir "%RUNNER_TEMP%\build" --skip_submodule_sync --parallel --test_parallel 4 --use_vcpkg --use_vcpkg_ms_internal_asset_cache --cmake_generator "Visual Studio 17 2022" --disable_memleak_checker --enable_address_sanitizer

tools/ci_build/build.py

@@ -205,6 +205,12 @@ def number_of_parallel_jobs(args):
     return os.cpu_count() if args.parallel == 0 else args.parallel
 
 
+def number_of_test_parallel_jobs(args):
+    if args.test_parallel is None:
+        return number_of_parallel_jobs(args)
+    return os.cpu_count() if args.test_parallel == 0 else args.test_parallel
+
+
 def number_of_nvcc_threads(args):
     if args.nvcc_threads >= 0:
         return args.nvcc_threads
@@ -1728,7 +1734,7 @@ def run_onnxruntime_tests(args, source_dir, ctest_path, build_dir, configs):
                 test_output = f"--gtest_output=xml:{cwd}/{exe}.{config}.results.xml"
                 run_subprocess([os.path.join(cwd, exe), test_output], cwd=cwd, dll_path=dll_path)
         else:
-            num_parallel_jobs = number_of_parallel_jobs(args)
+            num_parallel_jobs = number_of_test_parallel_jobs(args)
             ctest_cmd = [
                 ctest_path,
                 "--build-config",
@@ -2574,6 +2580,9 @@ def main():
         build_targets(args, cmake_path, build_dir, configs, num_parallel_jobs, args.targets)
 
     if args.test:
+        if args.test_parallel is not None and args.test_parallel < 0:
+            raise BuildError(f"Invalid test parallel job count: {args.test_parallel}")
+
         if args.enable_onnx_tests:
             source_onnx_model_dir = "C:\\local\\models" if is_windows() else "/data/models"
             setup_test_data(source_onnx_model_dir, "models", build_dir, configs)

tools/ci_build/build_args.py

@@ -228,6 +228,12 @@ def add_testing_args(parser: argparse.ArgumentParser) -> None:
     parser.add_argument("--skip_winml_tests", action="store_true", help="Explicitly disable WinML related tests.")
     parser.add_argument("--skip_nodejs_tests", action="store_true", help="Explicitly disable Node.js binding tests.")
     parser.add_argument("--ctest_timeout", default="10800", help="Timeout provided to CTest --timeout (seconds).")
+    parser.add_argument(
+        "--test_parallel",
+        default=None,
+        type=int,
+        help="Max CTest parallel jobs. Defaults to --parallel. Optional value 0 uses num CPUs.",
+    )
     parser.add_argument("--enable_transformers_tool_test", action="store_true", help="Enable transformers tool test.")
     parser.add_argument("--build_micro_benchmarks", action="store_true", help="Build ONNXRuntime micro-benchmarks.")
     parser.add_argument("--code_coverage", action="store_true", help="Generate code coverage report (Android only).")