[ALMIOPEN-1856] Simplify dnn-benchmarking engine comparison CLI (#7866)

## Summary

This PR replaces the old A/B-specific dnn-benchmarking workflow with an
explicit ordered multi-engine path. It keeps engine selection on
`--engine`, supports repeated engine IDs as distinct selections, and
allows comma-delimited plugin paths that map positionally to the
selected engines.

## Risk Assessment

Medium risk. This changes CLI behavior and removes legacy A/B flags in a
benchmark tool, but the new path is covered by parser, config,
execution, reporting, full Python tests, and gfx90a hardware smoke
validation.

## Testing Summary

- Static Python compile validation covered the changed source and tests.
- Full dnn-benchmarking pytest coverage ran on Alola gfx90a with Python
3.12 and ROCm PyTorch, including GPU-marked tests.
- Focused regression coverage includes CLI parsing, config validation,
ordered engine/plugin selection, same-engine/different-plugin paths,
absolute per-engine plugin loading, per-engine setup failures, output
shape, internal profiling plugin-path forwarding, and serialization.
- Pre-commit ran on all PR-changed files and passed.
- Alola gfx90a validation used the installed MIOpen provider from
`/opt/rocm/lib/hipdnn_plugins/engines` and ran dnn-benchmarking smoke
tests for single-engine, two-engine, and comma-delimited plugin-path
flows.

## Testing Checklist

- [x] Python compile - `python3 -m compileall -q
projects/hipdnn/tools/dnn-benchmarking/src/dnn_benchmarking
projects/hipdnn/tools/dnn-benchmarking/tests` - Status: Passed
- [x] Pre-commit on PR-changed files - `pre-commit run --files <PR
changed files>` - Status: Passed
- [x] Full dnn-benchmarking test suite -
`workspace/scripts/alola-session run -- 'cd
/home/AMD/sareeder/worktrees/rocmlibs-benchmarking-alola-test/projects/hipdnn/tools/dnn-benchmarking
&& source /workspace/.venv/bin/activate &&
DNN_BENCH_WORKSPACE=/workspace python -m pytest tests'` - ASICs: gfx90a
- Status: Passed, 683 passed, 6 skipped, 8 xfailed
- [x] Alola engine listing - `hipdnn_list_engines --plugin-dir
/opt/rocm/lib/hipdnn_plugins/engines` - ASICs: gfx90a - Status: Passed,
found `MIOPEN_ENGINE` and `MIOPEN_ENGINE_DETERMINISTIC`
- [x] Alola single-engine smoke - `python -m dnn_benchmarking --graph
graphs/sample_conv_fwd.json --warmup 1 --iters 2 --engine
1563989756945604898 --plugin-path /opt/rocm/lib/hipdnn_plugins/engines`
- ASICs: gfx90a - Status: Passed
- [x] Alola two-engine smoke - `python -m dnn_benchmarking --graph
graphs/sample_conv_fwd.json --warmup 1 --iters 5 --engine
1563989756945604898,-6748551569128940061 --plugin-path
/opt/rocm/lib/hipdnn_plugins/engines` - ASICs: gfx90a - Status: Passed
- [x] Alola comma-delimited plugin-path smoke - `python -m
dnn_benchmarking --graph graphs/sample_conv_fwd.json --warmup 1 --iters
1 --engine 1563989756945604898,-6748551569128940061 --plugin-path
/opt/rocm/lib/hipdnn_plugins/engines,/opt/rocm/lib/hipdnn_plugins/engines`
- ASICs: gfx90a - Status: Passed
- [x] PR CI - GitHub PR checks - Status: Pending

Example comparison:
```
Results:
  engine                       status  plugin_path                           kernel_mean_ms  kernel_median_ms  e2e_mean_ms  e2e_median_ms
  ---------------------------  ------  ------------------------------------  --------------  ----------------  -----------  -------------
  MIOPEN_ENGINE                passed  /opt/rocm/lib/hipdnn_plugins/engines  0.028           0.027             0.069        0.041
  MIOPEN_ENGINE_DETERMINISTIC  passed  /opt/rocm/lib/hipdnn_plugins/engines  0.026           0.026             0.042        0.037
```

## Technical Changes

- Removes legacy A/B runner/configuration code and old `--AId`, `--BId`,
`--APath`, and `--BPath` flags.
- Adds comma-delimited `--plugin-path` parsing with validation for one
shared plugin path or one path per selected engine.
- Preserves duplicate `--engine` IDs and models each requested run as an
ordered engine selection with its own plugin path.
- Uses absolute plugin loading mode for per-engine plugin-path setup so
plugin comparisons are isolated.
- Records per-engine plugin setup and handle creation failures as error
rows instead of aborting the suite.
- Adds per-engine summary rows with raw kernel/E2E means and medians;
percent deltas are intentionally not emitted because they are derivable
from the raw columns.
- Serializes median timing data and plugin paths for JSON consumers.
- Documents `hipdnn_list_engines --plugin-dir <path>` in CLI help so
users can discover engine IDs and names.
- Raises dnn-benchmarking's Python minimum to 3.12 and uses Python 3.12
tarfile data filters directly for tarball graph extraction.

---------

Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>

Author: SamuelReeder

projects/hipdnn/tools/dnn-benchmarking/pyproject.toml

@@ -3,7 +3,7 @@ name = "dnn-benchmarking"
 version = "0.1.0"
 description = "Benchmarking and validation tool for hipDNN graphs"
 readme = "README.md"
-requires-python = ">=3.9"
+requires-python = ">=3.12"
 license = {text = "MIT"}
 authors = [
     {name = "Advanced Micro Devices, Inc."},
@@ -13,9 +13,6 @@ classifiers = [
     "Intended Audience :: Developers",
     "License :: OSI Approved :: MIT License",
     "Programming Language :: Python :: 3",
-    "Programming Language :: Python :: 3.9",
-    "Programming Language :: Python :: 3.10",
-    "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",
     "Programming Language :: Python :: 3.13",
     "Topic :: Scientific/Engineering",

projects/hipdnn/tools/dnn-benchmarking/src/dnn_benchmarking/cli/ab_runner_cli.py

@@ -24,7 +24,7 @@
 
 from ..common.exceptions import GraphLoadError
 from ..config.benchmark_config import MetricsConfig, SuiteConfig
-from ..execution.suite_runner import _run_single_provider_engine
+from ..execution.suite_runner import run_single_provider_engine, set_plugin_path
 from ..graph.loader import GraphLoader
 
 
@@ -49,9 +49,18 @@ def run_internal_profiling(args: argparse.Namespace) -> int:
         )
         return 1
 
+    plugin_path = None
+    if args.plugin_path:
+        if len(args.plugin_path) != 1:
+            print(
+                "internal-profiling-run: expected exactly one --plugin-path",
+                file=sys.stderr,
+            )
+            return 1
+        plugin_path = args.plugin_path[0]
+
     try:
-        if args.plugin_path is not None:
-            hipdnn.set_engine_plugin_paths([str(args.plugin_path)])
+        set_plugin_path(hipdnn, plugin_path)
         handle = hipdnn.Handle()
     except RuntimeError as e:
         print(
@@ -74,11 +83,9 @@ def run_internal_profiling(args: argparse.Namespace) -> int:
     # the inner pass; the parent already collected basic metrics on the
     # timed pass.
     #
-    # `plugin_path` is forwarded so the child's SuiteConfig matches the
-    # parent's. hipdnn.set_engine_plugin_paths above is what actually
-    # loads the plugin today, but any future code that reads
-    # config.plugin_path from inside _run_single_provider_engine would
-    # otherwise silently see None in the child.
+    # `plugin_path` is forwarded so the child SuiteConfig matches the
+    # parent's selected engine/plugin row. The outer suite runner passes
+    # exactly one plugin path for this single-engine subprocess.
     suite_config = SuiteConfig(
         warmup_iters=args.warmup,
         benchmark_iters=args.iters,
@@ -88,11 +95,11 @@ def run_internal_profiling(args: argparse.Namespace) -> int:
         reference_provider="none",
         verbose=False,
         metrics=MetricsConfig(tier="off"),
-        plugin_path=args.plugin_path,
+        plugin_paths=[plugin_path] if plugin_path is not None else None,
     )
 
     try:
-        result = _run_single_provider_engine(
+        result = run_single_provider_engine(
             graph_path=graph_path,
             graph_json_str=json.dumps(graph_json),
             graph_name=graph_json.get("name", graph_path.stem),
@@ -101,6 +108,7 @@ def run_internal_profiling(args: argparse.Namespace) -> int:
             handle=handle,
             provider="profiling-inner",
             engine_id=engine_id,
+            plugin_path=plugin_path,
             ref_provider=None,
             validation_requested=False,
             graph_json=graph_json,

projects/hipdnn/tools/dnn-benchmarking/src/dnn_benchmarking/cli/internal_profiling.py

@@ -26,7 +26,6 @@
 
 from ..common.exceptions import GraphLoadError
 from ..reporting.reporter import Reporter
-from .ab_runner_cli import run_ab_cli
 from .internal_profiling import run_internal_profiling
 from .parser import create_parser
 from .pytorch_runner_cli import run_pytorch_cli
@@ -82,15 +81,7 @@ def main() -> int:
         return 1
 
     try:
-        if args.AId is not None or args.BId is not None:
-            if len(resolved_files) > 1:
-                reporter.print_error(
-                    "A/B testing requires a single graph file, not a glob pattern"
-                )
-                return 1
-            return run_ab_cli(args, Path(resolved_files[0]), reporter)
-
-        elif args.backend == "pytorch":
+        if args.backend == "pytorch":
             if len(resolved_files) > 1:
                 reporter.print_error(
                     "Suite mode is not supported with --backend pytorch"