Add benchmarking script (#20188) (#20188)

Summary:

Adds a standalone microbenchmark for the ImageProcessor reuse APIs and a
companion script to diff two runs, so kernel/pipeline changes (e.g. the
NEON deinterleave switch) can be measured reproducibly.

New directory xplat/executorch/extension/image/benchmark/:

* image_processor_benchmark.cpp (cxx_binary) — times process_into
(BGRA/RGBA) and process_yuv_into (NV12/NV21) over a sweep of common
input sizes × target sizes. Per cell it runs variants covering execution
path (CPU / GPU / size-default), resize mode (stretch / letterbox),
orientation (upright + 90°), cropped ROI, and the allocating process()
vs process_into(). Each row reports mean/median/p95/stddev over 100
iters (10 warmup) on a synthetic gradient input; a row that fails is
reported as ERROR rather than timed.
* Flags (all optional): --format=bgra|rgba|nv12|nv21,
--unit=cpu|gpu|default (both default to all), --out=PATH (writes a clean
results table; the input-size sweep and rotation always run). Output is
grouped under === API-section banners with a column legend, and ---
per-cell separators.
* compare_benchmarks.py (python_binary, stdlib-only) — matches rows by
(API section, input→target cell, variant) and prints per-row base / new
speedup plus a summary bucketed by execution path (CPU / GPU / default).
* README.md — usage, the build-mode caveat, and the capture→compare
workflow.
* BUCK / TARGETS / targets.bzl — build defs.

Note: benchmark only with an optimized build (-c
cxx.extra_cxxflags=-Os); the default buck2 run is -O0 and
unrepresentative.

Differential Revision: D108048181

Author: metascroy

extension/image/benchmark/BUCK

@@ -0,0 +1,5 @@
+load(":targets.bzl", "define_common_targets")
+
+oncall("executorch")
+
+define_common_targets()

extension/image/benchmark/README.md

@@ -0,0 +1,73 @@
+# ImageProcessor benchmark
+
+A microbenchmark for the `ImageProcessor` reuse APIs (`process_into` and
+`process_yuv_into`) plus a companion script to compare two runs.
+
+## What it measures
+
+`image_processor_benchmark` sweeps common input sizes × target sizes and, per
+cell, times a set of variants:
+
+- **API**: `process_into` (BGRA/RGBA) and `process_yuv_into` (NV12/NV21)
+- **execution path**: CPU, GPU, and the size-threshold default
+- **resize mode**: stretch, letterbox
+- **orientation**: upright and 90° rotate
+- **other**: cropped ROI, and the allocating `process()` vs `process_into()`
+
+Each row reports mean / median / p95 / stddev over 100 measured iterations
+(10 warmup).
+
+## Build mode matters
+
+Always benchmark an **optimized** build. The default `buck2 run` compiles at
+`-O0`, where the hand-written NEON kernels are unrepresentative. Pass `-c cxx.extra_cxxflags=-Os` to match
+how ExecuTorch ships:
+
+```bash
+buck2 run -c cxx.extra_cxxflags=-Os \
+  fbsource//xplat/executorch/extension/image/benchmark:image_processor_benchmark
+```
+
+## Options
+
+| Flag | Default | Meaning |
+|------|---------|---------|
+| `--format=bgra\|rgba\|nv12\|nv21` | all | restrict to one color / YUV format |
+| `--unit=cpu\|gpu\|default` | all | restrict to one execution path |
+| `--out=PATH` | stdout | write the results table to PATH |
+
+The input-size sweep and the rotation variant always run. Writing with `--out`
+keeps the file free of buck build-log lines (which go to stderr).
+
+## Comparing two runs
+
+Capture a baseline and a candidate, then diff them:
+
+```bash
+TARGET=fbsource//xplat/executorch/extension/image/benchmark:image_processor_benchmark
+buck2 run -c cxx.extra_cxxflags=-Os $TARGET -- --out=/tmp/base.txt
+# ... make your change ...
+buck2 run -c cxx.extra_cxxflags=-Os $TARGET -- --out=/tmp/new.txt
+
+python3 xplat/executorch/extension/image/benchmark/compare_benchmarks.py \
+  /tmp/base.txt /tmp/new.txt
+# or via buck:
+buck2 run fbsource//xplat/executorch/extension/image/benchmark:compare_benchmarks \
+  -- /tmp/base.txt /tmp/new.txt
+```
+
+`compare_benchmarks.py` matches rows by (API section, input→target cell, variant)
+and prints the per-row `base / new` speedup plus a summary bucketed by execution
+path (CPU / GPU / default). Cross-run and thermal drift shift all rows together,
+so compare the buckets against each other rather than reading any single ratio
+absolutely.
+
+For a clean A/B, capture both files back-to-back on an otherwise idle machine.
+
+## Files
+
+- `image_processor_benchmark.cpp` — the benchmark binary; buck target
+  `:image_processor_benchmark` (run with `buck2 run`)
+- `compare_benchmarks.py` — compares two result files (stdlib only); buck target
+  `:compare_benchmarks` (run with `buck2 run …:compare_benchmarks -- BASE NEW`)
+- `BUCK` / `TARGETS` / `targets.bzl` — build definitions

extension/image/benchmark/TARGETS

@@ -0,0 +1,5 @@
+load(":targets.bzl", "define_common_targets")
+
+oncall("executorch")
+
+define_common_targets()

extension/image/benchmark/compare_benchmarks.py

@@ -0,0 +1,122 @@
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+#
+# This source code is licensed under the BSD-style license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""Compare two image_processor_benchmark result files.
+
+Each input is the output of `image_processor_benchmark --out=PATH` (or its
+stdout). Rows are matched by (API section, input->target cell, variant label)
+and the per-row speedup base/new is reported.
+
+The summary buckets rows by execution path (CPU / GPU / default). Cross-run and
+thermal drift shift all rows together, so compare the buckets against each other
+rather than reading any single ratio absolutely.
+
+Usage:
+  compare_benchmarks.py BASE.txt NEW.txt [--metric=median|mean]
+"""
+
+import argparse
+import re
+import statistics
+import sys
+
+ROW_RE = re.compile(
+    r"^(?P<label>.*?)\s+mean=\s*(?P<mean>[\d.]+) ms\s+"
+    r"median=\s*(?P<median>[\d.]+) ms"
+)
+CELL_RE = re.compile(r"^\[(?P<cell>.+?)\]\s*$")
+
+
+def path_bucket(label):
+    """Bucket a variant by execution path for the summary, or None to skip."""
+    if "GPU" in label:
+        return "GPU"
+    if "def" in label:
+        return "Default"
+    if "CPU" in label:
+        return "CPU"
+    return None
+
+
+def parse(path, metric):
+    """Return {(section, cell, label): value} for the chosen metric."""
+    rows = {}
+    section = None
+    cell = None
+    with open(path) as f:
+        for line in f:
+            stripped = line.strip()
+            if "ImageProcessor::process_yuv_into" in stripped:
+                section = "process_yuv_into"
+                continue
+            if "ImageProcessor::process_into" in stripped:
+                section = "process_into"
+                continue
+            cell_m = CELL_RE.match(stripped)
+            if cell_m and "->" in stripped:
+                cell = cell_m.group("cell")
+                continue
+            row_m = ROW_RE.match(line)
+            if row_m:
+                key = (section, cell, row_m.group("label").strip())
+                rows[key] = float(row_m.group(metric))
+    return rows
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument("base", help="baseline results file")
+    ap.add_argument("new", help="new results file")
+    ap.add_argument("--metric", choices=["median", "mean"], default="median")
+    args = ap.parse_args()
+
+    base = parse(args.base, args.metric)
+    new = parse(args.new, args.metric)
+
+    keys = [k for k in base if k in new]
+    if not keys:
+        print("no matching rows between the two files", file=sys.stderr)
+        return 1
+    only = set(base) ^ set(new)
+    if only:
+        print(f"note: {len(only)} row(s) present in only one file (ignored)\n")
+
+    buckets = {"CPU": [], "GPU": [], "Default": []}
+    for section in ("process_into", "process_yuv_into"):
+        sect_keys = [k for k in keys if k[0] == section]
+        if not sect_keys:
+            continue
+        print(f"=== {section} ({args.metric}, speedup = base / new) ===")
+        print(f"{'cell':<26}{'variant':<24}{'base':>9}{'new':>9}{'speedup':>9}")
+        print("-" * 77)
+        for k in sect_keys:
+            _, cell, label = k
+            b, n = base[k], new[k]
+            sp = b / n if n else float("nan")
+            bucket = path_bucket(label)
+            if bucket is not None:
+                buckets[bucket].append(sp)
+            print(f"{cell:<26}{label:<24}{b:>9.3f}{n:>9.3f}{sp:>8.2f}x")
+        print()
+
+    def summary(name, xs):
+        if not xs:
+            return
+        print(
+            f"{name:<14} n={len(xs):<4} "
+            f"median={statistics.median(xs):.2f}x  "
+            f"min={min(xs):.2f}x  max={max(xs):.2f}x"
+        )
+
+    print("=== summary (speedup = base / new, by execution path) ===")
+    for name in ("CPU", "GPU", "Default"):
+        summary(f"{name} rows", buckets[name])
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())