Skip to content

Commit 7729804

Browse files
quic-boyucclaude
andcommitted
observatory: add Qualcomm ADB lens and --lens-recipe multi-select
Introduces AdbLens, a Qualcomm-specific Observatory lens that captures SimpleADB activity (push/pull/execute) and surfaces it in the report: - Run dashboard: Device Info (serial, host, soc, htp_arch, workspace, build_path) + compact Transfer Summary per push/pull group. - Left-panel record per execute() call: full qnn_executor_runner command (copyable), PASS/FAIL badge, scrollable monospace log with error-line highlighting, logcat -d and dmesg panels (default ON for inference, configurable via adb.fetch_logcat / adb.fetch_dmesg). - JS renderers inlined via Frontend.resources() - no template files. Also renames --lens_recipe to --lens-recipe in both QNN and XNNPACK Observatory CLIs and switches to multi-value (action='append' + comma-separated support), enabling e.g. --lens-recipe adb accuracy. Verified with 13 unit tests (mocked subprocess) and an end-to-end run of swin_transformer on SM8850 device 5382e6d2. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 1d9a56a commit 7729804

8 files changed

Lines changed: 1716 additions & 23 deletions

File tree

backends/qualcomm/debugger/README.md

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -96,13 +96,15 @@ A new, review-focused Observatory implementation is available under:
9696

9797
`backends/qualcomm/debugger/observatory`
9898

99-
Use the Observatory CLI to wrap any Qualcomm AOT export script. Use `--lens_recipe=accuracy`
100-
to enable accuracy lenses.
99+
Use the Observatory CLI to wrap any Qualcomm AOT export script. Pass
100+
one or more lens recipes via `--lens-recipe` (space-separated):
101+
`accuracy` enables per-stage / per-layer accuracy lenses; `adb`
102+
enables device info, transfer summary, and inference log capture.
101103

102104
```bash
103105
python -m executorch.backends.qualcomm.debugger.observatory \
104106
--output-html obs_report.html \
105-
--lens_recipe=accuracy \
107+
--lens-recipe accuracy \
106108
{original script and args}
107109
```
108110

@@ -111,7 +113,7 @@ For example:
111113
```bash
112114
python -m executorch.backends.qualcomm.debugger.observatory \
113115
--output-html obs_report.html \
114-
--lens_recipe=accuracy \
116+
--lens-recipe accuracy --lens-recipe adb \
115117
examples/qualcomm/oss_scripts/mobilevit_v2.py \
116118
--backend htp --model SM8650 -d ./imagenet-mini-val/ -b build-android/ --compile_only
117119
```

backends/qualcomm/debugger/observatory/README.md

Lines changed: 56 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -13,15 +13,28 @@ python -m executorch.backends.qualcomm.debugger.observatory \
1313
[--output-html PATH] [--output-json PATH] SCRIPT [SCRIPT_ARGS...]
1414
```
1515

16-
### With accuracy debugging
16+
### With one or more lens recipes
17+
18+
Repeat the flag or comma-separate the values:
1719

1820
```bash
1921
python -m executorch.backends.qualcomm.debugger.observatory \
20-
--lens_recipe=accuracy \
22+
--lens-recipe adb --lens-recipe accuracy \
2123
[--output-html PATH] [--output-json PATH] \
2224
SCRIPT [SCRIPT_ARGS...]
25+
26+
python -m executorch.backends.qualcomm.debugger.observatory \
27+
--lens-recipe adb,accuracy \
28+
SCRIPT [SCRIPT_ARGS...]
2329
```
2430

31+
Available recipes:
32+
33+
- `accuracy``AccuracyLens` + `PerLayerAccuracyLens` (per-stage and
34+
per-layer accuracy on the graph).
35+
- `adb``AdbLens` (device info, transfer summary, inference command +
36+
logs with logcat/dmesg). See the *ADB lens* section below.
37+
2538
### Visualize mode (JSON → HTML, no re-execution)
2639

2740
```bash
@@ -43,7 +56,7 @@ source $QNN_SDK_ROOT/bin/envsetup.sh
4356
python -m executorch.backends.qualcomm.debugger.observatory \
4457
--output-html /tmp/obs_vit/report.html \
4558
--output-json /tmp/obs_vit/report.json \
46-
--lens_recipe=accuracy \
59+
--lens-recipe accuracy \
4760
examples/qualcomm/scripts/torchvision_vit.py \
4861
-m SM8650 -b ./build-android \
4962
--dataset imagenet-mini-val/ \
@@ -56,7 +69,7 @@ python -m executorch.backends.qualcomm.debugger.observatory \
5669
```bash
5770
python -m executorch.backends.qualcomm.debugger.observatory \
5871
--output-html /tmp/obs_roberta/report.html \
59-
--lens_recipe=accuracy \
72+
--lens-recipe accuracy \
6073
examples/qualcomm/oss_scripts/roberta.py \
6174
-m SM8650 -b ./build-android \
6275
-H mlgtw-linux -s <device_serial> \
@@ -97,7 +110,7 @@ Dataset: Wikipedia sentences (`wikisent2.txt`). Pass with `-d <path>`.
97110
Common flags: `-m <SOC_MODEL>` (e.g. `SM8650`), `-b <build_folder>`, `-H <host>`,
98111
`-s <device_serial>`, `-a <artifact_dir>`, `--compile_only`.
99112

100-
## Accuracy lenses (`--lens_recipe=accuracy`)
113+
## Accuracy lenses (`--lens-recipe accuracy`)
101114

102115
Registers `AccuracyLens` and `PerLayerAccuracyLens` (with QNN dataset patches) on top of the
103116
default `PipelineGraphCollectorLens`. These produce:
@@ -109,6 +122,44 @@ default `PipelineGraphCollectorLens`. These produce:
109122
QNN dataset patches (`lenses/qnn_dataset_patches.py`) wire the on-device inference output back
110123
into the accuracy lens so metrics reflect true QNN outputs, not emulated CPU results.
111124

125+
## ADB lens (`--lens-recipe adb`)
126+
127+
Captures the device-side life of a QNN run by patching `SimpleADB`:
128+
every `push`, `pull`, and the inference invocation is recorded with
129+
exit code, duration, and (for the inference) full stdout. The HTML
130+
report gains:
131+
132+
- A **Device Info** dashboard block (serial, host, soc, htp arch,
133+
workspace, build path).
134+
- A compact **Transfers** dashboard block (one row per push/pull
135+
group) so 20+ file transfers do not crowd out the inference detail.
136+
- A left-panel **`adb.execute`** record per inference call, showing
137+
the full `qnn_executor_runner ...` command with one-click copy, a
138+
scrollable log with errors highlighted in red, and collapsible
139+
`logcat -d` / `adb shell dmesg` panels fetched after the run.
140+
141+
Config (override via `Observatory.enable_context(config={"adb": {...}})`):
142+
143+
| Key | Default | Effect |
144+
|---|---|---|
145+
| `enabled` | `True` | Master switch. `False` disables patching. |
146+
| `forward_to_stdout` | `True` | Tee captured inference stdout to terminal. |
147+
| `max_stdout_bytes` | `4 * 1024 * 1024` | Cap on captured streams. |
148+
| `fetch_logcat` | `"auto"` | `"auto"`: only after `execute()`; `True`/`False` force. |
149+
| `fetch_dmesg` | `"auto"` | Same shape as `fetch_logcat`. |
150+
151+
Example combining both recipes:
152+
153+
```bash
154+
python -m executorch.backends.qualcomm.debugger.observatory \
155+
--output-html /tmp/obs/report.html \
156+
--lens-recipe accuracy --lens-recipe adb \
157+
examples/qualcomm/oss_scripts/swin_transformer.py \
158+
-m SM8850 -b ./build-android \
159+
--dataset imagenet-mini-val/ \
160+
-H weilhuan-linux -s <device_serial>
161+
```
162+
112163
## Two-step workflow
113164

114165
Collect on-device in CI, visualize locally without re-running:

backends/qualcomm/debugger/observatory/cli.py

Lines changed: 43 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -10,9 +10,11 @@
1010
python -m executorch.backends.qualcomm.debugger.observatory \\
1111
[--output-html PATH] [--output-json PATH] SCRIPT [SCRIPT_ARGS...]
1212
13-
With accuracy debugging:
13+
With one or more lens recipes (repeat the flag or comma-separate):
1414
python -m executorch.backends.qualcomm.debugger.observatory \\
15-
--lens_recipe=accuracy SCRIPT [SCRIPT_ARGS...]
15+
--lens-recipe adb --lens-recipe accuracy SCRIPT [SCRIPT_ARGS...]
16+
python -m executorch.backends.qualcomm.debugger.observatory \\
17+
--lens-recipe adb,accuracy SCRIPT [SCRIPT_ARGS...]
1618
1719
Visualize mode (JSON -> HTML):
1820
python -m executorch.backends.qualcomm.debugger.observatory visualize \\
@@ -24,6 +26,26 @@
2426
import sys
2527

2628

29+
_RECIPE_CHOICES = ["accuracy", "adb"]
30+
31+
32+
def _parse_recipe_values(values):
33+
"""Split repeated and comma-separated --lens-recipe values."""
34+
recipes: set = set()
35+
for v in values or []:
36+
for token in str(v).split(","):
37+
token = token.strip()
38+
if not token:
39+
continue
40+
if token not in _RECIPE_CHOICES:
41+
raise SystemExit(
42+
"argument --lens-recipe: invalid choice: "
43+
f"{token!r} (choose from {', '.join(repr(c) for c in _RECIPE_CHOICES)})"
44+
)
45+
recipes.add(token)
46+
return recipes
47+
48+
2749
def main():
2850
from executorch.devtools.observatory.cli import (
2951
make_collect_parser,
@@ -42,12 +64,19 @@ def main():
4264
prog="python -m executorch.backends.qualcomm.debugger.observatory"
4365
)
4466
parser.add_argument(
45-
"--lens_recipe",
46-
choices=["accuracy"],
47-
default=None,
48-
help="Lens recipe to enable (e.g. accuracy)",
67+
"--lens-recipe",
68+
action="append",
69+
default=[],
70+
metavar="RECIPE",
71+
help=(
72+
"Lens recipe to enable. Repeat the flag or comma-separate to "
73+
"enable multiple (e.g. '--lens-recipe adb --lens-recipe accuracy' "
74+
"or '--lens-recipe adb,accuracy'). "
75+
f"Choices: {', '.join(_RECIPE_CHOICES)}"
76+
),
4977
)
5078
args = parser.parse_args(sys.argv[1:])
79+
recipes = _parse_recipe_values(args.lens_recipe)
5180

5281
from executorch.devtools.observatory.observatory import Observatory
5382
from executorch.devtools.observatory.lenses.pipeline_graph_collector import (
@@ -59,7 +88,7 @@ def main():
5988
PipelineGraphCollectorLens.register_backend_patches(install_qnn_patches)
6089
Observatory.register_lens(PipelineGraphCollectorLens)
6190

62-
if args.lens_recipe == "accuracy":
91+
if "accuracy" in recipes:
6392
from executorch.devtools.observatory.lenses.accuracy import AccuracyLens
6493
from .lenses.qnn_dataset_patches import install_qnn_dataset_patches
6594

@@ -72,6 +101,13 @@ def main():
72101

73102
Observatory.register_lens(PerLayerAccuracyLens)
74103

104+
if "adb" in recipes:
105+
from .lenses.adb import AdbLens
106+
from .lenses.adb_patches import install_adb_patches
107+
108+
AdbLens.register_adb_patches(install_adb_patches)
109+
Observatory.register_lens(AdbLens)
110+
75111
run_observatory(
76112
args.script, args.script_args, Observatory, args.output_html, args.output_json
77113
)

0 commit comments

Comments
 (0)