Skip to content

Commit 6a3afad

Browse files
quic-boyucquic-boyuc
committed
Add backend specific readme.md
1 parent f02dd78 commit 6a3afad

2 files changed

Lines changed: 265 additions & 0 deletions

File tree

‎backends/qualcomm/debugger/observatory/README.md‎

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
# Qualcomm Observatory CLI
2+
3+
Qualcomm-specific Observatory CLI that wraps `devtools/observatory` with QNN backend patches and
4+
accuracy lenses. Requires a QNN SDK environment (source `$QNN_SDK_ROOT/bin/envsetup.sh` before
5+
running on-device jobs).
6+
7+
## Usage
8+
9+
### Collection mode (default)
10+
11+
```bash
12+
python -m executorch.backends.qualcomm.debugger.observatory \
13+
[--output-html PATH] [--output-json PATH] SCRIPT [SCRIPT_ARGS...]
14+
```
15+
16+
### With accuracy debugging
17+
18+
```bash
19+
python -m executorch.backends.qualcomm.debugger.observatory \
20+
--lens_recipe=accuracy \
21+
[--output-html PATH] [--output-json PATH] \
22+
SCRIPT [SCRIPT_ARGS...]
23+
```
24+
25+
### Visualize mode (JSON → HTML, no re-execution)
26+
27+
```bash
28+
python -m executorch.backends.qualcomm.debugger.observatory visualize \
29+
--input-json report.json --output-html report.html
30+
```
31+
32+
## Qualcomm examples
33+
34+
Qualcomm example scripts use only absolute imports and live in directories without `__init__.py`,
35+
so the Observatory CLI runs them as plain scripts via `runpy.run_path` (no special invocation
36+
needed).
37+
38+
### Vision model (ImageNet)
39+
40+
```bash
41+
source $QNN_SDK_ROOT/bin/envsetup.sh
42+
43+
python -m executorch.backends.qualcomm.debugger.observatory \
44+
--output-html /tmp/obs_vit/report.html \
45+
--output-json /tmp/obs_vit/report.json \
46+
--lens_recipe=accuracy \
47+
examples/qualcomm/scripts/torchvision_vit.py \
48+
-m SM8650 -b ./build-android \
49+
--dataset imagenet-mini-val/ \
50+
-H mlgtw-linux -s <device_serial> \
51+
-a /tmp/obs_vit --seed 1126 --compile_only
52+
```
53+
54+
### NLP model (Wikipedia sentences)
55+
56+
```bash
57+
python -m executorch.backends.qualcomm.debugger.observatory \
58+
--output-html /tmp/obs_roberta/report.html \
59+
--lens_recipe=accuracy \
60+
examples/qualcomm/oss_scripts/roberta.py \
61+
-m SM8650 -b ./build-android \
62+
-H mlgtw-linux -s <device_serial> \
63+
-a /tmp/obs_roberta --compile_only
64+
```
65+
66+
### Compile-only (no device required)
67+
68+
Add `--compile_only` to any Qualcomm script to export and lower without pushing to device.
69+
This is useful for inspecting the compilation pipeline in CI or on a dev machine.
70+
71+
## Available example scripts
72+
73+
### `examples/qualcomm/scripts/` — vision models
74+
75+
| Script | Model |
76+
|---|---|
77+
| `torchvision_vit.py` | Vision Transformer |
78+
| `mobilenet_v2.py` | MobileNetV2 |
79+
| `mobilenet_v3.py` | MobileNetV3 |
80+
| `inception_v3.py` | InceptionV3 |
81+
| `inception_v4.py` | InceptionV4 |
82+
83+
Dataset: ImageNet (pass with `--dataset <path>` or `-d <path>`).
84+
85+
### `examples/qualcomm/oss_scripts/` — NLP/open-source models
86+
87+
| Script | Model |
88+
|---|---|
89+
| `roberta.py` | RoBERTa |
90+
| `bert.py` | BERT |
91+
| `albert.py` | ALBERT |
92+
| `distilbert.py` | DistilBERT |
93+
| `eurobert.py` | EuroBERT |
94+
95+
Dataset: Wikipedia sentences (`wikisent2.txt`). Pass with `-d <path>`.
96+
97+
Common flags: `-m <SOC_MODEL>` (e.g. `SM8650`), `-b <build_folder>`, `-H <host>`,
98+
`-s <device_serial>`, `-a <artifact_dir>`, `--compile_only`.
99+
100+
## Accuracy lenses (`--lens_recipe=accuracy`)
101+
102+
Registers `AccuracyLens` and `PerLayerAccuracyLens` (with QNN dataset patches) on top of the
103+
default `PipelineGraphCollectorLens`. These produce:
104+
105+
- Per-stage accuracy metrics (PSNR, cosine similarity, MSE, top-k)
106+
- Per-layer accuracy heat-map overlaid on the graph
107+
- Cross-stage diff labels in the left panel of the HTML report
108+
109+
QNN dataset patches (`lenses/qnn_dataset_patches.py`) wire the on-device inference output back
110+
into the accuracy lens so metrics reflect true QNN outputs, not emulated CPU results.
111+
112+
## Two-step workflow
113+
114+
Collect on-device in CI, visualize locally without re-running:
115+
116+
```bash
117+
# Step 1 — collect (e.g., in CI with device attached)
118+
python -m executorch.backends.qualcomm.debugger.observatory \
119+
--output-html /tmp/obs/report.html \
120+
--output-json /tmp/obs/report.json \
121+
examples/qualcomm/scripts/torchvision_vit.py \
122+
-m SM8650 -b ./build-android -d imagenet-mini-val/ \
123+
-H mlgtw-linux -s <device_serial> -a /tmp/obs
124+
125+
# Step 2 — re-generate HTML from JSON (e.g., locally after lens update)
126+
python -m executorch.backends.qualcomm.debugger.observatory visualize \
127+
--input-json /tmp/obs/report.json \
128+
--output-html /tmp/obs/report_v2.html
129+
```
130+
131+
## Backend patches
132+
133+
`lenses/qnn_patches.py` installs a monkey-patch on `ptq_calibrate` so the
134+
`PipelineGraphCollectorLens` can intercept the QNN quantization calibration stage and capture
135+
the graph at that point. The patch is active only while the Observatory context is open.
136+
137+
`lenses/qnn_dataset_patches.py` wires on-device inference results into `AccuracyLens` so that
138+
accuracy metrics use real QNN outputs.
139+
140+
## See also
141+
142+
- `backends/qualcomm/debugger/README.md` — broader Qualcomm debugger overview (QAIRT visualizer,
143+
intermediate output debugger)
144+
- `devtools/observatory/README.md` — framework overview, Python API, custom lens guide
145+
- `devtools/observatory/USAGE.md` — full CLI reference
146+
- `devtools/observatory/lenses/LENSES.md` — built-in lens details

‎backends/xnnpack/debugger/observatory/README.md‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
# XNNPack Observatory CLI
2+
3+
XNNPack-specific Observatory CLI that wraps `devtools/observatory` with XNNPack backend patches and accuracy lenses.
4+
5+
## Usage
6+
7+
### Collection mode (default)
8+
9+
```bash
10+
python -m executorch.backends.xnnpack.debugger.observatory \
11+
[--output-html PATH] [--output-json PATH] SCRIPT [SCRIPT_ARGS...]
12+
```
13+
14+
### With accuracy debugging
15+
16+
```bash
17+
python -m executorch.backends.xnnpack.debugger.observatory \
18+
--lense_recipe=accuracy \
19+
[--output-html PATH] [--output-json PATH] \
20+
SCRIPT [SCRIPT_ARGS...]
21+
```
22+
23+
### Visualize mode (JSON → HTML, no re-execution)
24+
25+
```bash
26+
python -m executorch.backends.xnnpack.debugger.observatory visualize \
27+
--input-json report.json --output-html report.html
28+
```
29+
30+
## XNNPack examples
31+
32+
`examples/xnnpack/aot_compiler.py` uses relative imports (`from . import MODEL_NAME_TO_OPTIONS`,
33+
`from ..models import ...`) and must be executed as a Python module. The CLI handles this
34+
automatically: when the supplied path ends in `.py` and its directory contains `__init__.py`, it
35+
uses `runpy.run_module` instead of `runpy.run_path`.
36+
37+
### File path (auto-detected as module)
38+
39+
```bash
40+
python -m executorch.backends.xnnpack.debugger.observatory \
41+
--output-html /tmp/mv2/report.html \
42+
--lense_recipe=accuracy \
43+
examples/xnnpack/aot_compiler.py \
44+
--model_name=mv2 --delegate --quantize --output_dir /tmp/mv2
45+
```
46+
47+
### Dotted module name (explicit)
48+
49+
```bash
50+
python -m executorch.backends.xnnpack.debugger.observatory \
51+
--output-html /tmp/mv2/report.html \
52+
--lense_recipe=accuracy \
53+
examples.xnnpack.aot_compiler \
54+
--model_name=mv2 --delegate --quantize --output_dir /tmp/mv2
55+
```
56+
57+
### Available model names
58+
59+
Pass `--model_name` with any of the models defined in `examples/xnnpack/__init__.py`:
60+
61+
| Model | Notes |
62+
|---|---|
63+
| `mv2` | MobileNetV2 — fast, quantizable |
64+
| `mv3` | MobileNetV3 |
65+
| `resnet18` | ResNet-18 |
66+
| `resnet50` | ResNet-50 |
67+
| `vit` | Vision Transformer |
68+
| `ic3` | InceptionV3 |
69+
| `ic4` | InceptionV4 |
70+
| `dl3` | DeepLabV3 |
71+
| `edsr` | Super-resolution |
72+
| `mobilebert` | MobileBERT |
73+
| `w2l` | Wav2Letter |
74+
| `linear` | Linear baseline |
75+
| `add` / `add_mul` | Arithmetic baselines |
76+
| `llama2` | Llama 2 (requires HuggingFace token) |
77+
| `emformer_join` / `emformer_transcribe` | Speech |
78+
79+
Common flags: `--delegate` (XNNPACK delegation, on by default), `--quantize` (8-bit PTQ),
80+
`--output_dir` (where the `.pte` is written).
81+
82+
## Accuracy lenses (`--lense_recipe=accuracy`)
83+
84+
Registers `AccuracyLens` and `PerLayerAccuracyLens` on top of the default
85+
`PipelineGraphCollectorLens`. These produce:
86+
87+
- Per-stage accuracy metrics (PSNR, cosine similarity, MSE, top-k)
88+
- Per-layer accuracy heat-map overlaid on the graph
89+
- Cross-stage diff labels in the left panel of the HTML report
90+
91+
## Two-step workflow
92+
93+
Collect in one environment, visualize in another:
94+
95+
```bash
96+
# Step 1 — collect
97+
python -m executorch.backends.xnnpack.debugger.observatory \
98+
--output-html /tmp/mv2/report.html \
99+
--output-json /tmp/mv2/report.json \
100+
examples/xnnpack/aot_compiler.py \
101+
--model_name=mv2 --delegate --quantize --output_dir /tmp/mv2
102+
103+
# Step 2 — re-generate HTML from JSON (e.g., after lens code update)
104+
python -m executorch.backends.xnnpack.debugger.observatory visualize \
105+
--input-json /tmp/mv2/report.json \
106+
--output-html /tmp/mv2/report_v2.html
107+
```
108+
109+
## Backend patches
110+
111+
`lenses/xnnpack_patches.py` installs XNNPack-specific monkey-patches so the
112+
`PipelineGraphCollectorLens` can intercept XNNPack-specific lowering steps. These patches are
113+
active only while the Observatory context is open and are removed when it closes.
114+
115+
## See also
116+
117+
- `devtools/observatory/README.md` — framework overview, Python API, custom lens guide
118+
- `devtools/observatory/USAGE.md` — full CLI reference
119+
- `devtools/observatory/lenses/LENSES.md` — built-in lens details

0 commit comments

Comments
 (0)