Boyuc/observatory draft demo

[quic-boyuc]

Observatory + fx_viewer — A Visualizer and Debugging Framework for ExecuTorch

Please read RFC first : #20618

This document covers the feature set, architecture, and API design with concrete examples and demos. For the higher-level motivation, the fair comparison against Model Explorer, and the open design questions, see the RFC at #20618

Motivation (brief)

ExecuTorch already has strong low-level capture tools (Inspector, ETRecord/ETDump), but no layer to coordinate them, and no embeddable, overlay-capable graph viewer for in-pipeline debugging. As a result each backend builds its own glue scripts, and debugging output ends up scattered across console prints, CSVs, and screenshots.

• Observatory provides the missing coordination + synthesis layer: write one Lens once, get a shareable report.
• fx_viewer provides the missing viewer: server-free, embeddable in one HTML file, with programmatic overlays and N-way compare.

The full motivation and the fair comparison against Model Explorer / Inspector live in the RFC ( #20618 §2). This document is about how it works.

---

What You Get

One command in, one HTML file out:

pip3 install 'fast-sugiyama[all]'   # requires python >= 3.11 — required by the graph lens

python -m executorch.backends.qualcomm.debugger.observatory \
    --output-html obs_report.html \
    --lens-recipe accuracy \
    examples/qualcomm/oss_scripts/mobilevit_v2.py \
    --backend htp --model SM8650 -d ./imagenet-mini-val/ \
    -b build-android/ --compile_only

Output: a self-contained HTML file — no server, no login, no external service. Attach it to an issue, PR, or email.

Features in the report:

• Session dashboard — per-session metadata, lens-contributed sections.
• Record tree — captured artifacts, time-ordered or grouped by region_stack.
• Interactive FX graph — pan, zoom, minimap, fuzzy search, N-way synchronized compare.
• Per-layer accuracy overlay — PSNR / cosine / MSE as a color gradient on graph nodes, from CPU simulation across intermediate snapshots at prepare_pt2e, convert_pt2e, and to_edge_transform_and_lower (stages not stored in ETRecord). Runtime/delegated accuracy via Inspector is a planned follow-up lens.

Two machine-readable outputs accompany the HTML:

• Archive (JSON) — raw sessions[] + records[], no analysis baked in; the input for compare subcommand and late re-analysis.
• Report (JSON) — analyzed summary for CI gates, dashboards, and LLM triage.

---

Architecture Overview

Observatory follows a three-layer design: Interface → Core → Lenses.

╔═════════════════════════════════════════════════════════════════════════════╗
║  INTERFACE LAYER                                                            ║
╠═════════════════════════════════════════════════════════════════════════════╣
║ ┌─ User interface ──────────┐ ┌─ Lens author ──────┐ ┌─ Artifacts ───────┐ ║
║ │ generic CLI               │ │ implements a Lens:  │ │ Report (HTML)     │ ║
║ │ backend CLI               │ │   on_session_start  │ │ Archive (JSON)    │ ║
║ │ `with` block (context)    │ │   on_session_end    │ │ Report (JSON)     │ ║
║ │ @observe_pass decorator   │ │   observe / digest  │ │                   │ ║
║ │ Observatory.collect(...)  │ │   analyze           │ │                   │ ║
║ │                           │ │   get_frontend_spec │ │                   │ ║
║ └───────────────────────────┘ └─────────────────────┘ └───────────────────┘ ║
╚═════════════════════════════════════════════════════════════════════════════╝
         │                              │                          ▲
         │ drives                       │ registers with           │ emits
         ▼                              ▼                          │
╔═════════════════════════════════════════════════════════════════════════════╗
║  OBSERVATORY CORE           (backend-agnostic)                              ║
╠═════════════════════════════════════════════════════════════════════════════╣
║ ┌─ Session manager ──────────┐   ┌─ Record store ─────────────────────┐    ║
║ │ region & config stack      │   │ per-collect fan-out to every lens  │    ║
║ │ Session open/close         │   │ time-ordered records, region_stack │    ║
║ │ lens registry              │   │ tagged with active session_id      │    ║
║ └────────────────────────────┘   └────────────────────────────────────┘    ║
║                                                                             ║
║ ┌─ Report assembly ────────────────────┐  ┌─ fx_viewer ──────────────────┐ ║
║ │ per-lens analyze over Archive        │  │ Python API (build-time)      │ ║
║ │ Frontend.dashboard / .record         │  │ JS API (browser run-time)    │ ║
║ │ base graph + extension overlays      │  │ Canvas renderer, no server   │ ║
║ └──────────────────────────────────────┘  └───────────────────────────────┘ ║
║                                                                             ║
║ ┌─ Export ─────────────────────────────────────────────────────────────────┐║
║ │ Report (HTML)   — self-contained, for human reviewers                    │║
║ │ Archive (JSON)  — raw sessions[] + records[]; CI input; reload/ --compare │║
║ │ Report (JSON)   — analyzed output for LLM triage / dashboards            │║
║ └──────────────────────────────────────────────────────────────────────────┘║
╚═════════════════════════════════════════════════════════════════════════════╝
         ▲
         │ registered at CLI entry; called by Core's lifecycle phases
         │
╔═════════════════════════════════════════════════════════════════════════════╗
║  LENSES                     (all implement the Lens protocol)               ║
╠═════════════════════════════════════════════════════════════════════════════╣
║ ┌─ Common lenses ─────────────────────┐ ┌─ Backend-specific lenses ──────┐ ║
║ │ graph, metadata, accuracy           │ │ qualcomm/ — QNN debug lenses   │ ║
║ │ per_layer_accuracy, stack_trace     │ │ xnnpack/  — XNNPACK lenses     │ ║
║ │ pipeline_graph_collector            │ │ arm/      — ARM lenses          │ ║
║ │ graph_color                         │ │                                 │ ║
║ └─────────────────────────────────────┘ └─────────────────────────────────┘ ║
╚═════════════════════════════════════════════════════════════════════════════╝

The Capture / Analysis Split

Observatory keeps capture (online, during the run) separate from analysis (offline, from the saved Archive). Only the Archive is raw and persisted; Reports are always derived from it.

  Run (live)                    Persist                    Derive
  ─────────                     ───────                    ──────
  Session hooks fire            Archive (JSON)             Report (HTML)
  collect() → Records           sessions[] + records[]     analyze + Frontend
  observe/digest per lens       no analysis baked in       per-lens derived views
        │                              │                          │
        └──────── written at ──────────┘                          │
                  end of run                                      │
                                       │                          │
                                       └─── reload path ──────────┘
                                            (re-analyze with new
                                             config, or combine
                                             via --compare)

Key property: the same Archive can be re-analyzed with different lens configs, or two archives compared via --compare, all without re-running the AOT compile.

---

Python vs. JavaScript API Boundaries

fx_viewer and Observatory meet at exactly two API surfaces — one in Python (build-time), one in JavaScript (run-time in the browser).

   BUILD TIME (Python API)
   ┌─ Observatory ──────────────────┐  calls  ┌─ fx_viewer Python API ────────┐
   │ graph / graph_color /          │ ──────► │ FXGraphExporter               │
   │ per_layer_accuracy lenses      │         │ GraphExtension / ColorRule    │
   │ Observatory core (emit)        │         └────────────────┬───────────────┘
   └────────────────┬───────────────┘                          │ produces
                    │ assembles + writes                        ▼
                    └──────────────► report.html ◄─────────────┘
                                     (embedded payload + JS runtime bundle)
                                     │ opened in browser
                                     ▼
   RUN TIME (JS API, all in browser)
   ┌─ Observatory report shell ─────┐  calls  ┌─ fx_viewer JS API ───────────┐
   │ 03_blocks.js                   │ ──────► │ FXGraphViewer.create({...})  │
   │   (mount viewer per Record)    │         │ FXGraphCompare.create({...}) │
   │ 04_actions.js                  │         │ setLayers / setColorBy /     │
   │   (theme + selection sync)     │         │  setTheme / selectNode       │
   └────────────────────────────────┘         └──────────────────────────────┘

Python API (Build-Time)

Component	Role
FXGraphExporter(gm)	Extract FX graph structure from a GraphModule, compute Sugiyama layout (x, y + edge routing), export to HTML or JSON
GraphExtension(id, name)	Declare an overlay layer — per-node data, coloring rules, labels, tooltips
NumericColorRule(attribute, cmap)	Map a continuous numeric field to a color gradient ("viridis", "reds", "blues", "greens")
CategoricalColorRule(attribute)	Map discrete string values to deterministic hues

Standalone usage (no Observatory dependency):

from executorch.devtools.fx_viewer import FXGraphExporter
FXGraphExporter(graph_module).export_html("my_graph.html")

JavaScript API (Run-Time)

Component	Role
FXGraphViewer.create(config)	Mount a single graph viewer on a DOM element
FXGraphCompare.create({viewers, layout, sync})	Mount N-way compare with cross-graph selection sync
setLayers(ids[]) / setColorBy(id) / setTheme(name)	Runtime layer/theme mutation
Selection sync via debug_handle or from_node	Cross-graph node matching across AOT pipeline stages, even after fusion/decomposition

---

The Lens Protocol

A Lens is the single extension unit — one Python class that owns one debugging concern end-to-end. Implement only the methods you need; the framework ignores the rest.

Protocol Methods (fired in lifecycle order)

class Lens:
    @classmethod
    def get_name(cls) -> str: ...          # config key: config[get_name()]

    @classmethod
    def setup(cls) -> None: ...            # one-time init at registration

    @classmethod
    def on_session_start(cls, context: ObservationContext) -> None:
        """Install instrumentation (e.g. monkey-patches). Fires at outermost enter_context."""

    @classmethod
    def observe(cls, artifact: Any, context: ObservationContext) -> Any:
        """Filter + transform the artifact at each collect() call. Return None to skip."""

    @classmethod
    def digest(cls, observation: Any, context: ObservationContext) -> Serializable:
        """Convert observation into a JSON-serializable form for the Record."""

    @classmethod
    def on_session_end(cls, context: ObservationContext) -> None:
        """Restore instrumentation. Guaranteed to fire even on exception."""

    @classmethod
    def clear(cls) -> None: ...            # reset state between runs

    @staticmethod
    def analyze(records: List[RecordDigest], config: Dict[str, Any]) -> AnalysisResult:
        """Derive insights across all records at report time (offline)."""

    @staticmethod
    def get_frontend_spec() -> Frontend:
        """Return the Frontend strategy that renders this lens's report blocks."""

Lens Lifecycle Diagram

  on_session_start  ─► instrumentation installed
        │
        ▼
  for each Observatory.collect(name, artifact):
    each lens: observe → digest  ─► serialized into Record.digests[lens_name]
        │
        ▼
  on_session_end    ─► instrumentation restored (guaranteed on exception)
        │
        ▼
  ═══ ARCHIVE (raw: sessions[] + records[]) ═══
        │
        ▼
  analyze(records, config)  ─► AnalysisResult
        │
        ▼
  get_frontend_spec() → Frontend
        ├── dashboard(session, records, analysis) → ViewList | None
        │     (session-level blocks: TableBlock, HtmlBlock, ...)
        └── record(digest, analysis, context)     → ViewList | None
              (per-record blocks: TableBlock, GraphBlock, ...)

Frontend and Block Types

get_frontend_spec() returns a Frontend instance. The framework calls its methods at report-emit time:

Method	When called	Returns
dashboard(session, records, analysis)	Once per (Session, lens) pair	ViewList of session-level blocks, or None
record(digest, analysis, context)	Once per (Record, lens) pair	ViewList of per-record blocks, or None
json_report(session, records, analysis)	Once per (Session, lens) pair	Dict for Report (JSON), or None
resources()	Once at report assembly	{"js": ..., "css": ...} for shared assets

Available block types in a ViewList:

Block	Purpose
TableBlock	Key-value or row data; auto side-by-side diff in compare view
HtmlBlock	Arbitrary HTML fragment
CustomBlock	Custom JS-rendered widget
GraphBlock	Interactive FX graph via fx_viewer; synchronized N-way compare

Shipped Common Lenses

Lens	Purpose
graph	Base FX graph extraction and layout via FXGraphExporter
metadata	Run-wide info: CLI command, env, model
accuracy	Graph-level accuracy metrics across lowering stages
per_layer_accuracy	Per-operator PSNR / cosine / MSE color overlay
pipeline_graph_collector	Auto-collect at prepare_pt2e, convert_pt2e, to_edge_transform_and_lower
stack_trace	User-code call stack provenance at each collection point
graph_color	Partition/delegation color overlay

---

Worked Example: AdbLogLens (Custom Lens)

This example shows the full extension surface — a lens that captures device-side logs from ADB without touching Observatory core:

from dataclasses import dataclass
from executorch.devtools.observatory import Observatory
from executorch.devtools.observatory.interfaces import (
    Lens, ObservationContext, Frontend, ViewList, TableBlock,
)
from executorch.backends.qualcomm.export_utils import SimpleADB

@dataclass
class AdbLogEntry:
    source: str        # "logcat" or "dmesg"
    cmd_label: str     # which on-device command produced this
    content: str

class AdbLogLens(Lens):
    _original = None

    @classmethod
    def get_name(cls) -> str:
        return "adb_log"

    @classmethod
    def on_session_start(cls, context: ObservationContext) -> None:
        """Patch SimpleADB.execute to capture logcat + dmesg after every call."""
        cls._original = SimpleADB.execute

        def patched(self, *args, **kw):
            cls._original(self, *args, **kw)
            cmd_label = kw.get("custom_runner_cmd") or "executor_runner"
            for source, shell_cmd in [
                ("logcat", "logcat -d"),
                ("dmesg",  "dmesg"),
            ]:
                Observatory.collect(
                    f"adb_log/{cmd_label}/{source}",
                    AdbLogEntry(source, cmd_label,
                                content=_adb_shell(self, shell_cmd)),
                )

        SimpleADB.execute = patched

    @classmethod
    def on_session_end(cls, context: ObservationContext) -> None:
        """Restore original — guaranteed even on exception."""
        SimpleADB.execute = cls._original
        cls._original = None

    @classmethod
    def observe(cls, artifact, context: ObservationContext):
        """Filter by type and live config — reads config per-call for nesting."""
        if not isinstance(artifact, AdbLogEntry):
            return None
        cfg = context.config.get("adb_log", {})
        if not cfg.get("enabled", True):
            return None
        if artifact.source not in cfg.get("sources", ["logcat"]):
            return None
        return {"source": artifact.source,
                "cmd":    artifact.cmd_label,
                "lines":  artifact.content.splitlines()}

    @classmethod
    def digest(cls, observation, context: ObservationContext):
        return observation   # already JSON-serializable

    @staticmethod
    def analyze(records, config):
        return AnalysisResult()   # no cross-record analysis needed

    @staticmethod
    def get_frontend_spec():
        return _AdbFrontend()

class _AdbFrontend(Frontend):
    def record(self, digest, analysis, context):
        if not digest:
            return None
        rows = [{"source": digest["source"], "cmd": digest["cmd"],
                 "lines": len(digest["lines"])}]
        return ViewList(blocks=[TableBlock(id="adb_log", rows=rows)])

Key design patterns demonstrated:

• Session hooks install/restore instrumentation (monkey-patching).
• observe() reads context.config at call time — nested enter_context overrides work correctly.
• Type filtering: lens returns None for artifacts it doesn't own.
• collect() is type-agnostic — multiple lenses coexist without knowing about each other.
• get_frontend_spec() returns a Frontend; record() returns a ViewList with a TableBlock.

Nested Config Scoping in Practice

with Observatory.enter_context(config={
    "accuracy": {"evaluator": my_evaluator},
    "adb_log":  {"sources": ["logcat"]},          # lightweight default
}):
    run_on_device(probe_cmd)      # only logcat captured
    run_on_device(smoke_cmd)      # only logcat captured

    with Observatory.enter_context(config={
        "accuracy": {"enabled": False},
        "adb_log":  {"sources": ["logcat", "dmesg"]},  # broaden for target
    }):
        device_result = run_on_device(target_cmd)       # logcat + dmesg
        Observatory.collect("post_device", device_result)

    run_on_device(teardown_cmd)   # back to logcat only

Config overrides are pushed on enter_context entry and popped on exit — even on exception.

---

Invocation Surfaces

There are two real surfaces: the CLI and the context + collection points pair. @observe_pass is syntactic sugar over the second.

1. CLI (zero-code-change)

Observatory parses its own leading flags, then runs your script exactly as written via runpy, forwarding all remaining arguments verbatim:

# Generic (framework lenses only)
python -m executorch.devtools.observatory \
    --output-html run.html \
    your_script.py --your-args

# Backend-specific
python -m executorch.backends.xnnpack.debugger.observatory \
    --lens-recipe=accuracy \
    examples/xnnpack/aot_compiler.py --model_name=mv2 --delegate --quantize

# Archive for CI (no HTML needed at capture time)
python -m executorch.backends.xnnpack.debugger.observatory \
    --output-archive nightly/${DATE}/mv2.json \
    --lens-recipe=accuracy \
    examples/xnnpack/aot_compiler.py --model_name=mv2

1a. compare Subcommand (overlay multiple archives)

compare is a subcommand of any Observatory CLI (generic or backend-specific). It ingests two or more Archive JSONs and produces one HTML report where records with the same name (e.g. Annotated Model) are placed side-by-side and label-prefixed.

# Cross-backend compare — XNNPack mv2 vs Qualcomm mobilenet_v2
python -m executorch.backends.qualcomm.debugger.observatory compare \
    --input-archive xnn/mv2/archive.json          --label XNNPACK/mv2 \
    --input-archive qnn/mobilenet_v2/archive.json --label Qualcomm/mobilenet_v2 \
    --output-html comparison.html \
    --lens-recipe accuracy,adb \
    --title "MobileNetV2 — XNNPACK vs Qualcomm"

Which backend CLI to invoke compare from — this matters. Each backend CLI pre-registers its own lens frontends (HTML/CSS/JS assets baked into the report). The generic python -m executorch.devtools.observatory compare only registers core lens frontends (graph, metadata, accuracy). To render backend-specific overlays (e.g. Qualcomm's ADB log panel) in the compared report, run compare under the backend CLI that owns those lens frontends — even when the input archives came from a different backend. In the example above we invoke QNN's CLI so the ADB log frontend is registered; XNNPack's CLI would produce the same compare geometry but omit that panel.

--lens-recipe on compare. Same syntax as collection mode — repeatable (--lens-recipe accuracy --lens-recipe adb) or comma-separated (--lens-recipe accuracy,adb). It only pre-registers frontends for rendering; the archives themselves already contain the captured lens data, so recipes here control which panels are drawn, not what is analyzed.

1b. visualize Subcommand (Archive JSON → HTML re-render)

visualize re-renders a single Archive JSON to HTML without re-executing the compile pipeline — useful after a lens code update, or when the report was captured on a headless CI runner:

python -m executorch.backends.xnnpack.debugger.observatory visualize \
    --input-archive archive.json --output-html report_v2.html \
    [--lens-recipe accuracy]

Same backend-CLI-registers-lens-frontends rule as compare.

2. Context + Collection Points

Activate the lenses you want, open a session, and mark the points to capture. Lenses are turned on (and tuned) by the config dict, keyed by lens name:

from executorch.devtools.observatory import Observatory
from executorch.devtools.observatory.lenses import PerLayerAccuracyLens

Observatory.register_lens(PerLayerAccuracyLens)

with Observatory.enter_context("quantization",
                               config={"per_layer_accuracy": {"enabled": True}}):
    Observatory.collect("before_quantize", gm)           # capture point 1

    with Observatory.enter_context("fast_passes",
                                   config={"per_layer_accuracy": {"enabled": False}}):
        gm = run_cheap_passes(gm)                        # lens off for this sub-step

    quantized_gm = quantize_model(gm)
    Observatory.collect("after_quantize", quantized_gm)  # capture point 2

# Stage 1 — persist the raw Archive JSON
Observatory.export_json("archive.json")

# Stage 2 — render HTML from the archive, any time, without re-running
Observatory.generate_html_from_json("archive.json", "report.html")

3. @observe_pass Decorator (sugar over context + collect)

@observe_pass wraps a pass so its input and output graphs are collect()-ed automatically — no manual collection points needed:

from executorch.devtools.observatory import Observatory, observe_pass

@observe_pass
class MyPass(ExportPass):
    def call(self, gm): ...

# Or wrap existing pass instances without modifying their class:
observed_passes = [observe_pass(p) for p in [FoldQDQ(), LayoutTransform()]]

with Observatory.enter_context("pipeline"):
    PassManager(observed_passes)(graph_module)

---

Collection-Point Mechanisms

Three distinct mechanisms produce collection points — all flow into the same collect(name, artifact) path:

Mechanism	How it works	Patch targets / scope
pipeline_graph_collector lens patches	Wraps standard pipeline functions at on_session_start; each patch calls collect() with the returned object, then restores at on_session_end	prepare_pt2e, convert_pt2e, to_edge_transform_and_lower, ETRecord.add_*
@observe_pass decorator	Tags any PassBase subclass/instance; wraps each pass call in its own Region and collects graph before + after	Any pass: user-defined or framework-provided
Manual Observatory.collect(name, artifact)	Direct call anywhere inside user code	Any artifact type, any moment

# What a pipeline_graph_collector patch looks like (simplified):
import torchao.quantization.pt2e.quantize_pt2e as qt

_original = qt.convert_pt2e

def _patched(model, *args, **kwargs):
    Observatory.collect("Calibrated Model", model)        # capture input graph
    result = _original(model, *args, **kwargs)            # call the real function
    Observatory.collect("Quantized Model", result)        # capture output graph
    return result

qt.convert_pt2e = _patched
# on_session_end: qt.convert_pt2e = _original  (always restored, even on exception)

All patches are installed in on_session_start and removed in on_session_end — the user's script runs unchanged.

---

Key Vocabulary

Term	Definition
Region	Named scope from enter_context(name). Nests. Pure labelling — no lens hooks fire at region boundaries.
Session	Outermost Region. Lens on_session_start/on_session_end fire here.
Record	One collect(name, artifact) item. Tagged with session_id + region_stack.
Archive	Raw state: sessions[] + records[]. The only thing persisted.
Report	Derived output from an Archive via analyze + Frontend. HTML or JSON.
Lens	One Python class that owns one debugging concern end-to-end.
Frontend	The visualization strategy returned by get_frontend_spec(); contributes dashboard() and record() blocks.

---

POC Implementation Structure (~/executorch)

devtools/
├── observatory/
│   ├── __init__.py              # Public API: Observatory, observe_pass
│   ├── observatory.py           # Core: Session manager, Record store, export
│   ├── interfaces.py            # Lens protocol, Frontend, ViewList, block types
│   ├── observe_pass.py          # @observe_pass decorator
│   ├── cli.py                   # Generic CLI entry point
│   ├── html_template.py         # Report HTML assembly
│   ├── lenses/
│   │   ├── graph.py             # Base FX graph extraction + layout
│   │   ├── metadata.py          # Run-wide metadata
│   │   ├── accuracy.py          # Graph-level accuracy metrics
│   │   ├── per_layer_accuracy.py # Per-operator PSNR/cosine/MSE
│   │   ├── pipeline_graph_collector.py  # Auto-collection at pipeline points
│   │   ├── stack_trace.py       # Call stack provenance
│   │   └── graph_color.py       # Partition color overlay
│   ├── templates/js/
│   │   ├── 03_blocks.js         # Mount viewer per Record graph block
│   │   └── 04_actions.js        # Theme + selection sync logic
│   └── tests/
│
├── fx_viewer/
│   ├── __init__.py              # Public API: FXGraphExporter
│   ├── exporter.py              # Graph extraction, Sugiyama layout, HTML/JSON export
│   ├── extension.py             # GraphExtension: add_node_data, set_color_rule, set_sync_key
│   ├── color_rules.py           # NumericColorRule, CategoricalColorRule
│   ├── models.py                # Data models: GraphNode, GraphEdge, GraphPayload
│   ├── templates/               # JS canvas runtime bundle
│   └── examples/                # Standalone usage examples
│
backends/
├── qualcomm/debugger/observatory/
│   ├── cli.py                   # QNN backend CLI
│   ├── lenses/                  # QNN-specific lenses (AdbLens, etc.)
│   └── tests/
│
└── xnnpack/debugger/observatory/
    ├── cli.py                   # XNNPACK backend CLI
    └── lenses/                  # XNNPACK-specific lenses

---

Governance and Merge Strategy

Recommended: Three-PR Stack

PR	Scope	Dependency	Reviewer Focus
PR 1	devtools/fx_viewer/	None	Standalone graph viewer; usable via FXGraphExporter(gm).export_html(...)
PR 2	devtools/observatory/	PR 1	Core framework + 7 common lenses + generic CLI
PR 3	Backend CLIs	PR 2	Qualcomm + XNNPACK CLIs with model-matrix validation

Ownership Model

• Core devtools reviewers → devtools/observatory/ core, devtools/fx_viewer/ core, common lenses.
• Backend teams → backends/<name>/debugger/observatory/, backend-specific lenses and backend CLI — no core sign-off needed.
• Cross-cutting changes (Lens protocol, GraphExtension API, JS runtime, JSON schemas) → core sign-off required.

Public API Surfaces (schema stability)

Surface	Stability	Change requires
Lens protocol (9 methods: get_name, setup, on_session_start, observe, digest, on_session_end, clear, analyze, get_frontend_spec)	Experimental (candidate stable)	Core sign-off
Frontend API (dashboard, record, json_report, resources)	Experimental (candidate stable)	Core sign-off
GraphExtension API (add_node_data, set_color_rule, set_sync_key, set_label_formatter)	Experimental (candidate stable)	Core sign-off
Archive (JSON) schema (sessions[] + records[])	Experimental (candidate stable)	Core sign-off
Report (JSON) schema	Experimental (candidate stable)	Core sign-off
FXGraphViewer.create / FXGraphCompare.create (JS)	Experimental (candidate stable)	Core sign-off
Backend CLI flags (--output-html, --lens-recipe, etc.)	Experimental (candidate stable)	Core sign-off
Per-lens config keys (e.g., accuracy.evaluator)	Backend-owned	Backend team
Backend-specific lenses	Backend-owned	Backend team

All surfaces are currently experimental. The "candidate stable" designation means they are designed for stability and will be promoted in a follow-up RFC after one release cycle with real external consumers.

Testing Strategy

• Core infra tests → devtools/observatory/tests/ (owned by core).
• Backend tests → owned by backend teams.
• CI invariant: every backend CLI smoke-tests on a representative model.

---

Implemented Capabilities (on the POC Branch Today)

Capability	Mechanism	Status
Compile-time per-layer accuracy (intermediate stages)	accuracy + per_layer_accuracy lenses; CPU simulation across prepare_pt2e, convert_pt2e, to_edge_transform_and_lower snapshots	✅ Implemented
Graph-state collection at pipeline points	pipeline_graph_collector lens patches; captures intermediate graph snapshots not stored in ETRecord	✅ Implemented
Pass diff (before/after)	@observe_pass decorator + graph compare mode	✅ Implemented
Collection provenance (stack trace)	stack_trace lens	✅ Implemented
Interactive FX graph view	fx_viewer: pan, zoom, minimap, fuzzy search, N-way compare	✅ Implemented
Run-metadata dashboard	metadata lens: CLI command, env, model	✅ Implemented
Region tree-view toggle	Left panel groups Records by region_stack; toggle flat/tree	✅ Implemented
Report (HTML) — self-contained	Single file, no server, attach anywhere	✅ Implemented
Archive (JSON) — raw persistence	--output-archive; reload for re-analysis	✅ Implemented
Report (JSON) via json_report	Structured analysis for LLM triage, CI, dashboards (--output-report-json)	✅ Implemented
Partition/delegation color overlay	graph_color lens	✅ Implemented
compare CLI subcommand	Load 2+ archives, emit regression Report	✅ Implemented
ADB log capture (logcat + dmesg)	AdbLens (Qualcomm backend-specific, not a default core lens)	✅ Implemented
Runtime / delegated-graph accuracy	On-device vs CPU comparison via debug_handle + Inspector	❌ Follow-up

---

Known Limitations

1. Per-layer accuracy is compile-time simulation only. The per_layer_accuracy lens runs CPU simulation across intermediate graph snapshots — stages not stored in ETRecord. It does not compare against actual on-device (delegated) execution. Runtime/delegated accuracy via Inspector integration is a planned follow-up lens.
2. fast-sugiyama requires Python ≥ 3.11 and is a hard requirement for the graph lens. The current fx_viewer layout stage raises ImportError when fast-sugiyama (and rectangle-packer, its multi-component packer) are missing, so records that go through GraphLens produce no graph payload. Since the interactive FX graph is the RFC's centrepiece feature, treat this as required. A future pure-Python fallback layout could downgrade this to optional (gated behind executorch[observatory-layout] to avoid raising the project's Python floor), but that fallback is not yet implemented.
3. Runtime / delegated-graph accuracy not yet implemented. Comparing on-device (delegated) execution against CPU requires a follow-up lens using debug_handle + Inspector.

---

Pre-Generated Demo Reports

Reports are available for 30+ models across XNNPACK and Qualcomm backends:

Backend	Model	Nodes	Report
xnnpack	mobilebert	2361	HTML
xnnpack	resnet50	550	HTML
qualcomm	inception_v4	1541	HTML
qualcomm	mobilenet_v2	521	HTML

Cross-backend comparisons (XNNPACK vs. Qualcomm QNN):

Model	Comparison
MobileNetV2	HTML
ViT	HTML

Full model matrix and walkthrough video: see rfc_review_real.md, §3.3.

---

Dependencies

• fast-sugiyama[all] (Python ≥ 3.11, required for the graph lens) — Sugiyama graph layout for fx_viewer. The graph lens raises ImportError at layout time if this (or its rectangle-packer extra) is missing. A pure-Python fallback that would demote this to optional is planned but not yet implemented; when it lands, gate the fast path behind executorch[observatory-layout].
• No JavaScript framework dependencies — the canvas renderer is plain HTML/JS with no npm dependencies bundled at runtime.
• Observatory is a client of existing ExecuTorch capture primitives (ETRecord, ETDump, Inspector) — it adds no new capture formats and requires no changes to Inspector or ETRecord.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19288

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⚠️ 12 Awaiting Approval

As of commit f7ddf43 with merge base d54a0c0 ():

AWAITING APPROVAL - The following workflows need approval before CI can run:

• Build documentation (gh)
• Build Presets (gh)
• Cadence Build & Test (gh)
• Lint (gh)
• periodic (gh)
• pull (gh)
• Test ARM Backend (gh)
• Test CoreML Backend (gh)
• Test QNN Backend (gh)
• Test Vulkan Backend (gh)
• Test WebGPU Backend (gh)
• Test XNNPACK Backend (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[linux-foundation-easycla]

• ✅ Unknown (1094ea2, 33dc37e, 3521c1a, 393802a, 6a431d5, 98b37ef, a0aff81, 187ab55, 23ab069, 7729804, 785f21b, 85299d1, 9ae38e8, 9f878fa, be19c24, 22e1c72, 3c71a4c, 58d16e1, 7845c90, 838e5f0, aac8ae8, ca046bc, f2ac791, 04534a4, 1a526e0, 217b2f9, 2774f3e, 3ccf3fe, 4ef1bad, 59c3d70, 698e9ea, 6c32093, 6c952f3, 6cad11d, 8ac6014, 8b9666a, 8d2d00f, 8fcaa3f, bdf1150, c523396, cd5d111, d095ccd, df5a245, e95395b, f9e9d0b)
• ✅ login: quic-boyuc / name: boyuc (01df26e, 04534a4, 05d3cf5, 06f28a0, 09c3af7, 1094ea2, 1428f31, 144154a, 187ab55, 18c4bab, 1a526e0, 1b5dfc3, 1bacd3b, 1d9a56a, 1f89d8b, 217b2f9, 21896af, 22e1c72, 23ab069, 23dfc41, 23e02bf, 2774f3e, 27a1c16, 3075729, 32a8bf4, 33dc37e, 3521c1a, 35486c1, 36f95e7, 391adbc, 393802a, 3c71a4c, 3ccf3fe, 4231ca2, 471b1bb, 4ef1bad, 53dc4d8, 58d16e1, 59c3d70, 698e9ea, 6a431d5, 6bff376, 6c32093, 6c952f3, 6cad11d, 76fb3e0, 7729804, 77b0e2d, 7845c90, 785f21b, 7e395a1, 838e5f0, 85299d1, 8ac6014, 8b9666a, 8d2d00f, 8fcaa3f, 901a477, 98b37ef, 9ae38e8, 9aec846, 9bc6c78, 9d70d66, 9f878fa, a0aff81, a94d9ff, a975de0, aa0262b, aac8ae8, ad5a35d, b039002, b11dc77, b32ed86, b3dd576, b49b1bb, bdf1150, be06904, be19c24, c255466, c523396, ca046bc, cd5d111, d095ccd, d38fb2f, dd93e9b, df5a245, e3ace6e, e95395b, ec4a29f, ecbb139, ef0b9d0, f2ac791, f8d6898, f9e9d0b, fe373d3)
• ❌ The email address for the commit (f7ddf43) is not linked to the GitHub account, preventing the EasyCLA check. Consult this Help Article and GitHub Help to resolve. (To view the commit's email address, add .patch at the end of this PR page's URL.) For further assistance with EasyCLA, please visit our EasyCLA portal and chat with our support bot.

One or more co-authors of this pull request were not found. You must specify co-authors in commit message trailer via:

Co-authored-by: name <email>

Supported Co-authored-by: formats include:

1. Anything <id+login@users.noreply.github.com> - it will locate your GitHub user by id part.
2. Anything <login@users.noreply.github.com> - it will locate your GitHub user by login part.
3. Anything <public-email> - it will locate your GitHub user by public-email part. Note that this email must be made public on Github.
4. Anything <other-email> - it will locate your GitHub user by other-email part but only if that email was used before for any other CLA as a main commit author.
5. login <any-valid-email> - it will locate your GitHub user by login part, note that login part must be at least 3 characters long.

Alternatively, if the co-author should not be included, remove the Co-authored-by: line from the commit message.

Please update your commit message(s) by doing git commit --amend and then git push [--force] and then request re-running CLA check via commenting on this pull request:

/easycla