Update on "[ET Device Support] Schema changes: device info on Tensor and buffer-level device array"

This diff adds device placement information to the ExecuTorch schema to support representing tensor-level device type information, which will be the basic requirement for the following tensor_parser updates.

This is part of the Phase 1 implementation to make ET device type work E2E without user-specified device placement.

Design doc: https://docs.google.com/document/d/1lwd9BlohmwkN5EEvRulO_b-XnZBwv1nMb5l2K3jfuwA/edit?tab=t.0#heading=h.o6anuvkix4bu

Differential Revision: [D93635657](https://our.internmc.facebook.com/intern/diff/D93635657/)

[ghstack-poisoned]

Author: Gasoonjia

.claude/skills/building/SKILL.md

@@ -1,23 +1,223 @@
 ---
 name: building
-description: Build ExecuTorch runners or C++ libraries. Use when compiling runners for Llama, Whisper, or other models, or building the C++ runtime.
+description: Build ExecuTorch from source — Python package, C++ runtime, runners, cross-compilation, and backend-specific builds. Use when compiling anything in the ExecuTorch repo, diagnosing build failures, or setting up platform-specific builds.
 ---
 
-# Building
+# Building ExecuTorch
 
-## Runners (Makefile)
+## Step 1: Ensure Python environment (detect and fix automatically)
+
+**Path A — conda (preferred):**
+```bash
+# Initialize conda for non-interactive shells (required in Claude Code / CI)
+eval "$(conda shell.bash hook 2>/dev/null)"
+
+# Check if executorch conda env exists; create if not
+conda env list 2>/dev/null | grep executorch || \
+  ls "$(conda info --base 2>/dev/null)/envs/" 2>/dev/null | grep executorch || \
+  conda create -yn executorch python=3.12
+
+# Activate
+conda activate executorch
+```
+
+**Path B — no conda (fall back to venv):**
+```bash
+# Find a compatible Python (3.10–3.13). On macOS with only Homebrew Python 3.14+,
+# install a compatible version first: brew install python@3.12
+python3.12 -m venv .executorch-venv   # or python3.11, python3.10, python3.13
+source .executorch-venv/bin/activate
+pip install --upgrade pip
+```
+
+**Then verify (either path):**
+
+Run `python --version` and `cmake --version`. Fix automatically:
+- **Python not 3.10–3.13**: recreate the env with a correct Python version.
+- **cmake missing or < 3.24**: run `pip install 'cmake>=3.24'` inside the env.
+- **cmake >= 4.0**: works in practice, no action needed.
+
+Parallel jobs: `$(sysctl -n hw.ncpu)` on macOS, `$(nproc)` on Linux.
+
+## Step 2: Build
+
+Route based on what the user asks for:
+- User mentions **Android** → skip to [Cross-compilation: Android](#cross-compilation)
+- User mentions **iOS** or **frameworks** → skip to [Cross-compilation: iOS](#cross-compilation)
+- User mentions a **model name** (llama, whisper, etc.) → skip to [LLM / ASR model runner](#llm--asr-model-runner-simplest-path-for-running-models)
+- User mentions **C++ runtime** or **cmake** → skip to [C++ runtime](#c-runtime-standalone)
+- Otherwise → default to **Python package** below
+
+### Python package (default)
 ```bash
-make help              # list all targets
-make llama-cpu         # Llama
-make whisper-metal     # Whisper on Metal
-make gemma3-cuda       # Gemma3 on CUDA
+conda activate executorch
+./install_executorch.sh --editable    # editable install from source
 ```
+This handles everything: submodules, deps, C++ build, Python install. Takes ~10 min on Apple Silicon.
+
+For subsequent rebuilds (deps already present): `pip install -e . --no-build-isolation`
+
+For minimal install (skip example deps): `./install_executorch.sh --minimal`
+
+Enable additional backends:
+```bash
+CMAKE_ARGS="-DEXECUTORCH_BUILD_COREML=ON -DEXECUTORCH_BUILD_MPS=ON" ./install_executorch.sh --editable
+```
+
+Verify: `python -c "from executorch.exir import to_edge_transform_and_lower; print('OK')"`
+
+### LLM / ASR model runner (simplest path for running models)
+
+```bash
+conda activate executorch
+make <model>-<backend>
+```
+
+Available targets (run `make help` for full list):
+
+| Target | Backend | macOS | Linux |
+|--------|---------|-------|-------|
+| `llama-cpu` | CPU | yes | yes |
+| `llama-cuda` | CUDA | — | yes |
+| `llama-cuda-debug` | CUDA (debug) | — | yes |
+| `llava-cpu` | CPU | yes | yes |
+| `whisper-cpu` | CPU | yes | yes |
+| `whisper-metal` | Metal | yes | — |
+| `whisper-cuda` | CUDA | — | yes |
+| `parakeet-cpu` | CPU | yes | yes |
+| `parakeet-metal` | Metal | yes | — |
+| `parakeet-cuda` | CUDA | — | yes |
+| `voxtral-cpu` | CPU | yes | yes |
+| `voxtral-cuda` | CUDA | — | yes |
+| `voxtral-metal` | Metal | yes | — |
+| `voxtral_realtime-cpu` | CPU | yes | yes |
+| `voxtral_realtime-cuda` | CUDA | — | yes |
+| `voxtral_realtime-metal` | Metal | yes | — |
+| `gemma3-cpu` | CPU | yes | yes |
+| `gemma3-cuda` | CUDA | — | yes |
+| `sortformer-cpu` | CPU | yes | yes |
+| `sortformer-cuda` | CUDA | — | yes |
+| `silero-vad-cpu` | CPU | yes | yes |
+| `clean` | — | yes | yes |
 
 Output: `cmake-out/examples/models/<model>/<runner>`
 
-## C++ Libraries (CMake)
+### C++ runtime (standalone)
+
+**With presets (recommended):**
+
+| Platform | Command |
+|----------|---------|
+| macOS | `cmake -B cmake-out --preset macos` (uses Xcode generator — requires Xcode) |
+| Linux | `cmake -B cmake-out --preset linux -DCMAKE_BUILD_TYPE=Release` |
+| Windows | `cmake -B cmake-out --preset windows -T ClangCL` |
+
+Then: `cmake --build cmake-out --config Release -j$(sysctl -n hw.ncpu)` (macOS) or `cmake --build cmake-out -j$(nproc)` (Linux)
+
+**LLM libraries via workflow presets** (configure + build + install in one command):
+```bash
+cmake --workflow --preset llm-release        # CPU
+cmake --workflow --preset llm-release-metal  # Metal (macOS)
+cmake --workflow --preset llm-release-cuda   # CUDA (Linux/Windows)
+```
+
+**Manual CMake (custom flags):**
+```bash
+cmake -B cmake-out \
+  -DCMAKE_BUILD_TYPE=Release \
+  -DEXECUTORCH_BUILD_XNNPACK=ON \
+  -DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
+  -DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
+  -DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
+  -DEXECUTORCH_BUILD_EXTENSION_NAMED_DATA_MAP=ON \
+  -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
+  -DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON
+cmake --build cmake-out --parallel "$(nproc 2>/dev/null || sysctl -n hw.ncpu)"
+```
+
+Run `cmake --list-presets` to see all available presets.
+
+### Cross-compilation
+
+**iOS/macOS frameworks:**
+```bash
+./scripts/build_apple_frameworks.sh --coreml --mps --xnnpack
+```
+Link in Xcode with `-all_load` linker flag.
+
+**Android:**
+
+Requires `ANDROID_NDK` on PATH (typically set by Android Studio or standalone NDK install).
 ```bash
-cmake --list-presets                    # list presets
-cmake --workflow --preset llm-release   # LLM CPU
-cmake --workflow --preset llm-release-metal  # LLM Metal
+# Verify NDK is available
+echo $ANDROID_NDK           # must point to NDK root, e.g. ~/Library/Android/sdk/ndk/<version>
+export ANDROID_ABIS=arm64-v8a BUILD_AAR_DIR=aar-out
+mkdir -p $BUILD_AAR_DIR && sh scripts/build_android_library.sh
 ```
+
+## Key build options
+
+Most commonly needed flags (full list: `CMakeLists.txt`):
+
+| Flag | What it enables |
+|------|-----------------|
+| `EXECUTORCH_BUILD_XNNPACK` | XNNPACK CPU backend |
+| `EXECUTORCH_BUILD_COREML` | Core ML (macOS/iOS) |
+| `EXECUTORCH_BUILD_MPS` | MPS GPU (macOS/iOS) |
+| `EXECUTORCH_BUILD_METAL` | Metal compute (macOS, requires EXTENSION_TENSOR) |
+| `EXECUTORCH_BUILD_CUDA` | CUDA GPU (Linux/Windows, requires EXTENSION_TENSOR) |
+| `EXECUTORCH_BUILD_KERNELS_OPTIMIZED` | Optimized kernels |
+| `EXECUTORCH_BUILD_KERNELS_QUANTIZED` | Quantized kernels |
+| `EXECUTORCH_BUILD_EXTENSION_MODULE` | Module extension (requires DATA_LOADER + FLAT_TENSOR + NAMED_DATA_MAP) |
+| `EXECUTORCH_BUILD_EXTENSION_LLM` | LLM extension |
+| `EXECUTORCH_BUILD_TESTS` | Unit tests (`ctest --test-dir cmake-out --output-on-failure`) |
+| `EXECUTORCH_BUILD_DEVTOOLS` | DevTools (Inspector, ETDump) |
+| `EXECUTORCH_OPTIMIZE_SIZE` | Size-optimized build (`-Os`, no exceptions/RTTI) |
+| `CMAKE_BUILD_TYPE` | `Release` or `Debug` (5-10x slower). Some presets (e.g. `llm-release`) set this; others require it explicitly. |
+
+## Troubleshooting
+
+| Symptom | Fix |
+|---------|-----|
+| Missing headers / `CMakeLists.txt not found` in third-party | `git submodule sync --recursive && git submodule update --init --recursive` |
+| Mysterious failures after `git pull` or branch switch | `rm -rf cmake-out/ pip-out/ && git submodule sync && git submodule update --init --recursive` |
+| `conda env list` PermissionError | Use `CONDA_NO_PLUGINS=true conda env list` or check env dir directly |
+| CMake >= 4.0 | Works in practice despite `< 4.0` in docs; only fix if build actually fails |
+| `externally-managed-environment` / PEP 668 error | You're using system Python, not conda. Activate conda env first. |
+| pip conflicts with torch versions | Fresh conda env; or `./install_executorch.sh --use-pt-pinned-commit` |
+| Missing `Python.h` (Linux) | `sudo apt install python3.X-dev` |
+| Missing operator registrations at runtime | Link kernel libs with `-Wl,-force_load,<lib>` (macOS) or `-Wl,--whole-archive <lib> -Wl,--no-whole-archive` (Linux) |
+| `install_executorch.sh` fails on Intel Mac | No prebuilt PyTorch wheels; use `--use-pt-pinned-commit --minimal` |
+| XNNPACK build errors about cpuinfo/pthreadpool | Ensure `EXECUTORCH_BUILD_CPUINFO=ON` and `EXECUTORCH_BUILD_PTHREADPOOL=ON` (both ON by default) |
+| Duplicate kernel registration abort | Only link one `gen_operators_lib` per target |
+
+## Build output
+
+**From `./install_executorch.sh` (Python package):**
+
+| Artifact | Location |
+|----------|----------|
+| Python package | `site-packages/executorch` |
+
+**From CMake builds** (`cmake --install` with `CMAKE_INSTALL_PREFIX=cmake-out`):
+
+| Artifact | Location |
+|----------|----------|
+| Core runtime | `cmake-out/lib/libexecutorch.a` |
+| XNNPACK backend | `cmake-out/lib/libxnnpack_backend.a` |
+| executor_runner | `cmake-out/executor_runner` (Ninja/Make) or `cmake-out/Release/executor_runner` (Xcode) |
+| Model runners | `cmake-out/examples/models/<model>/<runner>` |
+
+**From cross-compilation:**
+
+| Artifact | Location |
+|----------|----------|
+| iOS frameworks | `cmake-out/*.xcframework` |
+| Android AAR | `aar-out/` |
+
+## Tips
+- Always use `Release` for benchmarking; `Debug` is 5–10x slower
+- `ccache` is auto-detected if installed (`brew install ccache`)
+- `Ninja` is faster than Make (`-G Ninja`) — but `--preset macos` uses Xcode generator
+- For LLM workflows, `make <model>-<backend>` is the simplest path
+- After `git pull`, clean and re-init submodules before rebuilding

backends/arm/_passes/__init__.py

@@ -136,6 +136,7 @@
 from .rewrite_le_lt_to_ge_gt_pass import RewriteLeLtToGeGtPass  # noqa
 from .rewrite_matmul import RewriteMatmulPass  # noqa
 from .rewrite_pad import RewritePadPass  # noqa
+from .rewrite_slice import RewriteSlicePass  # noqa
 from .rewrite_upsample import RewriteUpsamplePass  # noqa
 from .scalars_to_attribute_pass import ScalarsToAttributePass  # noqa
 from .size_adjust_input_pass import SizeAdjustInputPass  # noqa

backends/arm/_passes/arm_pass_manager.py

@@ -121,6 +121,7 @@
     RewriteLeLtToGeGtPass,
     RewriteMatmulPass,
     RewritePadPass,
+    RewriteSlicePass,
     RewriteUpsamplePass,
     ScalarsToAttributePass,
     SizeAdjustInputPass,
@@ -374,6 +375,7 @@ def _tosa_pipeline(
                 RewriteConvPass(exported_program),
                 RewriteMatmulPass(),
                 RewritePadPass(),
+                RewriteSlicePass(),
             ]
         )
 

backends/arm/_passes/arm_pass_utils.py

@@ -8,7 +8,7 @@
 
 import traceback
 from inspect import isclass
-from typing import Optional, Sequence
+from typing import List, Optional, Sequence, Tuple
 
 import torch
 import torch.fx
@@ -17,6 +17,10 @@
 from executorch.exir import ExportedProgram
 from executorch.exir.dialects._ops import ops as exir_ops
 from executorch.exir.dialects.edge._ops import EdgeOpOverload
+from executorch.exir.graph_module import (
+    _get_control_flow_submodules,
+    get_control_flow_submodules,
+)
 
 from torch._export.utils import (
     get_buffer,
@@ -29,6 +33,7 @@
 from torch._ops import OpOverload
 from torch._subclasses.fake_tensor import FakeTensor
 from torch.export.graph_signature import InputKind
+from torch.fx import GraphModule, Node
 
 
 def is_submodule_node(node: torch.fx.Node):
@@ -284,3 +289,45 @@ def set_node_arg(node: torch.fx.Node, i: int | str, value):
 def get_output_dim_orders(graph_module):
     output_node = graph_module.graph.output_node()
     return [get_first_fake_tensor(node).dim_order() for node in output_node.args[0]]
+
+
+def is_nested_control_flow_graph(graph_module: GraphModule) -> bool:
+    """Returns True if graph_module is a nested control-flow graph."""
+
+    # Find all top-level control-flow submodules
+    top_cf = get_control_flow_submodules(graph_module)
+    # For each submodule, see if it itself has control-flow inside
+    for _, submod, _ in top_cf:
+        if get_control_flow_submodules(submod):
+            return True
+    return False
+
+
+def get_cond_while_submodules_nested(
+    graph_module: GraphModule,
+    apply_quantization: bool = False,
+) -> List[Tuple[str, GraphModule, Node]]:
+    """Recursively find cond/while_loop submodules in an GraphModule.
+
+    In nested control flow graphs, FX records the submodule functions
+    (true/false or cond/body) in reverse order compared to top-level graphs. We
+    must swap the indices when nested so that cond (first) and body/true_fn
+    (second) are consistently identified across all nesting levels.
+
+    """
+
+    # Determine arg indices based on nesting and whether only cond branch is needed
+    nested = is_nested_control_flow_graph(graph_module)
+    # cond: [true_fn, false_fn] or swapped if nested
+    cond_indices = [2, 1] if nested else [1, 2]
+    # while_loop: [cond_fn, body_fn] or swapped if nested
+    while_indices = [1, 0] if nested else [0, 1]
+    if apply_quantization:
+        # only keep the cond_fn for while_loop (first index) when quantizing.
+        while_indices = [while_indices[0]]
+    mapping = {
+        torch.ops.higher_order.cond: cond_indices,
+        torch.ops.higher_order.while_loop: while_indices,
+    }
+    # collect cond/while submodules (using mapping indices)
+    return _get_control_flow_submodules(graph_module, mapping)

backends/arm/_passes/control_flow_const_inline.py

@@ -5,11 +5,14 @@
 
 from typing import Set, Type
 
+import torch
 from executorch.backends.arm._passes.arm_pass import ArmPass
+from executorch.backends.arm._passes.arm_pass_utils import (
+    get_cond_while_submodules_nested,
+    is_submodule_node,
+)
 from executorch.backends.transforms.utils import is_get_attr_node
 from executorch.exir.dialects._ops import ops as exir_ops
-from executorch.exir.graph_module import get_cond_while_submodules
-
 from executorch.exir.pass_base import ExportPass, PassResult
 from torch.fx import GraphModule
 
@@ -27,15 +30,23 @@ class ControlFlowConstInlinePass(ArmPass):
 
     _passes_required_after: Set[Type[ExportPass]] = set()
 
-    def __init__(self, *args, **kwargs):
-        super().__init__(*args, **kwargs)
+    _targeted_ops = {
+        torch.ops.higher_order.cond,
+        torch.ops.higher_order.while_loop,
+    }
 
-    def call(self, graph_module: GraphModule) -> PassResult:
+    def _convert_getattr(self, graph_module):
         modified = False
-
-        for _, submodule, _ in get_cond_while_submodules(graph_module):
+        for _, submodule, _ in get_cond_while_submodules_nested(graph_module):
             for submodule_node in submodule.graph.nodes:
-                if is_get_attr_node(submodule_node):
+                if submodule_node.target in self._targeted_ops:
+                    self._convert_getattr(submodule)
+
+                # For nested control flow, a "node" may be may actually be GraphModule.
+                # Enure we are only checking for nodes here.
+                if is_get_attr_node(submodule_node) and not is_submodule_node(
+                    submodule_node
+                ):
                     val = getattr(
                         submodule_node.graph.owning_module, submodule_node.target
                     )
@@ -53,6 +64,14 @@ def call(self, graph_module: GraphModule) -> PassResult:
                     submodule_node.replace_all_uses_with(const_node)
                     submodule.graph.erase_node(submodule_node)
                     modified = True
+        return modified
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+
+    def call(self, graph_module: GraphModule) -> PassResult:
+
+        modified = self._convert_getattr(graph_module)
 
         if modified:
             graph_module.recompile()