|
| 1 | +--- |
| 2 | +name: webgpu-local-testing |
| 3 | +description: Build and run ONNX Runtime WebGPU provider tests on Linux WITHOUT a real GPU, using a software Vulkan adapter (Mesa lavapipe). Use when you need to exercise WebGPU EP kernels off-Mac — the Linux webgpu CI leg is build-only, so software Vulkan is how you actually run WebGPU correctness tests locally. SCOPE: lavapipe only validates host-side enforce/shape bugs and MatMul-free kernels; any graph containing MatMul (including the expanded-Attention node tests) crashes lavapipe and runs ONLY on macOS-arm64 Metal, which is the source of truth for those. Covers install (dnf on Azure Linux), the --use_webgpu build flag, the onnxruntime_provider_test target, VK_ICD_FILENAMES, and the lavapipe MatMul crash gotcha. |
| 4 | +--- |
| 5 | + |
| 6 | +# Running ONNX Runtime WebGPU Tests Locally on Linux (No GPU) |
| 7 | + |
| 8 | +Reusable knowledge for exercising the **WebGPU execution provider** on a Linux box |
| 9 | +with no physical GPU. |
| 10 | + |
| 11 | +> **Scope**: Linux ORT WebGPU only. macOS uses the Metal backend and a real GPU; |
| 12 | +> this skill is the off-Mac story for running WebGPU EP kernels in CI-less dev loops. |
| 13 | +
|
| 14 | +## 1. Why software Vulkan is enough |
| 15 | + |
| 16 | +On Linux, ORT's WebGPU EP runs on **Dawn** with the **Vulkan** backend. Vulkan does |
| 17 | +not require a hardware GPU — **Mesa lavapipe** is a software (CPU) Vulkan adapter that |
| 18 | +Dawn enumerates like any other device. For **EP correctness tests** this is sufficient |
| 19 | +because: |
| 20 | + |
| 21 | +- Many host-side validation paths (shape/broadcast checks, `ORT_ENFORCE`s) fire |
| 22 | + **before** any shader is dispatched. E.g. the WebGPU broadcast `ORT_ENFORCE` runs |
| 23 | + host-side, so the failure is observable on a software adapter without ever touching |
| 24 | + the GPU. |
| 25 | +- Element-wise and broadcasting kernels that do dispatch run correctly (if slowly) on |
| 26 | + lavapipe, so their numeric output can be validated against the CPU reference. |
| 27 | + |
| 28 | +You are trading speed for not needing hardware. It is **not** a substitute for a real |
| 29 | +GPU on perf-sensitive or driver-specific paths — but for kernel correctness it is the |
| 30 | +practical local loop. |
| 31 | + |
| 32 | +> **Scope — what lavapipe can and cannot validate.** Software Vulkan covers (a) |
| 33 | +> **host-side** failures (shape/broadcast `ORT_ENFORCE`s that fire *before* any shader |
| 34 | +> dispatch) and (b) **MatMul-free** kernels that dispatch. It does **NOT** cover any |
| 35 | +> graph that contains a **MatMul** — the MatMul family crashes lavapipe's LLVM JIT (see |
| 36 | +> §5). This explicitly includes the motivating expanded-Attention node tests |
| 37 | +> (`test_attention_4d_softcap_neginf_mask_expanded`): they decompose to |
| 38 | +> `softmax(Q·Kᵀ + bias)·V`, which **contains MatMuls**, so they **cannot run on |
| 39 | +> lavapipe**. For those, **macOS-arm64 Metal is the source of truth**. Concretely, the |
| 40 | +> #28969 WebGPU broadcast-underflow fix was validated on lavapipe via a standalone |
| 41 | +> Add-broadcast `OpTester` proxy (a host-side enforce/shape path) — **NOT** via the |
| 42 | +> expanded-Attention node test. **Never** run lavapipe green and conclude an |
| 43 | +> Attention/MatMul fix is validated off-Mac. |
| 44 | +
|
| 45 | +## 2. Install the software Vulkan stack (Azure Linux — use `dnf`, NOT `apt`) |
| 46 | + |
| 47 | +```bash |
| 48 | +dnf install -y mesa-vulkan-drivers vulkan-loader |
| 49 | +# optional, for sanity-checking the adapter: |
| 50 | +dnf install -y vulkan-tools && vulkaninfo | head |
| 51 | +``` |
| 52 | + |
| 53 | +`mesa-vulkan-drivers` provides lavapipe; `vulkan-loader` provides the ICD loader. |
| 54 | +The lavapipe ICD manifest lands at `/usr/share/vulkan/icd.d/lvp_icd.<arch>.json` — |
| 55 | +`lvp_icd.x86_64.json` on x86_64, `lvp_icd.aarch64.json` on arm64. The examples below |
| 56 | +use the x86_64 name; substitute your arch, or glob it: |
| 57 | +`VK_ICD_FILENAMES=$(echo /usr/share/vulkan/icd.d/lvp_icd.*.json)`. |
| 58 | + |
| 59 | +## 3. Build with `--use_webgpu` |
| 60 | + |
| 61 | +```bash |
| 62 | +./build.sh --config Release --parallel --use_webgpu |
| 63 | +``` |
| 64 | + |
| 65 | +See the `ort-build` skill for general build phases and flags. |
| 66 | + |
| 67 | +## 4. Run the WebGPU provider tests |
| 68 | + |
| 69 | +WebGPU operator/kernel tests are **provider op tests** — they build into the |
| 70 | +`onnxruntime_provider_test` target (**NOT** `onnxruntime_test_all`; see the `ort-test` |
| 71 | +skill for the executable taxonomy). Point the Vulkan loader at the lavapipe ICD and |
| 72 | +select a subset with `--gtest_filter`: |
| 73 | + |
| 74 | +```bash |
| 75 | +cd build/Linux/Release |
| 76 | +VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/lvp_icd.x86_64.json \ |
| 77 | + ./onnxruntime_provider_test --gtest_filter="*WebGPU*" |
| 78 | +``` |
| 79 | + |
| 80 | +`VK_ICD_FILENAMES` forces Vulkan to load **only** lavapipe, so the run is |
| 81 | +deterministic regardless of what else is installed. |
| 82 | + |
| 83 | +## 5. Gotcha: lavapipe crashes on the MatMul family |
| 84 | + |
| 85 | +`MathOpTest.MatMulFloatType` (and other MatMul-family tests) crash lavapipe with: |
| 86 | + |
| 87 | +``` |
| 88 | +LLVM ERROR: Instruction Combining did not reach a fixpoint after 1 iterations |
| 89 | +``` |
| 90 | + |
| 91 | +This is a **pre-existing limitation of software Vulkan (Mesa lavapipe's LLVM JIT)**, |
| 92 | +not an ORT bug. **Exclude the MatMul family** from broad lavapipe runs: |
| 93 | + |
| 94 | +```bash |
| 95 | +VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/lvp_icd.x86_64.json \ |
| 96 | + ./onnxruntime_provider_test --gtest_filter="*WebGPU*:-*MatMul*" |
| 97 | +``` |
| 98 | + |
| 99 | +## 6. Why this matters: the Linux webgpu CI leg is build-only |
| 100 | + |
| 101 | +The Linux webgpu CI leg (`py-linux-webgpu-stage.yml`) **only builds** — it does not run |
| 102 | +WebGPU kernels. A green Linux webgpu leg therefore does **not** mean any WebGPU test |
| 103 | +actually executed. The macOS-arm64 webgpu leg is the only CI leg that runs WebGPU |
| 104 | +backend node tests. So a local lavapipe run is the practical way to **actually exercise |
| 105 | +WebGPU kernels off-Mac** before you push. |
| 106 | + |
| 107 | +But mind the §1 scope: lavapipe covers host-side enforce/shape paths and **MatMul-free** |
| 108 | +kernels only. Any **MatMul-containing** graph — including the expanded-Attention node |
| 109 | +tests (`test_attention_4d_softcap_neginf_mask_expanded`) — crashes lavapipe and runs |
| 110 | +**only** on the macOS-arm64 Metal leg, which is the source of truth for those. A green |
| 111 | +lavapipe run never validates a MatMul/Attention fix off-Mac. |
0 commit comments