clean-up

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: namgyu-youn

.claude/skills/ci-fails-ort/SKILL.md

@@ -0,0 +1,70 @@
+---
+name: ci-fails-ort
+description: Fetch and diagnose ONNX Runtime CI failures (GitHub Actions + Azure DevOps pipelines) on a PR, with focus on the CUDA/TensorRT workflows. Use when investigating failing CI checks on a PR, when the user pastes a GitHub Actions or dev.azure.com URL, or asks to fetch/diagnose CI logs.
+---
+
+# Diagnosing ONNX Runtime CI Failures
+
+ORT PRs run two CI systems, both public (no login needed for logs):
+
+1. **GitHub Actions** — `.github/workflows/*.yml` (the CUDA-relevant ones:
+   `linux_cuda_ci.yml`, `windows_cuda.yml`, `linux_tensorrt_ci.yml`,
+   `linux_cuda_plugin_ci.yml`, `lint.yml`).
+2. **Azure DevOps pipelines** — defined in
+   `tools/ci_build/github/azure-pipelines/`, shown as "Azure Pipelines" checks
+   on the PR; results live at `dev.azure.com/onnxruntime/onnxruntime/_build`.
+
+## Step 1 — enumerate failing checks
+
+```bash
+gh pr checks <PR> -R microsoft/onnxruntime | grep -v pass
+```
+
+Each line carries the check name and a URL — the URL's host tells you which
+system to fetch from.
+
+## Step 2a — GitHub Actions logs
+
+```bash
+# run id is the number in .../actions/runs/<run-id>/...
+gh run view <run-id> -R microsoft/onnxruntime --log-failed > ci-<run-id>.log
+
+# or per-job (job id from the check URL's /job/<job-id> suffix):
+gh run view -R microsoft/onnxruntime --job <job-id> --log-failed
+```
+
+## Step 2b — Azure DevOps logs
+
+From a check URL like `.../results?buildId=<id>`, the REST API serves logs
+anonymously:
+
+```bash
+curl -s "https://dev.azure.com/onnxruntime/onnxruntime/_apis/build/builds/<id>/logs?api-version=7.1" \
+  | python3 -c "import json,sys; [print(l['id'], l['lineCount']) for l in json.load(sys.stdin)['value']]"
+curl -s "https://dev.azure.com/onnxruntime/onnxruntime/_apis/build/builds/<id>/logs/<log-id>?api-version=7.1" > ci-azp-<id>.log
+```
+
+Maintainers can rerun individual Azure pipelines with a PR comment
+(`/azp run <pipeline name>`); external contributors need a maintainer to
+trigger it.
+
+## Step 3 — classify the failure
+
+Work from the bottom of the log up; the build.py invocation echoes the exact
+failing command. Common ORT failure classes:
+
+| Symptom | Likely cause |
+|---|---|
+| `lintrunner` diff in `lint.yml` | run `lintrunner -a` locally, commit the fixes |
+| `OperatorKernels.md` / `ContribOperators.md` mismatch | regenerate the docs (`tools/python/gen_doc` scripts referenced in the failing step) after kernel/schema changes |
+| gtest failure only in the CUDA workflow | arch/driver-dependent kernel dispatch — check which attention/gemm backend the CI GPU selects vs yours (see [write-ort-test](../write-ort-test/SKILL.md) on pinning backends) |
+| fp16 tolerance failures on one GPU generation | tolerance too tight for a different kernel path, not necessarily your bug — compare against the baseline run on main |
+| timeout in `onnxruntime_test_all` | usually a hang in a newly added test's EP wait, not slowness |
+
+Reproduce locally with the same flags CI used: the workflow yml shows the
+`tools/ci_build/build.py` arguments; run the failing gtest via
+`onnxruntime_test_all --gtest_filter=<failing test>`.
+
+Flaky-looking failures: check whether the same check fails on recent main
+(`gh run list -R microsoft/onnxruntime --workflow <workflow> --branch main -L 10`)
+before bisecting your own change.

.claude/skills/debug-ep-placement/SKILL.md

@@ -0,0 +1,111 @@
+---
+name: debug-ep-placement
+description: Debug ONNX Runtime node placement and CUDA kernel dispatch for LLM/VLM models — which nodes fall back to CPU, where Memcpy nodes appear, and which attention/gemm kernel a contrib op actually dispatches to. Use when a CUDA EP run is slower than expected, outputs differ across machines, or you need to confirm which kernel path executed.
+---
+
+# Debugging EP Placement and CUDA Kernel Dispatch
+
+Two distinct questions, two toolsets:
+
+1. **Graph level** — which nodes run on `CUDAExecutionProvider` vs fall back
+   to CPU, and where `MemcpyToHost`/`MemcpyFromHost` transfer nodes got
+   inserted at the boundary.
+2. **Kernel level** — which backend an LLM contrib op (Attention,
+   MultiHeadAttention, GroupQueryAttention) actually dispatched to: flash
+   attention, memory-efficient (cutlass), cuDNN flash, TRT fused, lean, or the
+   unfused fallback.
+
+## Step 1 — Node placement (graph level)
+
+Verbose session logging prints a "Node placements" section listing every node
+grouped by EP (`onnxruntime/core/framework/session_state.cc`):
+
+```python
+import onnxruntime as ort
+so = ort.SessionOptions()
+so.log_severity_level = 0      # VERBOSE
+so.optimized_model_filepath = "/tmp/optimized.onnx"   # inspect in Netron
+sess = ort.InferenceSession(model, so,
+    providers=["CUDAExecutionProvider", "CPUExecutionProvider"])
+```
+
+Read the placements for:
+
+- Compute nodes on `CPUExecutionProvider` — each one forces a device sync +
+  transfer. For LLM decode loops this is usually the whole regression.
+- `MemcpyToHost` / `MemcpyFromHost` nodes — inserted automatically at EP
+  boundaries; their count and position show exactly where data bounces.
+
+A profile gives the same answer quantitatively: every `Node` event carries
+`args.provider`, and the [ort-profile-analysis skill](../ort-profile-analysis/SKILL.md)
+prints a per-provider table plus Memcpy share.
+
+Common causes of CPU fallback for transformer models:
+
+- No CUDA kernel for the op/dtype combination — check `docs/OperatorKernels.md`
+  (CUDA column) for the op and opset actually in the graph.
+- Shape/attribute constraints a kernel doesn't support (e.g. int64 compute,
+  dynamic axes a fused op rejects).
+- The graph wasn't fused: run the transformer optimizer first
+  (`python -m onnxruntime.transformers.optimizer --model_type <t> ...` or the
+  fusion logic in `onnxruntime/python/tools/transformers/`) so Attention/
+  SkipLayerNorm/MatMulNBits contrib ops exist for CUDA to claim.
+
+## Step 2 — Attention kernel dispatch (kernel level)
+
+Attention backend selection is centralized in
+`onnxruntime/contrib_ops/cuda/bert/attention_kernel_options.cc`, driven by the
+CUDA provider option `sdpa_kernel` and the `ORT_*` env vars declared in
+`onnxruntime/contrib_ops/cpu/bert/attention_common.h`.
+
+To see what actually ran:
+
+```bash
+export ORT_ENABLE_ATTENTION_KERNEL_DEBUG_INFO=1   # prints chosen kernel per op
+```
+
+To pin or ablate a backend (bisection of a perf/precision difference):
+
+```bash
+export ORT_DISABLE_FLASH_ATTENTION=1
+export ORT_DISABLE_MEMORY_EFFICIENT_ATTENTION=1
+export ORT_ENABLE_CUDNN_FLASH_ATTENTION=1
+export ORT_ENABLE_LEAN_ATTENTION=1
+# thresholds that silently change dispatch with sequence length:
+export ORT_MIN_SEQ_LEN_FLASH_ATTENTION_PACKED_QKV=<n>
+export ORT_MIN_SEQ_LEN_EFFICIENT_ATTENTION_FP32=<n>
+```
+
+Remember dispatch also depends on SM architecture, dtype, head size, and
+past/present KV layout — a result that differs between two GPUs is often just
+two different backends. Always record the resolved kernel (debug-info output)
+next to any benchmark number.
+
+## Step 3 — Confirm with a profile
+
+Capture a profile (see [generate-profile](../generate-profile/SKILL.md)); with
+a `--enable_cuda_profiling` build the `Kernel`-category events name the actual
+CUDA kernels (e.g. `flash_fwd_kernel` vs `fmha_cutlassF_*`), which is ground
+truth when the debug-info flag isn't available in the installed wheel.
+`nsys profile` + `onnxruntime/test/python/transformers/parse_nsys.py` is the
+lower-overhead alternative for kernel-level timing.
+
+## Step 4 — Node input/output dumping (debug builds)
+
+For numerical divergence (not placement), build with
+`--cmake_extra_defines onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS=1` and use the
+`ORT_DEBUG_NODE_IO_*` env vars
+(`onnxruntime/core/framework/debug_node_inputs_outputs_utils.h`):
+`ORT_DEBUG_NODE_IO_DUMP_NODE_PLACEMENT=1`, `ORT_DEBUG_NODE_IO_DUMP_OUTPUT_DATA=1`,
+`ORT_DEBUG_NODE_IO_NAME_FILTER=<substr>` to bisect the first diverging node
+between two configurations.
+
+## Quick Reference
+
+| Question | Tool |
+|---|---|
+| Which nodes fell back to CPU? | `log_severity_level=0` → "Node placements"; profile per-provider table |
+| Where do device↔host copies happen? | `MemcpyToHost/FromHost` nodes in optimized model / profile |
+| Which attention backend ran? | `ORT_ENABLE_ATTENTION_KERNEL_DEBUG_INFO=1`; Kernel events in profile |
+| Force a specific backend | `ORT_DISABLE_*` / `ORT_ENABLE_*` env vars, `sdpa_kernel` provider option |
+| First numerically diverging node | debug build + `ORT_DEBUG_NODE_IO_*` |

.claude/skills/env-var-conventions/SKILL.md

@@ -0,0 +1,86 @@
+---
+name: env-var-conventions
+description: Conventions for ONNX Runtime configuration knobs — session/run config keys vs provider options vs ORT_* environment variables, parsing helpers, and naming. Use when adding, renaming, or reviewing any ORT_* environment variable, CUDA provider option, or session config entry.
+---
+
+# Configuration Knobs — Conventions
+
+ORT has **no central env-var registry** (nothing like a single `envs.py`).
+The order of preference for a new knob is fixed; env vars are the last resort.
+
+## Rule 1 — Prefer config surfaces over env vars
+
+| If the knob is… | Use |
+|---|---|
+| Per-session behavior (optimization, memory, debug dump) | Session config entry: string key in `include/onnxruntime/core/session/onnxruntime_session_options_config_keys.h` (`kOrtSessionOptions*`, keys namespaced like `"session.x"`, `"ep.x"`), read via `session_options.config_options.GetConfigOrDefault(...)` |
+| Per-run behavior | Run config entry: `onnxruntime_run_options_config_keys.h` |
+| EP-specific (CUDA arena, TF32, `sdpa_kernel`, TRT engine cache) | Provider option: `include/onnxruntime/core/providers/cuda/cuda_provider_options.h` + parsing in `cuda_execution_provider_info.cc`; exposed to Python via `providers=[("CUDAExecutionProvider", {...})]` |
+| Process-wide expert toggle, kill-switch, or debug/CI-only hook with no session in scope | `ORT_*` env var (rules below) |
+
+The TensorRT EP's `ORT_TENSORRT_*` env vars are the cautionary precedent:
+they predate provider options and survive only as legacy fallbacks — don't
+copy that pattern for new work. When both surfaces exist, the provider option
+must win over the env var.
+
+## Rule 2 — Parsing
+
+Always parse through `onnxruntime/core/platform/env_var_utils.h`, never raw
+`getenv` arithmetic:
+
+```cpp
+#include "core/platform/env_var_utils.h"
+
+// optional<T>; empty when unset/unparseable
+auto v = ParseEnvironmentVariable<int>("ORT_MY_KNOB");
+// or with default:
+bool flag = ParseEnvironmentVariableWithDefault<bool>("ORT_MY_KNOB", false);
+// test-only knobs log a warning when used:
+auto t = ParseTestOnlyEnvironmentVariable<int>("ORT_TEST_ONLY_KNOB", ...);
+```
+
+Read the var **once at session/provider/kernel-options initialization**, cache
+it in the options struct, and pass it down — never call `ParseEnvironmentVariable`
+inside `Compute()`/per-Run paths. `attention_kernel_options.cc` is the model:
+env vars are parsed once into `AttentionKernelOptions`, and the CUDA provider
+option (`sdpa_kernel`) takes precedence over the env-var defaults.
+
+## Rule 3 — Naming
+
+- Prefix `ORT_`; feature families share a token so they grep together:
+  the attention set uses `ORT_DISABLE_FLASH_ATTENTION`,
+  `ORT_ENABLE_LEAN_ATTENTION`, `ORT_MIN_SEQ_LEN_*`
+  (all declared as `constexpr const char*` in
+  `onnxruntime/contrib_ops/cpu/bert/attention_common.h` — declare names as
+  constants next to their consumers like this, never inline string literals).
+- `ORT_DISABLE_X` for opting out of a default-on path, `ORT_ENABLE_X` for
+  opting into a default-off one — don't create double negatives.
+- Debug-build instrumentation follows the `ORT_DEBUG_NODE_IO_*` family style
+  (`onnxruntime/core/framework/debug_node_inputs_outputs_utils.h`).
+- EP-specific vars carry the EP in the name (`ORT_TENSORRT_*` — legacy only,
+  see Rule 1).
+
+## Rule 4 — Kernel-dispatch knobs affect benchmarks
+
+Any env var that changes CUDA kernel selection (attention backend, min-seq-len
+thresholds, fused-op toggles) invalidates perf comparisons that don't hold it
+fixed. When adding one:
+
+- default must preserve current dispatch on all architectures;
+- document it in the same header as the constant;
+- mention it in the PR description so benchmark scripts can pin it — and
+  record it in any runbook this workdir produces (see
+  [debug-ep-placement](../debug-ep-placement/SKILL.md)).
+
+## Renames and removals
+
+There is no alias machinery. For a rename, read the old name as a fallback at
+the single parse site, log a deprecation warning there, and note the removal
+target. Never flip a default in the same change as a rename.
+
+## Review checklist
+
+- [ ] Could this be a session config key or provider option instead? (usually yes)
+- [ ] Name declared as a constant next to the consumer, `ORT_` prefixed
+- [ ] Parsed once at init via `env_var_utils.h`, not per-Run
+- [ ] Provider option / config entry takes precedence if both exist
+- [ ] Dispatch-affecting knob documented for benchmark pinning

.claude/skills/generate-profile/SKILL.md

@@ -0,0 +1,79 @@
+---
+name: generate-profile
+description: Capture an ONNX Runtime profile of a CUDA EP run for an LLM/VLM model — session profiling (chrome-trace JSON), onnxruntime_perf_test -p, CUPTI kernel events via --enable_cuda_profiling, and Nsight Systems. Returns the profile path for analysis.
+---
+
+# Generate a Profile of an ORT CUDA Run
+
+ORT's built-in profiler writes a chrome-trace JSON per session. Reference
+implementation: `onnxruntime/core/common/profiler.cc`. Profiling adds
+overhead — keep the profiled run short, and never compare a profiled number
+against an unprofiled one.
+
+## Option A — Python session profiling (LLM/VLM scripts)
+
+```python
+import onnxruntime as ort
+
+so = ort.SessionOptions()
+so.enable_profiling = True
+# so.profile_file_prefix = "gqa_fused"       # optional
+
+sess = ort.InferenceSession(model_path, so,
+    providers=[("CUDAExecutionProvider", {"device_id": 0}), "CPUExecutionProvider"])
+
+for _ in range(3 + 10):  # first iterations include first-run allocations and
+    sess.run(None, feeds)  # kernel autotuning — aggregate over the later ones
+
+path = sess.end_profiling()   # onnxruntime_profile_<prefix>_<ts>.json
+print(path)
+```
+
+For decode-loop workloads, profile prefill and decode as separate sessions/
+runs where possible so per-op aggregates aren't a blend of the two regimes.
+Record next to the trace: GPU model, wheel version, and any `ORT_*` attention
+env vars in effect (they change kernel dispatch — see
+[debug-ep-placement](../debug-ep-placement/SKILL.md)).
+
+## Option B — onnxruntime_perf_test
+
+For a model file with generated inputs, no script needed:
+
+```bash
+./build/Linux/RelWithDebInfo/onnxruntime_perf_test \
+    -e cuda -m times -r 100 -p /tmp/myprofile model.onnx
+# -p <prefix> enables profiling and dumps the profile file
+# -I generates dummy inputs when you have no input data
+```
+
+## Kernel-level events — CUPTI build
+
+A stock build's `Node` events time the host-side kernel launch. For actual
+GPU kernel names/durations (`Kernel`-category events in the same JSON), the
+wheel must be built with:
+
+```bash
+./build.sh --config RelWithDebInfo --use_cuda --enable_cuda_profiling ...
+```
+
+## Nsight Systems alternative (lower overhead)
+
+```bash
+nsys profile -o gqa_run python benchmark_script.py
+python onnxruntime/test/python/transformers/parse_nsys.py gqa_run.nsys-rep
+```
+
+The in-tree LLM kernel benchmarks (`onnxruntime/test/python/transformers/`:
+`benchmark_mha.py`, `benchmark_gqa.py`, `profile_gqa.py`,
+`profile_matmul_nbits.py`) already wire this up — extend those before writing
+a new harness.
+
+## Viewing and analyzing
+
+- **Perfetto UI**: <https://ui.perfetto.dev/> (drag-and-drop) or `chrome://tracing`.
+- For op/provider/kernel aggregate tables, hand the JSON to the
+  [ort-profile-analysis skill](../ort-profile-analysis/SKILL.md).
+
+Trace anatomy: events carry `cat` ∈ {`Session`, `Node`, `Kernel`, `Api`};
+per-node timing is the `<node_name>_kernel_time` events, whose `args` include
+`op_name` and `provider`.