Add structured stats reporting and GPU memory tracking to Qwen3.5 MoE runner by digantdesai · Pull Request #19190 · pytorch/executorch

digantdesai · 2026-04-28T15:56:53Z

Stack from ghstack (oldest at bottom):

Runner now uses llm::Stats with proper timestamps for model load, prefill,
decode, and GPU memory (via cudaMemGetInfo). Output matches stats.h
print_report format: PyTorchObserver JSON line plus human-readable table.

This commit was authored with the assistance of Claude Code.

… runner Runner now uses llm::Stats with proper timestamps for model load, prefill, decode, and GPU memory (via cudaMemGetInfo). Output matches stats.h print_report format: PyTorchObserver JSON line plus human-readable table. This commit was authored with the assistance of Claude Code. [ghstack-poisoned]

pytorch-bot · 2026-04-28T15:56:57Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19190

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 6 New Failures, 2 Cancelled Jobs, 1 Unrelated Failure

As of commit 87c9947 with merge base cb4e5ae ():

NEW FAILURES - The following jobs have failed:

MLX / test-mlx-qwen35-moe / test-mlx-qwen35-moe (gh)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
Test CUDA Builds / unittest-cuda / linux-job (gh)
backends/cuda/tests/test_int4_matmul.py::TestDequantThenMatmul::test_prefill_short
Test CUDA Windows Export and E2E / export-model-cuda-windows-artifact (facebook, dinov2-small-imagenet1k-1-layer, non-quantized) / linux-job (gh)
RuntimeError: Command docker exec -t 53ee62f849a0c9273d64f249c8795de5db237c07ac2f10513cc3228267cda395 /exec failed with exit code 1
Test CUDA Windows Export and E2E / export-model-cuda-windows-artifact (nvidia, parakeet-tdt, non-quantized) / linux-job (gh)
RuntimeError: Command docker exec -t d1c06e37d65b638ab1ce9ece07329dc128e36c7bff68507bfd65aaba3ef2f576 /exec failed with exit code 1
Test CUDA Windows Export and E2E / export-model-cuda-windows-artifact (nvidia, parakeet-tdt, quantized-int4-weight-only) / linux-job (gh)
RuntimeError: Command docker exec -t bf7922dc5bfeedd22891b25a857b035855f43684d6b63646edc6dcf6a562a8b4 /exec failed with exit code 1
Test Metal Backend / test-metal-qwen35-moe-tiny / macos-job (gh)
/Users/ec2-user/runner/_work/executorch/executorch/pytorch/executorch/examples/models/qwen3_5_moe/main.cpp:367:42: error: use of undeclared identifier 'cudaSuccess'

CANCELLED JOBS - The following jobs were cancelled. Please retry:

pull / test-multimodal-linux (gemma3-4b) / linux-job (gh)
##[error]The operation was canceled.
Test CUDA Builds / export-model-cuda-artifact (SocialLocalMobile, Qwen3.5-35B-A3B-HQQ-INT4, quantized-int4-tile-packed) / linux-job (gh)

FLAKY - The following job failed but was likely due to flakiness present on trunk:

pull / test-qnn-wheel-packages-linux (3.13) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions · 2026-04-28T15:57:50Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

…Qwen3.5 MoE runner" Runner now uses llm::Stats with proper timestamps for model load, prefill, decode, and GPU memory (via cudaMemGetInfo). Output matches stats.h print_report format: PyTorchObserver JSON line plus human-readable table. This commit was authored with the assistance of Claude Code. [ghstack-poisoned]

… runner Runner now uses llm::Stats with proper timestamps for model load, prefill, decode, and GPU memory (via cudaMemGetInfo). Output matches stats.h print_report format: PyTorchObserver JSON line plus human-readable table. This commit was authored with the assistance of Claude Code. ghstack-source-id: c6f86d3 Pull Request resolved: #19190

…Qwen3.5 MoE runner" Runner now uses llm::Stats with proper timestamps for model load, prefill, decode, and GPU memory (via cudaMemGetInfo). Output matches stats.h print_report format: PyTorchObserver JSON line plus human-readable table. This commit was authored with the assistance of Claude Code. [ghstack-poisoned]

… runner Runner now uses llm::Stats with proper timestamps for model load, prefill, decode, and GPU memory (via cudaMemGetInfo). Output matches stats.h print_report format: PyTorchObserver JSON line plus human-readable table. This commit was authored with the assistance of Claude Code. ghstack-source-id: b04dd37 Pull Request resolved: #19190

…Qwen3.5 MoE runner" Runner now uses llm::Stats with proper timestamps for model load, prefill, decode, and GPU memory (via cudaMemGetInfo). Output matches stats.h print_report format: PyTorchObserver JSON line plus human-readable table. This commit was authored with the assistance of Claude Code. [ghstack-poisoned]

… runner Runner now uses llm::Stats with proper timestamps for model load, prefill, decode, and GPU memory (via cudaMemGetInfo). Output matches stats.h print_report format: PyTorchObserver JSON line plus human-readable table. This commit was authored with the assistance of Claude Code. ghstack-source-id: 1c45a1e Pull Request resolved: #19190

…Qwen3.5 MoE runner" Runner now uses llm::Stats with proper timestamps for model load, prefill, decode, and GPU memory (via cudaMemGetInfo). Output matches stats.h print_report format: PyTorchObserver JSON line plus human-readable table. This commit was authored with the assistance of Claude Code. [ghstack-poisoned]

… runner Runner now uses llm::Stats with proper timestamps for model load, prefill, decode, and GPU memory (via cudaMemGetInfo). Output matches stats.h print_report format: PyTorchObserver JSON line plus human-readable table. This commit was authored with the assistance of Claude Code. ghstack-source-id: fea9eb8 Pull Request resolved: #19190

Gasoonjia

Thansk for adding so much details! My major concern is it may crash on non-cuda scenerio. Added ci labels to confirm

Gasoonjia · 2026-04-29T18:40:57Z



-def _set_batched_moe(model, enabled, moe_moe_moe_activation_dtype="bf16"):
+def _set_batched_moe(model, enabled, moe_moe_moe_moe_activation_dtype="bf16"):


why another extra MoE lol

Gasoonjia · 2026-04-29T18:42:15Z

+  // GPU memory: before load
+  {
+    size_t free = 0, total = 0;
+    if (cudaMemGetInfo(&free, &total) == cudaSuccess) {


right now all cuda functions are not under EXECUTORCH_BUILD_CUDA marco; i think it will crash when mps or metal backend using this script

Gasoonjia · 2026-04-29T18:42:48Z

  double decode_ms =
      (double)(stats.inference_end_ms - stats.prompt_eval_end_ms);
+  printf(
+      "Prefill: %" PRId64 " tokens in %.1f ms (%.1f tok/s)\n",


why do we want to print prefill twice?

…Qwen3.5 MoE runner" Runner now uses llm::Stats with proper timestamps for model load, prefill, decode, and GPU memory (via cudaMemGetInfo). Output matches stats.h print_report format: PyTorchObserver JSON line plus human-readable table. This commit was authored with the assistance of Claude Code. [ghstack-poisoned]

… runner Runner now uses llm::Stats with proper timestamps for model load, prefill, decode, and GPU memory (via cudaMemGetInfo). Output matches stats.h print_report format: PyTorchObserver JSON line plus human-readable table. This commit was authored with the assistance of Claude Code. ghstack-source-id: 9227519 Pull Request resolved: #19190

digantdesai requested a review from lucylq as a code owner April 28, 2026 15:56

This was referenced Apr 28, 2026

Add W4A8 INT8 activation kernels for batched MoE prefill #19187

Open

Add Triton INT4 dense kernels with dequant prefill path for Qwen3.5 MoE #19188

Open

digantdesai mentioned this pull request Apr 28, 2026

Remove benchmark scripts from git tracking #19189

Open

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 28, 2026

digantdesai requested review from Gasoonjia and mergennachin April 28, 2026 21:07

Gasoonjia reviewed Apr 29, 2026

View reviewed changes

Gasoonjia added ciflow/metal ciflow/mlx ciflow/cuda labels Apr 29, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add structured stats reporting and GPU memory tracking to Qwen3.5 MoE runner#19190

Add structured stats reporting and GPU memory tracking to Qwen3.5 MoE runner#19190
digantdesai wants to merge 6 commits intogh/digantdesai/53/basefrom
gh/digantdesai/53/head

digantdesai commented Apr 28, 2026 •

edited

Loading

Uh oh!

pytorch-bot Bot commented Apr 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Apr 28, 2026

Uh oh!

Gasoonjia left a comment •

edited

Loading

Uh oh!

Gasoonjia Apr 29, 2026

Uh oh!

Gasoonjia Apr 29, 2026

Uh oh!

Gasoonjia Apr 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants



		def _set_batched_moe(model, enabled, moe_moe_moe_activation_dtype="bf16"):
		def _set_batched_moe(model, enabled, moe_moe_moe_moe_activation_dtype="bf16"):

Conversation

digantdesai commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot Bot commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19190

❌ 6 New Failures, 2 Cancelled Jobs, 1 Unrelated Failure

Uh oh!

github-actions Bot commented Apr 28, 2026

This PR needs a release notes: label

Uh oh!

Gasoonjia left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Gasoonjia Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

Gasoonjia Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

Gasoonjia Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

digantdesai commented Apr 28, 2026 •

edited

Loading

pytorch-bot Bot commented Apr 28, 2026 •

edited

Loading

This PR needs a `release notes:` label

Gasoonjia left a comment •

edited

Loading