Skip to content

Add dynamic shape tests for xnnpack model tests#18701

Merged
GregoryComer merged 2 commits intopytorch:mainfrom
Lidang-Jiang:fix/xnnpack-dynamism-tests
Apr 10, 2026
Merged

Add dynamic shape tests for xnnpack model tests#18701
GregoryComer merged 2 commits intopytorch:mainfrom
Lidang-Jiang:fix/xnnpack-dynamism-tests

Conversation

@Lidang-Jiang
Copy link
Copy Markdown
Contributor

@Lidang-Jiang Lidang-Jiang commented Apr 5, 2026

Summary

Add dynamic shape tests for xnnpack model tests that previously only tested with static inputs, as requested in #11585.

Models covered:

  • DeepLab V3 — dynamic height/width with DynamicWrapper + interpolate
  • EDSR — dynamic height/width with DynamicWrapper + interpolate
  • Inception V3/V4 — dynamic height/width with DynamicWrapper + interpolate
  • Emformer RNNT — dynamic batch dimension for joiner and transcriber
  • MobileBERT — dynamic sequence length

All dynamic tests follow the established pattern from resnet.py (DynamicResNet) and torchvision_vit.py (DynamicViT): a wrapper module that resizes dynamic spatial inputs to fixed dimensions via F.interpolate before feeding into the model.

Models already covered by existing dynamic tests (no changes):

  • MobileNet V2, MobileNet V3, ResNet, ViT

Models skipped:

  • llama2_et_example.py — LLM, requires separate dynamic shape strategy
  • very_big_model.py — synthetic test model
Before (only static tests)
$ python -m pytest backends/xnnpack/test/models/deeplab_v3.py -v
backends/xnnpack/test/models/deeplab_v3.py::TestDeepLabV3::test_fp32_dl3 PASSED

(no dynamic shape test exists)
After (static + dynamic tests pass)
$ python -m pytest backends/xnnpack/test/models/deeplab_v3.py::TestDeepLabV3::test_fp32_dl3_dynamic -v
backends/xnnpack/test/models/deeplab_v3.py::TestDeepLabV3::test_fp32_dl3_dynamic PASSED [100%]
======================== 1 passed in 28.84s ==============================

$ python -m pytest backends/xnnpack/test/models/mobilebert.py::TestMobilebert::test_fp32_mobilebert_dynamic -v
backends/xnnpack/test/models/mobilebert.py::TestMobilebert::test_fp32_mobilebert_dynamic PASSED [100%]
======================== 1 passed in 160.79s (0:02:40) ===================

Test plan

  • test_fp32_dl3_dynamic passes locally
  • test_fp32_mobilebert_dynamic passes locally
  • CI xnnpack test suite passes

cc @GregoryComer @digantdesai @cbilgin

@pytorch-bot
Copy link
Copy Markdown

pytorch-bot Bot commented Apr 5, 2026

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/18701

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

❌ 1 New Failure, 2 Unrelated Failures

As of commit 3e2193a with merge base ac68932 (image):

NEW FAILURE - The following job has failed:

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 5, 2026
@Lidang-Jiang
Copy link
Copy Markdown
Contributor Author

@pytorchbot label "release notes: xnnpack"

@pytorch-bot pytorch-bot Bot added the release notes: xnnpack Changes to the XNNPack backend delegate label Apr 6, 2026
@nil-is-all nil-is-all added the module: xnnpack Issues related to xnnpack delegation and the code under backends/xnnpack/ label Apr 7, 2026
@GregoryComer
Copy link
Copy Markdown
Member

@Lidang-Jiang Thanks for the contribution. Looks good. Can you resolve the lint failure (run link)? I can go ahead and merge once the lint job is green.

Add dynamic shape support to xnnpack model tests that previously
only tested with static inputs. This covers DeepLab V3, EDSR,
Inception V3/V4, Emformer RNNT, and MobileBERT using the same
DynamicWrapper + interpolate pattern established by ResNet and ViT.

Addresses pytorch#11585

Signed-off-by: Lidang-Jiang <lidangjiang@gmail.com>
Signed-off-by: Lidang-Jiang <lidangjiang@gmail.com>
@Lidang-Jiang Lidang-Jiang force-pushed the fix/xnnpack-dynamism-tests branch from d3be8be to 3e2193a Compare April 8, 2026 02:37
@Lidang-Jiang
Copy link
Copy Markdown
Contributor Author

Good catch, fixed in 3e2193a. I rebased onto latest main, applied the UFMT formatting fix in mobilebert.py, and force-pushed to rerun lint/CI.

@GregoryComer
Copy link
Copy Markdown
Member

I'll go ahead and merge. Thanks for the contribution!

@GregoryComer GregoryComer merged commit 9a10876 into pytorch:main Apr 10, 2026
159 of 162 checks passed
jpiat pushed a commit to jpiat/executorch that referenced this pull request Apr 14, 2026
## Summary

Add dynamic shape tests for xnnpack model tests that previously only
tested with static inputs, as requested in pytorch#11585.

**Models covered:**
- **DeepLab V3** — dynamic height/width with DynamicWrapper +
interpolate
- **EDSR** — dynamic height/width with DynamicWrapper + interpolate
- **Inception V3/V4** — dynamic height/width with DynamicWrapper +
interpolate
- **Emformer RNNT** — dynamic batch dimension for joiner and transcriber
- **MobileBERT** — dynamic sequence length

All dynamic tests follow the established pattern from `resnet.py`
(`DynamicResNet`) and `torchvision_vit.py` (`DynamicViT`): a wrapper
module that resizes dynamic spatial inputs to fixed dimensions via
`F.interpolate` before feeding into the model.

**Models already covered by existing dynamic tests (no changes):**
- MobileNet V2, MobileNet V3, ResNet, ViT

**Models skipped:**
- `llama2_et_example.py` — LLM, requires separate dynamic shape strategy
- `very_big_model.py` — synthetic test model

<details>
<summary>Before (only static tests)</summary>

```
$ python -m pytest backends/xnnpack/test/models/deeplab_v3.py -v
backends/xnnpack/test/models/deeplab_v3.py::TestDeepLabV3::test_fp32_dl3 PASSED

(no dynamic shape test exists)
```

</details>

<details>
<summary>After (static + dynamic tests pass)</summary>

```
$ python -m pytest backends/xnnpack/test/models/deeplab_v3.py::TestDeepLabV3::test_fp32_dl3_dynamic -v
backends/xnnpack/test/models/deeplab_v3.py::TestDeepLabV3::test_fp32_dl3_dynamic PASSED [100%]
======================== 1 passed in 28.84s ==============================

$ python -m pytest backends/xnnpack/test/models/mobilebert.py::TestMobilebert::test_fp32_mobilebert_dynamic -v
backends/xnnpack/test/models/mobilebert.py::TestMobilebert::test_fp32_mobilebert_dynamic PASSED [100%]
======================== 1 passed in 160.79s (0:02:40) ===================
```

</details>

## Test plan
- [x] `test_fp32_dl3_dynamic` passes locally
- [x] `test_fp32_mobilebert_dynamic` passes locally
- [ ] CI xnnpack test suite passes

cc @GregoryComer @digantdesai @cbilgin

---------

Signed-off-by: Lidang-Jiang <lidangjiang@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. module: xnnpack Issues related to xnnpack delegation and the code under backends/xnnpack/ release notes: xnnpack Changes to the XNNPack backend delegate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants