Arm backend: Make pytest collection seeding deterministic by zingo · Pull Request #20621 · pytorch/executorch

zingo · 2026-06-30T13:53:41Z

Summary

Seed the Arm test RNG during pytest configuration in OSS/local test runs so module-level random inputs are deterministic before collection imports test modules.

This will make tests behave less random and hopefully fix some flakeynes in testing/CI.
For example test_dl3_arm.py failes about 5% of the time before this fix due to it getting non seeded random numbers.

Extra cleanup/fixes:
Use TEST_SEED for the Arm test session seed, replacing the old ARM_TEST_SEED environment variable name.

Keep packaged non-OSS Buck behavior unchanged by skipping that early torch import there, while preserving the per-test seeding fixture.

Test plan

This is tested by all Arm backend pytests

cc @digantdesai @freddan80 @per @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani

Seed the Arm test RNG during pytest configuration in OSS/local test runs so module-level random inputs are deterministic before collection imports test modules. This will make tests behave less random and hopefully fix some flakeynes in testing/CI. For example test_dl3_arm.py failes about 5% of the time before this fix due to it getting non seeded random numbers. Extra cleanup/fixes: Use TEST_SEED for the Arm test session seed, replacing the old ARM_TEST_SEED environment variable name. Keep packaged non-OSS Buck behavior unchanged by skipping that early torch import there, while preserving the per-test seeding fixture. Signed-off-by: Zingo Andersen <Zingo.Andersen@arm.com> Change-Id: If8597f79e0e59bf92d6f00f1b3f12097f08ce21e

pytorch-bot · 2026-06-30T13:53:46Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20621

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures

As of commit a764734 with merge base b1ef9a5 ():

NEW FAILURES - The following jobs have failed:

pull / unittest / linux / linux-job (gh)
RuntimeError: Command docker exec -t d145c10aa5b72be05ce46c7038bd09d554f2db9fa973cbc89d8d023ab6979273 /exec failed with exit code 1
pull / unittest-editable / linux / linux-job (gh)
RuntimeError: Command docker exec -t 9f6910c087aa91b6b48cb7919a3a8b1fcfe89bbe895aac3dcbc4d8c1a9d9124d /exec failed with exit code 1
pull / unittest-nxp-neutron / linux-job (gh)
RuntimeError: Command docker exec -t 7197876fa1b22e758489134ab78ac969d708ac35585b65686ccdfea3427f0a26 /exec failed with exit code 1
trunk / unittest-release / linux / linux-job (gh)
RuntimeError: Command docker exec -t 302a2574ab90eee3ad405bc371599348983dc4191534ec2e69fe77ebaee72f0f /exec failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Copilot

Pull request overview

This PR makes Arm backend pytest runs more deterministic by selecting a single session seed during pytest_configure (early enough to affect module-import-time randomness during collection in OSS/local runs), while preserving the packaged/non-OSS Buck behavior by skipping the early torch import there.

Changes:

Add session-level seed setup in pytest_configure, report the chosen seed in the pytest header, and seed torch/python RNG early for OSS/local runs.
Switch Arm test seeding env var to TEST_SEED (from ARM_TEST_SEED) and update the autouse fixture to re-seed each test using the session seed.
Pass a new Buck test env flag (TEST_RUNTIME_IS_NOT_OSS) to control whether early seeding is safe in that environment.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
backends/arm/test/targets.bzl	Adds `TEST_RUNTIME_IS_NOT_OSS` and refactors env construction for Arm pytest Buck targets.
backends/arm/test/conftest.py	Implements deterministic session seeding during pytest configuration, updates env var name to `TEST_SEED`, and reports seed in the pytest header.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

zingo · 2026-06-30T13:59:42Z

Hi @Gasoonjia @digantdesai @rascani Here is a new version of a patch that got reverted due to torch was used to early in you buck setup. This version try to guard this with the runtime.is_oss so hopefully this works both on GutHub and internally, but internally you will not get the same nice locked down randomness in the tests but it will work the same way as today.
As we can test this we need help with that.
I have a follow up PR doing the same for cortex-m tests if this works. #20623

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

-        raise TypeError(
-            "ARM_TEST_SEED env variable must be integers or the string RANDOM"
-        )
+        raise TypeError("TEST_SEED env variable must be integers or the string RANDOM")


+    seed, seed_label = _setup_random_seed()
+    config._test_seed = seed
+    config._test_seed_label = seed_label
+
+    if os.environ.get("TEST_RUNTIME_IS_NOT_OSS", "0") != "1":


…eterministic

zingo · 2026-07-07T10:11:08Z

Hi @digantdesai / @rascani is this OK to merge and test if internal Meta build still works? Or do you want to pre-run it?
I got an approve and just merged the Cortex-M code doing the same.

rascani · 2026-07-07T16:24:59Z

Hi @digantdesai / @rascani is this OK to merge and test if internal Meta build still works? Or do you want to pre-run it? I got an approve and just merged the Cortex-M code doing the same.

I'll do an import just in case.

meta-codesync · 2026-07-07T16:29:34Z

@rascani has imported this pull request. If you are a Meta employee, you can view this in D110912183.

rascani · 2026-07-07T21:34:03Z

Hi @digantdesai / @rascani is this OK to merge and test if internal Meta build still works? Or do you want to pre-run it? I got an approve and just merged the Cortex-M code doing the same.

I'll do an import just in case.

Internal tests were green. I unlinked the diff, but it might need a fresh test run.

zingo · 2026-07-07T22:06:58Z

Testfails are unrelated test only changes test under backends/arm/tests

Same/similar fails are visible on other trunk jobs.

Copilot AI review requested due to automatic review settings June 30, 2026 13:53

zingo requested a review from digantdesai as a code owner June 30, 2026 13:53

zingo added partner: arm For backend delegation, kernels, demo, etc. from the 3rd-party partner, Arm release notes: arm Changes to the ARM backend delegate labels Jun 30, 2026

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jun 30, 2026

github-actions Bot added ciflow/trunk module: arm Issues related to arm backend labels Jun 30, 2026

Copilot started reviewing on behalf of zingo June 30, 2026 13:54 View session

zingo requested a review from Gasoonjia June 30, 2026 13:54

zingo added the help wanted Extra attention is needed label Jun 30, 2026

Copilot AI reviewed Jun 30, 2026

View reviewed changes

Comment thread backends/arm/test/conftest.py Outdated

Comment thread backends/arm/test/conftest.py

Potential fix for pull request finding

f485cbc

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings June 30, 2026 14:00

Copilot started reviewing on behalf of zingo June 30, 2026 14:01 View session

Copilot AI reviewed Jun 30, 2026

View reviewed changes

Merge branch 'main' into Arm-backend-Make-pytest-collection-seeding-d…

a764734

…eterministic

rascani approved these changes Jul 7, 2026

View reviewed changes

zingo merged commit dcc04ef into pytorch:main Jul 7, 2026
487 of 496 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Arm backend: Make pytest collection seeding deterministic#20621

Arm backend: Make pytest collection seeding deterministic#20621
zingo merged 3 commits into
pytorch:mainfrom
zingo:Arm-backend-Make-pytest-collection-seeding-deterministic

zingo commented Jun 30, 2026 •

edited by pytorch-bot Bot

Loading

Uh oh!

pytorch-bot Bot commented Jun 30, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

zingo commented Jun 30, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

zingo commented Jul 7, 2026 •

edited

Loading

Uh oh!

rascani commented Jul 7, 2026

Uh oh!

meta-codesync Bot commented Jul 7, 2026

Uh oh!

rascani commented Jul 7, 2026

Uh oh!

zingo commented Jul 7, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

zingo commented Jun 30, 2026 • edited by pytorch-bot Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

pytorch-bot Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20621

❌ 4 New Failures

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

zingo commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

zingo commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rascani commented Jul 7, 2026

Uh oh!

meta-codesync Bot commented Jul 7, 2026

Uh oh!

rascani commented Jul 7, 2026

Uh oh!

zingo commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zingo commented Jun 30, 2026 •

edited by pytorch-bot Bot

Loading

pytorch-bot Bot commented Jun 30, 2026 •

edited

Loading

zingo commented Jun 30, 2026 •

edited

Loading

zingo commented Jul 7, 2026 •

edited

Loading

zingo commented Jul 7, 2026 •

edited

Loading