Introduce CSE pass to ExecuTorch

[metascroy]

This introduces a CSE pass to ExecuTorch, which eliminates common subexpressions that occur in exported programs.

This pass was first developed as part of the MLX delegate (#16718) to optimize transformers, but I'm introducing it to ExecuTorch more generally because I believe it could benefit many other backends.

Examples of common subexpressions that occur in transformers:

• Repeated mask constructions per layer (only needs to be done once)
• Repeated extraction of symints from 1d tensors for cache position (emits .item calls, which cause tensor materialization)

This pass eliminates these inefficiencies without having to rewrite the model.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/17752

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 2 Pending

As of commit c0031e8 with merge base b5dbfb7 ():

NEW FAILURES - The following jobs have failed:

• Lint / lintrunner-mypy (gh)
• pull / unittest / macos / macos-job (gh)
  backends/xnnpack/test/ops/test_conv2d.py::TestConv2d::test_fp16_conv2d

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[metascroy]

@manuelcandales @mergennachin I think you can use this in the metal delegate

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[meta-codesync]

@metascroy has imported this pull request. If you are a Meta employee, you can view this in D94581516.

[larryliu0820]

This is great! Should we turn it on by default?

[metascroy]

This is great! Should we turn it on by default?

Could do as follow up

[mergennachin]

@metascroy

There's already CSE in torch fx that we can use from torch.fx.passes.dialect.common.cse_pass import CSEPass

See a draft PR on how I using (before running another fusion pass)

#15813

[metascroy]

@metascroy

There's already CSE in torch fx that we can use from torch.fx.passes.dialect.common.cse_pass import CSEPass

See a draft PR on how I using (before running another fusion pass)

#15813

Oh sorry, I merged this already. I will check of the CSE pass in torch and unify with ExecuTorch pass as follow-up.