[devtools] Add reference_graph parameter to calculate_numeric_gap by Gasoonjia · Pull Request #17459 · pytorch/executorch

Gasoonjia · 2026-02-13T20:41:33Z

Stack from ghstack (oldest at bottom):

This change adds a reference_graph parameter to the calculate_numeric_gap API, allowing users to explicitly select which graph to use as the golden reference for numeric gap calculation, which enables backends like Qualcomm to use the post-custom-transform graph as the golden reference for numeric gap calculation.

This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381).

Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud

Differential Revision: D93266779

This change adds a `reference_graph` parameter to the `calculate_numeric_gap` API, allowing users to explicitly select which graph to use as the golden reference for numeric gap calculation, which enables backends like Qualcomm to use the post-custom-transform graph as the golden reference for numeric gap calculation. This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381). Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud Differential Revision: [D93266779](https://our.internmc.facebook.com/intern/diff/D93266779/) [ghstack-poisoned]

pytorch-bot · 2026-02-13T20:41:38Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/17459

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Cancelled Job

As of commit 099ec34 with merge base 63f9724 ():

CANCELLED JOB - The following job was cancelled. Please retry:

pull / test-samsung-models-linux / linux-job (gh)
##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

This change adds a `reference_graph` parameter to the `calculate_numeric_gap` API, allowing users to explicitly select which graph to use as the golden reference for numeric gap calculation, which enables backends like Qualcomm to use the post-custom-transform graph as the golden reference for numeric gap calculation. This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381). Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud Differential Revision: [D93266779](https://our.internmc.facebook.com/intern/diff/D93266779/) ghstack-source-id: 341228298 Pull Request resolved: #17459

github-actions · 2026-02-13T20:42:34Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

…ric_gap" This change adds a `reference_graph` parameter to the `calculate_numeric_gap` API, allowing users to explicitly select which graph to use as the golden reference for numeric gap calculation, which enables backends like Qualcomm to use the post-custom-transform graph as the golden reference for numeric gap calculation. This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381). Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud Differential Revision: [D93266779](https://our.internmc.facebook.com/intern/diff/D93266779/) [ghstack-poisoned]

Pull Request resolved: #17459 This change adds a `reference_graph` parameter to the `calculate_numeric_gap` API, allowing users to explicitly select which graph to use as the golden reference for numeric gap calculation, which enables backends like Qualcomm to use the post-custom-transform graph as the golden reference for numeric gap calculation. This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381). Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud Differential Revision: [D93266779](https://our.internmc.facebook.com/intern/diff/D93266779/) ghstack-source-id: 341245956

winskuo-quic

LGTM. Thanks for the PR.

…ric_gap" This change adds a `reference_graph` parameter to the `calculate_numeric_gap` API, allowing users to explicitly select which graph to use as the golden reference for numeric gap calculation, which enables backends like Qualcomm to use the post-custom-transform graph as the golden reference for numeric gap calculation. This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381). Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud Differential Revision: [D93266779](https://our.internmc.facebook.com/intern/diff/D93266779/) [ghstack-poisoned]

Pull Request resolved: #17459 This change adds a `reference_graph` parameter to the `calculate_numeric_gap` API, allowing users to explicitly select which graph to use as the golden reference for numeric gap calculation, which enables backends like Qualcomm to use the post-custom-transform graph as the golden reference for numeric gap calculation. This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381). Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud ghstack-source-id: 344105834 Differential Revision: [D93266779](https://our.internmc.facebook.com/intern/diff/D93266779/)

…ric_gap" This change adds a `reference_graph` parameter to the `calculate_numeric_gap` API, allowing users to explicitly select which graph to use as the golden reference for numeric gap calculation, which enables backends like Qualcomm to use the post-custom-transform graph as the golden reference for numeric gap calculation. This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381). Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud Differential Revision: [D93266779](https://our.internmc.facebook.com/intern/diff/D93266779/) [ghstack-poisoned]

Pull Request resolved: #17459 This change adds a `reference_graph` parameter to the `calculate_numeric_gap` API, allowing users to explicitly select which graph to use as the golden reference for numeric gap calculation, which enables backends like Qualcomm to use the post-custom-transform graph as the golden reference for numeric gap calculation. This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381). Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud ghstack-source-id: 344454839 Differential Revision: [D93266779](https://our.internmc.facebook.com/intern/diff/D93266779/)

…ric_gap" This change adds a `reference_graph` parameter to the `calculate_numeric_gap` API, allowing users to explicitly select which graph to use as the golden reference for numeric gap calculation, which enables backends like Qualcomm to use the post-custom-transform graph as the golden reference for numeric gap calculation. This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381). Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud Differential Revision: [D93266779](https://our.internmc.facebook.com/intern/diff/D93266779/) [ghstack-poisoned]

Pull Request resolved: #17459 This change adds a `reference_graph` parameter to the `calculate_numeric_gap` API, allowing users to explicitly select which graph to use as the golden reference for numeric gap calculation, which enables backends like Qualcomm to use the post-custom-transform graph as the golden reference for numeric gap calculation. This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381). Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud ghstack-source-id: 344497033 Differential Revision: [D93266779](https://our.internmc.facebook.com/intern/diff/D93266779/)

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom): * #17459 * #17434 * __->__ #17433 This change extends NumericalComparatorBase to support custom preprocessing before numeric gap calculation, enabling backends like Qualcomm to apply necessary tensor transformations (e.g., dequantization, layout conversion) before comparison. Key changes: - Extended NumericalComparatorBase with: - `__init__(inspector)` to store optional Inspector reference for accessing graph metadata - `preprocessing(mapping)` method that can be overridden for custom tensor transformations (default: identity) - `element_compare(a, b)` abstract method for element-level tensor comparison - `compare(mapping, ...)` method that orchestrates the full pipeline: preprocessing → element-wise compare → aggregate to DataFrame This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381). Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud Differential Revision: [D93169813](https://our.internmc.facebook.com/intern/diff/D93169813/)

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom): * #17459 * __->__ #17434 * #17433 When `to_edge_transform_and_lower` is called with `generate_etrecord=True` and custom `transform_passes` are applied, the after-transform graph is now automatically recorded in the ETRecord's `graph_map` under the key `"edge_after_transform"`. This enables backends like Qualcomm to use the post-custom-transform graph as the golden reference for numeric gap calculation, while have zero impact for regular exportation. This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381). Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud Differential Revision: [D93176563](https://our.internmc.facebook.com/intern/diff/D93176563/)

@Gasoonjia

…7697) This PR was created by the merge bot to help merge the original PR into the main branch. ghstack PR number: #17459 by @Gasoonjia ^ Please use this as the source of truth for the PR details, comments, and reviews ghstack PR base: https://github.com/pytorch/executorch/tree/gh/gasoonjia/121/base ghstack PR head: https://github.com/pytorch/executorch/tree/gh/gasoonjia/121/head Merge bot PR base: https://github.com/pytorch/executorch/tree/gh/gasoonjia/120/orig Merge bot PR head: https://github.com/pytorch/executorch/tree/gh/gasoonjia/121/orig Differential Revision: [D93266779](https://our.internmc.facebook.com/intern/diff/D93266779/) @diff-train-skip-merge Co-authored-by: gasoonjia <gasoonjia@icloud.com>

This was referenced Feb 13, 2026

[devtools] Add preprocessing support to NumericalComparatorBase #17433

Merged

[devtools] Auto-record after-transform graph in ETRecord #17434

Merged

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 13, 2026

meta-codesync Bot added fb-exported meta-exported labels Feb 13, 2026

winskuo-quic reviewed Feb 24, 2026

View reviewed changes

larryliu0820 approved these changes Feb 24, 2026

View reviewed changes

Gasoonjia merged commit 30c37c7 into gh/gasoonjia/121/base Feb 25, 2026
202 of 203 checks passed

Gasoonjia deleted the gh/gasoonjia/121/head branch February 25, 2026 08:02

Gasoonjia temporarily deployed to cherry-pick-bot February 25, 2026 08:02 — with GitHub Actions Inactive

pytorchbot mentioned this pull request Feb 25, 2026

[devtools] Add reference_graph parameter to calculate_numeric_gap #17697

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[devtools] Add reference_graph parameter to calculate_numeric_gap#17459

[devtools] Add reference_graph parameter to calculate_numeric_gap#17459
Gasoonjia merged 5 commits intogh/gasoonjia/121/basefrom
gh/gasoonjia/121/head

Gasoonjia commented Feb 13, 2026 •

edited

Loading

Uh oh!

pytorch-bot Bot commented Feb 13, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Feb 13, 2026

Uh oh!

winskuo-quic left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Gasoonjia commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot Bot commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/17459

❌ 1 Cancelled Job

Uh oh!

github-actions Bot commented Feb 13, 2026

This PR needs a release notes: label

Uh oh!

winskuo-quic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Gasoonjia commented Feb 13, 2026 •

edited

Loading

pytorch-bot Bot commented Feb 13, 2026 •

edited

Loading

This PR needs a `release notes:` label