[devtools] Add preprocessing support to NumericalComparatorBase by Gasoonjia · Pull Request #17433 · pytorch/executorch

Gasoonjia · 2026-02-13T00:54:25Z

Stack from ghstack (oldest at bottom):

This change extends NumericalComparatorBase to support custom preprocessing before numeric gap calculation, enabling backends like Qualcomm to apply necessary tensor transformations (e.g., dequantization, layout conversion) before comparison.

Key changes:

Extended NumericalComparatorBase with:
- __init__(inspector) to store optional Inspector reference for accessing graph metadata
- preprocessing(mapping) method that can be overridden for custom tensor transformations (default: identity)
- element_compare(a, b) abstract method for element-level tensor comparison
- compare(mapping, ...) method that orchestrates the full pipeline: preprocessing → element-wise compare → aggregate to DataFrame

This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381).

Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud

Differential Revision: D93169813

This change extends NumericalComparatorBase to support custom preprocessing before numeric gap calculation, enabling backends like Qualcomm to apply necessary tensor transformations (e.g., dequantization, layout conversion) before comparison. Key changes: - Extended NumericalComparatorBase with: - `__init__(inspector)` to store optional Inspector reference for accessing graph metadata - `preprocessing(mapping)` method that can be overridden for custom tensor transformations (default: identity) - `element_compare(a, b)` abstract method for element-level tensor comparison - `compare(mapping, ...)` method that orchestrates the full pipeline: preprocessing → element-wise compare → aggregate to DataFrame This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381). Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud Differential Revision: [D93169813](https://our.internmc.facebook.com/intern/diff/D93169813/) [ghstack-poisoned]

pytorch-bot · 2026-02-13T00:54:29Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/17433

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Cancelled Job

As of commit 2c27e4d with merge base 63f9724 ():

CANCELLED JOB - The following job was cancelled. Please retry:

pull / test-samsung-models-linux / linux-job (gh)
##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

This change extends NumericalComparatorBase to support custom preprocessing before numeric gap calculation, enabling backends like Qualcomm to apply necessary tensor transformations (e.g., dequantization, layout conversion) before comparison. Key changes: - Extended NumericalComparatorBase with: - `__init__(inspector)` to store optional Inspector reference for accessing graph metadata - `preprocessing(mapping)` method that can be overridden for custom tensor transformations (default: identity) - `element_compare(a, b)` abstract method for element-level tensor comparison - `compare(mapping, ...)` method that orchestrates the full pipeline: preprocessing → element-wise compare → aggregate to DataFrame This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381). Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud Differential Revision: [D93169813](https://our.internmc.facebook.com/intern/diff/D93169813/) ghstack-source-id: 340935651 Pull Request resolved: #17433

github-actions · 2026-02-13T00:55:17Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

…rBase" This change extends NumericalComparatorBase to support custom preprocessing before numeric gap calculation, enabling backends like Qualcomm to apply necessary tensor transformations (e.g., dequantization, layout conversion) before comparison. Key changes: - Extended NumericalComparatorBase with: - `__init__(inspector)` to store optional Inspector reference for accessing graph metadata - `preprocessing(mapping)` method that can be overridden for custom tensor transformations (default: identity) - `element_compare(a, b)` abstract method for element-level tensor comparison - `compare(mapping, ...)` method that orchestrates the full pipeline: preprocessing → element-wise compare → aggregate to DataFrame This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381). Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud Differential Revision: [D93169813](https://our.internmc.facebook.com/intern/diff/D93169813/) [ghstack-poisoned]

Pull Request resolved: #17433 This change extends NumericalComparatorBase to support custom preprocessing before numeric gap calculation, enabling backends like Qualcomm to apply necessary tensor transformations (e.g., dequantization, layout conversion) before comparison. Key changes: - Extended NumericalComparatorBase with: - `__init__(inspector)` to store optional Inspector reference for accessing graph metadata - `preprocessing(mapping)` method that can be overridden for custom tensor transformations (default: identity) - `element_compare(a, b)` abstract method for element-level tensor comparison - `compare(mapping, ...)` method that orchestrates the full pipeline: preprocessing → element-wise compare → aggregate to DataFrame This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381). Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud Differential Revision: [D93169813](https://our.internmc.facebook.com/intern/diff/D93169813/) ghstack-source-id: 340943620

…rBase" This change extends NumericalComparatorBase to support custom preprocessing before numeric gap calculation, enabling backends like Qualcomm to apply necessary tensor transformations (e.g., dequantization, layout conversion) before comparison. Key changes: - Extended NumericalComparatorBase with: - `__init__(inspector)` to store optional Inspector reference for accessing graph metadata - `preprocessing(mapping)` method that can be overridden for custom tensor transformations (default: identity) - `element_compare(a, b)` abstract method for element-level tensor comparison - `compare(mapping, ...)` method that orchestrates the full pipeline: preprocessing → element-wise compare → aggregate to DataFrame This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381). Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud Differential Revision: [D93169813](https://our.internmc.facebook.com/intern/diff/D93169813/) [ghstack-poisoned]

Pull Request resolved: #17433 This change extends NumericalComparatorBase to support custom preprocessing before numeric gap calculation, enabling backends like Qualcomm to apply necessary tensor transformations (e.g., dequantization, layout conversion) before comparison. Key changes: - Extended NumericalComparatorBase with: - `__init__(inspector)` to store optional Inspector reference for accessing graph metadata - `preprocessing(mapping)` method that can be overridden for custom tensor transformations (default: identity) - `element_compare(a, b)` abstract method for element-level tensor comparison - `compare(mapping, ...)` method that orchestrates the full pipeline: preprocessing → element-wise compare → aggregate to DataFrame This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381). Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud ghstack-source-id: 340952320 Differential Revision: [D93169813](https://our.internmc.facebook.com/intern/diff/D93169813/)

…rBase" This change extends NumericalComparatorBase to support custom preprocessing before numeric gap calculation, enabling backends like Qualcomm to apply necessary tensor transformations (e.g., dequantization, layout conversion) before comparison. Key changes: - Extended NumericalComparatorBase with: - `__init__(inspector)` to store optional Inspector reference for accessing graph metadata - `preprocessing(mapping)` method that can be overridden for custom tensor transformations (default: identity) - `element_compare(a, b)` abstract method for element-level tensor comparison - `compare(mapping, ...)` method that orchestrates the full pipeline: preprocessing → element-wise compare → aggregate to DataFrame This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381). Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud Differential Revision: [D93169813](https://our.internmc.facebook.com/intern/diff/D93169813/) [ghstack-poisoned]

winskuo-quic · 2026-02-14T03:39:19Z

+            custom_comparator = ScalingComparator(scale_factor=2.0)
+
+            # Test with custom comparator
+            df = inspector_instance.calculate_numeric_gap(distance=custom_comparator)


Hi @Gasoonjia,
Thank you so much for the PR to support the preprocessing.
Sorry I am now having limited access to laptop and may not be able to try out this PR until next week.

For this unit test, could you add a case where we will be performing multiple comparisons? For example, perform custom_operator and then perform a snr comparator immediately, and both the comparator requires that same preprocessing function. I believe this would be a UT that can closely align with Qualcomm's use case, which also helps demonstrate how inspector expects users to handle the intermediate tensor map between comparison.
Thanks

larryliu0820

Review automatically exported from Phabricator review in Meta.

winskuo-quic · 2026-02-23T09:25:11Z

Hi @Gasoonjia,
Thanks again for the PR.
I have left a comment above related to unit test. Besides, this PR looks good to me.
Thanks

…rBase" This change extends NumericalComparatorBase to support custom preprocessing before numeric gap calculation, enabling backends like Qualcomm to apply necessary tensor transformations (e.g., dequantization, layout conversion) before comparison. Key changes: - Extended NumericalComparatorBase with: - `__init__(inspector)` to store optional Inspector reference for accessing graph metadata - `preprocessing(mapping)` method that can be overridden for custom tensor transformations (default: identity) - `element_compare(a, b)` abstract method for element-level tensor comparison - `compare(mapping, ...)` method that orchestrates the full pipeline: preprocessing → element-wise compare → aggregate to DataFrame This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381). Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud Differential Revision: [D93169813](https://our.internmc.facebook.com/intern/diff/D93169813/) [ghstack-poisoned]

Gasoonjia · 2026-02-24T06:33:49Z

thansk for comment @winskuo-quic
I've updated the existing test to support two consecutive numeric comparator with same preprocessor.
Plz let me know if there's any concern, or i will land them in recent days.

…rBase" This change extends NumericalComparatorBase to support custom preprocessing before numeric gap calculation, enabling backends like Qualcomm to apply necessary tensor transformations (e.g., dequantization, layout conversion) before comparison. Key changes: - Extended NumericalComparatorBase with: - `__init__(inspector)` to store optional Inspector reference for accessing graph metadata - `preprocessing(mapping)` method that can be overridden for custom tensor transformations (default: identity) - `element_compare(a, b)` abstract method for element-level tensor comparison - `compare(mapping, ...)` method that orchestrates the full pipeline: preprocessing → element-wise compare → aggregate to DataFrame This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381). Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud Differential Revision: [D93169813](https://our.internmc.facebook.com/intern/diff/D93169813/) [ghstack-poisoned]

Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom): * #17459 * __->__ #17434 * #17433 When `to_edge_transform_and_lower` is called with `generate_etrecord=True` and custom `transform_passes` are applied, the after-transform graph is now automatically recorded in the ETRecord's `graph_map` under the key `"edge_after_transform"`. This enables backends like Qualcomm to use the post-custom-transform graph as the golden reference for numeric gap calculation, while have zero impact for regular exportation. This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381). Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud Differential Revision: [D93176563](https://our.internmc.facebook.com/intern/diff/D93176563/)

…7459) Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at bottom): * __->__ #17459 * #17434 * #17433 This change adds a `reference_graph` parameter to the `calculate_numeric_gap` API, allowing users to explicitly select which graph to use as the golden reference for numeric gap calculation, which enables backends like Qualcomm to use the post-custom-transform graph as the golden reference for numeric gap calculation. This is part of the operator-level numeric discrepancy detector project for ExecuTorch Qualcomm backend (#16381). Design doc: https://docs.google.com/document/d/1GaCHiy9InytOsUrl2BKEgOiP1iKTfpCVdWg6QDh0N2E/edit?tab=t.0#heading=h.fcrpnrtb6cud Differential Revision: [D93266779](https://our.internmc.facebook.com/intern/diff/D93266779/)

@Gasoonjia

This PR was created by the merge bot to help merge the original PR into the main branch. ghstack PR number: #17433 by @Gasoonjia ^ Please use this as the source of truth for the PR details, comments, and reviews ghstack PR base: https://github.com/pytorch/executorch/tree/gh/gasoonjia/119/base ghstack PR head: https://github.com/pytorch/executorch/tree/gh/gasoonjia/119/head Merge bot PR base: https://github.com/pytorch/executorch/tree/main Merge bot PR head: https://github.com/pytorch/executorch/tree/gh/gasoonjia/119/orig Differential Revision: [D93169813](https://our.internmc.facebook.com/intern/diff/D93169813/) @diff-train-skip-merge --------- Co-authored-by: gasoonjia <gasoonjia@icloud.com>

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 13, 2026

meta-codesync Bot added fb-exported meta-exported labels Feb 13, 2026

Gasoonjia requested a review from kimishpatel February 13, 2026 00:55

Gasoonjia requested a review from winskuo-quic February 13, 2026 00:55

Gasoonjia mentioned this pull request Feb 13, 2026

[devtools] Auto-record after-transform graph in ETRecord #17434

Merged

Gasoonjia mentioned this pull request Feb 13, 2026

[devtools] Add reference_graph parameter to calculate_numeric_gap #17459

Merged

winskuo-quic reviewed Feb 14, 2026

View reviewed changes

larryliu0820 approved these changes Feb 17, 2026

View reviewed changes

Gasoonjia added 2 commits February 24, 2026 16:10

Gasoonjia merged commit b3f0a27 into gh/gasoonjia/119/base Feb 25, 2026
202 of 203 checks passed

Gasoonjia deleted the gh/gasoonjia/119/head branch February 25, 2026 08:02

Gasoonjia temporarily deployed to cherry-pick-bot February 25, 2026 08:02 — with GitHub Actions Inactive

pytorchbot mentioned this pull request Feb 25, 2026

[devtools] Add preprocessing support to NumericalComparatorBase #17695

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[devtools] Add preprocessing support to NumericalComparatorBase#17433

[devtools] Add preprocessing support to NumericalComparatorBase#17433
Gasoonjia merged 8 commits intogh/gasoonjia/119/basefrom
gh/gasoonjia/119/head

Gasoonjia commented Feb 13, 2026 •

edited

Loading

Uh oh!

pytorch-bot Bot commented Feb 13, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Feb 13, 2026

Uh oh!

winskuo-quic Feb 14, 2026

Uh oh!

larryliu0820 left a comment

Uh oh!

winskuo-quic commented Feb 23, 2026

Uh oh!

Gasoonjia commented Feb 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Gasoonjia commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot Bot commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/17433

❌ 1 Cancelled Job

Uh oh!

github-actions Bot commented Feb 13, 2026

This PR needs a release notes: label

Uh oh!

winskuo-quic Feb 14, 2026

Choose a reason for hiding this comment

Uh oh!

larryliu0820 left a comment

Choose a reason for hiding this comment

Uh oh!

winskuo-quic commented Feb 23, 2026

Uh oh!

Gasoonjia commented Feb 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Gasoonjia commented Feb 13, 2026 •

edited

Loading

pytorch-bot Bot commented Feb 13, 2026 •

edited

Loading

This PR needs a `release notes:` label