Qualcomm AI Engine Direct - LPAI Support Partition#19835
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19835
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 New FailuresAs of commit 863cebb with merge base ff2bf9c ( NEW FAILURES - The following jobs have failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
winskuo-quic
force-pushed
the
dev1/winskuo/gh/lpai/support_partition
branch
from
4e89627 to
b5c9b66
Compare
psiddh
added
module: qnn
| if node.target in q_ops or node.target in dq_ops: | ||
| continue | ||
| if node.meta.get(QCOM_FALLBACK_NODE): | ||
| assert all( |
There was a problem hiding this comment.
It looks like insert_partition_qdq only wraps an input with a Q/DQ pair when that input is a call_function that has QCOM_QUANT_ATTRS (and isn't already a dq op). So only quantized tensor inputs get a DQ in front of them which makes sense, since you can't dequantize something like an int index or a shape arg.
But here the assert seems to require that every input of the fallback node is a dq op. So if a fallback node has an input that wasn't quantized, e.g. an integer/index input, an fp32 graph input, or a weight, wouldn't that be a problem ?
Do we have tests that actually validate fallback ops that are not fully-quantized tensors ?
There was a problem hiding this comment.
Thanks for the code review and catching this.
This indeed will cause error when the tensor is other dtype such as int.
I have added a new UT and fix this issue.
The UT will fallback a node that is type integer and ensured it is working with the newest commit.
|
Can you resolve easy CLA and rebase to resolve conflicts please ? |
winskuo-quic
force-pushed
the
dev1/winskuo/gh/lpai/support_partition
branch
from
b5c9b66 to
863cebb
Compare
Hi @psiddh, |
…d fix for D108707519) (#20324) Summary: Forward fix for D108707519 (Qualcomm AI Engine Direct - LPAI Support Partition, #19835). The new LpaiPartitionFallbackSupport pass crashes when run without QNN compiler specs: lpai_partition_fallback_support.py:153 get_unsupported_nodes op_validator = QnnOperatorSupport(compiler_specs=self.compiler_specs, ...) TypeError: 'NoneType' object is not iterable The edge-transform pass framework injects compiler_specs into passes but defaults it to None (QnnPassManager.get_to_edge_transform_passes(compiler_specs= None)). Lowering paths that run the edge transform without partitioner-supplied specs — e.g. the modai LPAI offline-compile path (modai/test:test_qualcomm_lpai_recipes::test_export_and_lower_8a8w_writes_pte) — reach this pass with compiler_specs=None, and QnnOperatorSupport then iterates the None specs and aborts the whole to_edge_transform_and_lower. The pass fundamentally needs compiler specs to decide which nodes are LPAI- unsupported, so with no specs there is nothing to validate against. Skip the pass (return PassResult(.., modified=False)) in that case, restoring the behavior from before this pass existed for spec-less paths. On-device paths that do supply compiler_specs are unaffected. Differential Revision: D108853837
…d fix for D108707519) (#20324) (#20324) Summary: Forward fix for D108707519 (Qualcomm AI Engine Direct - LPAI Support Partition, #19835). The new LpaiPartitionFallbackSupport pass crashes when run without QNN compiler specs: lpai_partition_fallback_support.py:153 get_unsupported_nodes op_validator = QnnOperatorSupport(compiler_specs=self.compiler_specs, ...) TypeError: 'NoneType' object is not iterable The edge-transform pass framework injects compiler_specs into passes but defaults it to None (QnnPassManager.get_to_edge_transform_passes(compiler_specs= None)). Lowering paths that run the edge transform without partitioner-supplied specs — e.g. the modai LPAI offline-compile path (modai/test:test_qualcomm_lpai_recipes::test_export_and_lower_8a8w_writes_pte) — reach this pass with compiler_specs=None, and QnnOperatorSupport then iterates the None specs and aborts the whole to_edge_transform_and_lower. The pass fundamentally needs compiler specs to decide which nodes are LPAI- unsupported, so with no specs there is nothing to validate against. Skip the pass (return PassResult(.., modified=False)) in that case, restoring the behavior from before this pass existed for spec-less paths. On-device paths that do supply compiler_specs are unaffected. Differential Revision: D108853837
2 participants
Summary
Traditionally, when it gets to partial delegation for quantized model, we need to wrap qdq nodes around the unsupported node. e.g., supported_node_1(u8) -> dq(fp32) -> cpu_node(fp32) -> q(u8) -> supported_node_2.
All the nodes besides from cpu_node(fp32) is run on HTP.
However, LPAI's accuracy when computing dq and q node isn't as good as HTP, so to achieve good accuracy, we want to fallback q and dq nodes back to cpu too.
This PR is trying to achieve this while ensuring edge graph is a valid graph to PyTorch.
Test plan
A series of test validating cases to fallback multi-output node, multi-input node, contiguous fallback, etc.
python backends/qualcomm/tests/test_qnn_delegate.py -k TestQNNQuantizedUtils.test_qnn_backend_skip_node_id_partitioner --model SM8850 --device $DEVICE --build_folder build-android --seed 42 --backend lpaipython backends/qualcomm/tests/test_qnn_delegate.py -k TestQNNQuantizedUtils.test_qnn_backend_skip_node_op_partitioner --model SM8850 --device $DEVICE --build_folder build-android --seed 42 --backend lpaicc @cccclai @cbilgin @abhinaykukkadapu