fix: bind ExecuTorch TRT engine inputs in delegate arg order#4400
fix: bind ExecuTorch TRT engine inputs in delegate arg order#4400shoumikhin wants to merge 1 commit into
Conversation
e934dec to
7cb6d0b
Compare
|
Does executorch have a deterministic input flattening like exported program does with pytrees? |
7cb6d0b to
1bc58ea
Compare
|
Yes, ExecuTorch's delegate input order is deterministic, but determinism is not the issue here; order agreement between two independently-produced orders is. There are two separate ordering sources that are each deterministic on their own:
Both are deterministic, but they are derived by different stages and need not agree. The runtime binds purely positionally ( On the pytree angle: torch.export's in/out pytree spec governs the program boundary flatten/unflatten, not the internal lowered delegate ABI. The Outputs are also bound positionally but are stable by construction (they are |
|
cc: @cehongwang who is working on binding metadata for other work |
## The problem
When a model with more than one input is compiled to a TensorRT engine and
run through ExecuTorch, the inputs can be delivered to the wrong engine slots.
The failure looks like this at runtime:
input 'timesteps' rank 4 does not match profile rank 1
The model is fine -- the wrong tensor is simply arriving at that slot.
Why it happens: the ExecuTorch TensorRT runtime backend matches inputs to
engine bindings *by position*. Argument `i` handed to `execute()` is fed to
`input_binding_names[i]`. There are no names attached to the incoming tensors
at runtime; the backend just trusts that the i-th tensor belongs to the i-th
binding name.
But those two lists are built by two different stages that never coordinate:
1. `input_binding_names` is recorded during TensorRT conversion, in the order
the interpreter visits the subgraph placeholders
(`_TRTInterpreter.placeholder`).
2. The tensors passed to `executorch_call_delegate` at runtime are ordered by
ExecuTorch's own lowering: its FX fuser sorts the delegate's placeholder
inputs by top-level graph order, which can be a different permutation.
For a single-input engine there is nothing to reorder, so it always works.
For a multi-input engine whose inputs get laid out differently after lowering,
the two orders disagree and a tensor lands in the wrong binding -- which either
fails loudly (rank/shape mismatch, as above) or, if two inputs happen to share
a shape, silently produces wrong results.
## Alternatives considered
- Match tensors to bindings by name at runtime. Not possible: the ExecuTorch
delegate ABI hands `execute()` only positional values, with no names. Adding
names would require changing the serialized blob format and the runtime ABI.
- Match names to placeholders by name at serialization time. Does not work: the
TensorRT binding names are the original semantic placeholder targets (e.g.
`timesteps`, `position_ids`) while the lowered ExecuTorch placeholders are
generic (`arg_0`, `arg_1`, ...). There is no reliable name correspondence.
- Fix the order earlier, during TensorRT conversion/export. Does not stick:
ExecuTorch performs another lowering/fusion pass afterward that can re-permute
the placeholders, so any alignment done before that pass is undone.
## The solution
Reorder `input_binding_names` inside the ExecuTorch backend `preprocess()` --
the first point where both the TensorRT binding order and the final delegate
argument order are visible. The permutation is recovered by *node identity*,
not by name: the engine node's first argument lists its input nodes in binding
order, and each of those nodes is a placeholder in the lowered graph whose
position is exactly the runtime argument slot it will occupy. So we sort the
binding names by each input node's placeholder position.
Single-input engines are naturally identity, so models that already worked are
unaffected. The runtime is unchanged, and the per-binding optimization-profile
bounds follow automatically because they are looked up by name at engine init
(`getProfileShape`).
Adds unit tests in `tests/py/dynamo/executorch/test_backend.py`: single-input
identity, node-identity reorder, generic-name multi-input permutation, and a
guard for a binding-count mismatch.
1bc58ea to
d375ceb
Compare
The problem
When a model with more than one input is compiled to a TensorRT engine and
run through ExecuTorch, the inputs can be delivered to the wrong engine slots.
The failure looks like this at runtime:
The model is fine -- the wrong tensor is simply arriving at that slot.
Why it happens: the ExecuTorch TensorRT runtime backend matches inputs to
engine bindings by position. Argument
ihanded toexecute()is fed toinput_binding_names[i]. There are no names attached to the incoming tensorsat runtime; the backend just trusts that the i-th tensor belongs to the i-th
binding name.
But those two lists are built by two different stages that never coordinate:
input_binding_namesis recorded during TensorRT conversion, in the orderthe interpreter visits the subgraph placeholders
(
_TRTInterpreter.placeholder).executorch_call_delegateat runtime are ordered byExecuTorch's own lowering: its FX fuser sorts the delegate's placeholder
inputs by top-level graph order, which can be a different permutation.
For a single-input engine there is nothing to reorder, so it always works.
For a multi-input engine whose inputs get laid out differently after lowering,
the two orders disagree and a tensor lands in the wrong binding -- which either
fails loudly (rank/shape mismatch, as above) or, if two inputs happen to share
a shape, silently produces wrong results.
Alternatives considered
Match tensors to bindings by name at runtime. Not possible: the ExecuTorch
delegate ABI hands
execute()only positional values, with no names. Addingnames would require changing the serialized blob format and the runtime ABI.
Match names to placeholders by name at serialization time. Does not work: the
TensorRT binding names are the original semantic placeholder targets (e.g.
timesteps,position_ids) while the lowered ExecuTorch placeholders aregeneric (
arg_0,arg_1, ...). There is no reliable name correspondence.Fix the order earlier, during TensorRT conversion/export. Does not stick:
ExecuTorch performs another lowering/fusion pass afterward that can re-permute
the placeholders, so any alignment done before that pass is undone.
The solution
Reorder
input_binding_namesinside the ExecuTorch backendpreprocess()--the first point where both the TensorRT binding order and the final delegate
argument order are visible. The permutation is recovered by node identity,
not by name: the engine node's first argument lists its input nodes in binding
order, and each of those nodes is a placeholder in the lowered graph whose
position is exactly the runtime argument slot it will occupy. So we sort the
binding names by each input node's placeholder position.
Single-input engines are naturally identity, so models that already worked are
unaffected. The runtime is unchanged, and the per-binding optimization-profile
bounds follow automatically because they are looked up by name at engine init
(
getProfileShape).Adds unit tests in
tests/py/dynamo/executorch/test_backend.py: single-inputidentity, node-identity reorder, generic-name multi-input permutation, and a
guard for a binding-count mismatch.