Commit 1bc58ea
committed
fix: bind ExecuTorch TRT engine inputs in delegate arg order
## The problem
When a model with more than one input is compiled to a TensorRT engine and
run through ExecuTorch, the inputs can be delivered to the wrong engine slots.
The failure looks like this at runtime:
input 'timesteps' rank 4 does not match profile rank 1
The model is fine -- the wrong tensor is simply arriving at that slot.
Why it happens: the ExecuTorch TensorRT runtime backend matches inputs to
engine bindings *by position*. Argument `i` handed to `execute()` is fed to
`input_binding_names[i]`. There are no names attached to the incoming tensors
at runtime; the backend just trusts that the i-th tensor belongs to the i-th
binding name.
But those two lists are built by two different stages that never coordinate:
1. `input_binding_names` is recorded during TensorRT conversion, in the order
the interpreter visits the subgraph placeholders
(`_TRTInterpreter.placeholder`).
2. The tensors passed to `executorch_call_delegate` at runtime are ordered by
ExecuTorch's own lowering: its FX fuser sorts the delegate's placeholder
inputs by top-level graph order, which can be a different permutation.
For a single-input engine there is nothing to reorder, so it always works.
For a multi-input engine whose inputs get laid out differently after lowering,
the two orders disagree and a tensor lands in the wrong binding -- which either
fails loudly (rank/shape mismatch, as above) or, if two inputs happen to share
a shape, silently produces wrong results.
## Alternatives considered
- Match tensors to bindings by name at runtime. Not possible: the ExecuTorch
delegate ABI hands `execute()` only positional values, with no names. Adding
names would require changing the serialized blob format and the runtime ABI.
- Match names to placeholders by name at serialization time. Does not work: the
TensorRT binding names are the original semantic placeholder targets (e.g.
`timesteps`, `position_ids`) while the lowered ExecuTorch placeholders are
generic (`arg_0`, `arg_1`, ...). There is no reliable name correspondence.
- Fix the order earlier, during TensorRT conversion/export. Does not stick:
ExecuTorch performs another lowering/fusion pass afterward that can re-permute
the placeholders, so any alignment done before that pass is undone.
## The solution
Reorder `input_binding_names` inside the ExecuTorch backend `preprocess()` --
the first point where both the TensorRT binding order and the final delegate
argument order are visible. The permutation is recovered by *node identity*,
not by name: the engine node's first argument lists its input nodes in binding
order, and each of those nodes is a placeholder in the lowered graph whose
position is exactly the runtime argument slot it will occupy. So we sort the
binding names by each input node's placeholder position.
Single-input engines are naturally identity, so models that already worked are
unaffected. The runtime is unchanged, and the per-binding optimization-profile
bounds follow automatically because they are looked up by name at engine init
(`getProfileShape`).
Adds unit tests in `tests/py/dynamo/executorch/test_backend.py`: single-input
identity, node-identity reorder, generic-name multi-input permutation, and a
guard for a binding-count mismatch.1 parent 2aac435 commit 1bc58ea
2 files changed
Lines changed: 296 additions & 25 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
58 | 58 | | |
59 | 59 | | |
60 | 60 | | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
61 | 72 | | |
62 | 73 | | |
63 | 74 | | |
| |||
71 | 82 | | |
72 | 83 | | |
73 | 84 | | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
81 | 88 | | |
82 | 89 | | |
83 | 90 | | |
| |||
198 | 205 | | |
199 | 206 | | |
200 | 207 | | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
201 | 250 | | |
202 | 251 | | |
203 | 252 | | |
| |||
223 | 272 | | |
224 | 273 | | |
225 | 274 | | |
226 | | - | |
| 275 | + | |
| 276 | + | |
227 | 277 | | |
228 | 278 | | |
229 | 279 | | |
| |||
240 | 290 | | |
241 | 291 | | |
242 | 292 | | |
243 | | - | |
244 | | - | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
245 | 297 | | |
246 | 298 | | |
247 | 299 | | |
| |||
0 commit comments