Skip to content

fix: bind ExecuTorch TRT engine inputs in delegate arg order#4400

Open
shoumikhin wants to merge 1 commit into
pytorch:mainfrom
shoumikhin:fix/executorch-input-binding-order
Open

fix: bind ExecuTorch TRT engine inputs in delegate arg order#4400
shoumikhin wants to merge 1 commit into
pytorch:mainfrom
shoumikhin:fix/executorch-input-binding-order

Conversation

@shoumikhin

Copy link
Copy Markdown
Contributor

The problem

When a model with more than one input is compiled to a TensorRT engine and
run through ExecuTorch, the inputs can be delivered to the wrong engine slots.
The failure looks like this at runtime:

input 'timesteps' rank 4 does not match profile rank 1

The model is fine -- the wrong tensor is simply arriving at that slot.

Why it happens: the ExecuTorch TensorRT runtime backend matches inputs to
engine bindings by position. Argument i handed to execute() is fed to
input_binding_names[i]. There are no names attached to the incoming tensors
at runtime; the backend just trusts that the i-th tensor belongs to the i-th
binding name.

But those two lists are built by two different stages that never coordinate:

  1. input_binding_names is recorded during TensorRT conversion, in the order
    the interpreter visits the subgraph placeholders
    (_TRTInterpreter.placeholder).
  2. The tensors passed to executorch_call_delegate at runtime are ordered by
    ExecuTorch's own lowering: its FX fuser sorts the delegate's placeholder
    inputs by top-level graph order, which can be a different permutation.

For a single-input engine there is nothing to reorder, so it always works.
For a multi-input engine whose inputs get laid out differently after lowering,
the two orders disagree and a tensor lands in the wrong binding -- which either
fails loudly (rank/shape mismatch, as above) or, if two inputs happen to share
a shape, silently produces wrong results.

Alternatives considered

  • Match tensors to bindings by name at runtime. Not possible: the ExecuTorch
    delegate ABI hands execute() only positional values, with no names. Adding
    names would require changing the serialized blob format and the runtime ABI.

  • Match names to placeholders by name at serialization time. Does not work: the
    TensorRT binding names are the original semantic placeholder targets (e.g.
    timesteps, position_ids) while the lowered ExecuTorch placeholders are
    generic (arg_0, arg_1, ...). There is no reliable name correspondence.

  • Fix the order earlier, during TensorRT conversion/export. Does not stick:
    ExecuTorch performs another lowering/fusion pass afterward that can re-permute
    the placeholders, so any alignment done before that pass is undone.

The solution

Reorder input_binding_names inside the ExecuTorch backend preprocess() --
the first point where both the TensorRT binding order and the final delegate
argument order are visible. The permutation is recovered by node identity,
not by name: the engine node's first argument lists its input nodes in binding
order, and each of those nodes is a placeholder in the lowered graph whose
position is exactly the runtime argument slot it will occupy. So we sort the
binding names by each input node's placeholder position.

Single-input engines are naturally identity, so models that already worked are
unaffected. The runtime is unchanged, and the per-binding optimization-profile
bounds follow automatically because they are looked up by name at engine init
(getProfileShape).

Adds unit tests in tests/py/dynamo/executorch/test_backend.py: single-input
identity, node-identity reorder, generic-name multi-input permutation, and a
guard for a binding-count mismatch.

@meta-cla meta-cla Bot added the cla signed label Jul 13, 2026
@github-actions github-actions Bot added component: tests Issues re: Tests component: api [Python] Issues re: Python API labels Jul 13, 2026
@github-actions github-actions Bot requested a review from narendasan July 13, 2026 06:04
@shoumikhin shoumikhin force-pushed the fix/executorch-input-binding-order branch from e934dec to 7cb6d0b Compare July 13, 2026 06:17
@narendasan

Copy link
Copy Markdown
Collaborator

Does executorch have a deterministic input flattening like exported program does with pytrees?

@shoumikhin shoumikhin force-pushed the fix/executorch-input-binding-order branch from 7cb6d0b to 1bc58ea Compare July 13, 2026 16:23
@shoumikhin

Copy link
Copy Markdown
Contributor Author

Yes, ExecuTorch's delegate input order is deterministic, but determinism is not the issue here; order agreement between two independently-produced orders is.

There are two separate ordering sources that are each deterministic on their own:

  1. TRT engine binding order is fixed early, at conversion time, from the TRT submodule's placeholder visitation order (_TRTInterpreter.placeholder appends to _input_names, which becomes TorchTensorRTModule.input_binding_names and is serialized into the engine blob).
  2. ExecuTorch delegate arg order is produced later, during lowering. The FX fuser sorts the delegate submodule's external placeholder inputs by top-level graph order (fuse_as_graphmodule in torch/fx/passes/utils/fuser_utils.py), and executorch_call_delegate then receives those args positionally.

Both are deterministic, but they are derived by different stages and need not agree. The runtime binds purely positionally (args[i] -> input_binding_names[i] in TensorRTBackend.cpp), so when the two orders disagree (multi-input engines where fusion permuted the placeholders) the wrong tensor lands in a binding and you get e.g. input 'timesteps' rank 4 does not match profile rank 1.

On the pytree angle: torch.export's in/out pytree spec governs the program boundary flatten/unflatten, not the internal lowered delegate ABI. The executorch_call_delegate node carries only positional value IDs, with no binding-name metadata pairing args to engine bindings, so there is no existing ExecuTorch mechanism to lean on. preprocess() is the first point where both the final delegate placeholder order and the TRT engine input-node order are visible, so aligning them there by node identity (rather than by name, since the lowered placeholders are generic arg_N) is the natural fix.

Outputs are also bound positionally but are stable by construction (they are getitem(engine_node, idx) nodes whose index order equals the engine output-binding order, and fusion only re-sorts external placeholder inputs), so only inputs need the reorder. I have added tests covering the multi-input permutation and the output-order-preservation cases.

@narendasan

Copy link
Copy Markdown
Collaborator

cc: @cehongwang who is working on binding metadata for other work

## The problem

When a model with more than one input is compiled to a TensorRT engine and
run through ExecuTorch, the inputs can be delivered to the wrong engine slots.
The failure looks like this at runtime:

    input 'timesteps' rank 4 does not match profile rank 1

The model is fine -- the wrong tensor is simply arriving at that slot.

Why it happens: the ExecuTorch TensorRT runtime backend matches inputs to
engine bindings *by position*. Argument `i` handed to `execute()` is fed to
`input_binding_names[i]`. There are no names attached to the incoming tensors
at runtime; the backend just trusts that the i-th tensor belongs to the i-th
binding name.

But those two lists are built by two different stages that never coordinate:

1. `input_binding_names` is recorded during TensorRT conversion, in the order
   the interpreter visits the subgraph placeholders
   (`_TRTInterpreter.placeholder`).
2. The tensors passed to `executorch_call_delegate` at runtime are ordered by
   ExecuTorch's own lowering: its FX fuser sorts the delegate's placeholder
   inputs by top-level graph order, which can be a different permutation.

For a single-input engine there is nothing to reorder, so it always works.
For a multi-input engine whose inputs get laid out differently after lowering,
the two orders disagree and a tensor lands in the wrong binding -- which either
fails loudly (rank/shape mismatch, as above) or, if two inputs happen to share
a shape, silently produces wrong results.

## Alternatives considered

- Match tensors to bindings by name at runtime. Not possible: the ExecuTorch
  delegate ABI hands `execute()` only positional values, with no names. Adding
  names would require changing the serialized blob format and the runtime ABI.

- Match names to placeholders by name at serialization time. Does not work: the
  TensorRT binding names are the original semantic placeholder targets (e.g.
  `timesteps`, `position_ids`) while the lowered ExecuTorch placeholders are
  generic (`arg_0`, `arg_1`, ...). There is no reliable name correspondence.

- Fix the order earlier, during TensorRT conversion/export. Does not stick:
  ExecuTorch performs another lowering/fusion pass afterward that can re-permute
  the placeholders, so any alignment done before that pass is undone.

## The solution

Reorder `input_binding_names` inside the ExecuTorch backend `preprocess()` --
the first point where both the TensorRT binding order and the final delegate
argument order are visible. The permutation is recovered by *node identity*,
not by name: the engine node's first argument lists its input nodes in binding
order, and each of those nodes is a placeholder in the lowered graph whose
position is exactly the runtime argument slot it will occupy. So we sort the
binding names by each input node's placeholder position.

Single-input engines are naturally identity, so models that already worked are
unaffected. The runtime is unchanged, and the per-binding optimization-profile
bounds follow automatically because they are looked up by name at engine init
(`getProfileShape`).

Adds unit tests in `tests/py/dynamo/executorch/test_backend.py`: single-input
identity, node-identity reorder, generic-name multi-input permutation, and a
guard for a binding-count mismatch.
@shoumikhin shoumikhin force-pushed the fix/executorch-input-binding-order branch from 1bc58ea to d375ceb Compare July 13, 2026 21:36
Comment thread py/torch_tensorrt/executorch/backend.py

@cehongwang cehongwang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed component: api [Python] Issues re: Python API component: tests Issues re: Tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants