executorch: fall back to the CUDA backend for ops TensorRT can't take

[Conarnar]

Description

When saving with output_format="executorch", ops that TensorRT does not convert (unsupported ops, or ops pinned via torch_executed_ops) are left on eager CPU kernels. This adds an opt-in CUDA fallback: save(..., cuda_fallback=True) appends an ExecuTorch CudaPartitioner as a catch-all after the TensorRTPartitioner, so those ops run on ExecuTorch's CUDA (AOTInductor) backend instead.

Opt-in (cuda_fallback=False by default). Enabling it requires ExecuTorch's CUDA backend and a CUDA toolkit (nvcc/ptxas) at export time, runs AOTInductor autotuning, and produces a .pte that needs a CUDA runtime at load. Leaving it off keeps the previous behavior — TRT-excluded ops stay non-delegated and the .pte loads on CPU — so partially-convertible models are unaffected. If cuda_fallback=True is requested but executorch.backends.cuda is unavailable, save() raises a clear error rather than silently degrading.

External weights. The CUDA backend emits its weights as external named data; save() now persists them (write_tensor_data_to_file) next to the .pte, so a fallback partition that carries weights doesn't lose its blob and fail to load.

Device consistency.

• The fallback's target_device is derived from the caller's compile_specs, so the CUDA partition lands on the same GPU as the TRT engines (previously hardcoded cuda:0).
• Fixes _exporter.py: lifted parameter/buffer placeholders now keep their real device and dtype. Previously torch.empty_strided(...) defaulted them to cpu/float32, which caused a FakeTensorDeviceMismatchError (cuda:0 vs cpu) when a weighted op was excluded from TensorRT.

Testing

Added tests/py/dynamo/executorch/test_cuda_fallback.py:

• test_erfinv_falls_back_to_cuda_backend — erfinv (no TRT converter, no torch_executed_ops pin needed) lands on the CUDA backend while tanh/cos stay on TensorRT.
• test_weighted_fallback_persists_external_data — a weighted, TRT-excluded mm produces a CUDA delegate and its external .ptd data is written next to the .pte.

Both are scoped to delegate composition + external-data persistence; a load-run-allclose check is deferred until the mixed ATen-mode runtime path (tensor_parser_aten) lands. Tests skip cleanly without a GPU / the CUDA backend / nvcc.

[Gasoonjia]

can we dynamic choose the method name, instead of statically use forward?
Also in the test lets add a multi-method one.

[Conarnar]

At the moment save only supports a single exported program which to_edge_transform_and_lower will default to the method name forward.

[Gasoonjia]

is the model runnable now? Can we handle the data transfer between different delegate blobs?

[Conarnar]

Should be runnable.

[shoumikhin]

A few points from a closer look, most important first.

1. Dropped external CUDA weights. _save_as_executorch only calls executorch_program.write_to_file(f), but the CUDA backend emits its weights as external named data (save_data_externally is True, tagged aoti_<device>_blob) that ExecuTorch serializes only through write_tensor_data_to_file. Since that is never called, any residual partition that carries weights loses its blob and the .pte cannot load. The new test does not catch this because cos(erfinv(tanh(x))) has no parameters, so please persist the external data next to the .pte and add a test model with weights in the fallback partition.

2. The import guard is too narrow. The try/except only catches ImportError on the module import, but the real CUDA compile runs later inside to_edge_transform_and_lower via CudaBackend.preprocess calling torch._inductor.aot_compile with max_autotune and the Triton backends, which needs nvcc/ptxas. On a host that has the executorch cuda wheel but no cuda toolkit, export now throws where before the residual op stayed non-delegated and save produced a working CPU .pte, so the docstring line about degrading gracefully only holds for the missing-import case. Consider gating the fallback on ptxas availability or making it opt-in, and narrowing the docstring.

3. Always on with no opt-out. Appending CudaPartitioner unconditionally means any model with even one non-TRT op now embeds a CudaBackend blob and requires a CUDA runtime at load, where before those ops stayed non-delegated and the .pte ran on CPU. Most real models are not fully TRT-convertible, and this also silently turns on max_autotune autotuning at save time, so please add a save() kwarg to keep the portable CPU-fallback behavior.

4. target_device is not propagated to the fallback. It is forwarded only to TensorRTPartitioner; the fallback is built as cuda_specs = [CudaBackend.generate_method_name_compile_spec("forward")] with no target_device, so CudaPartitioner defaults to cuda:0. If a user pins TRT to cuda:1 the two delegates end up on different devices, so derive the fallback target_device from the same compile_specs.

5. The test proves composition, not runnability. It only deserializes the program and checks that CudaBackend and TensorRTBackend appear in the delegate ids; it never builds a Method, runs, or compares to eager. The mixed ATen-mode roundtrip is also still gated on the tensor_parser_aten fix (planned CUDA buffers get a CPU data ptr so Method::init fails in getDeviceFromPtr), so scope the assertion to "produces both delegates" and add a real load-run-allclose once that lands.

[Gasoonjia]

Always enjoy reading Anthony's comment

1. for external weight lets make it as part of pte to in line with tensorrt type., instead of an individual ptd file like standard cuda-only delegated model.
2. we should make sure it is executable. I would encourage to add a pybinding test as guard.