fix(qa): TRT 11 strongly-typed cast for plan_float* output dtype - #8836
Merged
Merged
Conversation
Follow-up to TRI-1406 / d898a0f. The previous TRT 11 migration only inserted an explicit cast layer when the target output dtype was uint8, leaving float<->float dtype changes routed through the now-removed ITensor.dtype setter. On TRT 11.0 that setter is a no-op, so the generated engines silently inherited the elementwise op's natural output dtype (matching the input dtype) instead of the declared output dtype. This produced "unexpected datatype TYPE_FP{16,32} for inference output 'OUTPUT{0,1}'" failures on plan_float32_uint8_float16, plan_float16_float32_float32, and their nobatch variants — observed across 11 L0_infer* and L0_server_status jobs on pipeline 54492155 (branch 26.06-devel) on SBSA A100/GB300, AGX-Thor, and RTX50-rhel. create_plan_dynamic_rf_modelfile and create_plan_fixed_rf_modelfile now insert trt_cast_tensor() whenever the elementwise op's output dtype differs from the declared output dtype, regardless of whether the target is uint8 or another float type. The dead .dtype setter try/except blocks are removed. Verified locally on TRT 11.0 (tensorrt:26.06-py3-base): engine output bindings dumped via trtexec --loadEngine now match the config.pbtxt declarations for all four previously-failing models.
whoisj
approved these changes
Jun 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does the PR do?
Fixes the TRT 11 strongly-typed-network migration of
gen_qa_models.pyso QAplan_*model engines come out with the declared output dtype.d898a0fe6(TRI-1406) only inserted an explicit cast layer when the target output dtype wasuint8. Float-to-float dtype changes were left to the now-removedITensor.dtypesetter, which silently no-ops on TRT 11.0. As a result, the generated engines silently inherited the elementwise op's natural output dtype (which equals the input dtype) instead of the declared output dtype, producing CI failures like:create_plan_dynamic_rf_modelfileandcreate_plan_fixed_rf_modelfilenow inserttrt_cast_tensor()whenever the elementwise op's output dtype differs from the declared output dtype — covers bothfloat<->floatanduint8cases. The dead.dtypesettertry/except AttributeError: passblocks are removed.Checklist
<commit_type>: <Title>Commit Type:
Related PRs:
d898a0fe6— original TRT 11 QA migration)Where should the reviewer start?
qa/common/gen_qa_models.py:create_plan_dynamic_rf_modelfile(around line 109): cast block replaces the uint8-only cast + dead.dtypesetter.create_plan_fixed_rf_modelfile(around line 412): cast block added, dead.dtypesetter removed.Test plan:
Reproduction (before patch) on
r26.06HEAD withtensorrt:26.06-py3-base(TRT 11.0):TENSORRT_IMAGE=gitlab-master.nvidia.com:5005/dl/dgx/tensorrt:26.06-py3-base \ PYTORCH_IMAGE=gitlab-master.nvidia.com:5005/dl/dgx/pytorch:26.06-py3-base \ bash -xe ./qa/common/gen_qa_model_repository # Then inspect engines via trtexec --loadEngine=<model.plan>Engines produced before patch (mismatch against
config.pbtxt):plan_float32_uint8_float16plan_float16_float32_float32Engines produced after patch (all aligned):
plan_float32_uint8_float16plan_nobatch_float32_uint8_float16plan_float16_float32_float32plan_nobatch_float16_float32_float3226.06_*QA repo is needed for end-to-end confirmation.)Caveats:
/data/inferenceserver/26.06_*mounts that CI consumes must be regenerated from this PR before the affectedL0_infer*/L0_server_statusjobs will go green. The fix is in the model-generation script; the published model artifacts on shared storage still hold the broken engines until republished.gen_qa_identity_models.py,gen_qa_sequence_models.py,gen_qa_implicit_models.py,gen_qa_dyna_sequence_models.py,gen_qa_dyna_sequence_implicit_models.py,gen_qa_reshape_models.py,gen_qa_sequence_models.py) still carry the same deadout.dtype = …/except AttributeError: passpattern. Audit shows most are harmless because input/output dtypes match in those generators, but a follow-up sweep is recommended.Background
TRT 11.0 introduced strongly-typed networks (TRT release notes). As part of that change:
ITensor.dtypeis no longer a writable setter — engine output dtype is inferred from the network graph.ILayer.set_output_typewas removed on identity layers in favour ofnetwork.add_cast().BuilderFlag.PREFER_PRECISION_CONSTRAINTS,FP16,INT8were removed (strongly-typed networks make them redundant).d898a0fe6added atrt_cast_tensor()helper togen_common.pyand used it foruint8conversion increate_plan_dynamic_rf_modelfile, but did not cover the float→float dtype change case in eithercreate_plan_dynamic_rf_modelfileorcreate_plan_fixed_rf_modelfile. This PR completes the migration for those two functions. The same audit was performed on the other QA generators — most do not change dtypes between input and output, so the dead setter is a no-op there; no functional bugs are blocking the immediate CI failures.Observed failure footprint on pipeline 54492155 (branch 26.06-devel): 11 affected jobs covering SBSA A100, SBSA GB300, AGX-Thor, RTX50-rhel, and the base
L0_server_status.Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)