Skip to content

fix(qa): TRT 11 strongly-typed cast for plan_float* output dtype - #8836

Merged
mc-nv merged 1 commit into
r26.06from
mchornyi/TRI-1417/fix-trt11-plan-output-dtype-cast
Jun 12, 2026
Merged

fix(qa): TRT 11 strongly-typed cast for plan_float* output dtype#8836
mc-nv merged 1 commit into
r26.06from
mchornyi/TRI-1417/fix-trt11-plan-output-dtype-cast

Conversation

@mc-nv

@mc-nv mc-nv commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

What does the PR do?

Fixes the TRT 11 strongly-typed-network migration of gen_qa_models.py so QA plan_* model engines come out with the declared output dtype.

d898a0fe6 (TRI-1406) only inserted an explicit cast layer when the target output dtype was uint8. Float-to-float dtype changes were left to the now-removed ITensor.dtype setter, which silently no-ops on TRT 11.0. As a result, the generated engines silently inherited the elementwise op's natural output dtype (which equals the input dtype) instead of the declared output dtype, producing CI failures like:

unexpected datatype TYPE_FP32 for inference output 'OUTPUT1', expecting TYPE_FP16 for plan_float32_uint8_float16_0
unexpected datatype TYPE_FP16 for inference output 'OUTPUT0', expecting TYPE_FP32 for plan_float16_float32_float32_0

create_plan_dynamic_rf_modelfile and create_plan_fixed_rf_modelfile now insert trt_cast_tensor() whenever the elementwise op's output dtype differs from the declared output dtype — covers both float<->float and uint8 cases. The dead .dtype setter try/except AttributeError: pass blocks are removed.

Checklist

  • PR title reflects the change and is of format <commit_type>: <Title>
  • Changes are described in the pull request.
  • Related issues are referenced.
  • Populated github labels field
  • Added test plan and verified test passes.
  • Verified that the PR passes existing CI.
  • Verified copyright is correct on all changed files.
  • Added succinct git squash message before merging.
  • All template sections are filled out.
  • Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

  • build
  • ci
  • docs
  • feat
  • fix
  • perf
  • refactor
  • revert
  • style
  • test

Related PRs:

Where should the reviewer start?

qa/common/gen_qa_models.py:

  • create_plan_dynamic_rf_modelfile (around line 109): cast block replaces the uint8-only cast + dead .dtype setter.
  • create_plan_fixed_rf_modelfile (around line 412): cast block added, dead .dtype setter removed.

Test plan:

Reproduction (before patch) on r26.06 HEAD with tensorrt:26.06-py3-base (TRT 11.0):

TENSORRT_IMAGE=gitlab-master.nvidia.com:5005/dl/dgx/tensorrt:26.06-py3-base \
PYTORCH_IMAGE=gitlab-master.nvidia.com:5005/dl/dgx/pytorch:26.06-py3-base \
bash -xe ./qa/common/gen_qa_model_repository

# Then inspect engines via trtexec --loadEngine=<model.plan>

Engines produced before patch (mismatch against config.pbtxt):

Model OUTPUT0 (config / engine) OUTPUT1 (config / engine)
plan_float32_uint8_float16 UINT8 / uint8 FP16 / fp32
plan_float16_float32_float32 FP32 / fp16 FP32 / fp16

Engines produced after patch (all aligned):

Model OUTPUT0 (config / engine) OUTPUT1 (config / engine)
plan_float32_uint8_float16 UINT8 / uint8 ✓ FP16 / fp16 ✓
plan_nobatch_float32_uint8_float16 UINT8 / uint8 ✓ FP16 / fp16 ✓
plan_float16_float32_float32 FP32 / fp32 ✓ FP32 / fp32 ✓
plan_nobatch_float16_float32_float32 FP32 / fp32 ✓ FP32 / fp32 ✓
  • CI Pipeline ID: (to be added — pipeline 54492155 originally exhibited the failure across 11 jobs; a new run from a regenerated 26.06_* QA repo is needed for end-to-end confirmation.)

Caveats:

  • QA /data/inferenceserver/26.06_* mounts that CI consumes must be regenerated from this PR before the affected L0_infer* / L0_server_status jobs will go green. The fix is in the model-generation script; the published model artifacts on shared storage still hold the broken engines until republished.
  • Other generator scripts (gen_qa_identity_models.py, gen_qa_sequence_models.py, gen_qa_implicit_models.py, gen_qa_dyna_sequence_models.py, gen_qa_dyna_sequence_implicit_models.py, gen_qa_reshape_models.py, gen_qa_sequence_models.py) still carry the same dead out.dtype = … / except AttributeError: pass pattern. Audit shows most are harmless because input/output dtypes match in those generators, but a follow-up sweep is recommended.

Background

TRT 11.0 introduced strongly-typed networks (TRT release notes). As part of that change:

  • ITensor.dtype is no longer a writable setter — engine output dtype is inferred from the network graph.
  • ILayer.set_output_type was removed on identity layers in favour of network.add_cast().
  • BuilderFlag.PREFER_PRECISION_CONSTRAINTS, FP16, INT8 were removed (strongly-typed networks make them redundant).

d898a0fe6 added a trt_cast_tensor() helper to gen_common.py and used it for uint8 conversion in create_plan_dynamic_rf_modelfile, but did not cover the float→float dtype change case in either create_plan_dynamic_rf_modelfile or create_plan_fixed_rf_modelfile. This PR completes the migration for those two functions. The same audit was performed on the other QA generators — most do not change dtypes between input and output, so the dead setter is a no-op there; no functional bugs are blocking the immediate CI failures.

Observed failure footprint on pipeline 54492155 (branch 26.06-devel): 11 affected jobs covering SBSA A100, SBSA GB300, AGX-Thor, RTX50-rhel, and the base L0_server_status.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

  • Relates to TRI-1406 (this is a follow-up fix to that work)
  • TRI-1417 (this PR)

Follow-up to TRI-1406 / d898a0f. The previous TRT 11 migration only
inserted an explicit cast layer when the target output dtype was uint8,
leaving float<->float dtype changes routed through the now-removed
ITensor.dtype setter. On TRT 11.0 that setter is a no-op, so the
generated engines silently inherited the elementwise op's natural
output dtype (matching the input dtype) instead of the declared output
dtype. This produced "unexpected datatype TYPE_FP{16,32} for inference
output 'OUTPUT{0,1}'" failures on plan_float32_uint8_float16,
plan_float16_float32_float32, and their nobatch variants — observed
across 11 L0_infer* and L0_server_status jobs on pipeline 54492155
(branch 26.06-devel) on SBSA A100/GB300, AGX-Thor, and RTX50-rhel.

create_plan_dynamic_rf_modelfile and create_plan_fixed_rf_modelfile
now insert trt_cast_tensor() whenever the elementwise op's output
dtype differs from the declared output dtype, regardless of whether
the target is uint8 or another float type. The dead .dtype setter
try/except blocks are removed.

Verified locally on TRT 11.0 (tensorrt:26.06-py3-base): engine
output bindings dumped via trtexec --loadEngine now match the
config.pbtxt declarations for all four previously-failing models.
@mc-nv mc-nv self-assigned this Jun 12, 2026
@mc-nv
mc-nv marked this pull request as ready for review June 12, 2026 19:46
@mc-nv
mc-nv merged commit d4d7275 into r26.06 Jun 12, 2026
3 checks passed
@mc-nv
mc-nv deleted the mchornyi/TRI-1417/fix-trt11-plan-output-dtype-cast branch June 12, 2026 19:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants