[None][fix] Add AutoDeploy post-merge stages by hnover-nv · Pull Request #15418 · NVIDIA/TensorRT-LLM

hnover-nv · 2026-06-16T16:09:56Z

Summary by CodeRabbit

Tests
- Enhanced post-merge testing pipeline with additional automated test stages for comprehensive validation.
- Updated performance testing configuration for improved accuracy across hardware variants.

Description

Add missing AutoDeploy post-merge Jenkins stages for test-db blocks that already declare stage: post_merge and backend: autodeploy:

H100_PCIe-AutoDeploy-Post-Merge-1 for l0_h100
DGX_B200-AutoDeploy-Post-Merge-1 for l0_b200
DGX_B200-4_GPUs-AutoDeploy-Post-Merge-1 for l0_dgx_b200

These mirror the existing pre-merge AutoDeploy stage configurations for the same YAML files and GPU counts. Without these stages, the corresponding post-merge AutoDeploy test-db blocks are present but not selected by Jenkins unless the same tests also appear in another reachable block.

Also remove one duplicate 4-GPU B200 AutoDeploy perf sanity entry from the post-merge block. That test remains in the pre-merge AutoDeploy block, which is also run during post-merge, so keeping it in both places would make it run twice after this PR makes the post-merge block reachable.

After enabling the previously unreachable B200 AutoDeploy post-merge blocks, PR CI exposed existing Nano NVFP4 failures:

accuracy/test_llm_api_autodeploy.py::TestNemotronNanoV3::test_accuracy[nvfp4-1-attn_dp_off-trtllm]: GSM8K accuracy below threshold.
accuracy/test_llm_api_autodeploy.py::TestNemotronNanoV3::test_accuracy[nvfp4-4-attn_dp_off-trtllm]: timeout / hang during executor shutdown.

This PR temporarily waives those exact tests so the stage wiring fix can land while the accuracy/hang regressions are investigated separately. The comparable pre-move single-GPU test previously ran as test_accuracy[nvfp4-1-trtllm]; the newer pytest ids come from the later attention-DP parameter split.

Test Coverage

Ran a local one-off reachability check over jenkins/L0_Test.groovy and tests/integration/test_lists/test-db/*.yml for AutoDeploy post-merge blocks.
- Before this fix: 3 missing AutoDeploy post-merge blocks.
- After this fix: 5 matched AutoDeploy post-merge blocks, 0 missing.
Checked pre/post duplicate overlap for the newly covered AutoDeploy stage shapes:
- l0_h100.yml, 1 GPU: 0 overlapping tests.
- l0_b200.yml, 1 GPU: 0 overlapping tests.
- l0_dgx_b200.yml, 4 GPUs: 0 overlapping tests after removing the duplicate perf sanity entry.
Parsed tests/integration/test_lists/test-db/l0_dgx_b200.yml with PyYAML after editing.
Ran ./scripts/check_test_list.py --check-duplicate-waives.
Remote PR CI was run with:
- /bot run --stage-list "H100_PCIe-AutoDeploy-Post-Merge-1,DGX_B200-AutoDeploy-Post-Merge-1,DGX_B200-4_GPUs-AutoDeploy-Post-Merge-1"
- Build #43691 selected the new post-merge stages and exposed the waived 1-GPU Nano NVFP4 accuracy failure.
- Build #44060 selected the new post-merge stages and exposed the waived 4-GPU Nano NVFP4 timeout/hang. The H100 stage passed, the B200 1-GPU stage passed with the 1-GPU waive applied, and the B200 4-GPU stage passed three GPT-OSS tests before timing out on the newly waived 4-GPU Nano test.

No runtime CI was run locally; this is a Jenkins stage mapping, test-list cleanup, and temporary waive change.

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

hnover-nv · 2026-06-16T18:31:47Z

/bot run --post-merge

coderabbitai · 2026-06-16T18:35:01Z

📝 Walkthrough

Walkthrough

Three new post-merge AutoDeploy stage mappings are added to launchTestJobs() in the Jenkins pipeline for H100 PCIe and DGX B200 (both single and 4-GPU configurations). Additionally, the 4-GPU DGX B200 AutoDeploy perf sanity test list replaces the super_ad aggregated upload variant with the super_mtp_ad variant.

Changes

AutoDeploy Post-Merge CI Configuration

Layer / File(s)	Summary
New Jenkins post-merge AutoDeploy stage mappings `jenkins/L0_Test.groovy`	Adds `H100_PCIe-AutoDeploy-Post-Merge-1` (x86, `h100-cr`, `l0_h100`, split `1,1`), `DGX_B200-AutoDeploy-Post-Merge-1` (x86 SLURM, `auto:dgx-b200-flex`, `l0_b200`, split `1,1,1,1`), and `DGX_B200-4_GPUs-AutoDeploy-Post-Merge-1` (x86 SLURM, `auto:dgx-b200-flex`, `l0_dgx_b200`, split `1,1,4,1`) to `launchTestJobs()`.
DGX B200 4-GPU perf sanity test variant swap `tests/integration/test_lists/test-db/l0_dgx_b200.yml`	Replaces the `aggr_upload-super_ad_blackwell-super_ad_ws4_1k1k` `test_e2e` variant with `aggr_upload-super_mtp_ad_blackwell-super_mtp_ad_ws4_1k1k` in the 4-GPU `post_merge` AutoDeploy perf sanity block.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The PR title clearly and concisely describes the main change: adding AutoDeploy post-merge stages. It follows the required format [None][fix] and accurately reflects the primary objective of the changeset.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The PR description comprehensively explains the changes, including what was added (three AutoDeploy post-merge Jenkins stages), why it was needed (to enable previously unreachable test-db blocks), and detailed test coverage validation performed by the author.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

tensorrt-cicd · 2026-06-16T18:38:35Z

PR_Github #54654 [ run ] triggered by Bot. Commit: df21810 Link to invocation

tburt-nv · 2026-06-16T19:06:54Z

/bot run --stage-list "H100_PCIe-AutoDeploy-Post-Merge-1,DGX_B200-AutoDeploy-Post-Merge-1,DGX_B200-4_GPUs-AutoDeploy-Post-Merge-1"

tburt-nv · 2026-06-16T19:07:49Z

#15418 (comment) would launch the entire post-merge pipeline, but we only need to run these stages.

tensorrt-cicd · 2026-06-16T19:14:04Z

PR_Github #54660 [ run ] triggered by Bot. Commit: df21810 Link to invocation

hnover-nv · 2026-06-16T19:14:12Z

#15418 (comment) would launch the entire post-merge pipeline, but we only need to run these stages.

I wanted to make sure the normal post-merge pipeline picked the stages up. While it should be enough the stages exist, it seemed possible something else was mis-wired so that they could be directly invoked but would not auto-run. but if you think it looks ok and we just need to check that the tests are still green (they were off for a week or two) then that's find with me, we can check manually next week or whatever that they ran.

tensorrt-cicd · 2026-06-16T19:17:35Z

PR_Github #54654 [ run ] completed with state ABORTED. Commit: df21810

Link to invocation

tensorrt-cicd · 2026-06-16T20:52:02Z

PR_Github #54660 [ run ] completed with state FAILURE. Commit: df21810
/LLM/main/L0_MergeRequest_PR pipeline #43691 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hnover-nv · 2026-06-16T23:50:06Z

/bot run --disable-fail-fast

github-actions · 2026-06-16T23:55:45Z

👎 Promotion blocked, new vulnerability found

Vulnerability report

Component	Vulnerability	Description	Severity
pytorch	CVE-2025-3000	A vulnerability classified as critical has been found in PyTorch 2.6.0. This affects the function torch.jit.script. The manipulation leads to memory corruption. It is possible to launch the attack on the local host. The exploit has been disclosed to the public and may be used.	MEDIUM

hnover-nv · 2026-06-22T19:05:41Z

/bot run --stage-list "H100_PCIe-AutoDeploy-Post-Merge-1,DGX_B200-AutoDeploy-Post-Merge-1,DGX_B200-4_GPUs-AutoDeploy-Post-Merge-1"

tensorrt-cicd · 2026-06-22T19:11:35Z

PR_Github #55073 [ run ] triggered by Bot. Commit: b0dbcce Link to invocation

tensorrt-cicd · 2026-06-22T22:41:12Z

PR_Github #55073 [ run ] completed with state FAILURE. Commit: b0dbcce
/LLM/main/L0_MergeRequest_PR pipeline #44060 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hnover-nv · 2026-06-23T04:08:06Z

/bot run --stage-list "H100_PCIe-AutoDeploy-Post-Merge-1,DGX_B200-AutoDeploy-Post-Merge-1,DGX_B200-4_GPUs-AutoDeploy-Post-Merge-1"

github-actions · 2026-06-23T04:08:12Z

⚠️ Bot command ignored: The /bot command must appear at the very beginning of the comment (no leading blank lines or spaces). Please post a new comment with /bot as the first character.

hnover-nv · 2026-06-23T04:18:27Z

/bot run --stage-list "H100_PCIe-AutoDeploy-Post-Merge-1,DGX_B200-AutoDeploy-Post-Merge-1,DGX_B200-4_GPUs-AutoDeploy-Post-Merge-1"

tensorrt-cicd · 2026-06-23T04:23:55Z

PR_Github #55152 [ run ] triggered by Bot. Commit: 000fbce Link to invocation

hnover-nv · 2026-06-26T19:38:07Z

Let's try this again!

hnover-nv · 2026-06-26T19:38:24Z

/bot run --stage-list "H100_PCIe-AutoDeploy-Post-Merge-1,DGX_B200-AutoDeploy-Post-Merge-1,DGX_B200-4_GPUs-AutoDeploy-Post-Merge-1"

tensorrt-cicd · 2026-06-26T19:47:46Z

PR_Github #56127 [ run ] triggered by Bot. Commit: 124f6e8 Link to invocation

tensorrt-cicd · 2026-06-26T23:58:31Z

PR_Github #56127 [ run ] completed with state SUCCESS. Commit: 124f6e8
/LLM/main/L0_MergeRequest_PR pipeline #44991 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

hnover-nv · 2026-06-29T03:02:33Z

CI passed but only "partly tested". I don't know what that means but github is taking it as not a pass? I'll try rerunning w/out post-merge stages and see what happens I guess.

hnover-nv · 2026-06-29T03:02:37Z

/bot run

hnover-nv · 2026-06-29T03:02:48Z

/bot run

tensorrt-cicd · 2026-06-29T03:08:12Z

PR_Github #56281 [ run ] triggered by Bot. Commit: 124f6e8 Link to invocation

tensorrt-cicd · 2026-06-29T03:08:45Z

PR_Github #56282 [ run ] triggered by Bot. Commit: 124f6e8 Link to invocation

tensorrt-cicd · 2026-06-29T03:08:48Z

PR_Github #56281 [ run ] completed with state ABORTED. Commit: 124f6e8

Link to invocation

tensorrt-cicd · 2026-06-29T03:47:04Z

PR_Github #56282 [ run ] completed with state FAILURE. Commit: 124f6e8
/LLM/main/L0_MergeRequest_PR pipeline #45135 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hnover-nv · 2026-06-29T04:59:43Z

/bot run

tensorrt-cicd · 2026-06-29T05:05:02Z

PR_Github #56303 [ run ] triggered by Bot. Commit: 124f6e8 Link to invocation

tensorrt-cicd · 2026-06-29T07:20:25Z

PR_Github #56303 [ run ] completed with state FAILURE. Commit: 124f6e8
/LLM/main/L0_MergeRequest_PR pipeline #45154 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hnover-nv · 2026-06-29T18:19:31Z

/bot run

tensorrt-cicd · 2026-06-29T18:25:53Z

PR_Github #56436 [ run ] triggered by Bot. Commit: 124f6e8 Link to invocation

hnover-nv · 2026-06-29T18:32:52Z

/bot kill

Signed-off-by: Harris Nover <249353502+hnover-nv@users.noreply.github.com>

tensorrt-cicd · 2026-06-29T18:39:25Z

PR_Github #56440 [ kill ] triggered by Bot. Commit: 0644801 Link to invocation

hnover-nv · 2026-06-29T18:39:52Z

/bot help

github-actions · 2026-06-29T18:40:00Z

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Details

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental) --high-priority]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Supports wildcard * for pattern matching (e.g., "*PerfSanity*" matches all stages containing PerfSanity). Examples: "A10-PyTorch-1, xxx", "PerfSanity". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Supports wildcard * for pattern matching. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx", --extra-stage "Post-Merge".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

--high-priority (OPTIONAL) : Run the pipeline with high priority. This option is restricted to authorized users only and will route the job to a high-priority queue.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

tensorrt-cicd · 2026-06-29T18:44:38Z

PR_Github #56436 [ run ] completed with state ABORTED. Commit: 124f6e8

Link to invocation

tensorrt-cicd · 2026-06-29T18:44:50Z

PR_Github #56440 [ kill ] completed with state SUCCESS. Commit: 0644801
Successfully killed previous jobs for commit 0644801

Link to invocation

hnover-nv · 2026-06-29T18:51:12Z

/bot run --stage-list "H100_PCIe-AutoDeploy-Post-Merge-1,DGX_B200-AutoDeploy-Post-Merge-1,DGX_B200-4_GPUs-AutoDeploy-Post-Merge-1"

tensorrt-cicd · 2026-06-29T18:57:32Z

PR_Github #56443 [ run ] triggered by Bot. Commit: 0644801 Link to invocation

tensorrt-cicd · 2026-06-29T20:41:31Z

PR_Github #56443 [ run ] completed with state FAILURE. Commit: 0644801
/LLM/main/L0_MergeRequest_PR pipeline #45285 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

github-actions Bot assigned hnover-nv Jun 16, 2026

hnover-nv changed the title ~~[None][infra] Add AutoDeploy post-merge stages~~ [None][fix] Add AutoDeploy post-merge stages Jun 16, 2026

hnover-nv force-pushed the hnover/ad-post-merge-stages branch 2 times, most recently from 37f9c2a to df21810 Compare June 16, 2026 18:18

hnover-nv marked this pull request as ready for review June 16, 2026 18:32

hnover-nv requested review from a team as code owners June 16, 2026 18:32

hnover-nv requested review from tburt-nv and yiqingy0 June 16, 2026 18:32

govind-ramnarayan approved these changes Jun 16, 2026

View reviewed changes

tburt-nv approved these changes Jun 16, 2026

View reviewed changes

hnover-nv force-pushed the hnover/ad-post-merge-stages branch from df21810 to 0b1f948 Compare June 16, 2026 23:49

hnover-nv closed this Jun 29, 2026

hnover-nv reopened this Jun 29, 2026

hnover-nv force-pushed the hnover/ad-post-merge-stages branch from 124f6e8 to cd4fe54 Compare June 29, 2026 18:37

[None][infra] Add AutoDeploy post-merge stages

0644801

Signed-off-by: Harris Nover <249353502+hnover-nv@users.noreply.github.com>

hnover-nv force-pushed the hnover/ad-post-merge-stages branch from cd4fe54 to 0644801 Compare June 29, 2026 18:39

Uh oh!

Conversation

hnover-nv commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

hnover-nv commented Jun 16, 2026

Uh oh!

coderabbitai Bot commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

tburt-nv commented Jun 16, 2026

Uh oh!

tburt-nv commented Jun 16, 2026

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

hnover-nv commented Jun 16, 2026

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

tensorrt-cicd commented Jun 16, 2026

Uh oh!

hnover-nv commented Jun 16, 2026

Uh oh!

github-actions Bot commented Jun 16, 2026

👎 Promotion blocked, new vulnerability found

Vulnerability report

Uh oh!

hnover-nv commented Jun 22, 2026

Uh oh!

tensorrt-cicd commented Jun 22, 2026

Uh oh!

tensorrt-cicd commented Jun 22, 2026

Uh oh!

hnover-nv commented Jun 23, 2026

Uh oh!

github-actions Bot commented Jun 23, 2026

Uh oh!

hnover-nv commented Jun 23, 2026

Uh oh!

tensorrt-cicd commented Jun 23, 2026

Uh oh!

hnover-nv commented Jun 26, 2026

Uh oh!

hnover-nv commented Jun 26, 2026

Uh oh!

tensorrt-cicd commented Jun 26, 2026

Uh oh!

tensorrt-cicd commented Jun 26, 2026

Uh oh!

hnover-nv commented Jun 29, 2026

Uh oh!

hnover-nv commented Jun 29, 2026

Uh oh!

hnover-nv commented Jun 29, 2026

Uh oh!

tensorrt-cicd commented Jun 29, 2026

Uh oh!

tensorrt-cicd commented Jun 29, 2026

Uh oh!

tensorrt-cicd commented Jun 29, 2026

Uh oh!

tensorrt-cicd commented Jun 29, 2026

Uh oh!

hnover-nv commented Jun 29, 2026

Uh oh!

tensorrt-cicd commented Jun 29, 2026

Uh oh!

tensorrt-cicd commented Jun 29, 2026

Uh oh!

hnover-nv commented Jun 16, 2026 •

edited

Loading

coderabbitai Bot commented Jun 16, 2026 •

edited

Loading