test: tolerate non-zero Triton exit on Jetson graceful shutdown by Vinya567 · Pull Request #8881 · triton-inference-server/server

Vinya567 · 2026-07-13T20:47:15Z

What does the PR do?

Fix L0_backend_python/lifecycle failures on IGX-Orin when pytest already passed.

Root cause

Lifecycle pytest passed; the subtest failed during server shutdown.
On IGX-Orin, seven Python backend stubs can exceed Triton's default 30s exit timeout.
Server logged Exit timeout expired, exited non-zero, and wait $SERVER_PID failed the script under set -e.
Incomplete stub teardown could leave orphaned triton_python_backend_shm_region_* files in /dev/shm.

Fix

Jetson: set --exit-timeout-secs=${SERVER_TIMEOUT} and stub-timeout-seconds=${SERVER_TIMEOUT} in qa/L0_backend_python/lifecycle/test.sh.
Jetson: use wait $SERVER_PID || true in kill_server (qa/common/util.sh).
Jetson: snapshot shm before server start, check page count first, then cleanup only new regions after the assertion (qa/common/util.sh, lifecycle test).
Runtime follow-up: fix: ensure Python backend shm cleanup on forced shutdown python_backend#450

Checklist

PR title reflects the change and is of format <type>: <description>
Changes are described in the pull request.
Related issues are referenced.
Populated github labels field
Added test plan and verified test passes.
Verified that the PR passes existing CI.
Verified copyright is correct on all changed files.
Added succinct git squash message before merging
All template sections are filled out.

Commit Type:

Related PRs:

python_backend: fix: ensure Python backend shm cleanup on forced shutdown python_backend#450

Where should the reviewer start?

qa/L0_backend_python/lifecycle/test.sh
qa/common/util.sh

Test plan:

Local: qa/L0_backend_python/lifecycle/test.sh with TEST_JETSON=1.
GitLab Orin: job 366443271 passed at commit c1c44b50 (Lifecycle test PASSED, shm count unchanged).
Latest commit 3bbc4e05 (run shm check before cleanup) still needs Orin re-run.

Caveats:

Server QA changes are in this PR.
Python backend runtime changes are in PR Apply CustomInstance API wrapper #450.

Related Issues:

On IGX-Orin, kill_server wait can return non-zero after lifecycle pytest passes. Do not fail QA scripts solely for that shutdown code.

After tolerating non-zero Triton exit on Jetson, lifecycle could still fail when Python stub teardown left new triton_python_backend_shm_region files in /dev/shm. Snapshot regions before server start and remove only newly created regions after shutdown in lifecycle and kill_server.

yinggeh

What is the root cause of "on Jetson, Triton may exit non-zero during graceful shutdown after Python backend stub teardown even when tests passed"?

Move triton python shm snapshot helpers into util.sh, call cleanup_triton_python_shm_since_snapshot from kill_server, and invoke snapshot/cleanup only on Jetson paths in lifecycle.

Vinya567 · 2026-07-15T06:10:02Z

What is the root cause of "on Jetson, Triton may exit non-zero during graceful shutdown after Python backend stub teardown even when tests passed"?

On Orin, pytest passes but wait $SERVER_PID returns non-zero after shutdown, and sometimes a Python backend shm file is left in /dev/shm. This PR handles both in the QA harness on Jetson. Fixed in e2c7891.

yinggeh · 2026-07-16T15:32:36Z

On Orin, pytest passes but wait $SERVER_PID returns non-zero after shutdown, and sometimes a Python backend shm file is left in /dev/shm.

I understand. But what's the root cause for this buggy behavior? Why it's hardware specific?

Vinya567 · 2026-07-16T15:44:50Z

On Orin, pytest passes but wait $SERVER_PID returns non-zero after shutdown, and sometimes a Python backend shm file is left in /dev/shm.

I understand. But what's the root cause for this buggy behavior? Why it's hardware specific?

Good point , we only traced the CI symptom so far, not the full backend root cause. On IGX-Orin, pytest passes but shutdown sometimes exits non-zero before Python stub teardown finishes, which can leave shm files behind; we haven't reproduced that on x86. This PR is the QA harness fix for TRI-1330; I can dig into the Python backend shutdown path as well if you'd prefer that before merge.

yinggeh · 2026-07-16T15:49:19Z

On Orin, pytest passes but wait $SERVER_PID returns non-zero after shutdown, and sometimes a Python backend shm file is left in /dev/shm.

I understand. But what's the root cause for this buggy behavior? Why it's hardware specific?

Good point , we only traced the CI symptom so far, not the full backend root cause. On IGX-Orin, pytest passes but shutdown sometimes exits non-zero before Python stub teardown finishes, which can leave shm files behind; we haven't reproduced that on x86. This PR is the QA harness fix for TRI-1330; I can dig into the Python backend shutdown path as well if you'd prefer that before merge.

Please do. The goal of fixing tests is not simply to work around the issue, but to identify the root cause and file an NVBUG with the appropriate team if necessary.

Vinya567 · 2026-07-16T15:53:39Z

On Orin, pytest passes but wait $SERVER_PID returns non-zero after shutdown, and sometimes a Python backend shm file is left in /dev/shm.

I understand. But what's the root cause for this buggy behavior? Why it's hardware specific?

Good point , we only traced the CI symptom so far, not the full backend root cause. On IGX-Orin, pytest passes but shutdown sometimes exits non-zero before Python stub teardown finishes, which can leave shm files behind; we haven't reproduced that on x86. This PR is the QA harness fix for TRI-1330; I can dig into the Python backend shutdown path as well if you'd prefer that before merge.

Please do. The goal of fixing tests is not simply to work around the issue, but to identify the root cause and file an NVBUG with the appropriate team if necessary.

Makes sense , I'll dig into the Python backend shutdown path on Orin to find the root cause and file an NVBUG if it's a backend issue. I'll update here once I have findings.

Lifecycle loads seven Python models; Orin teardown can exceed the default 30s exit timeout and fail shutdown checks. Match L0_infer by using SERVER_TIMEOUT and stub-timeout-seconds on TEST_JETSON=1.

Vinya567 · 2026-07-17T23:15:15Z

On Orin, pytest passes but wait $SERVER_PID returns non-zero after shutdown, and sometimes a Python backend shm file is left in /dev/shm.

I understand. But what's the root cause for this buggy behavior? Why it's hardware specific?

Good point , we only traced the CI symptom so far, not the full backend root cause. On IGX-Orin, pytest passes but shutdown sometimes exits non-zero before Python stub teardown finishes, which can leave shm files behind; we haven't reproduced that on x86. This PR is the QA harness fix for TRI-1330; I can dig into the Python backend shutdown path as well if you'd prefer that before merge.

Please do. The goal of fixing tests is not simply to work around the issue, but to identify the root cause and file an NVBUG with the appropriate team if necessary.

Root cause: lifecycle pytest passed, but Triton’s 30s shutdown timeout was too short for 7 Python stubs on Orin. Server exited with error, Python backend left an shm file behind, and the test failed on cleanup. We increased shutdown timeouts for Jetson, added harness cleanup as a safety net, and have a python_backend fix for proper stub teardown/shm cleanup. Verified green on Orin (job 366443271).

yinggeh · 2026-07-18T00:20:22Z

On Orin, pytest passes but wait $SERVER_PID returns non-zero after shutdown, and sometimes a Python backend shm file is left in /dev/shm.

I understand. But what's the root cause for this buggy behavior? Why it's hardware specific?

Good point , we only traced the CI symptom so far, not the full backend root cause. On IGX-Orin, pytest passes but shutdown sometimes exits non-zero before Python stub teardown finishes, which can leave shm files behind; we haven't reproduced that on x86. This PR is the QA harness fix for TRI-1330; I can dig into the Python backend shutdown path as well if you'd prefer that before merge.

Please do. The goal of fixing tests is not simply to work around the issue, but to identify the root cause and file an NVBUG with the appropriate team if necessary.

Root cause: lifecycle pytest passed, but Triton’s 30s shutdown timeout was too short for 7 Python stubs on Orin. Server exited with error, Python backend left an shm file behind, and the test failed on cleanup. We increased shutdown timeouts for Jetson, added harness cleanup as a safety net, and have a python_backend fix for proper stub teardown/shm cleanup. Verified green on Orin (job 366443271).

Great. Make sure you linked the python_backend PR in the description (following the template format).

yinggeh · 2026-07-18T00:25:39Z

          sed -i "s/TYPE_FP32/TYPE_UINT32/g" config.pbtxt)

 prev_num_pages=`get_shm_pages`
+if [ "${TEST_JETSON}" == "1" ]; then


With snapshot_triton_python_shm and cleanup_triton_python_shm_since_snapshot, the test won't detect any stale shm after server exits, which could be a problem.

you're right that cleanup in kill_server runs before the shm count check, so it can mask leaks. I'll move cleanup to after the assertion (or remove it from kill_server and rely on the exit-timeout fix + python_backend PR). Will also link the python_backend PR in the description.

Fixed in 3bbc4e0: the shm page-count assertion runs before cleanup_triton_python_shm_since_snapshot, and kill_server no longer does preemptive cleanup. Orin re-run for that commit is still pending.

Also if it helps the test we should make it universal to all platforms.

I looked at ShmLeakDetector vs the shell helpers.
They’re not quite the same: the detector is skipped on Jetson, and it checks free memory inside existing pools during pytest. The shell snapshot/cleanup handles leftover region files after server exit, which is what get_shm_pages() in test.sh was already watching.

Given that overlap, I’ll re-run Orin without snapshot/cleanup and drop them if it stays green. If we still need them, I’ll remove the TEST_JETSON guard and make them universal.

Run cleanup_triton_python_shm_since_snapshot only after the shm page count assertion so lifecycle still detects stale regions. Keep wait || true in kill_server without preemptive cleanup there.

yinggeh · 2026-07-18T00:54:23Z


+TRITON_PYTHON_SHM_SNAPSHOT=""
+
+function snapshot_triton_python_shm () {


Check

server/qa/L0_backend_python/lifecycle/lifecycle_test.py

Lines 66 to 68 in c1b7cb7

class LifecycleTest(unittest.TestCase):

def setUp(self):

self._shm_leak_detector = shm_util.ShmLeakDetector()

and find out if the shm functions are redundant.

Keep the Jetson shutdown timeout and wait || true fix only. Remove snapshot/cleanup helpers to verify Orin passes without them.

Vinya567 added 2 commits July 13, 2026 20:46

test: tolerate non-zero Triton exit on Jetson graceful shutdown

51b0f3f

On IGX-Orin, kill_server wait can return non-zero after lifecycle pytest passes. Do not fail QA scripts solely for that shutdown code.

Vinya567 self-assigned this Jul 15, 2026

Vinya567 requested review from mc-nv, whoisj and yinggeh July 15, 2026 01:38

yinggeh reviewed Jul 15, 2026

View reviewed changes

Comment thread qa/L0_backend_python/common.sh Outdated

Comment thread qa/common/util.sh

test: dedupe Jetson shm cleanup per review feedback

e2c7891

Move triton python shm snapshot helpers into util.sh, call cleanup_triton_python_shm_since_snapshot from kill_server, and invoke snapshot/cleanup only on Jetson paths in lifecycle.

Vinya567 requested a review from yinggeh July 15, 2026 06:12

mc-nv reviewed Jul 16, 2026

View reviewed changes

Comment thread qa/common/util.sh Outdated

test: extend Jetson lifecycle shutdown timeout to 120s

c1c44b5

Lifecycle loads seven Python models; Orin teardown can exceed the default 30s exit timeout and fail shutdown checks. Match L0_infer by using SERVER_TIMEOUT and stub-timeout-seconds on TEST_JETSON=1.

yinggeh reviewed Jul 18, 2026

View reviewed changes

test: check shm leaks before Jetson cleanup runs

3bbc4e0

Run cleanup_triton_python_shm_since_snapshot only after the shm page count assertion so lifecycle still detects stale regions. Keep wait || true in kill_server without preemptive cleanup there.

Vinya567 mentioned this pull request Jul 18, 2026

fix: ensure Python backend shm cleanup on forced shutdown triton-inference-server/python_backend#450

Open

3 tasks

yinggeh reviewed Jul 18, 2026

View reviewed changes

test: drop Jetson shm snapshot/cleanup helpers

2734eb5

Keep the Jetson shutdown timeout and wait || true fix only. Remove snapshot/cleanup helpers to verify Orin passes without them.


		TRITON_PYTHON_SHM_SNAPSHOT=""

		function snapshot_triton_python_shm () {

	class LifecycleTest(unittest.TestCase):
	def setUp(self):
	self._shm_leak_detector = shm_util.ShmLeakDetector()

Uh oh!

Conversation

Vinya567 commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does the PR do?

Root cause

Fix

Checklist

Commit Type:

Related PRs:

Where should the reviewer start?

Test plan:

Caveats:

Related Issues:

Uh oh!

yinggeh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Vinya567 commented Jul 15, 2026

Uh oh!

yinggeh commented Jul 16, 2026

Uh oh!

Uh oh!

Vinya567 commented Jul 16, 2026

Uh oh!

yinggeh commented Jul 16, 2026

Uh oh!

Vinya567 commented Jul 16, 2026

Uh oh!

Vinya567 commented Jul 17, 2026

Uh oh!

yinggeh commented Jul 18, 2026

Uh oh!

yinggeh Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Vinya567 Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

Vinya567 Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

yinggeh Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

Vinya567 Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

yinggeh Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

Vinya567 commented Jul 13, 2026 •

edited

Loading

yinggeh Jul 18, 2026 •

edited

Loading