fix(serve): verify process identity and bound stop termination (EAI-7410)#117
Open
michaelroy-amd wants to merge 1 commit into
Open
fix(serve): verify process identity and bound stop termination (EAI-7410)#117michaelroy-amd wants to merge 1 commit into
michaelroy-amd wants to merge 1 commit into
Conversation
…410) Stopping a served model previously signalled a PID read from a state file with no identity check and reported success without confirming the process exited. A recycled PID could make `stop` SIGTERM an unrelated process tree, and `stopped`/`graceful` were claimed even when the process was still alive or already gone. Add rocm-core `proc_lifecycle`: a (PID + kernel start-time) identity that survives PID recycling, plus `terminate_verified` — verify identity, SIGTERM the scope, wait for actual exit up to a bounded grace, escalate to SIGKILL only under force, and report a truthful outcome. Tree scope snapshots and waits on every descendant so a lingering GPU worker is never reported as a graceful stop. Persist start-ticks in the vLLM/lemonade engine state and in the managed-service record, and verify identity in both the engine `stop_service` paths and the CLI managed-service stop before signalling. ComfyUI and the rocmd supervised-record stop paths share the same store-then-kill pattern and are left as tracked follow-ups. Signed-off-by: Michael Roy <michael.roy@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes EAI-7410: the serve stop path signalled a PID read from a state file with no identity check and reported success without confirming the process actually exited.
Two defects, shared by both engine adapters and the CLI managed-service layer:
stopread a persisted PID andSIGTERMed it (vLLM signalled the whole/procdescendant tree). Linux recycles PIDs, so a stale PID could belong to an unrelated process by stop time — killing an innocent process (and its subtree).SIGTERMwas merely delivered (ESRCH/already-gone counted as success). No bounded wait, no confirmed exit, noSIGKILLescalation — sostopped/gracefulwere claimed even while the process was still alive or already gone.What changed
New
rocm-core::proc_lifecycle:ProcessIdentity=(pid, kernel start-time)from/proc/<pid>/stat(field 22). A recycled PID has a different start-time, so the pair is a recycle-proof identity.identity_state→Matches/Recycled/Indeterminate/Gone(zombie-aware; refuses when a recorded identity can't be confirmed).terminate_verified(id, scope, grace, force)→ verify identity,SIGTERM, poll for actual exit up to a boundedgrace, escalate toSIGKILLonly underforce, and return a truthfulTerminationOutcome. Tree scope snapshots every descendant and waits for all to exit, so a lingering GPU worker is never reported as a graceful stop; forced escalation only targets survivors it can positively re-verify.Wiring:
start_ticks/server_start_ticksin engine state and verify identity instop_service(vLLMTree, lemonadeSingle); lemonade also refreshes the token at every state PID-swap.stop_internal_managed_serviceverifiesManagedServiceRecordidentities before signalling; the supervisor (launcher) and engine server carry their own tokens (supervisor_start_ticks/engine_start_ticks, the latter adopted inrefresh_from_engine_state). A stop claims"stopped"only when every recorded process is confirmed gone; otherwise the prior status stands and the existing liveness refresh reconciles it.signal_process_tree.Behavior degrades gracefully: legacy state files without a token, and non-Linux platforms, fall back to documented best-effort (unchanged from prior behavior).
Testing
cargo test --workspace→ 1459 passed, 0 failed (--test-threads=1).cargo clippy --workspace --all-targets -- -D warningsclean;cargo fmt --checkclean.identity_from_state(incl. legacy/no-token), and managed-layer stop (refuse recycled PID, verified kill, distinct supervisor/engine PIDs).Follow-ups (out of scope — separate subsystems, same pattern)
apps/rocm/src/comfyui.rsand therocmdsupervised-record stop path still use the store-then-kill pattern; recommend routing them throughterminate_verifiedin a follow-up.