Skip to content

fix(serve): verify process identity and bound stop termination (EAI-7410)#117

Open
michaelroy-amd wants to merge 1 commit into
mainfrom
michaelroy-amd/eai-7410-process-identity
Open

fix(serve): verify process identity and bound stop termination (EAI-7410)#117
michaelroy-amd wants to merge 1 commit into
mainfrom
michaelroy-amd/eai-7410-process-identity

Conversation

@michaelroy-amd

Copy link
Copy Markdown
Member

Summary

Fixes EAI-7410: the serve stop path signalled a PID read from a state file with no identity check and reported success without confirming the process actually exited.

Two defects, shared by both engine adapters and the CLI managed-service layer:

  1. PID reuse → wrong-process kill. stop read a persisted PID and SIGTERMed it (vLLM signalled the whole /proc descendant tree). Linux recycles PIDs, so a stale PID could belong to an unrelated process by stop time — killing an innocent process (and its subtree).
  2. Untruthful, unbounded termination. Success was reported when SIGTERM was merely delivered (ESRCH/already-gone counted as success). No bounded wait, no confirmed exit, no SIGKILL escalation — so stopped/graceful were claimed even while the process was still alive or already gone.

What changed

New rocm-core::proc_lifecycle:

  • ProcessIdentity = (pid, kernel start-time) from /proc/<pid>/stat (field 22). A recycled PID has a different start-time, so the pair is a recycle-proof identity.
  • identity_stateMatches / Recycled / Indeterminate / Gone (zombie-aware; refuses when a recorded identity can't be confirmed).
  • terminate_verified(id, scope, grace, force) → verify identity, SIGTERM, poll for actual exit up to a bounded grace, escalate to SIGKILL only under force, and return a truthful TerminationOutcome. Tree scope snapshots every descendant and waits for all to exit, so a lingering GPU worker is never reported as a graceful stop; forced escalation only targets survivors it can positively re-verify.

Wiring:

  • vLLM & lemonade persist start_ticks/server_start_ticks in engine state and verify identity in stop_service (vLLM Tree, lemonade Single); lemonade also refreshes the token at every state PID-swap.
  • CLI stop_internal_managed_service verifies ManagedServiceRecord identities before signalling; the supervisor (launcher) and engine server carry their own tokens (supervisor_start_ticks / engine_start_ticks, the latter adopted in refresh_from_engine_state). A stop claims "stopped" only when every recorded process is confirmed gone; otherwise the prior status stands and the existing liveness refresh reconciles it.
  • Removed the now-unused unverified signal_process_tree.

Behavior degrades gracefully: legacy state files without a token, and non-Linux platforms, fall back to documented best-effort (unchanged from prior behavior).

Testing

  • cargo test --workspace1459 passed, 0 failed (--test-threads=1).
  • cargo clippy --workspace --all-targets -- -D warnings clean; cargo fmt --check clean.
  • New focused lifecycle tests: identity match/recycle/gone/indeterminate, refuse-to-signal on mismatch, graceful/timed-out/forced, tree waits for descendants, forced tree kill of a SIGTERM-ignoring reparented descendant, engine identity_from_state (incl. legacy/no-token), and managed-layer stop (refuse recycled PID, verified kill, distinct supervisor/engine PIDs).
  • Reviewed across three rounds (reliability + security + OSS + dual fresh adversarial); all confirmed findings resolved.

Follow-ups (out of scope — separate subsystems, same pattern)

  • apps/rocm/src/comfyui.rs and the rocmd supervised-record stop path still use the store-then-kill pattern; recommend routing them through terminate_verified in a follow-up.

…410)

Stopping a served model previously signalled a PID read from a state file
with no identity check and reported success without confirming the process
exited. A recycled PID could make `stop` SIGTERM an unrelated process tree,
and `stopped`/`graceful` were claimed even when the process was still alive
or already gone.

Add rocm-core `proc_lifecycle`: a (PID + kernel start-time) identity that
survives PID recycling, plus `terminate_verified` — verify identity, SIGTERM
the scope, wait for actual exit up to a bounded grace, escalate to SIGKILL
only under force, and report a truthful outcome. Tree scope snapshots and
waits on every descendant so a lingering GPU worker is never reported as a
graceful stop. Persist start-ticks in the vLLM/lemonade engine state and in
the managed-service record, and verify identity in both the engine
`stop_service` paths and the CLI managed-service stop before signalling.

ComfyUI and the rocmd supervised-record stop paths share the same
store-then-kill pattern and are left as tracked follow-ups.

Signed-off-by: Michael Roy <michael.roy@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant