Skip to content

Env-server pool hard shutdown can orphan worker subprocesses #1992

Description

@morluto

Problem

EnvServerPool._shutdown() first calls terminate() for each worker, waits up to ten seconds, and then calls kill() if the worker remains alive. Both signals target only the worker PID.

The normal SIGTERM path lets a responsive worker unwind its async contexts and clean up runtimes. The hard-kill path cannot do that. A subprocess started below the worker can therefore remain alive after the worker has exited.

Reproduction

A worker that ignored SIGTERM was given a live subprocess in a separate process group and passed through the current _shutdown() implementation. After escalation:

  • the worker exited with -SIGKILL;
  • the subprocess was still alive.

Design constraint

Putting the worker and every child in one process group would break the per-rollout cleanup contract: SubprocessRuntime needs a distinct group so it can terminate one harness tree without affecting other concurrent rollouts.

Expected outcome

  • graceful SIGTERM cleanup remains the first shutdown step;
  • every pool worker has an OS ownership boundary that includes its framework-created subprocess groups;
  • hard shutdown reaps remaining members even if the worker exits during the grace period;
  • cleanup does not rely on stale descendant PIDs that may be reused.

Provider-side sandbox and tunnel cleanup requires a broker-visible resource registry rather than an OS signal and is tracked separately in #1994.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions