Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 60 additions & 9 deletions packages/envd/internal/services/process/handler/handler.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,13 @@
cmd *exec.Cmd
tty *os.File

// processGroup is true when the command was started in its own process
// group (Setpgid). When set, signals are delivered to the whole group so
// the command's child processes are terminated together with the leader.
// PTY processes are excluded — the pty package already puts them in their
// own session and we leave their signal handling untouched.
processGroup bool

cancel context.CancelFunc

outCtx context.Context //nolint:containedctx // todo: refactor so this can be removed
Expand Down Expand Up @@ -134,6 +141,25 @@
}
applyCgroupFD(cmd.SysProcAttr, cgroupFD, ok)

// Non-PTY commands run in their own process group so that SendSignal can
// reach the command's entire process tree (the leader plus any children it
// spawned), not just the single process envd manages. envd's per-process
// handle could not otherwise terminate those children, leaving them running
// — and consuming resources — after the command was killed.
// PTY processes are left alone: the pty package starts them in their own
// session already, and we keep their existing signal behavior.
isPTY := req.GetPty() != nil
if !isPTY {
cmd.SysProcAttr.Setpgid = true

// exec.CommandContext kills only the leader when the context (e.g. the
// command timeout) fires. Override the canceller to signal the whole
// process group instead, matching SendSignal's behavior.
cmd.Cancel = func() error {
return signalProcessGroup(cmd, syscall.SIGKILL)
}
}

resolvedPath, err := permissions.ExpandAndResolve(req.GetProcess().GetCwd(), user, defaults.Workdir)
if err != nil {
return nil, connect.NewError(connect.CodeInvalidArgument, err)
Expand Down Expand Up @@ -178,15 +204,16 @@
outCtx, outCancel := context.WithCancel(ctx)

h := &Handler{
Config: req.GetProcess(),
cmd: cmd,
Tag: req.Tag,
DataEvent: outMultiplex,
cancel: cancel,
outCtx: outCtx,
outCancel: outCancel,
EndEvent: NewMultiplexedChannel[rpc.ProcessEvent_End](0),
logger: logger,
Config: req.GetProcess(),
cmd: cmd,
Tag: req.Tag,
DataEvent: outMultiplex,
cancel: cancel,
outCtx: outCtx,
outCancel: outCancel,
EndEvent: NewMultiplexedChannel[rpc.ProcessEvent_End](0),
logger: logger,
processGroup: !isPTY,
}

if req.GetPty() != nil {
Expand Down Expand Up @@ -358,9 +385,33 @@
p.outCancel()
}

// For non-PTY commands the signal is delivered to the whole process group so
// the command's children are terminated together with the leader. PTY
// processes keep the original single-process behavior.
if p.processGroup {
return signalProcessGroup(p.cmd, signal)
}

return p.cmd.Process.Signal(signal)
}

// signalProcessGroup delivers signal to every process in cmd's process group.
// The command is started with Setpgid, so the process group id equals the
// leader's pid; signaling the negative pid targets the entire group. An already
// gone group (ESRCH) is treated as success — there is nothing left to kill.
func signalProcessGroup(cmd *exec.Cmd, signal syscall.Signal) error {
if cmd.Process == nil {
return errors.New("process not started")
}

err := syscall.Kill(-cmd.Process.Pid, signal)
if err != nil && !errors.Is(err, syscall.ESRCH) {
return err
}

return nil
}

Check failure on line 414 in packages/envd/internal/services/process/handler/handler.go

View check run for this annotation

Claude / Claude Code Review

SendSignal can target a recycled PID after the process is reaped

After this PR, `SendSignal` on a non-PTY handler bypasses Go's `ErrProcessDone` guard: `signalProcessGroup` calls `syscall.Kill(-cmd.Process.Pid, signal)` directly. There is a window after `cmd.Wait()` reaps the leader (freeing its PID) but before the deferred `s.processes.Delete(pid)` runs in `start.go:184-188` — a stale `SendSignal` arriving in that window can target a recycled PID and silently kill an unrelated command's whole process group. Fix: gate `signalProcessGroup` on `cmd.ProcessState
Comment on lines +401 to +414
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 After this PR, SendSignal on a non-PTY handler bypasses Go's ErrProcessDone guard: signalProcessGroup calls syscall.Kill(-cmd.Process.Pid, signal) directly. There is a window after cmd.Wait() reaps the leader (freeing its PID) but before the deferred s.processes.Delete(pid) runs in start.go:184-188 — a stale SendSignal arriving in that window can target a recycled PID and silently kill an unrelated command's whole process group. Fix: gate signalProcessGroup on cmd.ProcessState != nil (or a handler-level atomic flag set around cmd.Wait()).

Extended reasoning...

What the bug is

Pre-PR, SendSignal delivered the signal via p.cmd.Process.Signal(signal). Go's os.Process keeps a done atomic (set inside Wait() before/after Wait4), and pidSignal (/usr/local/go/src/os/exec_unix.go) returns os.ErrProcessDone once the process has been reaped. That guard makes signaling a reaped leader a no-op — important because the kernel is free to reuse a reaped PID immediately.

This PR introduces signalProcessGroup:

func signalProcessGroup(cmd *exec.Cmd, signal syscall.Signal) error {
    if cmd.Process == nil {
        return errors.New("process not started")
    }
    err := syscall.Kill(-cmd.Process.Pid, signal)
    if err != nil && !errors.Is(err, syscall.ESRCH) {
        return err
    }
    return nil
}

It bypasses Go's done guard entirely and issues a raw kill(-pid, sig) on whatever PID cmd.Process.Pid still records — which is exactly the PID that may have just been reassigned by the kernel.

The race window

start.go:184-188 reaps and unregisters like this:

go func() {
    defer s.processes.Delete(pid)
    proc.Wait()
}()

handler.Wait() then does, in order: <-p.outCtx.Done()p.cmd.Wait() (leader is reaped here — kernel can reuse the PID from this point) → p.tty.Close() → build endEvent → p.EndEvent.Source <- event (buffer-0 multiplex channel — blocks until the handleStart consumer reads) → log → p.cancel(). Only after all of that returns does s.processes.Delete(pid) run.

During that window:

  1. The handler is still discoverable via PID and via tag in getProcess (service.go:47).
  2. p.cmd.Process.Pid still holds the now-free leader PID.
  3. A new non-PTY command started in this PR sets SysProcAttr.Setpgid = true (handler.go:151), so if its leader gets the recycled PID it becomes a process-group leader with pgid == pid.

A stale SendSignal(SIGKILL) (client retry, tag lookup, UI race) arriving against the old handler in that window calls signalProcessGroupkill(-recycled_pid, SIGKILL), which lands on the brand-new, unrelated command's entire process group.

Step-by-step proof

  1. Client starts command A (non-PTY). Leader gets PID 1234. Setpgid=true makes it pgid=1234.
  2. Command A finishes. Inside handler.Wait(), p.cmd.Wait() returns — kernel marks PID 1234 reusable. Handler is still in s.processes because the defer s.processes.Delete(pid) hasn't fired yet.
  3. The buffer-0 send p.EndEvent.Source <- event blocks: the SDK consumer is slow / coalescing reads. Window stretches into ms+ territory.
  4. Concurrently, the same envd starts command B (non-PTY, also Setpgid=true). The kernel reassigns PID 1234 to B; B becomes pgid=1234 leader.
  5. A stale SendSignal(SIGKILL) for command A arrives — getProcess(pid=1234) (or by tag) still returns A's handler.
  6. A.SendSignal calls signalProcessGroup(A.cmd, SIGKILL)syscall.Kill(-1234, SIGKILL) → entire process group 1234 (= command B's leader and all its children) is killed.

Pre-PR step 6 would have returned ErrProcessDone and made no syscall.

Why existing code doesn't prevent it

  • The if cmd.Process == nil check only covers "not started yet"; it does nothing about "already reaped".
  • The errors.Is(err, syscall.ESRCH) swallow makes things worse, not better: it only triggers when the group is truly gone — which is exactly not the case when the PID has been recycled.
  • The PTY branch still goes through p.cmd.Process.Signal and retains Go's guard, so only non-PTY commands regress.
  • Linux's large pid_max doesn't save us: PIDs are recycled FIFO within the in-use set, and a sandbox that churns short-lived commands can wrap quickly (much faster in PID namespaces with smaller ranges).

Impact

A stale signal can silently kill a different command's whole process tree — exactly the blast radius this PR was supposed to introduce for legitimate kills, now applied to the wrong target. The error returned to the caller is nil, so the caller has no signal that anything went wrong; the victim command just dies. Likelihood is low (small window × PID reuse), but the failure mode is silent cross-command termination, which is hard to diagnose after the fact.

Fix

Restore the equivalent of ErrProcessDone. Simplest:

func signalProcessGroup(cmd *exec.Cmd, signal syscall.Signal) error {
    if cmd.Process == nil {
        return errors.New("process not started")
    }
    if cmd.ProcessState != nil {
        return os.ErrProcessDone
    }
    err := syscall.Kill(-cmd.Process.Pid, signal)
    ...
}

or track a handler-level atomic.Bool set immediately around p.cmd.Wait() and consult it in SendSignal before falling through. The cmd.Cancel callback (also new in this PR) has the same hazard and should use the same guard.

func (p *Handler) ResizeTty(size *pty.Winsize) error {
if p.tty == nil {
return errors.New("tty not assigned to process")
Expand Down
111 changes: 111 additions & 0 deletions packages/envd/internal/services/process/start_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@
"net/http/httptest"
"os"
"os/user"
"slices"
"strconv"
"strings"
"syscall"
"testing"
"time"

Expand Down Expand Up @@ -84,6 +88,113 @@
assert.NotNil(t, events[len(events)-1].GetEnd(), "last event should be End")
}

// TestSendSignal_KillsProcessTree verifies that killing a command also
// terminates the child processes it spawned. envd's per-process handle only
// signaled the leader, so children kept running after a kill; non-PTY commands
// now run in their own process group and SIGKILL reaches the whole tree.
func TestSendSignal_KillsProcessTree(t *testing.T) {
t.Parallel()

client, cleanup := newTestService(t)
defer cleanup()

ctx, cancel := context.WithTimeout(t.Context(), 20*time.Second)
defer cancel()

// The leader spawns two long-lived children and waits on them.
stream, err := client.Start(ctx, connect.NewRequest(&rpc.StartRequest{
Process: &rpc.ProcessConfig{
Cmd: "/bin/sh",
Args: []string{"-c", "sleep 120 & sleep 120 & wait"},
},
}))
require.NoError(t, err)

// Read the start event to capture the leader pid.
require.True(t, stream.Receive(), "expected a start event")
pid := stream.Msg().GetEvent().GetStart().GetPid()
require.NotZero(t, pid)

// Wait for the children to come up, then capture their pids.
var childPids []int
require.Eventually(t, func() bool {
childPids = childrenOf(t, int(pid))

return len(childPids) == 2
}, 5*time.Second, 50*time.Millisecond, "expected leader to spawn two children")

// Kill the command — this should take down the whole process group.
_, err = client.SendSignal(ctx, connect.NewRequest(&rpc.SendSignalRequest{
Process: &rpc.ProcessSelector{
Selector: &rpc.ProcessSelector_Pid{Pid: pid},
},
Signal: rpc.Signal_SIGNAL_SIGKILL,
}))
require.NoError(t, err)

// Drain the stream so the leader is reaped.
for stream.Receive() {
}
_ = stream.Close()

// The leader and every child must be gone.
require.Eventually(t, func() bool {
return !slices.ContainsFunc(childPids, processAlive)
}, 5*time.Second, 50*time.Millisecond, "child processes should be killed with the leader")
}

// childrenOf returns the pids whose parent is ppid, read from /proc.
func childrenOf(t *testing.T, ppid int) []int {
t.Helper()

entries, err := os.ReadDir("/proc")
require.NoError(t, err)

var children []int
for _, entry := range entries {
if !entry.IsDir() {
continue
}

pid, err := strconv.Atoi(entry.Name())
if err != nil {
continue
}

data, err := os.ReadFile("/proc/" + entry.Name() + "/stat")
if err != nil {
// The process may have exited between listing and reading.
continue
}

// /proc/<pid>/stat fields: pid (comm) state ppid ...
// comm can contain spaces/parens, so parse after the closing paren.
stat := string(data)
idx := strings.LastIndex(stat, ")")
if idx == -1 {
continue
}

fields := strings.Fields(stat[idx+1:])
// fields[0]=state, fields[1]=ppid
if len(fields) < 2 {
continue
}

if parsed, err := strconv.Atoi(fields[1]); err == nil && parsed == ppid {
children = append(children, pid)
}
}

return children
}

// processAlive reports whether the given pid still exists.
func processAlive(pid int) bool {
// Signal 0 performs error checking without actually sending a signal.
return syscall.Kill(pid, 0) == nil
}

Check warning on line 197 in packages/envd/internal/services/process/start_test.go

View check run for this annotation

Claude / Claude Code Review

processAlive treats zombies as alive — test can flake in init-less containers

processAlive uses syscall.Kill(pid, 0) which succeeds for zombies — kill(2) explicitly notes the existence check passes "as long as the process still exists" (i.e. has a /proc entry, even if unreaped). After the test SIGKILLs the process group, the two sleep children are orphaned to PID 1 / nearest subreaper; if the test ever runs in a container where PID 1 is the Go test binary itself (no tini/dumb-init), the orphans stay zombies, processAlive returns true forever, and the 5s Eventually times o
Comment on lines +193 to +197
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 processAlive uses syscall.Kill(pid, 0) which succeeds for zombies — kill(2) explicitly notes the existence check passes "as long as the process still exists" (i.e. has a /proc entry, even if unreaped). After the test SIGKILLs the process group, the two sleep children are orphaned to PID 1 / nearest subreaper; if the test ever runs in a container where PID 1 is the Go test binary itself (no tini/dumb-init), the orphans stay zombies, processAlive returns true forever, and the 5s Eventually times out. Test-only nit — childrenOf already parses /proc//stat, so reading fields[0] and treating '''Z''' as not-alive is a one-line fix.

Extended reasoning...

What the bug is

processAlive at start_test.go:194-197 uses syscall.Kill(pid, 0) to probe process existence. Per kill(2): "If sig is 0, then no signal is sent, but existence and permission checks are still performed; this can be used to check for the existence of a process ID..." — and a zombie is still an existing process (it has a /proc entry until the parent reaps it). Only when the entry is gone does kill(pid, 0) return ESRCH.

Why the test can flake

TestSendSignal_KillsProcessTree runs /bin/sh -c 'sleep 120 & sleep 120 & wait'. The shell (the leader) is the parent of both sleeps. The test then SIGKILLs the process group, which kills shell + both sleeps simultaneously. envd reaps the shell via cmd.Wait() in Handler.Wait(). But the orphaned sleeps get reparented (by the kernel, at the moment their parent dies) to PID 1 or the nearest PR_SET_CHILD_SUBREAPER. envd does not set itself as a subreaper, so reaping is up to whatever PID 1 happens to be in the test environment.

Why the refutation is partly right but not a reason to skip

The refutation is correct that in the environments this code is normally tested in (dev host, go test as a child of a shell, CI with docker run --init, systemd as PID 1), reaping happens within microseconds and the 5s Eventually window is enormous. The test will pass.

The issue is what happens in the one environment where it doesn't: docker run golang go test ./... (or any image built with ENTRYPOINT ["go","test",...]) — here the Go test binary itself is PID 1, and Go's runtime does not perform automatic SIGCHLD reaping for non-direct children. The zombies persist for the lifetime of the test process, processAlive keeps returning true, and the require.Eventually assertion times out. This is exactly the kind of environment infra-style projects sometimes adopt for sandboxed CI, and it's reasonable to harden against it preemptively — especially since the fix is essentially free.

Step-by-step proof

  1. Test starts leader sh -c 'sleep 120 & sleep 120 & wait' (pid L). Kernel creates children C1, C2 with ppid=L.
  2. childrenOf(L) returns [C1, C2].
  3. Client sends SIGKILL → envd calls syscall.Kill(-L, SIGKILL). Kernel marks L, C1, C2 as terminated.
  4. envd's cmd.Wait() reaps L → L's /proc entry disappears. C1, C2 still have entries (state = Z), now reparented to whatever process is PID 1 in this namespace.
  5. If PID 1 = the Go test binary (no reaper), no wait4() is ever called for C1, C2.
  6. slices.ContainsFunc(childPids, processAlive): syscall.Kill(C1, 0) → returns nil (zombie still exists, permission check passes since same uid), so processAlive(C1) == true.
  7. The Eventually predicate returns false for 5s, then require.Eventually fails: "child processes should be killed with the leader".

How to fix

childrenOf already parses /proc/<pid>/stat and even extracts fields[0] (the state) but discards it. Reuse it in processAlive:

func processAlive(pid int) bool {
    data, err := os.ReadFile("/proc/" + strconv.Itoa(pid) + "/stat")
    if err != nil {
        return false // gone from /proc → definitely not alive
    }
    idx := strings.LastIndex(string(data), ")")
    if idx == -1 {
        return false
    }
    fields := strings.Fields(string(data)[idx+1:])
    return len(fields) > 0 && fields[0] != "Z"
}

This makes the test robust regardless of who PID 1 is.

// TestStart_ClientDisconnectMidStream verifies that when a client
// cancels mid-stream, the handler returns without racing. This is
// the scenario that caused the nil-pointer panic in production:
Expand Down
2 changes: 1 addition & 1 deletion packages/envd/pkg/version.go
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
package pkg

const Version = "0.6.1"
const Version = "0.6.2"
Loading