Skip to content

[RayJob]Refactor the sidecar container failure detection logic and extract common functions#4976

Closed
zhuyanhuazhuyanhua wants to merge 0 commit into
ray-project:masterfrom
zhuyanhuazhuyanhua:master
Closed

[RayJob]Refactor the sidecar container failure detection logic and extract common functions#4976
zhuyanhuazhuyanhua wants to merge 0 commit into
ray-project:masterfrom
zhuyanhuazhuyanhua:master

Conversation

@zhuyanhuazhuyanhua

Copy link
Copy Markdown

Why are these changes needed?

In SidecarMode, the ray-job-submitter runs as a container inside the head Pod with the head Pod's RestartPolicy set to Never. If the submitter exits non-zero before the RayCluster reaches Ready (e.g. the job fails while workers are still being scheduled), the terminated submitter container keeps the head Pod Ready=False, so RayCluster.Status.State never becomes Ready.

The RayJob Initializing -> Running transition is gated on RayCluster.Status.State == Ready, and the only submitter-failure detector (checkSubmitterAndUpdateStatusIfNeeded) runs only in the Running state. As a result the RayJob is wedged in Initializing forever: it never transitions to Failed, shutdownAfterJobFinishes never runs, and the autoscaler keeps retrying workers — even though the submitter failure is already observable.

This PR detects a terminal submitter failure during Initializing (SidecarMode) and fails fast. The feature-gate-aware detection (SidecarSubmitterRestart) is extracted into checkSidecarSubmitterFailedAndUpdateStatus and shared by the Initializing and Running states; the Initializing call skips when the head Pod does not exist yet.

Related issue number

Closes #4945 #4946

Checks

  • I've made sure the tests are passing.

Testing:

  • go build ./controllers/ray/..., go vet ./controllers/ray/, gofumpt -l — clean
  • New unit test TestCheckSidecarSubmitterFailedAndUpdateStatus: submitter Terminated exit != 0 → Failed/SubmissionFailed; same with JobStatus already FailedAppFailed; exit 0 → no-op; still Running → no-op; SidecarSubmitterRestart enabled → routes to the restart-count check
  • go test ./controllers/ray/ -run 'TestCheckSidecarSubmitterFailedAndUpdateStatus|TestCheckIsRestartCountExceeded' — pass

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] SidecarMode RayJob stuck in Initializing forever when the submitter fails before the RayCluster is Ready

1 participant