Skip to content

fix: spot node termination case, failed workflow marked as succedded #242

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Shivam-nagar23 merged 7 commits into from
Apr 29, 2025
Merged

fix: spot node termination case, failed workflow marked as succedded #242

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
2dddcb3
failed update event mark as succedded
Shivam-nagar23 Apr 28, 2025
1ec688e
temp checks
Shivam-nagar23 Apr 28, 2025
bd6e99c
Revert "temp checks"
Shivam-nagar23 Apr 28, 2025
f5948d0
pod deleted
Shivam-nagar23 Apr 28, 2025
1147b13
ci failed
Shivam-nagar23 Apr 28, 2025
d757768
comments
Shivam-nagar23 Apr 29, 2025
54a9a81
log added
Shivam-nagar23 Apr 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion kubewatch/pkg/informer/cluster/systemExec/helper.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,13 @@ func (impl *InformerImpl) inferFailedReason(eventType string, pod *coreV1.Pod) (
// were SIGKILL'd. The executor may have had to forcefully terminate the sidecar (kill -9),
// resulting in a 137 exit code (which we had ignored earlier). If failMessages is empty, it
// indicates that this is the case and we return Success instead of Failure.
return v1alpha1.NodeSucceeded, ""

// Update diff from argo workflow here as we only have one main container.
// Handling this for case of spot interruption where containers are in running state (no termination state found), in that case
// it was marking it successful, doing this as it will be skipped at upper level, and delete event will handle it.
Comment thread
Shivam-nagar23 marked this conversation as resolved.
// ticket - you can find debug logs/details here - https://github.com/devtron-labs/sprint-tasks/issues/2092
impl.logger.Infow("Pod phase was Failed but no container had terminated state, marking it as failed now", "podName", pod.Name, "status", pod.Status)
return v1alpha1.NodeFailed, ""
}

func getFailedReasonFromPodConditions(conditions []coreV1.PodCondition) string {
Expand Down
Loading