Skip to content

fix: Build get Stuck Indefinitely in case of K8s Driver When Node Termination Occurs #303

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Ash-exp merged 27 commits into from
Jul 10, 2025
Merged

fix: Build get Stuck Indefinitely in case of K8s Driver When Node Termination Occurs #303

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
a450ac3
fix: added liveness dailer for buildx k8s driver pods
Ash-exp Jun 18, 2025
f7040f0
fix: updated CommandExecutor signatures
Ash-exp Jun 18, 2025
0f23fa6
chore: refactoring for ent
Ash-exp Jun 18, 2025
0ce837b
fix: ctx handling for parallel exec
Ash-exp Jun 19, 2025
523e607
fix: error return statement
Ash-exp Jun 19, 2025
4d21f9d
chore: minor refactoring
Ash-exp Jun 19, 2025
c9c65a9
feat: retry build on pod termination
Ash-exp Jun 19, 2025
4f5f5d8
feat: retry build on pod termination optimised
Ash-exp Jun 19, 2025
121e245
feat: retry build on pod termination
Ash-exp Jun 19, 2025
d63af5c
fix: retry build on pod termination
Ash-exp Jun 19, 2025
59fed4c
fix: retry build on pod termination
Ash-exp Jun 19, 2025
a463f10
fix: retry build on pod termination
Ash-exp Jun 20, 2025
4c433a3
fix: retry build on pod termination
Ash-exp Jun 20, 2025
ee81260
fix: restart all builders before restart build
Ash-exp Jun 20, 2025
3b6ad3f
feat: added configurable retry for docker rebuild stage
Ash-exp Jun 23, 2025
f565cfa
chore: dependency updated to 'origin/fix/buildx-liveness-dailer'
Ash-exp Jun 23, 2025
b56d6b4
fix: logger issues
Ash-exp Jun 23, 2025
85cba69
chore: pretty logger added
Ash-exp Jun 24, 2025
16267b7
fix: context with cancel cause
Ash-exp Jun 24, 2025
d171d63
fix: error handling for build image stage
Ash-exp Jun 24, 2025
93cfce6
updated log message
Ash-exp Jun 24, 2025
e46c966
fix: re build stage failed for 1st attempt
Ash-exp Jun 24, 2025
138ff4a
added debug logs
Ash-exp Jun 24, 2025
d55c3f3
Merge remote-tracking branch 'origin/develop' into fix/buildx-livenes…
Ash-exp Jul 2, 2025
e93b081
chore: dependency updated to 'origin/fix/buildx-liveness-dailer'
Ash-exp Jul 2, 2025
b8a70bd
Merge remote-tracking branch 'origin/develop' into fix/buildx-livenes…
Ash-exp Jul 8, 2025
d0285b3
chore: dependency updated to 'origin/fix/buildx-liveness-dailer'
Ash-exp Jul 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion chart-sync/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ go 1.24.0

toolchain go1.24.3

replace github.com/devtron-labs/common-lib => github.com/devtron-labs/devtron-services/common-lib v0.0.0-20250703034313-6e26d7880667
replace github.com/devtron-labs/common-lib => github.com/devtron-labs/devtron-services/common-lib v0.0.0-20250708132702-b8a70bd2d48e

require (
github.com/caarlos0/env v3.5.0+incompatible
Expand Down
4 changes: 2 additions & 2 deletions chart-sync/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,8 @@ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/devtron-labs/devtron-services/common-lib v0.0.0-20250703034313-6e26d7880667 h1:mV+gq9St5soJPS7vww0cWTW6ayKxDVKWmJVkhbkrtjM=
github.com/devtron-labs/devtron-services/common-lib v0.0.0-20250703034313-6e26d7880667/go.mod h1:/Ciy9tD9OxZOWBDPIasM448H7uvSo4+ZJiExpfwBZpA=
github.com/devtron-labs/devtron-services/common-lib v0.0.0-20250708132702-b8a70bd2d48e h1:iJg144GwEMX7vUuNqIYBWbBniw5BvJiFw5ZXG8LuH3c=
github.com/devtron-labs/devtron-services/common-lib v0.0.0-20250708132702-b8a70bd2d48e/go.mod h1:/Ciy9tD9OxZOWBDPIasM448H7uvSo4+ZJiExpfwBZpA=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
github.com/distribution/distribution/v3 v3.0.0 h1:q4R8wemdRQDClzoNNStftB2ZAfqOiN6UX90KJc4HjyM=
Expand Down
4 changes: 2 additions & 2 deletions chart-sync/vendor/modules.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ github.com/containerd/platforms
# github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc
## explicit
github.com/davecgh/go-spew/spew
# github.com/devtron-labs/common-lib v0.19.1 => github.com/devtron-labs/devtron-services/common-lib v0.0.0-20250703034313-6e26d7880667
# github.com/devtron-labs/common-lib v0.19.1 => github.com/devtron-labs/devtron-services/common-lib v0.0.0-20250708132702-b8a70bd2d48e
## explicit; go 1.24.0
github.com/devtron-labs/common-lib/constants
github.com/devtron-labs/common-lib/fetchAllEnv
Expand Down Expand Up @@ -967,4 +967,4 @@ sigs.k8s.io/structured-merge-diff/v4/value
sigs.k8s.io/yaml
sigs.k8s.io/yaml/goyaml.v2
sigs.k8s.io/yaml/goyaml.v3
# github.com/devtron-labs/common-lib => github.com/devtron-labs/devtron-services/common-lib v0.0.0-20250703034313-6e26d7880667
# github.com/devtron-labs/common-lib => github.com/devtron-labs/devtron-services/common-lib v0.0.0-20250708132702-b8a70bd2d48e
6 changes: 3 additions & 3 deletions ci-runner/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ go 1.24.0

toolchain go1.24.3

replace github.com/devtron-labs/common-lib => github.com/devtron-labs/devtron-services/common-lib v0.0.0-20250703034313-6e26d7880667
replace github.com/devtron-labs/common-lib => github.com/devtron-labs/devtron-services/common-lib v0.0.0-20250708132702-b8a70bd2d48e

require (
github.com/Knetic/govaluate v3.0.0+incompatible
Expand All @@ -16,8 +16,10 @@ require (
github.com/otiai10/copy v1.14.1
golang.org/x/sync v0.14.0
gopkg.in/yaml.v2 v2.4.0
k8s.io/api v0.33.1
k8s.io/apimachinery v0.33.1
k8s.io/client-go v0.33.1
k8s.io/utils v0.0.0-20250502105355-0f33e8f1c979
)

require (
Expand Down Expand Up @@ -149,10 +151,8 @@ require (
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
gotest.tools/v3 v3.5.1 // indirect
k8s.io/api v0.33.1 // indirect
k8s.io/klog/v2 v2.130.1 // indirect
k8s.io/kube-openapi v0.0.0-20250318190949-c8a335a9a2ff // indirect
k8s.io/utils v0.0.0-20250502105355-0f33e8f1c979 // indirect
mellium.im/sasl v0.3.2 // indirect
sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 // indirect
sigs.k8s.io/randfill v1.0.0 // indirect
Expand Down
4 changes: 2 additions & 2 deletions ci-runner/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,8 +114,8 @@ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/devtron-labs/devtron-services/common-lib v0.0.0-20250703034313-6e26d7880667 h1:mV+gq9St5soJPS7vww0cWTW6ayKxDVKWmJVkhbkrtjM=
github.com/devtron-labs/devtron-services/common-lib v0.0.0-20250703034313-6e26d7880667/go.mod h1:/Ciy9tD9OxZOWBDPIasM448H7uvSo4+ZJiExpfwBZpA=
github.com/devtron-labs/devtron-services/common-lib v0.0.0-20250708132702-b8a70bd2d48e h1:iJg144GwEMX7vUuNqIYBWbBniw5BvJiFw5ZXG8LuH3c=
github.com/devtron-labs/devtron-services/common-lib v0.0.0-20250708132702-b8a70bd2d48e/go.mod h1:/Ciy9tD9OxZOWBDPIasM448H7uvSo4+ZJiExpfwBZpA=
github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk=
github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E=
github.com/docker/cli v28.1.1+incompatible h1:eyUemzeI45DY7eDPuwUcmDyDj1pM98oD5MdSpiItp8k=
Expand Down
3 changes: 3 additions & 0 deletions ci-runner/helper/CiStageError.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
package helper

import (
"errors"
"fmt"
"github.com/devtron-labs/common-lib/utils/workFlow"
)
Expand Down Expand Up @@ -125,3 +126,5 @@ func (err *CdStageError) ErrorMessage() string {
func (err *CdStageError) Unwrap() error {
return err.stageErr
}

var BuilderPodDeletedError = errors.New("Buildx pod(s) was terminated in response to imminent node shutdown.")
16 changes: 15 additions & 1 deletion ci-runner/helper/CommandExecuter.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,15 @@
package helper

import (
cicxt "github.com/devtron-labs/ci-runner/executor/context"
"os/exec"

cicxt "github.com/devtron-labs/ci-runner/executor/context"
"github.com/devtron-labs/ci-runner/util"
)

type CommandExecutor interface {
RunCommand(ctx cicxt.CiContext, cmd *exec.Cmd) error
RunCommandWithCtx(ctx cicxt.CiContext, cmd *exec.Cmd) error
}

type CommandExecutorImpl struct {
Expand All @@ -21,3 +22,16 @@ func NewCommandExecutorImpl() *CommandExecutorImpl {
func (c *CommandExecutorImpl) RunCommand(ctx cicxt.CiContext, cmd *exec.Cmd) error {
return util.RunCommand(cmd)
}

func (c *CommandExecutorImpl) RunCommandWithCtx(ctx cicxt.CiContext, cmd *exec.Cmd) error {
return util.RunCommand(c.GetCommandWithCtx(ctx, cmd))
}

func (c *CommandExecutorImpl) GetCommandWithCtx(ctx cicxt.CiContext, cmd *exec.Cmd) *exec.Cmd {
// Ensure the command is run with the provided context
var args []string
if len(cmd.Args) > 1 {
args = cmd.Args[1:]
}
return exec.CommandContext(ctx, cmd.Path, args...)
}
Loading