Skip to content

fix(errors): retry transient "failed calling webhook" apiserver 500s#16259

Open
HsiuChuanHsu wants to merge 2 commits into
argoproj:mainfrom
HsiuChuanHsu:fix/14220-update
Open

fix(errors): retry transient "failed calling webhook" apiserver 500s#16259
HsiuChuanHsu wants to merge 2 commits into
argoproj:mainfrom
HsiuChuanHsu:fix/14220-update

Conversation

@HsiuChuanHsu

Copy link
Copy Markdown
Contributor

Fixes #14220

Motivation

A webhook can be briefly unreachable during normal operation. While it is unreachable, the kube-apiserver
rejects the request with an HTTP 500 InternalError:

Internal error occurred: failed calling webhook "..."

An earlier attempt (PR #16038) treated all 500s as transient which was not that correct. Blindly retrying them would mask real failures and pile load onto an already-unhealthy apiserver.
So this task is to retry only this known-transient case while leaving every other 500 classified as permanent.

Modifications

Added a single targeted classifier in util/errors/errors.go, same style as the existing isResourceQuotaTimeoutErr:

func isTransientWebhookErr(err error) bool {
    return apierr.IsInternalError(err) && strings.Contains(err.Error(), "failed calling webhook")
}

And added into isTransientErr` alongside the other transient checks.

Verification

Documentation

AI

Claude Sonnet 4.8

  • Debug core logic.
  • Refine comment wording
  • Unit tests

Signed-off-by: HsiuChuanHsu <hchsu2106@gmail.com>
Signed-off-by: HsiuChuanHsu <hchsu2106@gmail.com>
@HsiuChuanHsu HsiuChuanHsu marked this pull request as ready for review June 12, 2026 03:29
@HsiuChuanHsu HsiuChuanHsu requested a review from a team as a code owner June 12, 2026 03:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Argo fails to retry workflows when kube-apiserver returns a transient Internal Server Error

1 participant