Skip to content

Retry on enablement failure#151

Merged
fwiesel merged 2 commits into
mainfrom
retry_on_enablement_failure
Oct 20, 2025
Merged

Retry on enablement failure#151
fwiesel merged 2 commits into
mainfrom
retry_on_enablement_failure

Conversation

@fwiesel

@fwiesel fwiesel commented Oct 14, 2025

Copy link
Copy Markdown
Contributor

No description provided.

@fwiesel fwiesel requested a review from notandy October 14, 2025 11:38
@fwiesel

fwiesel commented Oct 14, 2025

Copy link
Copy Markdown
Contributor Author

We need to retry, if there is some transient error. My understanding is your ( @notandy ) preference is not to let the error get up to the controller loop, so I return now a retry and requeue the reconciliation manually.

@fwiesel fwiesel force-pushed the retry_on_enablement_failure branch from b44993f to 7db3c79 Compare October 14, 2025 15:48
Comment thread internal/controller/eviction_controller.go Outdated
Comment thread internal/controller/eviction_controller.go Outdated
@notandy

notandy commented Oct 15, 2025

Copy link
Copy Markdown
Contributor

Alternatively, if gophercloud is encapsulating all openstack related errors, you could just check if the bubbling up error is an gophercloud error to retry instead, and throw everything else.

@fwiesel fwiesel force-pushed the retry_on_enablement_failure branch from 7db3c79 to 0802f57 Compare October 15, 2025 16:02
@fwiesel fwiesel requested a review from notandy October 15, 2025 16:03
@fwiesel

fwiesel commented Oct 15, 2025

Copy link
Copy Markdown
Contributor Author

I went with the RetryError.

Comment thread internal/controller/eviction_controller.go Outdated
@fwiesel fwiesel force-pushed the retry_on_enablement_failure branch from 0802f57 to f6bdd81 Compare October 16, 2025 10:17
@fwiesel fwiesel requested a review from notandy October 16, 2025 10:18

@notandy notandy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor nitpick

Comment thread internal/controller/utils.go Outdated
@fwiesel fwiesel force-pushed the retry_on_enablement_failure branch from f6bdd81 to 5bdc12f Compare October 20, 2025 10:08
@fwiesel fwiesel requested a review from notandy October 20, 2025 10:09
We will likely encounter transient errors, and we should
not stop deleting the eviction because of that.
r.addCondition should aways be true when we don't have the
status condition. Still, let's return here a requeue to make sure
we do not end up in a dead-end by accident.
@fwiesel

fwiesel commented Oct 20, 2025

Copy link
Copy Markdown
Contributor Author

I start to think, returning the wish to requeue the reconciliation is not a good idea....
It started off with not needing to change the signature of an internal function, but if I continue going down that path, then I end up with something like

type errorRequeue struct {
	RequeueAfter time.Duration
}

func (e errorRequeue) Error() string {
	return "someone didn't handle the errorRetry case"
}

func requeueAfter(t time.Duration) errorRequeue {
	return errorRequeue{RequeueAfter: t}
}

func RetryOnConflict(backoff wait.Backoff, fn func() (ctrl.Result, error)) (ctrl.Result, error) {
	err := retry.OnError(backoff, k8serrors.IsConflict, func() error {
		result, err := fn()
		if err != nil {
			return err
		}
		if result.IsZero() {
			return nil
		}

		return requeueAfter(result.RequeueAfter)
	})

	var requeueError errorRequeue
	if errors.As(err, &requeueError); requeueError.RequeueAfter > 0 {
		return ctrl.Result{RequeueAfter: requeueError.RequeueAfter}, nil
	}

	return ctrl.Result{}, nil
}

And somehow it doesn't sit right with me.

@notandy

notandy commented Oct 20, 2025

Copy link
Copy Markdown
Contributor

You're right, if you really want to propagate a duration, it makes no sense anymore to marshal it into an error.
I am fine with the current pr although, but it's up to you if you want to build it differently.

@fwiesel fwiesel merged commit a3eb7db into main Oct 20, 2025
6 checks passed
@fwiesel fwiesel deleted the retry_on_enablement_failure branch October 20, 2025 12:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants