fix: requeue based on drain delay when old deployments are still active#109
Closed
mt-bt wants to merge 1 commit into
Closed
fix: requeue based on drain delay when old deployments are still active#109mt-bt wants to merge 1 commit into
mt-bt wants to merge 1 commit into
Conversation
When old deployment versions still have active invocations, the cleanup function returned next_removal=None, causing the controller to fall back to its default 5-minute requeue interval. This meant drainDelaySeconds had no effect on how quickly the operator re-checked drain status. Now, when active old deployments exist but no removal is scheduled, next_removal is set to now + drainDelaySeconds. With drainDelaySeconds=0 this results in an immediate requeue instead of a 5-minute wait. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cleanup_old_replicasets/cleanup_old_configurationsreturnednext_removal = None, causing the controller to fall back to the default 5-minute requeue intervaldrainDelaySeconds(from Make drain delay configurable via drainDelaySeconds field #96) had no effect on how quickly the operator polled for drain completion — evendrainDelaySeconds: 0resulted in a ~5 minute waitnext_removalis set tonow + drainDelaySeconds, so the controller requeues appropriatelyProblem
The reconcile flow after a new version is deployed:
cleanup_old_replicasetsruns — old RS hasdeployment_active = true(in-flight invocations) →continuewithout contributing tonext_removalnext_removal = None→ controller requeues in5 * 60seconds (hardcoded default)With
drainDelaySeconds: 0, step 3 should requeue immediately, not in 5 minutes.Fix
After the loop in both
cleanup_old_replicasets(ReplicaSet mode) andcleanup_old_configurations(Knative mode), if there are active old deployments butnext_removalis stillNone, set it tonow + drain_delay_seconds. This causes the controller to requeue based on the configured drain delay rather than the hardcoded 5-minute interval.Test plan
drainDelaySeconds: 0and verify old ReplicaSets are cleaned up within seconds (one reconcile cycle) rather than ~5 minutesdrainDelaySeconds(300) and verify no behavior change🤖 Generated with Claude Code