Skip to content

fix: requeue based on drain delay when old deployments are still active#109

Closed
mt-bt wants to merge 1 commit into
restatedev:mainfrom
mt-bt:fix/requeue-on-active-drain
Closed

fix: requeue based on drain delay when old deployments are still active#109
mt-bt wants to merge 1 commit into
restatedev:mainfrom
mt-bt:fix/requeue-on-active-drain

Conversation

@mt-bt

@mt-bt mt-bt commented Mar 26, 2026

Copy link
Copy Markdown
Contributor

Summary

  • When old deployment versions have active invocations (still draining), cleanup_old_replicasets / cleanup_old_configurations returned next_removal = None, causing the controller to fall back to the default 5-minute requeue interval
  • This meant drainDelaySeconds (from Make drain delay configurable via drainDelaySeconds field #96) had no effect on how quickly the operator polled for drain completion — even drainDelaySeconds: 0 resulted in a ~5 minute wait
  • Now, when active old deployments exist but no removal is yet scheduled, next_removal is set to now + drainDelaySeconds, so the controller requeues appropriately

Problem

The reconcile flow after a new version is deployed:

  1. New ReplicaSet created + registered with Restate
  2. cleanup_old_replicasets runs — old RS has deployment_active = true (in-flight invocations) → continue without contributing to next_removal
  3. next_removal = None → controller requeues in 5 * 60 seconds (hardcoded default)
  4. ~5 minutes later, invocations have drained, old deployment becomes inactive → removal annotation set → cleanup proceeds

With drainDelaySeconds: 0, step 3 should requeue immediately, not in 5 minutes.

Fix

After the loop in both cleanup_old_replicasets (ReplicaSet mode) and cleanup_old_configurations (Knative mode), if there are active old deployments but next_removal is still None, set it to now + drain_delay_seconds. This causes the controller to requeue based on the configured drain delay rather than the hardcoded 5-minute interval.

Test plan

  • Deploy with drainDelaySeconds: 0 and verify old ReplicaSets are cleaned up within seconds (one reconcile cycle) rather than ~5 minutes
  • Deploy with default drainDelaySeconds (300) and verify no behavior change
  • Verify Knative mode has equivalent behavior

🤖 Generated with Claude Code

When old deployment versions still have active invocations, the cleanup
function returned next_removal=None, causing the controller to fall back
to its default 5-minute requeue interval. This meant drainDelaySeconds
had no effect on how quickly the operator re-checked drain status.

Now, when active old deployments exist but no removal is scheduled,
next_removal is set to now + drainDelaySeconds. With drainDelaySeconds=0
this results in an immediate requeue instead of a 5-minute wait.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@mt-bt mt-bt closed this Mar 26, 2026
@github-actions github-actions Bot locked and limited conversation to collaborators Mar 26, 2026
@mt-bt mt-bt deleted the fix/requeue-on-active-drain branch March 26, 2026 18:25
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant