WIP: lease controller: cancel in-flight sync from Stop to prevent shutdown… by willie-yao · Pull Request #139176 · kubernetes/kubernetes

willie-yao · 2026-05-20T00:13:15Z

… deadlock

What type of PR is this?

/kind flake

What this PR does / why we need it:

Fixes a deadlock in the apiserver-identity-lease controller's shutdown sequence when sync is stuck on an API call blocked by a misbehaving admission webhook.

Which issue(s) this PR is related to:

Flake issue: #138608

Background

TestBrokenWebhook registers a fail-closed ValidatingWebhookConfiguration that points at a non-existent service, then restarts the apiserver. The webhook intercepts all mutating API operations, including the writes the apiserver-identity-lease controller makes against coordination.k8s.io/leases.

If the lease controller's first Create (issued during apiserver startup) races against the test installing the webhook and loses, latestLease stays nil. From that point on, every subsequent sync call enters backoffEnsureLease, which loops forever calling Create until either the call succeeds or its context cancels. With the broken webhook, the call never succeeds.

The deadlock chain at shutdown:

Stop() waits on reconcilingLock
  └─ sync holds reconcilingLock, stuck in backoffEnsureLease
       └─ backoffEnsureLease only exits on ctx.Done()
            └─ ctx is derived from RunPostStartHooks's ctx, which uses
               WithoutCancel of the parent — only cancels when
               notAcceptingNewRequestCh signals
                 └─ notAcceptingNewRequestCh signals only after
                    preShutdownHooksHasStoppedCh signals
                      └─ preShutdownHooksHasStoppedCh signals only after
                         RunPreShutdownHooks returns
                           └─ RunPreShutdownHooks waits for Stop() to return
                              └─ back to the top

TestBrokenWebhook times out at 10m as a result. See verified failing run ci-kubernetes-integration-1-36/2056150753738756096: controller.go:201 "Failed to ensure lease exists, will retry" log lines continue every 7s from 23:14:47 (Restarting apiserver) through the test timeout, with no "Cleaning up lease on exit" ever appearing.

What this PR changes

Adds a context.CancelFunc field on the controller, wires it up in Start() via context.WithCancel(wait.ContextForChannel(stopCh)), and invokes it in Stop() before acquiring reconcilingLock. This gives Stop() a direct way to interrupt sync's in-flight API call so the lock is released promptly, without depending on caller-side cancellation.

The cancel function is protected by a mutex so Start() and Stop() can safely race during apiserver shutdown. If Stop() is called before Start() finishes wiring the cancel function, Start() cancels the run context immediately after installing it.

This same pattern is seen in decorated_watcher.go: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/registry/generic/registry/decorated_watcher.go#L33

Special notes for your reviewer:

Reviewers may want to spot-check that Stop()'s invariants are preserved: cleanup Delete still runs after reconcilingLock is acquired, sync's serialization is unchanged, and the Stop/sync race fix from #119083 still holds.

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot · 2026-05-20T00:13:22Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: willie-yao
Once this PR has been reviewed and has the lgtm label, please assign deads2k for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

staging/src/k8s.io/component-helpers/apimachinery/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2026-05-20T00:13:23Z

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

willie-yao · 2026-05-20T17:20:15Z

/retest
/test pull-kubernetes-integration

willie-yao · 2026-05-20T18:48:43Z