feat: support Kubernetes Coscheduling Plugin for PodGroup#1833
feat: support Kubernetes Coscheduling Plugin for PodGroup#1833lin121291 wants to merge 3 commits into
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: lin121291 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
…ling - Retry LockNode with backoff on lock contention to allow concurrent PodGroup bind operations to serialize gracefully - Evict pending pods with stale annotations in onAddPod to prevent phantom GPU occupancy when Coscheduling denies a PodGroup - Call podManager.DelPod in Bind failure path to prevent ghost cache entries on failed binds - Add --node-lock-retry-timeout flag (default: 30s) Signed-off-by: lin121291 <4jp33f9e@gmail.com>
2db663e to
5b5ee96
Compare
There was a problem hiding this comment.
Code Review
This pull request introduces a retry mechanism for node lock acquisition to better handle concurrent bind operations, such as those occurring with coscheduling. It adds a configurable node-lock-retry-timeout flag and environment variable, implements an idempotent lock check, and adds logic to evict pods with stale annotations from the scheduler cache. Feedback indicates that the new eviction logic in onAddPod is too aggressive and may incorrectly remove pods currently in the allocation phase. Additionally, it is recommended to replace the fixed 1-second polling interval in the lock retry logic with an exponential backoff strategy to prevent excessive load on the Kubernetes API server.
Signed-off-by: lin121291 <4jp33f9e@gmail.com>
Signed-off-by: lin121291 <4jp33f9e@gmail.com>
e6d53d9 to
3634601
Compare
Codecov Report❌ Patch coverage is
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 9 files with indirect coverage changes 🚀 New features to boost your workflow:
|
|
Note: As of today, this issue/PR has been flagged under our new activity policy. If there is no response or update by July 18, 2026, it will be closed. It can be reopened at any time if still relevant. |
What type of PR is this?
/kind feature
What this PR does / why we need it:
Fixes two behaviors in HAMi that break when used with the Kubernetes Coscheduling Plugin for PodGroup:
Concurrent Bind failures — When Coscheduling releases a PodGroup, kube-scheduler binds all member pods near-simultaneously. Multiple
/bindcalls targeting the same node race onLockNodeand fail immediately.LockNodenow retries with backoff within a configurable timeout window (--node-lock-retry-timeout, default 30s).Phantom GPU occupancy — When Coscheduling denies a PodGroup, pods return to Pending with stale HAMi annotations.
onAddPodre-adds them topodManagercache, creating ghost GPU reservations. Pods in Pending state with stale annotations are now evicted from cache instead.A third fix cleans up
podManagercache in theBindfailure path, which previously leaked ghost entries on failed binds.Which issue(s) this PR fixes:
Fixes #1832
Does this PR introduce a user-facing change?:
Yes. New flag:
--node-lock-retry-timeout(default: 30s).