Skip to content

feat: support Kubernetes Coscheduling Plugin for PodGroup#1833

Closed
lin121291 wants to merge 3 commits into
Project-HAMi:masterfrom
lin121291:feat/coscheduling-podgroup-support
Closed

feat: support Kubernetes Coscheduling Plugin for PodGroup#1833
lin121291 wants to merge 3 commits into
Project-HAMi:masterfrom
lin121291:feat/coscheduling-podgroup-support

Conversation

@lin121291

Copy link
Copy Markdown
Contributor

What type of PR is this?

/kind feature

What this PR does / why we need it:

Fixes two behaviors in HAMi that break when used with the Kubernetes Coscheduling Plugin for PodGroup:

  1. Concurrent Bind failures — When Coscheduling releases a PodGroup, kube-scheduler binds all member pods near-simultaneously. Multiple /bind calls targeting the same node race on LockNode and fail immediately. LockNode now retries with backoff within a configurable timeout window (--node-lock-retry-timeout, default 30s).

  2. Phantom GPU occupancy — When Coscheduling denies a PodGroup, pods return to Pending with stale HAMi annotations. onAddPod re-adds them to podManager cache, creating ghost GPU reservations. Pods in Pending state with stale annotations are now evicted from cache instead.

A third fix cleans up podManager cache in the Bind failure path, which previously leaked ghost entries on failed binds.

Which issue(s) this PR fixes:

Fixes #1832

Does this PR introduce a user-facing change?:

Yes. New flag: --node-lock-retry-timeout (default: 30s).

@hami-robot

hami-robot Bot commented May 4, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: lin121291
Once this PR has been reviewed and has the lgtm label, please assign wawa0210 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hami-robot hami-robot Bot requested review from archlitchi and lengrongfu May 4, 2026 10:13
@hami-robot hami-robot Bot added the size/L label May 4, 2026
…ling

- Retry LockNode with backoff on lock contention to allow concurrent
  PodGroup bind operations to serialize gracefully
- Evict pending pods with stale annotations in onAddPod to prevent
  phantom GPU occupancy when Coscheduling denies a PodGroup
- Call podManager.DelPod in Bind failure path to prevent ghost cache
  entries on failed binds
- Add --node-lock-retry-timeout flag (default: 30s)

Signed-off-by: lin121291 <4jp33f9e@gmail.com>
@lin121291 lin121291 force-pushed the feat/coscheduling-podgroup-support branch from 2db663e to 5b5ee96 Compare May 4, 2026 10:15

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a retry mechanism for node lock acquisition to better handle concurrent bind operations, such as those occurring with coscheduling. It adds a configurable node-lock-retry-timeout flag and environment variable, implements an idempotent lock check, and adds logic to evict pods with stale annotations from the scheduler cache. Feedback indicates that the new eviction logic in onAddPod is too aggressive and may incorrectly remove pods currently in the allocation phase. Additionally, it is recommended to replace the fixed 1-second polling interval in the lock retry logic with an exponential backoff strategy to prevent excessive load on the Kubernetes API server.

Comment thread pkg/scheduler/scheduler.go Outdated
Comment thread pkg/util/nodelock/nodelock.go Outdated
Signed-off-by: lin121291 <4jp33f9e@gmail.com>
Signed-off-by: lin121291 <4jp33f9e@gmail.com>
@lin121291 lin121291 force-pushed the feat/coscheduling-podgroup-support branch from e6d53d9 to 3634601 Compare May 9, 2026 05:22
@codecov

codecov Bot commented May 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 61.76471% with 13 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
pkg/util/nodelock/nodelock.go 72.41% 7 Missing and 1 partial ⚠️
pkg/scheduler/scheduler.go 0.00% 4 Missing and 1 partial ⚠️
Flag Coverage Δ
unittests 58.51% <61.76%> (+0.45%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
pkg/scheduler/config/config.go 78.16% <ø> (ø)
pkg/scheduler/scheduler.go 48.66% <0.00%> (-0.86%) ⬇️
pkg/util/nodelock/nodelock.go 63.92% <72.41%> (+1.04%) ⬆️

... and 9 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mesutoezdil

Copy link
Copy Markdown
Contributor

Note: As of today, this issue/PR has been flagged under our new activity policy. If there is no response or update by July 18, 2026, it will be closed. It can be reopened at any time if still relevant.

@lin121291 lin121291 closed this Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: support Kubernetes Coscheduling Plugin for PodGroup

2 participants