Skip to content

feat(operator): scaffold DGD placement score status wiring#11057

Open
ashnamehrotra wants to merge 6 commits into
ai-dynamo:mainfrom
ashnamehrotra:grove-placement-score
Open

feat(operator): scaffold DGD placement score status wiring#11057
ashnamehrotra wants to merge 6 commits into
ai-dynamo:mainfrom
ashnamehrotra:grove-placement-score

Conversation

@ashnamehrotra

@ashnamehrotra ashnamehrotra commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Part of DEP: Grove status observability in DynamoGraphDeployment #10064
  • Adds DGD-level placement score API fields:
    • status.placementScore
    • status.placementScoreState
  • Mirrors these fields across v1beta1 and v1alpha1, including conversion wiring and conversion field-set acknowledgment.
  • Wires placement score state update into DGD reconcile on the Grove pathway.
  • Adds placement score aggregation scaffolding in Grove integration:
    • Returns explicit non-failing states for current upstream limitations.
    • Leaves placementScore unset until Grove exposes a score field in a consumable API shape.
  • Adds focused tests:
    • DGD status conversion round-trip includes placement fields.
    • Grove aggregation behavior is pinned for current states (Unsupported / NotReported).

Validation

  • GOCACHE=/tmp/dynamo-go-cache go test ./api/v1alpha1 ./api/v1beta1 -count=1
  • GOCACHE=/tmp/dynamo-go-cache go test ./api/v1alpha1 ./internal/dynamo -count=1
  • GOCACHE=/tmp/dynamo-go-cache go build ./... (from deploy/operator)

Scope Notes

  • This PR intentionally does not implement low-score metrics/events yet.
  • This PR intentionally does not emit Reported / Partial yet, because current Grove API does not expose placementScore where Dynamo can read it.
  • Ready condition diagnosis behavior is unchanged.

Open in Devin Review

Summary by CodeRabbit

  • New Features

    • Added placement score reporting to deployment status, including a state that indicates whether the score is reported, partial, unsupported, not reported, or unknown.
    • Deployment reconciliation now surfaces placement score information when available for the Grove path.
  • Bug Fixes

    • Preserved placement score data during status conversions and round-trips between versions.
    • Updated status copying so placement score values are retained correctly.

Signed-off-by: ashnamehrotra <ashnamehrotra@gmail.com>
@ashnamehrotra ashnamehrotra requested a review from a team as a code owner June 29, 2026 18:20
@copy-pr-bot

copy-pr-bot Bot commented Jun 29, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@ashnamehrotra ashnamehrotra temporarily deployed to external_collaborator June 29, 2026 18:20 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot added feat external-contribution Pull request is from an external contributor labels Jun 29, 2026
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test b29bb9a

@github-actions github-actions Bot added the deployment::k8s Relates to dynamo deployment in kubernetes label Jun 29, 2026
@datadog-official

datadog-official Bot commented Jun 29, 2026

Copy link
Copy Markdown

Pipelines

⚠️ Warnings

🚦 2 Pipeline jobs failed

Pre Merge | operator   View in Datadog   GitHub Actions

Pre Merge | pre-merge-status-check   View in Datadog   GitHub Actions

ℹ️ Info

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 42.78% (-3.01%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e3b76e7 | Docs | Give us feedback!

@coderabbitai

coderabbitai Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

Adds PlacementScore (*float64) and PlacementScoreState (new enum) fields to DynamoGraphDeploymentStatus in both v1beta1 and v1alpha1 APIs. A new AggregatePlacementScore helper computes these from a PodCliqueSet; the controller's Grove-pathway reconcile loop is wired to call it. Conversion, deepcopy, and tests are updated accordingly.

Changes

Placement Score for DynamoGraphDeployment

Layer / File(s) Summary
API type definitions (v1beta1 and v1alpha1)
deploy/operator/api/v1beta1/common.go, deploy/operator/api/v1beta1/dynamographdeployment_types.go, deploy/operator/api/v1beta1/zz_generated.deepcopy.go, deploy/operator/api/v1alpha1/dynamographdeployment_types.go, deploy/operator/api/v1alpha1/zz_generated.deepcopy.go
Introduces the PlacementScoreState string enum with five constants in both API versions and adds PlacementScore *float64 and PlacementScoreState fields to DynamoGraphDeploymentStatus, with generated deepcopy for the pointer field.
AggregatePlacementScore helper and tests
deploy/operator/internal/dynamo/grove.go, deploy/operator/internal/dynamo/grove_test.go
Adds AggregatePlacementScore(pcs *PodCliqueSet) returning (nil, Unsupported) for nil/missing score and (nil, NotReported) for empty pod gangs; table-driven tests cover all three current code paths.
Controller reconcile wiring
deploy/operator/internal/controller/dynamographdeployment_controller.go
On the Grove pathway in Reconcile, fetches the existing PodCliqueSet, calls AggregatePlacementScore, and writes results to dynamoDeployment.Status; fetch errors are logged non-fatally.
Conversion round-trip and coverage
deploy/operator/api/v1alpha1/dynamographdeployment_conversion.go, deploy/operator/api/v1alpha1/dynamographdeployment_conversion_test.go, deploy/operator/api/v1alpha1/conversion_field_coverage_test.go
Extends both directions of DynamoGraphDeploymentStatus conversion to preserve PlacementScore and PlacementScoreState; round-trip test and field-coverage acknowledgment set are updated to match.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Description check ⚠️ Warning It covers summary, validation, and scope notes, but misses the required template sections and explicit issue-link choice. Add Overview, Details, Where should reviewer start?, and the required Related Issues section with either the linked issue or no-issue confirmation.
Docstring Coverage ⚠️ Warning Docstring coverage is 71.43% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and accurately summarizes the main change: wiring DGD placement score status.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.12.2)

level=error msg="[linters_context] typechecking error: pattern ./...: directory prefix . does not contain main module or its selected dependencies"


Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@deploy/operator/internal/controller/dynamographdeployment_controller.go`:
- Around line 292-300: Reset the placement status fields in
dynamographdeployment_controller.go so stale values do not survive non-Grove or
lookup-failure reconciles. In the reconcile logic around r.isGrovePathway and
getExistingGrovePodCliqueSet, explicitly clear or set
dynamoDeployment.Status.PlacementScore and
dynamoDeployment.Status.PlacementScoreState on the non-Grove branch and when
pcsErr is returned, instead of only updating them after AggregatePlacementScore
succeeds. Use the existing symbols r.isGrovePathway,
getExistingGrovePodCliqueSet, and dynamo.AggregatePlacementScore to place the
fix in the same status-update path.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 03ddbb02-5731-442a-8efc-7bfe4bb28257

📥 Commits

Reviewing files that changed from the base of the PR and between 47f19f2 and b29bb9a.

📒 Files selected for processing (11)
  • deploy/operator/api/v1alpha1/conversion_field_coverage_test.go
  • deploy/operator/api/v1alpha1/dynamographdeployment_conversion.go
  • deploy/operator/api/v1alpha1/dynamographdeployment_conversion_test.go
  • deploy/operator/api/v1alpha1/dynamographdeployment_types.go
  • deploy/operator/api/v1alpha1/zz_generated.deepcopy.go
  • deploy/operator/api/v1beta1/common.go
  • deploy/operator/api/v1beta1/dynamographdeployment_types.go
  • deploy/operator/api/v1beta1/zz_generated.deepcopy.go
  • deploy/operator/internal/controller/dynamographdeployment_controller.go
  • deploy/operator/internal/dynamo/grove.go
  • deploy/operator/internal/dynamo/grove_test.go

Comment on lines +292 to +300
if r.isGrovePathway(dynamoDeployment) {
pcs, pcsErr := r.getExistingGrovePodCliqueSet(ctx, dynamoDeployment)
if pcsErr != nil {
logger.V(1).Info("Failed to get PodCliqueSet for placement score", "error", pcsErr)
} else {
score, state := dynamo.AggregatePlacementScore(pcs)
dynamoDeployment.Status.PlacementScore = score
dynamoDeployment.Status.PlacementScoreState = state
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Reset placement status on non-Grove and lookup-error paths.

Lines 292-300 only overwrite these fields on the happy Grove path, so a DGD can keep stale status.placementScore* values after Grove is disabled or after a transient getExistingGrovePodCliqueSet failure. Please clear or explicitly set them on every reconcile path instead of leaving the previous status intact.

Suggested fix
-	if r.isGrovePathway(dynamoDeployment) {
+	dynamoDeployment.Status.PlacementScore = nil
+	dynamoDeployment.Status.PlacementScoreState = nvidiacomv1beta1.PlacementScoreStateUnsupported
+	if r.isGrovePathway(dynamoDeployment) {
 		pcs, pcsErr := r.getExistingGrovePodCliqueSet(ctx, dynamoDeployment)
 		if pcsErr != nil {
 			logger.V(1).Info("Failed to get PodCliqueSet for placement score", "error", pcsErr)
+			dynamoDeployment.Status.PlacementScoreState = nvidiacomv1beta1.PlacementScoreStateUnknown
 		} else {
 			score, state := dynamo.AggregatePlacementScore(pcs)
 			dynamoDeployment.Status.PlacementScore = score
 			dynamoDeployment.Status.PlacementScoreState = state
 		}
 	}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if r.isGrovePathway(dynamoDeployment) {
pcs, pcsErr := r.getExistingGrovePodCliqueSet(ctx, dynamoDeployment)
if pcsErr != nil {
logger.V(1).Info("Failed to get PodCliqueSet for placement score", "error", pcsErr)
} else {
score, state := dynamo.AggregatePlacementScore(pcs)
dynamoDeployment.Status.PlacementScore = score
dynamoDeployment.Status.PlacementScoreState = state
}
dynamoDeployment.Status.PlacementScore = nil
dynamoDeployment.Status.PlacementScoreState = nvidiacomv1beta1.PlacementScoreStateUnsupported
if r.isGrovePathway(dynamoDeployment) {
pcs, pcsErr := r.getExistingGrovePodCliqueSet(ctx, dynamoDeployment)
if pcsErr != nil {
logger.V(1).Info("Failed to get PodCliqueSet for placement score", "error", pcsErr)
dynamoDeployment.Status.PlacementScoreState = nvidiacomv1beta1.PlacementScoreStateUnknown
} else {
score, state := dynamo.AggregatePlacementScore(pcs)
dynamoDeployment.Status.PlacementScore = score
dynamoDeployment.Status.PlacementScoreState = state
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/operator/internal/controller/dynamographdeployment_controller.go`
around lines 292 - 300, Reset the placement status fields in
dynamographdeployment_controller.go so stale values do not survive non-Grove or
lookup-failure reconciles. In the reconcile logic around r.isGrovePathway and
getExistingGrovePodCliqueSet, explicitly clear or set
dynamoDeployment.Status.PlacementScore and
dynamoDeployment.Status.PlacementScoreState on the non-Grove branch and when
pcsErr is returned, instead of only updating them after AggregatePlacementScore
succeeds. Use the existing symbols r.isGrovePathway,
getExistingGrovePodCliqueSet, and dynamo.AggregatePlacementScore to place the
fix in the same status-update path.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

Signed-off-by: ashnamehrotra <ashnamehrotra@gmail.com>
@ashnamehrotra ashnamehrotra temporarily deployed to external_collaborator June 30, 2026 13:40 — with GitHub Actions Inactive
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test 0c56c9e

Signed-off-by: ashnamehrotra <ashnamehrotra@gmail.com>
@ashnamehrotra ashnamehrotra requested a review from a team as a code owner June 30, 2026 16:33
@ashnamehrotra ashnamehrotra temporarily deployed to external_collaborator June 30, 2026 16:33 — with GitHub Actions Inactive
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test de57d0f

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Jun 30, 2026
Comment thread deploy/operator/api/v1alpha1/dynamographdeployment_types.go Outdated
Comment thread deploy/operator/internal/controller/dynamographdeployment_controller.go Outdated
Comment thread deploy/operator/api/v1alpha1/dynamographdeployment_types.go
- Drop the NotReported PlacementScoreState (removed by DEP; not
  operationally distinct from Unknown). Empty PodGang statuses now
  report Unknown.
- Enforce the [0.0, 1.0] score contract at the CRD boundary via
  Minimum=0 / Maximum=1 kubebuilder markers on the placementScore
  field in both v1alpha1 and v1beta1.
- Document the score contract on the API field: normalized [0,1],
  higher-is-better, minimum aggregation as a worst-placement signal,
  comparability only within the same scheduler scoring contract.
- Fix the reporting lifecycle so every reconcile leaves status
  consistent:
    * non-Grove pathway sets Unsupported and clears any stale score,
    * PCS read failure sets Unknown and clears any stale score,
    * successful aggregation writes through unchanged.
- Regenerate CRDs, api-reference, and the operator Helm chart CRD copy.

Signed-off-by: ashnamehrotra <ashnamehrotra@gmail.com>
@ashnamehrotra ashnamehrotra temporarily deployed to external_collaborator July 8, 2026 14:54 — with GitHub Actions Inactive
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test d86e8fd

Address review feedback from @sttts on PR ai-dynamo#11057:

- Introduce PlacementStatus struct (score + state) on both v1alpha1
  and v1beta1 DGDStatus, replacing the flat placementScore /
  placementScoreState fields. This future-proofs the schema against
  new placement signals (scheduler contract version, last-report
  timestamp, per-unit reports) that would otherwise be a schema break
  to add later.
- Update the operator helper to read logr.Logger from ctx via
  log.FromContext(ctx) instead of taking it as a parameter, and drop
  the now-unused go-logr/logr import.
- Rename helper to updatePlacementStatus and write through a fresh
  PlacementStatus value so stale fields are never carried across
  reconciles.
- Regenerate CRDs, api-reference, helm CRD copy, and deepcopy.

Signed-off-by: ashnamehrotra <ashnamehrotra@gmail.com>
@ashnamehrotra ashnamehrotra temporarily deployed to external_collaborator July 9, 2026 20:01 — with GitHub Actions Inactive
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test a93fe91

Resolve conflicts in the DGD controller (keep placement-status write,
drop the duplicate reconcile-error check that main removed) and in
grove.go (keep AggregatePlacementScore alongside main's shortened
comment on specToGroveTopologyConstraint). Regenerate deepcopy, CRDs,
api-reference, helm docs, and helm CRD copy to pick up both trees.

Signed-off-by: ashnamehrotra <ashnamehrotra@gmail.com>
@ashnamehrotra ashnamehrotra temporarily deployed to external_collaborator July 9, 2026 20:10 — with GitHub Actions Inactive
@dynamo-ops

Copy link
Copy Markdown
Contributor

/ok to test e3b76e7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deployment::k8s Relates to dynamo deployment in kubernetes documentation Improvements or additions to documentation external-contribution Pull request is from an external contributor feat size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants