[slice] NodeSelector in STS webhook#1176
[slice] NodeSelector in STS webhook#1176pajakd wants to merge 1 commit intoAI-Hypercomputer:slice-mainfrom
Conversation
pajakd
changed the title
| sts.Spec.Template.Spec.NodeSelector = make(map[string]string) | ||
| } | ||
| if _, ok := sts.Spec.Template.Spec.NodeSelector[core.TPUTopologyAnnotation]; !ok { | ||
| sts.Spec.Template.Spec.NodeSelector[core.TPUTopologyAnnotation] = tpuTopology |
There was a problem hiding this comment.
The health labels are always at the nodes (unlike the topology labels that are added only after the slice is Active), so it is not interfering with the scheduling to just add them in the LWS/Job/Jobset webhook.
There was a problem hiding this comment.
I was asking whether we should add node selector for healthy/degraded nodes or not.
There was a problem hiding this comment.
We add this healthy/degraded selector in LWS webhook and it is propagated to kueue workload, so TAS knows which nodes to exclude. I think it should be propagated from LWS to STS but even if its not then is it a problem?
| if len(w.Spec.LeaderWorkerTemplate.WorkerTemplate.Spec.Containers) == 0 { | ||
| w.Spec.LeaderWorkerTemplate.WorkerTemplate.Spec.Containers = []corev1.Container{{}} | ||
| } | ||
| w.Spec.LeaderWorkerTemplate.WorkerTemplate.Spec.Containers[0].Name = name |
There was a problem hiding this comment.
What if there is more than 1 container?
There was a problem hiding this comment.
Hmm but this is only for e2e tests. Making it work for any number of containers would introduce additional complexity and in kueue also we have 1 container hardcoded in such wrappers (https://github.com/kubernetes-sigs/kueue/blob/90375d4a3e8c363f7187512c8f5ad4145d041610/pkg/util/testingjobs/leaderworkerset/wrappers.go#L189-L211)
pajakd
changed the title
2 participants
Description
Add a StatefulSet webhook, that (for relevant tpuv7 pods) makes sure that the topology NodeSelector is present.
If the selector is not there, it adds it with the value taken from the "cloud.google.com/gke-tpu-topology" annotation.
This is to make sure that LWS workloads are not blocked by the GKE warden.
Issue
Testing