Improve KIND deployment reliability - #318
Merged
Merged
Conversation
KIND clusters fail to deploy out of the box due to three issues: webhook race conditions, a removed container image, and missing third-party services. - Wait for cert-manager and DWS webhooks to be functional before deploying dependent modules (cmd/main.go) - Update submodules to include gcr.io -> registry.k8s.io fix for kube-rbac-proxy - Auto-inject system CA certificates into KIND nodes for corporate proxy environments (tools/kind.sh) - Copy overlay-legacy.yaml-template so init installs cert-manager and mpi-operator without modifying shared config Signed-off-by: Anthony Floeder <anthony.floeder@hpe.com>
ajfloeder
force-pushed
the
wait-for-prerequisites-kind
branch
from
March 16, 2026 21:55
e76641f to
8f7fde1
Compare
bdevcich
approved these changes
Mar 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
KIND clusters fail to deploy out of the box due to webhook race conditions and a removed container image. Corporate networks also need CA certificates injected into KIND nodes.
Changes
Wait for webhooks during deploy (
cmd/main.go): Before deploying modules that depend on cert-manager or DWS, verify their webhooks are functional — not just that the deployment has available replicas. This closes the race condition where the pod is running but the webhook endpoint isn't serving yet.Update submodules: Pick up the
gcr.io->registry.k8s.iofix forkube-rbac-proxy(merged via Fix kube-rbac-proxy image registry: gcr.io -> registry.k8s.io DataWorkflowServices/dws#276, Fix kube-rbac-proxy image registry: gcr.io -> registry.k8s.io lustre-fs-operator#124, Fix kube-rbac-proxy image registry: gcr.io -> registry.k8s.io nnf-sos#599, Fix kube-rbac-proxy image registry: gcr.io -> registry.k8s.io nnf-dm#356).Auto-inject CA certificates (
tools/kind.sh): Newinject_ca_certsfunction extracts system CA certs (macOS keychain or Linux CA bundle) and injects them into KIND nodes so containerd can pull images behind a corporate TLS proxy. Override withKIND_CA_CERTSenv var.Testing
Verified clean end-to-end:
bash tools/kind.sh createfollowed bycp config/overlay-legacy.yaml-template overlay-legacy.yaml && ./nnf-deploy init && ./nnf-deploy deploy— no manual workarounds needed.