Create replacement nodes when CritControlPlane nodes are deleted

[ChrisRx]

When creating a CritControlPlane with multiple nodes (i.e 3 or 5), it creates all of the nodes initially and won't create anymore nodes unless an upgrade must happen (signified by a change to the CritControlPlane resource). This will not cover cases where, for example, an AWS machine is terminated, and therefore removed as a node by the aws cloud controller manager, and there is still a majority of nodes that are healthy.

This could be accomplished several ways, but I believe this should happen whenever ownedMachines < ccp.Spec.Replicas here (instead of when ownedMachines is 0):

cluster-api-bootstrap-provider-crit/controllers/controlplane/controlplane.go

Lines 112 to 129 in 793f4c8

	if len(ownedMachines) == 0 {
	logger.Info("Initializing control plane", "Replicas", ccp.Spec.Replicas, "Existing", len(ownedMachines))
	bootstrapSpec := ccp.Spec.CritConfigSpec.DeepCopy()
	// TODO(chrism): replace with NFailureDomainsWithFewestMachines so that
	// created control plane will be spread out over failure domains
	fd := controlPlane.FailureDomainWithFewestMachines()
	for i := 0; i < ccp.Spec.Replicas; i++ {
	if err := r.cloneConfigsAndGenerateMachine(ctx, cluster, ccp, bootstrapSpec, fd); err != nil {
	logger.Error(err, "failed to create initial control plane Machine")
	r.recorder.Eventf(ccp, corev1.EventTypeWarning, "FailedInitialization", "Failed to create initial control plane Machine for cluster %s/%s control plane: %v", cluster.Namespace, cluster.Name, err)
	return ctrl.Result{}, err
	}
	}
	// requeue the control plane, in case there are additional operations to perform
	return ctrl.Result{Requeue: true}, nil
	}

and should ensure that when creating replacement nodes that it did not lose quorum (if it did, it should do nothing as the cluster has become degraded).