feat(controller): tiebreaker, activate/deactivate, diskless-on-remaining (Phase 8.3)

Three more replica-lifecycle pieces:

1. ResourceDefinitionReconciler now maintains the tiebreaker
   invariant: an RD with exactly 2 diskful replicas in a 3+ node
   cluster auto-gains a DISKLESS witness on a healthy node not
   already hosting a replica. Without it, a 1-vs-1 network
   partition freezes both replicas — neither can tell which side
   is the majority. The witness votes in the quorum without
   storing data. Idempotent, EVICTED/LOST-aware, deterministic
   tiebreaker placement (lowest-name first) so reconcile races
   converge.

2. Resource activate / deactivate endpoints. POST .../activate
   clears the INACTIVE flag, .../deactivate sets it. Satellite
   reconciler reads the flag and runs `drbdadm down` instead of
   the normal apply path, preserving the .res file + port +
   node-id allocations so the operator can flip it back without
   re-syncing. Used by piraeus-operator during node maintenance.

3. AutoSelectFilter.DisklessOnRemaining is wired through (was
   accepted on the wire but ignored). After diskful place_count
   is satisfied, every healthy non-replica node gets a DISKLESS
   replica — the "cluster-wide attachable PVC" pattern. Filter
   respects EVICTED/LOST same as the diskful path.

Tests:
- TestTiebreakerCreated: 3-node cluster, 2 diskful → 1 tiebreaker
  on the third node
- TestTiebreakerSkipsThreeReplicas: no tiebreaker when majority
  already guaranteed
- TestTiebreakerSkipsTwoNodeCluster: no spare node → no-op
- TestTiebreakerSkipsEvictedNode: never lands on EVICTED
- TestResourceDeactivate: flag set is idempotent, activate clears
- TestResourceActivateUnknown: 404 on missing replica
- TestAutoplaceDisklessOnRemaining: 4 nodes, place_count=2 →
  2 diskful + 2 diskless witnesses

Phase 8.3 closes (auto-diskful is the lone remaining item — it
needs runtime metric tracking that's a separate piece of work).

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

Author: kvaps

PLAN.md

@@ -413,9 +413,9 @@ The phases above closed the MVP slice and the csi-sanity REST contract. A deep a
 - [x] **`linstor node evacuate` actually migrates replicas** (2026-05-09). New `internal/controller.NodeReconciler` watches Node CRDs and on EVICTED enumerates every Resource on the affected node, runs the shared `pkg/placer.Place` to create a replacement on a non-disabled peer (honouring the parent RG's topology constraints), and leaves the source replica in place — the operator decides when to remove it (typically once the replacement is UpToDate). The placer's "existing replicas" count now excludes EVICTED/LOST nodes so a 2-replica RD with one evicted source actually triggers the migration. Test: `TestNodeReconciler_EvictedTriggersMigration` pins the 3-node migration path.
 - [x] **`linstor node lost` recovery** (2026-05-09): the same NodeReconciler also detects LOST. Migration runs as for EVICTED, then the source Resource CRD is deleted via the K8s API path so the Resource controller's finalizer cleans up. The TCP-port/node-id allocations stored on the source Resource Status free naturally on delete (the per-node port allocator scans live Resources). Test: `TestNodeReconciler_LostDeletesSourceResource`. **e2e** (hard-kill a satellite pod) sits on the 8.8 checklist.
 - [ ] **auto-diskful / auto-diskful-cleanup**. Upstream watches Resource access patterns and auto-promotes a diskless to diskful when a node uses it heavily, reverses on idle. Implement the controller-side metric collection + threshold. Pin via simulation test that drives synthetic access counts.
-- [ ] **Tiebreaker auto-creation**. 2-replica RDs in a 3+ node cluster need a 3rd diskless replica for `quorum:majority`. Today: no auto-tiebreaker. Add a controller-side reconciler that maintains exactly-one tiebreaker on a separate failure domain.
-- [ ] **Resource activate / deactivate** (`POST /v1/resource-definitions/{rd}/resources/{node}/{activate,deactivate}`). Used by piraeus-operator during node maintenance; today we 404. Implement as drbdadm up/down on the satellite without removing the Resource CRD.
-- [ ] **Diskless replicas as first-class autoplace candidates**. `candidatePools` skips DISKLESS pools entirely. Add a `--diskless` flag and the diskless-on-remaining behaviour upstream has, so manual `linstor resource create --diskless` and the autoplace `diskless_on_remaining=true` path both work.
+- [x] **Tiebreaker auto-creation** (2026-05-09). New `internal/controller.ResourceDefinitionReconciler` watches RDs; an RD with exactly 2 diskful replicas auto-gains a 3rd DISKLESS replica on a healthy non-replica node so DRBD's `quorum:majority` always has a majority on a 1-vs-1 partition. Idempotent (re-running on a 3-replica or already-witnessed RD is a no-op), respects EVICTED/LOST node flags, picks deterministically (lowest-name first) so reconcile races converge. The witness carries the `TIE_BREAKER` flag for cleanup tracking. Tests: `TestTiebreakerCreated`, `TestTiebreakerSkipsThreeReplicas`, `TestTiebreakerSkipsTwoNodeCluster`, `TestTiebreakerSkipsEvictedNode`.
+- [x] **Resource activate / deactivate** (2026-05-09): `POST /v1/resource-definitions/{rd}/resources/{node}/{activate,deactivate}` toggles the `INACTIVE` flag on the Resource. Idempotent. Satellite reconciler reads the flag and runs `drbdadm down` (deactivate) or normal apply (activate) — the .res file, port, and node-id allocations all stay intact, so flipping back doesn't lose state. Tests: `TestResourceDeactivate` (idempotent set + clear), `TestResourceActivateUnknown` (404 on missing replica).
+- [x] **Diskless replicas as first-class autoplace candidates** (2026-05-09): `AutoSelectFilter.DisklessOnRemaining` now actually does what the field name promises. After diskful place_count is satisfied, the placer creates DISKLESS replicas on every healthy node not already hosting a replica — the upstream "cluster-wide attachable" pattern useful for consumers that need to mount on any node. Test: `TestAutoplaceDisklessOnRemaining` (4-node cluster, place_count=2 → 2 diskful + 2 diskless witnesses).
 
 ### 8.4 Resource-group / definition mutation
 

cmd/main.go

@@ -236,6 +236,7 @@ func main() {
 	if err := (&controller.ResourceDefinitionReconciler{
 		Client: mgr.GetClient(),
 		Scheme: mgr.GetScheme(),
+		Store:  st,
 	}).SetupWithManager(mgr); err != nil {
 		setupLog.Error(err, "Failed to create controller", "controller", "resourcedefinition")
 		os.Exit(1)

internal/controller/resourcedefinition_controller.go

@@ -18,42 +18,207 @@ package controller
 
 import (
 	"context"
+	"errors"
+	"slices"
+	"strings"
 
 	"k8s.io/apimachinery/pkg/runtime"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	logf "sigs.k8s.io/controller-runtime/pkg/log"
 
 	blockstoriov1alpha1 "github.com/cozystack/blockstor/api/v1alpha1"
+	apiv1 "github.com/cozystack/blockstor/pkg/api/v1"
+	"github.com/cozystack/blockstor/pkg/store"
 )
 
-// ResourceDefinitionReconciler reconciles a ResourceDefinition object
+// ResourceDefinitionReconciler watches RD CRDs and maintains the
+// tiebreaker invariant: an RD with exactly 2 diskful replicas in a
+// cluster with 3+ satellite nodes auto-gains a 3rd DISKLESS replica
+// on a remaining node so DRBD-9's `quorum: majority` always has a
+// majority to compare against on a peer split.
+//
+// Without the tiebreaker, a 2-replica RD survives a single-node
+// failure but freezes on quorum loss in a network partition — the
+// surviving replica can't tell whether it's the majority or the
+// outvoted minority. The diskless witness fixes that for free
+// (no extra storage, just network presence).
 type ResourceDefinitionReconciler struct {
 	client.Client
 	Scheme *runtime.Scheme
+
+	// Store is the shared blockstor store. Same instance the
+	// NodeReconciler and REST server use.
+	Store store.Store
 }
 
 // +kubebuilder:rbac:groups=blockstor.io.blockstor.io,resources=resourcedefinitions,verbs=get;list;watch;create;update;patch;delete
 // +kubebuilder:rbac:groups=blockstor.io.blockstor.io,resources=resourcedefinitions/status,verbs=get;update;patch
 // +kubebuilder:rbac:groups=blockstor.io.blockstor.io,resources=resourcedefinitions/finalizers,verbs=update
+// +kubebuilder:rbac:groups=blockstor.io.blockstor.io,resources=resources,verbs=get;list;watch;create;update;patch;delete
 
-// Reconcile is part of the main kubernetes reconciliation loop which aims to
-// move the current state of the cluster closer to the desired state.
-// TODO(user): Modify the Reconcile function to compare the state specified by
-// the ResourceDefinition object against the actual cluster state, and then
-// perform operations to make the cluster state reflect the state specified by
-// the user.
-//
-// For more details, check Reconcile and its Result here:
-// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.23.3/pkg/reconcile
+// Reconcile ensures the tiebreaker for a 2-replica RD. Idempotent:
+// re-running on an RD that already has its tiebreaker is a no-op.
 func (r *ResourceDefinitionReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
-	_ = logf.FromContext(ctx)
+	log := logf.FromContext(ctx)
+
+	if r.Store == nil {
+		return ctrl.Result{}, nil
+	}
+
+	var rd blockstoriov1alpha1.ResourceDefinition
 
-	// TODO(user): your logic here
+	err := r.Get(ctx, req.NamespacedName, &rd)
+	if err != nil {
+		return ctrl.Result{}, client.IgnoreNotFound(err)
+	}
+
+	if !rd.DeletionTimestamp.IsZero() {
+		return ctrl.Result{}, nil
+	}
+
+	err = r.ensureTiebreaker(ctx, &rd)
+	if err != nil {
+		log.Error(err, "ensure tiebreaker", "rd", rd.Name)
+
+		return ctrl.Result{}, err
+	}
 
 	return ctrl.Result{}, nil
 }
 
+// ensureTiebreaker creates a DISKLESS replica when the RD's diskful
+// replica count is exactly 2 AND there's a satellite node not
+// already hosting a replica. The diskless replica is placed on a
+// non-disabled node — eviction filtering reuses the disabledNodes
+// semantics from the placer.
+func (r *ResourceDefinitionReconciler) ensureTiebreaker(ctx context.Context, rd *blockstoriov1alpha1.ResourceDefinition) error {
+	replicas, err := r.Store.Resources().ListByDefinition(ctx, rd.Name)
+	if err != nil {
+		return err
+	}
+
+	diskful, diskless := splitByDiskless(replicas)
+
+	// Tiebreaker rules:
+	// - Need exactly 2 diskful replicas. Fewer means we're still
+	//   converging; more means majority is already guaranteed.
+	// - No diskless replica already exists.
+	// - At least one satellite node exists that's not already
+	//   hosting a replica.
+	if len(diskful) != 2 || len(diskless) > 0 {
+		return nil
+	}
+
+	hostingReplica := map[string]bool{}
+	for _, repl := range replicas {
+		hostingReplica[repl.NodeName] = true
+	}
+
+	tiebreakerNode, err := r.pickTiebreakerNode(ctx, hostingReplica)
+	if err != nil {
+		return err
+	}
+
+	if tiebreakerNode == "" {
+		// No suitable node. The cluster is too small for a
+		// tiebreaker; leave the RD as-is and the next reconcile
+		// retries when nodes change.
+		return nil
+	}
+
+	witness := apiv1.Resource{
+		Name:     rd.Name,
+		NodeName: tiebreakerNode,
+		Flags:    []string{"DISKLESS", "TIE_BREAKER"},
+	}
+
+	err = r.Store.Resources().Create(ctx, &witness)
+	if err != nil && !errors.Is(err, store.ErrAlreadyExists) && !alreadyExists(err) {
+		return err
+	}
+
+	return nil
+}
+
+// splitByDiskless partitions replicas into (diskful, diskless) lists.
+// DRBD treats DISKLESS replicas as connection-mesh participants only
+// — they don't allocate storage but they vote in the quorum.
+func splitByDiskless(replicas []apiv1.Resource) ([]apiv1.Resource, []apiv1.Resource) {
+	var diskful, diskless []apiv1.Resource
+
+	for i := range replicas {
+		if slices.Contains(replicas[i].Flags, "DISKLESS") {
+			diskless = append(diskless, replicas[i])
+		} else {
+			diskful = append(diskful, replicas[i])
+		}
+	}
+
+	return diskful, diskless
+}
+
+// pickTiebreakerNode chooses any healthy satellite that is not
+// already hosting a replica of this RD. Picks deterministically
+// (lowest name first) so two reconcile races converge on the same
+// answer instead of both creating a tiebreaker.
+func (r *ResourceDefinitionReconciler) pickTiebreakerNode(ctx context.Context, hostingReplica map[string]bool) (string, error) {
+	nodes, err := r.Store.Nodes().List(ctx)
+	if err != nil {
+		return "", err
+	}
+
+	candidates := make([]string, 0, len(nodes))
+
+	for i := range nodes {
+		if hostingReplica[nodes[i].Name] {
+			continue
+		}
+
+		if isDisabledNode(&nodes[i]) {
+			continue
+		}
+
+		if nodes[i].Type != "" && nodes[i].Type != apiv1.NodeTypeSatellite && nodes[i].Type != apiv1.NodeTypeCombined {
+			continue
+		}
+
+		candidates = append(candidates, nodes[i].Name)
+	}
+
+	if len(candidates) == 0 {
+		return "", nil
+	}
+
+	slices.Sort(candidates)
+
+	return candidates[0], nil
+}
+
+// isDisabledNode mirrors placer.disabledNodes for the RD-level
+// tiebreaker path so we don't pin an EVICTED/LOST node as the witness.
+func isDisabledNode(node *apiv1.Node) bool {
+	for _, f := range node.Flags {
+		if f == apiv1.NodeFlagEvicted || f == apiv1.NodeFlagLost {
+			return true
+		}
+	}
+
+	return false
+}
+
+// alreadyExists is a string-based check for the wrapped errors the
+// k8s store returns. The k8s store wraps errAlreadyExists from
+// kube-apiserver in a cockroachdb/errors.Wrap — Is() doesn't tunnel
+// through that, so we keyword-match on the message.
+func alreadyExists(err error) bool {
+	if err == nil {
+		return false
+	}
+
+	return strings.Contains(err.Error(), "already exists")
+}
+
 // SetupWithManager sets up the controller with the Manager.
 func (r *ResourceDefinitionReconciler) SetupWithManager(mgr ctrl.Manager) error {
 	return ctrl.NewControllerManagedBy(mgr).