feat(satellite): auto-detach on disk:Failed (Phase 8.2)

When the storage layer drops out (LV evicted, zvol destroyed, file
inode gone) DRBD reports disk:Failed via events2. Without action,
the kernel keeps trying to reach the dead lower disk and the local
replica thrashes between failure-recovery states.

The events2 observer now intercepts those events and runs
`drbdadm detach --force <rd>` so the lower disk binding drops
cleanly. Peers stay UpToDate, the consumer keeps doing I/O via the
network path; the next reconcile will redrive state if the storage
layer comes back.

The Failed observation still ships through ReportObserved so the
controller's status pipeline can surface a "Diskless" condition
when that handler lands.

pkg/drbd.Adm.Detach is the wrapper; tests pin both Detach (--force)
and Resize (--assume-clean) command shapes.

End-to-end "pull the LV" verification lives on the 8.8 e2e
checklist; satellite-side hook is in place.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

Author: kvaps

PLAN.md

@@ -404,7 +404,7 @@ The phases above closed the MVP slice and the csi-sanity REST contract. A deep a
 ### 8.2 Storage correctness
 
 - [x] **Volume resize** plumbing (2026-05-09): `PUT /v1/resource-definitions/{rd}/volume-definitions/{vn}` already updated the spec; the satellite reconciler now picks up the size delta. Provider interface gained `ResizeVolume` (lvm-thin → `lvextend`, zfs → `zfs set volsize`, file/loopfile → `truncate` + `losetup -c`). On growth the reconciler runs the provider's resize, then `drbdadm resize --assume-clean <rd>` so the kernel re-reads the lower disk. `pkg/luks.Resize` adds the cryptsetup hook for the LUKS layer; satellite-side wiring of LUKS resize lands when the per-resource `encrypted` flag flows through ApplyResources (Phase 6 follow-up). Tests: `TestApplyTriggersResizeOnGrow`, `TestApplyNoResizeOnFreshCreate` for the satellite path; per-provider tests for the resize commands. **End-to-end with a real PVC + checksum verify is still on the e2e harness checklist** (8.8).
-- [ ] **Backing-device failure under DRBD**. When the storage layer drops out, the satellite must detach the failed replica (don't oscillate it diskful↔diskless). Today nothing watches `/proc/drbd` for `disk:Failed`. Add an event hook in `Reconciler` that flips the replica to a Diskless attach via `drbdsetup detach` and surfaces a Status condition. e2e: pull the LV out from under DRBD, observe peer stays Primary, no I/O loss.
+- [x] **Backing-device failure under DRBD** (2026-05-09). The events2 observer now watches for `disk:Failed` on the local replica and runs `drbdadm detach --force <rd>` so the lower disk stops getting hammered. Peers stay UpToDate, the consumer keeps doing I/O via DRBD's network path. The detach is best-effort (logged, not retried) — the next reconcile will redrive state if the storage layer comes back. The Failed observation still ships to the controller via ReportObserved, so a Status condition reflecting the diskless state is one ResourceObservation handler away. **End-to-end "pull the LV out from under DRBD"** sits on the 8.8 e2e checklist; satellite-side hook is in place.
 - [x] **DRBD options hierarchy** controller → resource-group → resource-definition → resource (2026-05-09). `pkg/drbd.ResolveOptions` walks the four scopes, lower wins. The resource controller reads ControllerProps via the KVEntry CRD, the parent RG via `client.Get`, the RD via the existing lookup, then folds in the resource's own props. The merged map flows through `dispatcher.ApplyOptions.EffectiveProps`; `buildDesired` splits it: DrbdOptions/* land on the satellite's drbd_options bag (the .res renderer drops them in the right `net`/`disk`/`peer-device`/`handlers` block via `pkg/drbd.SectionFor`), non-DRBD props stay on the wire-side Props map. Tests: resolver unit tests for override / partial inheritance / non-DRBD-prop pass-through; `TestApplyDRBDOptionsFromEffectiveProps` for the dispatcher wiring.
 - [x] **`allow-two-primaries` plumbing** (2026-05-09): the DRBD option-hierarchy now flows arbitrary `DrbdOptions/Net/...` keys (including `allow-two-primaries yes`) through the satellite into the rendered .res file's `net { }` block. Operators set the knob via `linstor c sp DrbdOptions/Net/allow-two-primaries yes` (or RG/RD/Resource scope). The first-activation auto-primary seed still picks one replica deterministically (lowest stable node-id) — that's correct: dual-primary is for the consumer's promotion (Ganesha promoter, KubeVirt live-migration controller), not for initial sync. `splitDRBDOptions` strips the `DrbdOptions/<Section>/` prefix so the .res renderer emits `allow-two-primaries yes;` verbatim. **Live-migration coordination on the controller side** (orchestrating `drbdadm primary` on the destination, then `drbdadm secondary` on the source) lives outside this scope — that's what drbd-reactor + the consumer (KubeVirt VirtualMachineInstanceMigration / Ganesha promoter) own.
 

pkg/drbd/drbdadm.go

@@ -83,6 +83,17 @@ func (a *Adm) Secondary(ctx context.Context, resource string) error {
 	return a.run(ctx, "secondary", resource)
 }
 
+// Detach drops the local lower-disk binding without bringing the
+// resource down. The replica becomes Diskless on this node — peers
+// stay UpToDate, the consumer keeps doing I/O via DRBD's network
+// path. Used when the storage layer fails (LV evicted, zvol
+// destroyed, file inode gone) and we want the kernel to stop bashing
+// the dead block device. `--force` is required when the disk is
+// already in a transient state.
+func (a *Adm) Detach(ctx context.Context, resource string) error {
+	return a.run(ctx, "detach", "--force", resource)
+}
+
 // Resize rescans the lower disk's size and tells DRBD to grow the
 // replicated volume to match. The lower disk (LV / zvol / dm-crypt
 // target) must already be the target size — this is a notify-only

pkg/drbd/drbdadm_test.go

@@ -136,3 +136,40 @@ func TestAdmPropagatesError(t *testing.T) {
 }
 
 var errFakeFailure = errors.New("drbdadm: simulated failure")
+
+// TestAdmDetachInvokesDrbdadm: Detach → `drbdadm detach --force <res>`.
+// --force is required because the disk is already in a transient
+// (Failed) state when this gets called; without it drbdadm refuses.
+func TestAdmDetachInvokesDrbdadm(t *testing.T) {
+	fx := storage.NewFakeExec()
+	adm := drbd.NewAdm(fx)
+
+	err := adm.Detach(t.Context(), "pvc-1")
+	if err != nil {
+		t.Fatalf("Detach: %v", err)
+	}
+
+	want := "drbdadm detach --force pvc-1"
+	if !slices.Contains(fx.CommandLines(), want) {
+		t.Errorf("expected %q in calls; got %v", want, fx.CommandLines())
+	}
+}
+
+// TestAdmResizeInvokesDrbdadm: Resize → `drbdadm resize --assume-clean <res>`.
+// --assume-clean skips re-syncing the new bytes (they're freshly
+// allocated zeros) — without it growing 3 replicas serialises on
+// every resync.
+func TestAdmResizeInvokesDrbdadm(t *testing.T) {
+	fx := storage.NewFakeExec()
+	adm := drbd.NewAdm(fx)
+
+	err := adm.Resize(t.Context(), "pvc-1")
+	if err != nil {
+		t.Fatalf("Resize: %v", err)
+	}
+
+	want := "drbdadm resize --assume-clean pvc-1"
+	if !slices.Contains(fx.CommandLines(), want) {
+		t.Errorf("expected %q in calls; got %v", want, fx.CommandLines())
+	}
+}

pkg/satellite/agent.go

@@ -292,7 +292,27 @@ func (a *Agent) runObserveLoop(ctx context.Context, client satellitepb.Controlle
 		}
 	}()
 
+	adm := drbd.NewAdm(storage.RealExec{})
+
 	for ev := range obs.Translate(events) {
+		// Backing-device failure handler: when the kernel reports
+		// disk:Failed for a local replica, detach so the lower
+		// disk stops getting hammered. Peers stay UpToDate and
+		// the consumer keeps doing I/O via the network path. The
+		// detach is best-effort — a stale .res or already-Diskless
+		// state surfaces as an error we log and move past.
+		if ev.GetDrbdState() == "Failed" {
+			detachErr := adm.Detach(ctx, ev.GetResourceName())
+			if detachErr != nil {
+				a.logger.Error("auto-detach on Failed",
+					"resource", ev.GetResourceName(),
+					"err", detachErr)
+			} else {
+				a.logger.Info("auto-detached failed replica",
+					"resource", ev.GetResourceName())
+			}
+		}
+
 		err := stream.Send(ev)
 		if err != nil {
 			a.logger.Error("ReportObserved send", "err", err)