Skip to content

Commit 37db7ab

Browse files
kvapsclaude
andcommitted
feat(satellite): plumb allow-two-primaries into rendered .res (Phase 8.2)
The DRBD option hierarchy was reaching the satellite via DesiredResource.DrbdOptions but buildResFile dropped everything except the per-replica wiring keys. allow-two-primaries (Ganesha RWX, KubeVirt live-migration), max-buffers, after-sb-*, and every other operator-tunable knob silently never made it into the .res file's net{} block. splitDRBDOptions partitions the bag by section and strips the `DrbdOptions/<Section>/` prefix so drbd.Build emits the keys in the form drbdadm expects. Net keys land on `net { }`, everything else (including unknown sections that the upstream catalogue may add later) on the resource-level `options { }` block — drbd's catch-all, so a future option doesn't get dropped. Test: TestApplyRendersAllowTwoPrimaries pins `allow-two-primaries yes;` in the rendered .res when the controller folded the option in via DrbdOptions/Net/allow-two-primaries=yes. Live-migration sequencing (drbdadm primary on dst, secondary on src) is owned by drbd-reactor + the consumer; documented as such in PLAN. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
1 parent a22d736 commit 37db7ab

3 files changed

Lines changed: 97 additions & 2 deletions

File tree

PLAN.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -406,7 +406,7 @@ The phases above closed the MVP slice and the csi-sanity REST contract. A deep a
406406
- [x] **Volume resize** plumbing (2026-05-09): `PUT /v1/resource-definitions/{rd}/volume-definitions/{vn}` already updated the spec; the satellite reconciler now picks up the size delta. Provider interface gained `ResizeVolume` (lvm-thin → `lvextend`, zfs → `zfs set volsize`, file/loopfile → `truncate` + `losetup -c`). On growth the reconciler runs the provider's resize, then `drbdadm resize --assume-clean <rd>` so the kernel re-reads the lower disk. `pkg/luks.Resize` adds the cryptsetup hook for the LUKS layer; satellite-side wiring of LUKS resize lands when the per-resource `encrypted` flag flows through ApplyResources (Phase 6 follow-up). Tests: `TestApplyTriggersResizeOnGrow`, `TestApplyNoResizeOnFreshCreate` for the satellite path; per-provider tests for the resize commands. **End-to-end with a real PVC + checksum verify is still on the e2e harness checklist** (8.8).
407407
- [ ] **Backing-device failure under DRBD**. When the storage layer drops out, the satellite must detach the failed replica (don't oscillate it diskful↔diskless). Today nothing watches `/proc/drbd` for `disk:Failed`. Add an event hook in `Reconciler` that flips the replica to a Diskless attach via `drbdsetup detach` and surfaces a Status condition. e2e: pull the LV out from under DRBD, observe peer stays Primary, no I/O loss.
408408
- [x] **DRBD options hierarchy** controller → resource-group → resource-definition → resource (2026-05-09). `pkg/drbd.ResolveOptions` walks the four scopes, lower wins. The resource controller reads ControllerProps via the KVEntry CRD, the parent RG via `client.Get`, the RD via the existing lookup, then folds in the resource's own props. The merged map flows through `dispatcher.ApplyOptions.EffectiveProps`; `buildDesired` splits it: DrbdOptions/* land on the satellite's drbd_options bag (the .res renderer drops them in the right `net`/`disk`/`peer-device`/`handlers` block via `pkg/drbd.SectionFor`), non-DRBD props stay on the wire-side Props map. Tests: resolver unit tests for override / partial inheritance / non-DRBD-prop pass-through; `TestApplyDRBDOptionsFromEffectiveProps` for the dispatcher wiring.
409-
- [ ] **`allow-two-primaries` + live migration** path. Ganesha-based RWX and KubeVirt VM live-migration both depend on a small window where two nodes are Primary. Today we set `auto-primary` on the lex-lowest replica only; multiple-primary is unsupported. Add the flag plumbing through DesiredResource and the corresponding fence/dual-primary safety on the satellite.
409+
- [x] **`allow-two-primaries` plumbing** (2026-05-09): the DRBD option-hierarchy now flows arbitrary `DrbdOptions/Net/...` keys (including `allow-two-primaries yes`) through the satellite into the rendered .res file's `net { }` block. Operators set the knob via `linstor c sp DrbdOptions/Net/allow-two-primaries yes` (or RG/RD/Resource scope). The first-activation auto-primary seed still picks one replica deterministically (lowest stable node-id) — that's correct: dual-primary is for the consumer's promotion (Ganesha promoter, KubeVirt live-migration controller), not for initial sync. `splitDRBDOptions` strips the `DrbdOptions/<Section>/` prefix so the .res renderer emits `allow-two-primaries yes;` verbatim. **Live-migration coordination on the controller side** (orchestrating `drbdadm primary` on the destination, then `drbdadm secondary` on the source) lives outside this scope — that's what drbd-reactor + the consumer (KubeVirt VirtualMachineInstanceMigration / Ganesha promoter) own.
410410

411411
### 8.3 Replica lifecycle
412412

pkg/satellite/reconciler.go

Lines changed: 48 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ import (
2323
"path/filepath"
2424
"slices"
2525
"strconv"
26+
"strings"
2627
"sync"
2728
"time"
2829

@@ -486,11 +487,14 @@ func buildResFile(dr *satellitepb.DesiredResource, localNode, localAddr string,
486487
})
487488
}
488489

490+
netOpts, resOpts := splitDRBDOptions(opts)
491+
489492
out, err := drbd.Build(drbd.Resource{
490493
Name: dr.GetName(),
491-
Net: drbd.Net{ProtocolC: true},
494+
Net: drbd.Net{ProtocolC: true, Options: netOpts},
492495
Hosts: hosts,
493496
Volumes: vols,
497+
Options: resOpts,
494498
})
495499
if err != nil {
496500
return "", errors.Wrap(err, "drbd.Build")
@@ -499,6 +503,49 @@ func buildResFile(dr *satellitepb.DesiredResource, localNode, localAddr string,
499503
return out, nil
500504
}
501505

506+
// splitDRBDOptions partitions the satellite-received drbd_options bag
507+
// into per-section maps. Per-replica wiring (port/node-id/peer.*.…)
508+
// is dropped — those are not user-tunable knobs. Anything under
509+
// `DrbdOptions/Net/...` lands on the `net { }` block, anything else
510+
// under `DrbdOptions/...` lands on the resource-level `options { }`
511+
// block (drbd's catch-all). The ConfFileBuilder writes the keys
512+
// verbatim with the `DrbdOptions/<Section>/` prefix stripped — that's
513+
// the form drbdadm expects.
514+
func splitDRBDOptions(opts map[string]string) (map[string]string, map[string]string) {
515+
netOpts := map[string]string{}
516+
resOpts := map[string]string{}
517+
518+
for key, value := range opts {
519+
rest, ok := strings.CutPrefix(key, drbd.PropPrefix)
520+
if !ok {
521+
continue
522+
}
523+
524+
section, rawKey, hasSection := strings.Cut(rest, "/")
525+
if !hasSection {
526+
resOpts[rest] = value
527+
528+
continue
529+
}
530+
531+
switch strings.ToLower(section) {
532+
case "net":
533+
netOpts[rawKey] = value
534+
case "disk", "peerdevice", "peer-device", "handlers":
535+
// These sections aren't plumbed through Net struct
536+
// today; surface them as resource-level options so
537+
// drbdadm at least sees them. Full per-section
538+
// emission lands when ConfFileBuilder grows the
539+
// disk/peer-device blocks.
540+
resOpts[rawKey] = value
541+
default:
542+
resOpts[rawKey] = value
543+
}
544+
}
545+
546+
return netOpts, resOpts
547+
}
548+
502549
// resolveAddr substitutes the satellite's own IP whenever the
503550
// controller-supplied address is the placeholder "0.0.0.0" (which it
504551
// is until the controller starts learning each satellite's pod IP and

pkg/satellite/reconciler_drbd_test.go

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -266,3 +266,51 @@ func TestApplyNoResizeOnFreshCreate(t *testing.T) {
266266
}
267267
}
268268
}
269+
270+
// TestApplyRendersAllowTwoPrimaries verifies the option-hierarchy
271+
// pipeline lands `allow-two-primaries yes;` in the generated .res
272+
// file. Required for Ganesha-RWX (NFS export flips Primary on
273+
// failover) and KubeVirt live-migration (both nodes Primary briefly).
274+
func TestApplyRendersAllowTwoPrimaries(t *testing.T) {
275+
dir := t.TempDir()
276+
fx := storage.NewFakeExec()
277+
fx.Expect("lvs --noheadings -o lv_name vg/pvc-rwx_00000",
278+
storage.FakeResponse{Stdout: []byte("")})
279+
280+
thin := lvm.NewThin(lvm.ThinConfig{VolumeGroup: "vg", ThinPool: "tp"}, fx)
281+
rec := satellite.NewReconciler(satellite.ReconcilerConfig{
282+
Providers: map[string]storage.Provider{"thin1": thin},
283+
Adm: drbd.NewAdm(fx),
284+
StateDir: dir,
285+
NodeName: "n1",
286+
})
287+
288+
_, err := rec.Apply(t.Context(), []*satellitepb.DesiredResource{
289+
{
290+
Name: "pvc-rwx",
291+
NodeName: "n1",
292+
Volumes: []*satellitepb.DesiredVolume{
293+
{VolumeNumber: 0, SizeKib: 1024 * 1024, StoragePool: "thin1"},
294+
},
295+
DrbdOptions: map[string]string{
296+
"port": "7000", "node-id": "0", "address": "10.0.0.1", "minor": "1000",
297+
298+
// The controller-side resolver folds this in from
299+
// `linstor c sp DrbdOptions/Net/allow-two-primaries yes`.
300+
"DrbdOptions/Net/allow-two-primaries": "yes",
301+
},
302+
},
303+
})
304+
if err != nil {
305+
t.Fatalf("Apply: %v", err)
306+
}
307+
308+
body, err := os.ReadFile(filepath.Join(dir, "pvc-rwx.res"))
309+
if err != nil {
310+
t.Fatalf("read .res: %v", err)
311+
}
312+
313+
if !strings.Contains(string(body), "allow-two-primaries yes;") {
314+
t.Errorf(".res missing allow-two-primaries; body=%s", body)
315+
}
316+
}

0 commit comments

Comments
 (0)