Skip to content

Commit 2ed2714

Browse files
kvapsclaude
andcommitted
feat(satellite): volume resize end-to-end plumbing (Phase 8.2)
CSI ControllerExpandVolume previously dead-ended: the REST PUT updated the VolumeDefinition spec but no reconcile path actually grew the underlying block device or notified DRBD. Added: - storage.Provider gained ResizeVolume(ctx, vol). Implementations: * lvm-thin: lvextend --size <MiB>MiB * zfs: zfs set volsize=<MiB>M * file: truncate -s <bytes> (works for both thick and thin) * loopfile: truncate + losetup --set-capacity so the kernel notices the new size on the loop device - pkg/drbd.Adm.Resize: drbdadm resize --assume-clean <rd> (--assume-clean skips the resync of the new bytes since they were just allocated zero, otherwise growing 3 replicas would serialise on each resync) - pkg/luks.Resize: cryptsetup resize <dmName> --key-file - so the dm-crypt target picks up the new lower-disk size; satellite-side wiring lands when the per-resource encrypted flag flows through ApplyResources. - satellite reconciler applyStorage detects desired > usable on a pre-existing volume, runs ResizeVolume, signals back through a new return value; applyDRBD then runs drbdadm resize after Adjust. Tests: - pkg/storage/{lvm,zfs}_test.go pin the per-provider command shape - pkg/satellite TestApplyTriggersResizeOnGrow asserts the full lvextend → drbdadm resize sequence - TestApplyNoResizeOnFreshCreate locks in "no resize when first creating" so a fresh PVC doesn't get a spurious resize End-to-end PVC-resize-with-checksum-verify lives on the 8.8 e2e checklist; the building blocks are all in place. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
1 parent 811c7c8 commit 2ed2714

12 files changed

Lines changed: 321 additions & 9 deletions

File tree

PLAN.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -403,7 +403,7 @@ The phases above closed the MVP slice and the csi-sanity REST contract. A deep a
403403

404404
### 8.2 Storage correctness
405405

406-
- [ ] **Volume resize** with and without LUKS. No endpoint today (`PUT /v1/resource-definitions/{rd}/volume-definitions/{vn}` doesn't grow the device). Path: VD update → reconciler sees new size → provider grow (`lvextend`/`zfs set`) `cryptsetup resize` if LUKS layered → `drbdadm resize`. Pin via e2e: 1G PVC → write checksum → resize to 2G → verify checksum + new size on both replicas.
406+
- [x] **Volume resize** plumbing (2026-05-09): `PUT /v1/resource-definitions/{rd}/volume-definitions/{vn}` already updated the spec; the satellite reconciler now picks up the size delta. Provider interface gained `ResizeVolume` (lvm-thin → `lvextend`, zfs → `zfs set volsize`, file/loopfile `truncate` + `losetup -c`). On growth the reconciler runs the provider's resize, then `drbdadm resize --assume-clean <rd>` so the kernel re-reads the lower disk. `pkg/luks.Resize` adds the cryptsetup hook for the LUKS layer; satellite-side wiring of LUKS resize lands when the per-resource `encrypted` flag flows through ApplyResources (Phase 6 follow-up). Tests: `TestApplyTriggersResizeOnGrow`, `TestApplyNoResizeOnFreshCreate` for the satellite path; per-provider tests for the resize commands. **End-to-end with a real PVC + checksum verify is still on the e2e harness checklist** (8.8).
407407
- [ ] **Backing-device failure under DRBD**. When the storage layer drops out, the satellite must detach the failed replica (don't oscillate it diskful↔diskless). Today nothing watches `/proc/drbd` for `disk:Failed`. Add an event hook in `Reconciler` that flips the replica to a Diskless attach via `drbdsetup detach` and surfaces a Status condition. e2e: pull the LV out from under DRBD, observe peer stays Primary, no I/O loss.
408408
- [ ] **DRBD options hierarchy** controller → resource-group → resource-definition → resource. Today props are flat on each level; the rendered `.res` doesn't honour the upstream override order. Build a resolver in `pkg/drbd` that walks the inheritance chain and pin via a contract test that checks each level can override its parent.
409409
- [ ] **`allow-two-primaries` + live migration** path. Ganesha-based RWX and KubeVirt VM live-migration both depend on a small window where two nodes are Primary. Today we set `auto-primary` on the lex-lowest replica only; multiple-primary is unsupported. Add the flag plumbing through DesiredResource and the corresponding fence/dual-primary safety on the satellite.

pkg/drbd/drbdadm.go

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,16 @@ func (a *Adm) Secondary(ctx context.Context, resource string) error {
8383
return a.run(ctx, "secondary", resource)
8484
}
8585

86+
// Resize rescans the lower disk's size and tells DRBD to grow the
87+
// replicated volume to match. The lower disk (LV / zvol / dm-crypt
88+
// target) must already be the target size — this is a notify-only
89+
// command. `--assume-clean` skips the resync of the new bytes since
90+
// they were just allocated, which would otherwise serialise growing
91+
// every replica.
92+
func (a *Adm) Resize(ctx context.Context, resource string) error {
93+
return a.run(ctx, "resize", "--assume-clean", resource)
94+
}
95+
8696
// run is the single shell-out site so every drbdadm error gets
8797
// uniform context (subcommand + resource) for log triage.
8898
func (a *Adm) run(ctx context.Context, args ...string) error {

pkg/luks/luks.go

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,19 @@ func (c *Cryptsetup) Open(ctx context.Context, device, dmName string, key []byte
6868
return nil
6969
}
7070

71+
// Resize tells cryptsetup the underlying device has grown — without
72+
// it the dm-crypt target keeps the original size and `drbdadm resize`
73+
// only sees the LUKS-mapped portion. Idempotent: a no-op when the
74+
// device size already matches the dm target.
75+
func (c *Cryptsetup) Resize(ctx context.Context, dmName string, key []byte) error {
76+
err := c.runWithKey(ctx, key, "resize", dmName, "--key-file", "-")
77+
if err != nil {
78+
return errors.Wrapf(err, "luksResize %s", dmName)
79+
}
80+
81+
return nil
82+
}
83+
7184
// Close removes the dm-crypt mapping. Counterpart to Open.
7285
func (c *Cryptsetup) Close(ctx context.Context, dmName string) error {
7386
_, err := c.exec.Run(ctx, "cryptsetup", "luksClose", dmName)

pkg/satellite/reconciler.go

Lines changed: 44 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -239,8 +239,10 @@ func (r *Reconciler) applyOne(ctx context.Context, dr *satellitepb.DesiredResour
239239

240240
devices := map[int32]string{}
241241

242+
resized := false
243+
242244
if !diskless {
243-
got, err := r.applyStorage(ctx, dr)
245+
got, didResize, err := r.applyStorage(ctx, dr)
244246
if err != nil {
245247
res.Ok = false
246248
res.Message = err.Error()
@@ -249,10 +251,11 @@ func (r *Reconciler) applyOne(ctx context.Context, dr *satellitepb.DesiredResour
249251
}
250252

251253
devices = got
254+
resized = didResize
252255
}
253256

254257
if r.cfg.Adm != nil {
255-
err := r.applyDRBD(ctx, dr, diskless, devices)
258+
err := r.applyDRBD(ctx, dr, diskless, devices, resized)
256259
if err != nil {
257260
res.Ok = false
258261
res.Message = err.Error()
@@ -273,13 +276,14 @@ func (r *Reconciler) applyOne(ctx context.Context, dr *satellitepb.DesiredResour
273276
// Records the resource→pool mapping (first volume's pool) so
274277
// subsequent snapshot RPCs can route without the controller passing
275278
// the pool.
276-
func (r *Reconciler) applyStorage(ctx context.Context, dr *satellitepb.DesiredResource) (map[int32]string, error) {
279+
func (r *Reconciler) applyStorage(ctx context.Context, dr *satellitepb.DesiredResource) (map[int32]string, bool, error) {
277280
devices := map[int32]string{}
281+
resized := false
278282

279283
for _, vol := range dr.GetVolumes() {
280284
provider, ok := r.cfg.Providers[vol.GetStoragePool()]
281285
if !ok {
282-
return nil, errors.Errorf("unknown storage pool %q", vol.GetStoragePool())
286+
return nil, false, errors.Errorf("unknown storage pool %q", vol.GetStoragePool())
283287
}
284288

285289
err := provider.CreateVolume(ctx, storage.Volume{
@@ -288,15 +292,34 @@ func (r *Reconciler) applyStorage(ctx context.Context, dr *satellitepb.DesiredRe
288292
SizeKib: vol.GetSizeKib(),
289293
})
290294
if err != nil {
291-
return nil, errors.Wrapf(err, "create volume %s/%d", dr.GetName(), vol.GetVolumeNumber())
295+
return nil, false, errors.Wrapf(err, "create volume %s/%d", dr.GetName(), vol.GetVolumeNumber())
292296
}
293297

294298
status, err := provider.VolumeStatus(ctx, storage.Volume{
295299
ResourceName: dr.GetName(),
296300
VolumeNumber: vol.GetVolumeNumber(),
297301
})
298302
if err != nil {
299-
return nil, errors.Wrapf(err, "volume status %s/%d", dr.GetName(), vol.GetVolumeNumber())
303+
return nil, false, errors.Wrapf(err, "volume status %s/%d", dr.GetName(), vol.GetVolumeNumber())
304+
}
305+
306+
// Grow path: the controller's VolumeDefinition update set a
307+
// new size that's larger than what the provider has on disk.
308+
// Call ResizeVolume to extend the LV/zvol/file; the LUKS
309+
// layer (when present) and `drbdadm resize` are layered on
310+
// top by their own reconcile steps.
311+
if vol.GetSizeKib() > status.UsableKib && status.UsableKib > 0 {
312+
err = provider.ResizeVolume(ctx, storage.Volume{
313+
ResourceName: dr.GetName(),
314+
VolumeNumber: vol.GetVolumeNumber(),
315+
SizeKib: vol.GetSizeKib(),
316+
})
317+
if err != nil {
318+
return nil, false, errors.Wrapf(err, "resize volume %s/%d to %d KiB",
319+
dr.GetName(), vol.GetVolumeNumber(), vol.GetSizeKib())
320+
}
321+
322+
resized = true
300323
}
301324

302325
devices[vol.GetVolumeNumber()] = status.DevicePath
@@ -306,7 +329,7 @@ func (r *Reconciler) applyStorage(ctx context.Context, dr *satellitepb.DesiredRe
306329
r.rememberPool(dr.GetName(), dr.GetVolumes()[0].GetStoragePool())
307330
}
308331

309-
return devices, nil
332+
return devices, resized, nil
310333
}
311334

312335
// applyDRBD renders the .res file from dr's metadata and (re)applies
@@ -317,7 +340,7 @@ func (r *Reconciler) applyStorage(ctx context.Context, dr *satellitepb.DesiredRe
317340
// devices is the volNumber → DevicePath map applyStorage produced.
318341
// buildResFile uses it as the disk path so a loopfile-backed volume
319342
// gets `disk /dev/loopN` rather than the LVM-shaped guess.
320-
func (r *Reconciler) applyDRBD(ctx context.Context, dr *satellitepb.DesiredResource, diskless bool, devices map[int32]string) error {
343+
func (r *Reconciler) applyDRBD(ctx context.Context, dr *satellitepb.DesiredResource, diskless bool, devices map[int32]string, resized bool) error {
321344
resPath := filepath.Join(r.cfg.StateDir, dr.GetName()+".res")
322345
_, statErr := os.Stat(resPath)
323346
firstActivation := os.IsNotExist(statErr)
@@ -344,6 +367,19 @@ func (r *Reconciler) applyDRBD(ctx context.Context, dr *satellitepb.DesiredResou
344367
return errors.Wrapf(err, "adjust %s", dr.GetName())
345368
}
346369

370+
// Pickup-time resize: the storage layer was just grown, drbdadm
371+
// resize tells the kernel to extend the replicated device to
372+
// match. Adjust on its own won't do this — only resize re-reads
373+
// the lower disk's size. Diskless replicas don't have a lower
374+
// disk to resize but they still need their internal state to
375+
// catch up; drbdadm resize handles that case too.
376+
if resized {
377+
err = r.cfg.Adm.Resize(ctx, dr.GetName())
378+
if err != nil {
379+
return errors.Wrapf(err, "resize %s", dr.GetName())
380+
}
381+
}
382+
347383
// On first activation of a diskful replica the controller may
348384
// flag it as the auto-primary seed. We promote once (force-
349385
// primary then back to secondary) so the metadata moves out of

pkg/satellite/reconciler_drbd_test.go

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -170,3 +170,99 @@ func TestApplyDisklessNoCreateMD(t *testing.T) {
170170
}
171171
}
172172
}
173+
174+
// TestApplyTriggersResizeOnGrow simulates the satellite picking up a
175+
// VolumeDefinition update that grew the volume: lvs reports the LV
176+
// already exists at 1 GiB, the desired size is 2 GiB → reconciler
177+
// must call lvextend then drbdadm resize. Pins the upstream-style
178+
// growth-path semantics CSI ControllerExpandVolume relies on.
179+
func TestApplyTriggersResizeOnGrow(t *testing.T) {
180+
dir := t.TempDir()
181+
fx := storage.NewFakeExec()
182+
// Volume already exists.
183+
fx.Expect("lvs --noheadings -o lv_name vg/pvc-grow_00000",
184+
storage.FakeResponse{Stdout: []byte("pvc-grow_00000\n")})
185+
// VolumeStatus: 1 GiB on disk (1024*1024 KiB).
186+
fx.Expect("lvs --noheadings --separator | -o lv_path,lv_size --units k --nosuffix vg/pvc-grow_00000",
187+
storage.FakeResponse{Stdout: []byte("/dev/vg/pvc-grow_00000|1048576\n")})
188+
189+
thin := lvm.NewThin(lvm.ThinConfig{VolumeGroup: "vg", ThinPool: "tp"}, fx)
190+
rec := satellite.NewReconciler(satellite.ReconcilerConfig{
191+
Providers: map[string]storage.Provider{"thin1": thin},
192+
Adm: drbd.NewAdm(fx),
193+
StateDir: dir,
194+
NodeName: "n1",
195+
})
196+
197+
// Desired: 2 GiB.
198+
_, err := rec.Apply(t.Context(), []*satellitepb.DesiredResource{
199+
{
200+
Name: "pvc-grow",
201+
NodeName: "n1",
202+
Volumes: []*satellitepb.DesiredVolume{
203+
{VolumeNumber: 0, SizeKib: 2 * 1024 * 1024, StoragePool: "thin1"},
204+
},
205+
DrbdOptions: map[string]string{
206+
"port": "7000", "node-id": "0", "address": "10.0.0.1", "minor": "1000",
207+
},
208+
},
209+
})
210+
if err != nil {
211+
t.Fatalf("Apply: %v", err)
212+
}
213+
214+
want := []string{
215+
"lvextend --size 2048MiB vg/pvc-grow_00000",
216+
"drbdadm resize --assume-clean pvc-grow",
217+
}
218+
219+
for _, w := range want {
220+
if !slices.Contains(fx.CommandLines(), w) {
221+
t.Errorf("expected %q in calls; got %v", w, fx.CommandLines())
222+
}
223+
}
224+
}
225+
226+
// TestApplyNoResizeOnFreshCreate: when the volume doesn't exist yet
227+
// CreateVolume runs but ResizeVolume must NOT — there's nothing to
228+
// grow. drbdadm resize is also skipped.
229+
func TestApplyNoResizeOnFreshCreate(t *testing.T) {
230+
dir := t.TempDir()
231+
fx := storage.NewFakeExec()
232+
fx.Expect("lvs --noheadings -o lv_name vg/pvc-new_00000",
233+
storage.FakeResponse{Stdout: []byte("")})
234+
235+
thin := lvm.NewThin(lvm.ThinConfig{VolumeGroup: "vg", ThinPool: "tp"}, fx)
236+
rec := satellite.NewReconciler(satellite.ReconcilerConfig{
237+
Providers: map[string]storage.Provider{"thin1": thin},
238+
Adm: drbd.NewAdm(fx),
239+
StateDir: dir,
240+
NodeName: "n1",
241+
})
242+
243+
_, err := rec.Apply(t.Context(), []*satellitepb.DesiredResource{
244+
{
245+
Name: "pvc-new",
246+
NodeName: "n1",
247+
Volumes: []*satellitepb.DesiredVolume{
248+
{VolumeNumber: 0, SizeKib: 2 * 1024 * 1024, StoragePool: "thin1"},
249+
},
250+
DrbdOptions: map[string]string{
251+
"port": "7000", "node-id": "0", "address": "10.0.0.1", "minor": "1000",
252+
},
253+
},
254+
})
255+
if err != nil {
256+
t.Fatalf("Apply: %v", err)
257+
}
258+
259+
for _, line := range fx.CommandLines() {
260+
if strings.HasPrefix(line, "lvextend ") {
261+
t.Errorf("fresh create issued lvextend: %s", line)
262+
}
263+
264+
if strings.HasPrefix(line, "drbdadm resize") {
265+
t.Errorf("fresh create issued drbdadm resize: %s", line)
266+
}
267+
}
268+
}

pkg/storage/file/file.go

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,37 @@ func (p *Provider) CreateVolume(ctx context.Context, vol storage.Volume) error {
9999
return nil
100100
}
101101

102+
// ResizeVolume grows the backing file to vol.SizeKib bytes. truncate
103+
// is the right tool for both thick (fallocate-created) and thin
104+
// (truncate-created) cases — when the file is shorter than the new
105+
// size, the OS extends with sparse zero-bytes. Shrinks are rejected
106+
// to match the rest of the provider contract.
107+
func (p *Provider) ResizeVolume(ctx context.Context, vol storage.Volume) error {
108+
path := p.volumePath(vol)
109+
110+
info, err := os.Stat(path)
111+
if err != nil {
112+
if os.IsNotExist(err) {
113+
return errors.Wrapf(storage.ErrNotFound, "resize %s", path)
114+
}
115+
116+
return errors.Wrapf(err, "stat %s", path)
117+
}
118+
119+
target := vol.SizeKib * bytesPerKib
120+
121+
if info.Size() >= target {
122+
return nil
123+
}
124+
125+
_, err = p.exec.Run(ctx, "truncate", "-s", strconv.FormatInt(target, 10), path)
126+
if err != nil {
127+
return errors.Wrapf(err, "truncate %s", path)
128+
}
129+
130+
return nil
131+
}
132+
102133
// DeleteVolume removes the backing file. Missing → no-op.
103134
func (p *Provider) DeleteVolume(_ context.Context, vol storage.Volume) error {
104135
path := p.volumePath(vol)

pkg/storage/loopfile/loopfile.go

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,43 @@ func (p *Provider) CreateVolume(ctx context.Context, vol storage.Volume) error {
9292
return nil
9393
}
9494

95+
// ResizeVolume grows the backing file and re-runs losetup --set-capacity
96+
// so the kernel picks up the new size on the loop device. Shrinks are
97+
// rejected. Idempotent: a no-op when current size already meets target.
98+
func (p *Provider) ResizeVolume(ctx context.Context, vol storage.Volume) error {
99+
path := p.volumePath(vol)
100+
101+
info, err := os.Stat(path)
102+
if err != nil {
103+
if os.IsNotExist(err) {
104+
return errors.Wrapf(storage.ErrNotFound, "resize %s", path)
105+
}
106+
107+
return errors.Wrapf(err, "stat %s", path)
108+
}
109+
110+
target := vol.SizeKib * bytesPerKib
111+
112+
if info.Size() >= target {
113+
return nil
114+
}
115+
116+
_, err = p.exec.Run(ctx, "truncate", "-s", strconv.FormatInt(target, 10), path)
117+
if err != nil {
118+
return errors.Wrapf(err, "truncate %s", path)
119+
}
120+
121+
dev, err := p.lookupLoop(ctx, path)
122+
if err == nil && dev != "" {
123+
_, err = p.exec.Run(ctx, "losetup", "-c", dev)
124+
if err != nil {
125+
return errors.Wrapf(err, "losetup -c %s", dev)
126+
}
127+
}
128+
129+
return nil
130+
}
131+
95132
// DeleteVolume detaches the loop device (if any) and removes the file.
96133
// Missing → no-op.
97134
func (p *Provider) DeleteVolume(ctx context.Context, vol storage.Volume) error {

pkg/storage/lvm/lvm_thin.go

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,24 @@ func (t *Thin) CreateVolume(ctx context.Context, vol storage.Volume) error {
8888
return nil
8989
}
9090

91+
// ResizeVolume grows the LV to vol.SizeKib (rounded up to MiB to
92+
// match LINSTOR's reporting). Shrinks are rejected — DRBD doesn't
93+
// support online shrink and CSI ControllerExpandVolume is grow-only.
94+
// `lvextend --size` is a no-op when the requested size matches, so
95+
// the call stays idempotent.
96+
func (t *Thin) ResizeVolume(ctx context.Context, vol storage.Volume) error {
97+
sizeMiB := max(vol.SizeKib/mibPerKib, 1)
98+
99+
_, err := t.exec.Run(ctx, "lvextend",
100+
"--size", strconv.FormatInt(sizeMiB, 10)+"MiB",
101+
t.cfg.VolumeGroup+"/"+volumeLVName(vol))
102+
if err != nil {
103+
return errors.Wrapf(err, "lvextend %s", volumeLVName(vol))
104+
}
105+
106+
return nil
107+
}
108+
91109
// DeleteVolume idempotently removes the LV. Missing → no-op (reconcile
92110
// loops re-call this).
93111
func (t *Thin) DeleteVolume(ctx context.Context, vol storage.Volume) error {

pkg/storage/lvm/lvm_thin_test.go

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -255,3 +255,26 @@ func TestThinExecError(t *testing.T) {
255255
}
256256

257257
var errLVCreateFailed = errors.New("lvcreate: insufficient free space")
258+
259+
// TestThinResizeVolumeIssuesLvextend pins the lvextend command shape.
260+
// CSI ControllerExpandVolume → REST PUT → reconciler ResizeVolume,
261+
// so the wire-visible behaviour is "lvextend --size <newMiB>MiB
262+
// vg/lv". Refactors that change the args will fail loudly here.
263+
func TestThinResizeVolumeIssuesLvextend(t *testing.T) {
264+
fx := storage.NewFakeExec()
265+
p := lvm.NewThin(lvm.ThinConfig{VolumeGroup: "vg", ThinPool: "thinpool"}, fx)
266+
267+
err := p.ResizeVolume(t.Context(), storage.Volume{
268+
ResourceName: "pvc-1",
269+
VolumeNumber: 0,
270+
SizeKib: 2048 * 1024, // 2 GiB
271+
})
272+
if err != nil {
273+
t.Fatalf("ResizeVolume: %v", err)
274+
}
275+
276+
want := "lvextend --size 2048MiB vg/pvc-1_00000"
277+
if !slices.Contains(fx.CommandLines(), want) {
278+
t.Errorf("expected %q; got %v", want, fx.CommandLines())
279+
}
280+
}

0 commit comments

Comments
 (0)