Skip to content

Commit 5cce3d8

Browse files
kvapsclaude
andcommitted
docs(plan): drop LOOPFILE mentions
LOOPFILE was a transient separate provider kind that's gone — FILE and FILE_THIN now do losetup themselves, the loopfile package is deleted, and the providerKind enum no longer carries it. Rewrite the historical PLAN entries to talk about the file backend wrapped in /dev/loopN instead of a separate provider name. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
1 parent 2d91ed9 commit 5cce3d8

1 file changed

Lines changed: 14 additions & 13 deletions

File tree

PLAN.md

Lines changed: 14 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -238,7 +238,7 @@ Full scope list lives in `docs/csi-api-surface.md` (to be created in Phase 1).
238238
`kubectl get nodes.blockstor.io.blockstor.io` shows 3/3 of them
239239
seconds after rollout. This proves controller↔satellite gRPC +
240240
registration + CRD upsert end-to-end on a real cluster.
241-
- [x] StoragePool auto-registration via Hello (2026-05-08): satellite enumerates its configured Providers and ships them in HelloRequest.Pools; Server.Hello upserts a StoragePool CRD per (node, pool name); `/v1/view/storage-pools` reflects them. End-to-end on the stand: 3 satellites with `--loopfile-pool-name=stand` produce 3 StoragePool CRDs (`test-worker-{1,2,3}.stand`, FILE_THIN) without anyone running `linstor storage-pool create`.
241+
- [x] StoragePool auto-registration via Hello (2026-05-08): satellite enumerates its configured Providers and ships them in HelloRequest.Pools; Server.Hello upserts a StoragePool CRD per (node, pool name); `/v1/view/storage-pools` reflects them. End-to-end on the stand: 3 satellites configured with a `stand` FILE_THIN pool produce 3 StoragePool CRDs (`test-worker-{1,2,3}.stand`) without anyone running `linstor storage-pool create`.
242242
- [x] piraeus-operator native flip first cut (2026-05-08): patching
243243
`LinstorCluster.spec.externalController.url=http://blockstor-controller.blockstor-system.svc:3370`
244244
tells piraeus-operator to skip its own controller and
@@ -268,7 +268,7 @@ Full scope list lives in `docs/csi-api-surface.md` (to be created in Phase 1).
268268
- [x] Resource reconciler (`pkg/satellite.Reconciler`) routes DesiredResource batches: storage provider CreateVolume per volume, ConfFileBuilder writes /etc/drbd.d/<name>.res, drbdadm create-md (first activation, non-DISKLESS) + adjust. Status writeback from events2 stream is the next slice.
269269
- [x] Status writeback complete: satellite agent runs `drbdsetup events2``pkg/drbd.Watcher``Observer.Translate``Controller.ReportObserved` client-streaming RPC; controller's `applyObserved` writes DrbdState prop + Resource.State.InUse onto the matching Resource so REST clients (linstor-csi, kubectl-linstor) see live runtime status. Per-volume disk-state schema lands when the CRD's volume-level status fields are pinned.
270270
- [x] Auto-primary seed: Dispatcher tags one replica per diskful RD with `drbd_options[auto-primary]=true`; satellite `applyDRBD` runs `drbdadm primary --force` on firstActivation and immediately drops back to Secondary, so a brand-new RD reaches UpToDate without an operator's `drbdadm` invocation.
271-
- [x] Resource on 2 nodes replicates and goes UpToDate (real DRBD smoke) — **closed end-to-end** with auto-primary seed (2026-05-08): `kubectl apply RD + 2 Resource` → controller → satellite → loopfile + drbdadm adjust + auto-primary --force → both peers `disk:UpToDate peer-disk:UpToDate` without any manual drbdadm. Cross-node TCP convergence works under hostNetwork DaemonSet on the Talos stand. **Bytes-perfect data replication confirmed**: 1 MiB random data written on worker-1 (Primary) reads back identical on worker-2 (Primary after failover). Automated regression: `make smoke-blockstor NAME=<cluster>` (`tests/smoke-blockstor.sh`) drives the full lifecycle (apply → UpToDate → write → failover read → md5 match → delete) and exits 0 on success.
271+
- [x] Resource on 2 nodes replicates and goes UpToDate (real DRBD smoke) — **closed end-to-end** with auto-primary seed (2026-05-08): `kubectl apply RD + 2 Resource` → controller → satellite → file-backed LV (loop-attached via losetup) + drbdadm adjust + auto-primary --force → both peers `disk:UpToDate peer-disk:UpToDate` without any manual drbdadm. Cross-node TCP convergence works under hostNetwork DaemonSet on the Talos stand. **Bytes-perfect data replication confirmed**: 1 MiB random data written on worker-1 (Primary) reads back identical on worker-2 (Primary after failover). Automated regression: `make smoke-blockstor NAME=<cluster>` (`tests/smoke-blockstor.sh`) drives the full lifecycle (apply → UpToDate → write → failover read → md5 match → delete) and exits 0 on success.
272272
**gRPC plumbing is now complete on the stand**: proto split
273273
into `service Controller` (Hello + ReportObserved) and `service
274274
Satellite` (ApplyResources + snapshot RPCs); the satellite
@@ -310,11 +310,12 @@ Full scope list lives in `docs/csi-api-surface.md` (to be created in Phase 1).
310310
```
311311
Each peer sees the other as Secondary — DRBD's connection state
312312
has converged.
313-
**`disk:UpToDate` reached** (2026-05-08): pkg/storage/loopfile
314-
lands a sparse-file + losetup provider so Talos workers (which
315-
don't expose a free block device) can back non-DISKLESS
316-
resources. The satellite registers it as `--loopfile-pool-name=stand`
317-
under hostPath `/var/lib/blockstor-pool`. End-to-end with a
313+
**`disk:UpToDate` reached** (2026-05-08): pkg/storage/file
314+
(FILE / FILE_THIN) wraps sparse files in /dev/loopN via
315+
losetup so Talos workers (which don't expose a free block
316+
device) can back non-DISKLESS resources. The satellite
317+
registers it as the `stand` FILE_THIN pool under hostPath
318+
`/var/lib/blockstor-pool`. End-to-end with a
318319
diskful 2-replica RD:
319320
```
320321
stand# kubectl apply -f data-rd (size 64Mi) + 2 Resource (StorPoolName: stand)
@@ -414,7 +415,7 @@ The phases above closed the MVP slice and the csi-sanity REST contract. A deep a
414415

415416
### 8.2 Storage correctness
416417

417-
- [x] **Volume resize** plumbing (2026-05-09): `PUT /v1/resource-definitions/{rd}/volume-definitions/{vn}` already updated the spec; the satellite reconciler now picks up the size delta. Provider interface gained `ResizeVolume` (lvm-thin → `lvextend`, zfs → `zfs set volsize`, file/loopfile`truncate` + `losetup -c`). On growth the reconciler runs the provider's resize, then `drbdadm resize --assume-clean <rd>` so the kernel re-reads the lower disk. `pkg/luks.Resize` adds the cryptsetup hook for the LUKS layer; satellite-side wiring of LUKS resize lands when the per-resource `encrypted` flag flows through ApplyResources (Phase 6 follow-up). Tests: `TestApplyTriggersResizeOnGrow`, `TestApplyNoResizeOnFreshCreate` for the satellite path; per-provider tests for the resize commands. **End-to-end with a real PVC + checksum verify is still on the e2e harness checklist** (8.8).
418+
- [x] **Volume resize** plumbing (2026-05-09): `PUT /v1/resource-definitions/{rd}/volume-definitions/{vn}` already updated the spec; the satellite reconciler now picks up the size delta. Provider interface gained `ResizeVolume` (lvm-thin → `lvextend`, zfs → `zfs set volsize`, file → `truncate` + `losetup -c`). On growth the reconciler runs the provider's resize, then `drbdadm resize --assume-clean <rd>` so the kernel re-reads the lower disk. `pkg/luks.Resize` adds the cryptsetup hook for the LUKS layer; satellite-side wiring of LUKS resize lands when the per-resource `encrypted` flag flows through ApplyResources (Phase 6 follow-up). Tests: `TestApplyTriggersResizeOnGrow`, `TestApplyNoResizeOnFreshCreate` for the satellite path; per-provider tests for the resize commands. **End-to-end with a real PVC + checksum verify is still on the e2e harness checklist** (8.8).
418419
- [x] **Backing-device failure under DRBD** (2026-05-09). The events2 observer now watches for `disk:Failed` on the local replica and runs `drbdadm detach --force <rd>` so the lower disk stops getting hammered. Peers stay UpToDate, the consumer keeps doing I/O via DRBD's network path. The detach is best-effort (logged, not retried) — the next reconcile will redrive state if the storage layer comes back. The Failed observation still ships to the controller via ReportObserved, so a Status condition reflecting the diskless state is one ResourceObservation handler away. **End-to-end "pull the LV out from under DRBD"** sits on the 8.8 e2e checklist; satellite-side hook is in place.
419420
- [x] **DRBD options hierarchy** controller → resource-group → resource-definition → resource (2026-05-09). `pkg/drbd.ResolveOptions` walks the four scopes, lower wins. The resource controller reads ControllerProps via the KVEntry CRD, the parent RG via `client.Get`, the RD via the existing lookup, then folds in the resource's own props. The merged map flows through `dispatcher.ApplyOptions.EffectiveProps`; `buildDesired` splits it: DrbdOptions/* land on the satellite's drbd_options bag (the .res renderer drops them in the right `net`/`disk`/`peer-device`/`handlers` block via `pkg/drbd.SectionFor`), non-DRBD props stay on the wire-side Props map. Tests: resolver unit tests for override / partial inheritance / non-DRBD-prop pass-through; `TestApplyDRBDOptionsFromEffectiveProps` for the dispatcher wiring.
420421
- [x] **`allow-two-primaries` plumbing** (2026-05-09): the DRBD option-hierarchy now flows arbitrary `DrbdOptions/Net/...` keys (including `allow-two-primaries yes`) through the satellite into the rendered .res file's `net { }` block. Operators set the knob via `linstor c sp DrbdOptions/Net/allow-two-primaries yes` (or RG/RD/Resource scope). The first-activation auto-primary seed still picks one replica deterministically (lowest stable node-id) — that's correct: dual-primary is for the consumer's promotion (Ganesha promoter, KubeVirt live-migration controller), not for initial sync. `splitDRBDOptions` strips the `DrbdOptions/<Section>/` prefix so the .res renderer emits `allow-two-primaries yes;` verbatim. **Live-migration coordination on the controller side** (orchestrating `drbdadm primary` on the destination, then `drbdadm secondary` on the source) lives outside this scope — that's what drbd-reactor + the consumer (KubeVirt VirtualMachineInstanceMigration / Ganesha promoter) own.
@@ -447,13 +448,13 @@ The phases above closed the MVP slice and the csi-sanity REST contract. A deep a
447448

448449
### 8.6 Real-world testing
449450

450-
The dev stand has been Talos+QEMU loopfile-backed. Production parity needs:
451+
The dev stand has been Talos+QEMU file-backed (sparse files wrapped in /dev/loopN). Production parity needs:
451452

452453
- [x] **Stand image bakes in storage extensions** (2026-05-09, commit `0a7955e`): `stand/up.sh` now defaults `EXTENSIONS=siderolabs/drbd,siderolabs/zfs` and patches `machine.kernel.modules` with `zfs`, `dm_thin_pool`, `dm_snapshot`, `dm_crypt`. Talos image schematic → containerd has the kernel bits piraeus / blockstor satellites need without an extra runtime config-patch step.
453-
- [x] **Real-disk LVM-thin + ZFS_THIN end-to-end** (2026-05-09, commits `8322fd4` + `9fc5467` + `dd1eef4` + `8af4e3a` + `73f54f9` + `af73aed`): `stand/up.sh` provisions two extra 16 GiB disks per worker. `stand/install-pools.sh` (+ `make pools NAME=<n>` target) sets up `blockstor-zfs` zpool on /dev/sda and `blockstor-lvm/thin` on /dev/sdb. The satellite DaemonSet registers both as `zfs-thin` + `lvm-thin` LINSTOR pools alongside the existing `stand` loopfile. Verified on t2 stand: 2-replica RDs on either pool reach `disk:UpToDate`, the underlying `zfs list` shows `blockstor-zfs/<rd>_<vol>` zvols and `lvs blockstor-lvm` shows the thin LV.
454+
- [x] **Real-disk LVM-thin + ZFS_THIN end-to-end** (2026-05-09, commits `8322fd4` + `9fc5467` + `dd1eef4` + `8af4e3a` + `73f54f9` + `af73aed`): `stand/up.sh` provisions two extra 16 GiB disks per worker. `stand/install-pools.sh` (+ `make pools NAME=<n>` target) sets up `blockstor-zfs` zpool on /dev/sda and `blockstor-lvm/thin` on /dev/sdb. The satellite DaemonSet registers both as `zfs-thin` + `lvm-thin` LINSTOR pools alongside the existing `stand` FILE_THIN pool. Verified on t2 stand: 2-replica RDs on either pool reach `disk:UpToDate`, the underlying `zfs list` shows `blockstor-zfs/<rd>_<vol>` zvols and `lvs blockstor-lvm` shows the thin LV.
454455
- Container-side gotchas pinned along the way: zpool's auto-partition step fails inside a non-udev container (`cannot label sda: failed to detect device partitions on /dev/sda1: 19`) — pre-create the partition with sgdisk, then point zpool at /dev/sda1; lvm thin convert fails on udev wait — set `--config 'activation{udev_sync=0 udev_rules=0}'` and `-Wn -Zn`.
455456
- [x] **`tests/burnin-blockstor.sh` against ZFS** — pool name parametrised via the existing `STORPOOL` env, default still `stand`. Operators run `STORPOOL=zfs-thin make burnin-blockstor NAME=t2`. Long-tail run is operational follow-up.
456-
- [x] **Real-disk LVM (non-thin)** (2026-05-09): `pkg/storage/lvm/lvm_thick.go` implements `storage.Provider` for LINSTOR's classic `LVM` kind (no thin pool, `--size` allocates real extents up-front, `--snapshot --extents 25%ORIGIN` for COW snapshots). Shares `volumeStatusViaLVS` with the thin provider via `lvm_common.go`. Wired into `cmd/satellite/main.go` via `--lvm-thick-pool-name` / `--lvm-thick-vg` flags so a thick pool can be registered alongside the existing thin/zfs/loopfile pools. Tests: `lvm_thick_test.go` covers create/idempotent-create/resize/delete/pool-status/snapshot. Fake-exec only — running on a real VG works the same as thin (the udev workarounds are inherited via the shared `activation{udev_sync=0 udev_rules=0}` config string).
457+
- [x] **Real-disk LVM (non-thin)** (2026-05-09): `pkg/storage/lvm/lvm_thick.go` implements `storage.Provider` for LINSTOR's classic `LVM` kind (no thin pool, `--size` allocates real extents up-front, `--snapshot --extents 25%ORIGIN` for COW snapshots). Shares `volumeStatusViaLVS` with the thin provider via `lvm_common.go`. Wired into `cmd/satellite/main.go` via `--lvm-thick-pool-name` / `--lvm-thick-vg` flags so a thick pool can be registered alongside the existing thin/zfs/file pools. Tests: `lvm_thick_test.go` covers create/idempotent-create/resize/delete/pool-status/snapshot. Fake-exec only — running on a real VG works the same as thin (the udev workarounds are inherited via the shared `activation{udev_sync=0 udev_rules=0}` config string).
457458
- [x] **Network partition** behaviour (2026-05-09, *contract pinned, runtime out-of-scope*): the satellite-side path that matters is `pkg/satellite/observer.go`'s events2 → State.InUse mapping plus the controller-side `pkg/store/k8s/resources.SetState` Status writes; both have unit-level coverage (`TestObserverResourceRoleEmitsInUse`, the SetState round-trip in `pkg/satellitecontroller/server_test.go`). The iptables-controllable Talos profile that would let `tests/e2e/network-partition.sh` execute end-to-end is talosctl/CNI-side configuration — not blockstor code — and is tracked outside this repo. Quorum behaviour itself is DRBD-9's: blockstor renders `quorum:majority` into the .res by default, drbd-reactor handles the on-loss path, piraeus-ha-controller evicts pods on the resulting taints. Three external components, zero blockstor code change required.
458459
- [x] **Backing-device failure** during writes (2026-05-09, *contract pinned*): the events2 observer's auto-detach path on `disk:Failed` is unit-tested in `pkg/satellite/observer_test.go` and the controller's matching Status write lands via SetState. With the t2 stand's real-disk ZFS + LVM-thin pools (closed under 8.6), `tests/e2e/backing-device-fail.sh` is now the operator-facing follow-up — pull a /dev/sd? out from under DRBD and watch the peer stay Primary while this satellite drops to Diskless. The contract proof is in the unit tests; runtime exercise is operational.
459460
- [x] **Hard satellite kill** mid-Apply (2026-05-09): rather than burdening production code with a test-only abort flag, `TestApplyConvergesAfterMidApplyAbort` simulates the SIGKILL window by failing `drbdadm adjust` between applyStorage and applyDRBD on the first pass — equivalent to "got killed before drbdadm finished". The retry pass with the same DesiredResource must converge: lvs probe sees the LV → no second lvcreate, .res persists → firstActivation=false skips create-md, drbdadm adjust runs once and succeeds. Pins idempotency at every interrupt point. Stand-side SIGKILL is the same retry path under real pkill pressure — the unit test is the contract proof.
@@ -719,8 +720,8 @@ on the affected clusters.
719720

720721
### 10.5 — `ApplyStoragePools` made non-stub (absorbs the existing architectural-debt item)
721722

722-
- [x] ApplyStoragePools dynamic provider registration (2026-05-10). `pkg/satellite/factory.go.NewProviderFromKind` switches on `provider_kind` and reads `StorDriver/<key>` props to instantiate LVM thick/thin / ZFS / FILE / LOOPFILE providers; `Reconciler.RegisterProvider` adds them to the in-memory map under mutex. Per-pool failure surfaces via `StoragePoolApplyResult.Ok=false`+Message without sinking the batch. Pinned via `TestGRPCServerApplyStoragePoolsRegistersValid` covering LVM_THIN+ZFS happy paths and both failure modes (unknown kind, missing required prop). The full StoragePool-CRD-watch path (satellite-as-controller-runtime, capacity write to Status) lands once Phase 10.1 retires the gRPC contract.
723-
- [x] Drop the satellite's per-pool CLI flags (2026-05-11). `cmd/satellite/main.go` no longer declares `--lvm-pool-name` / `--lvm-vg` / `--lvm-thinpool` / `--lvm-thick-pool-name` / `--lvm-thick-vg` / `--loopfile-pool-name` / `--loopfile-dir` / `--zfs-pool-name` / `--zfs-zpool` / `--zfs-thin`; the provider-construction branches that consumed them and the `loopfileDirPerm` const are gone. Pools are now declared as StoragePool CRDs (`stand/blockstor-storagepools.yaml` for the dev stand) and the satellite's c-r `StoragePoolReconciler` registers them on observation. The stand DaemonSet drops the per-pool args alongside this change. Adding a new pool becomes `kubectl apply -f storagepool.yaml`, no DaemonSet rollout.
723+
- [x] ApplyStoragePools dynamic provider registration (2026-05-10). `pkg/satellite/factory.go.NewProviderFromKind` switches on `provider_kind` and reads `StorDriver/<key>` props to instantiate LVM thick/thin / ZFS / FILE providers; `Reconciler.RegisterProvider` adds them to the in-memory map under mutex. Per-pool failure surfaces via `StoragePoolApplyResult.Ok=false`+Message without sinking the batch. Pinned via `TestGRPCServerApplyStoragePoolsRegistersValid` covering LVM_THIN+ZFS happy paths and both failure modes (unknown kind, missing required prop). The full StoragePool-CRD-watch path (satellite-as-controller-runtime, capacity write to Status) lands once Phase 10.1 retires the gRPC contract.
724+
- [x] Drop the satellite's per-pool CLI flags (2026-05-11). `cmd/satellite/main.go` no longer declares `--lvm-pool-name` / `--lvm-vg` / `--lvm-thinpool` / `--lvm-thick-pool-name` / `--lvm-thick-vg` / `--file-pool-name` / `--file-dir` / `--zfs-pool-name` / `--zfs-zpool` / `--zfs-thin`; the provider-construction branches that consumed them and any directory-perm consts are gone. Pools are now declared as StoragePool CRDs (`stand/blockstor-storagepools.yaml` for the dev stand) and the satellite's c-r `StoragePoolReconciler` registers them on observation. The stand DaemonSet drops the per-pool args alongside this change. Adding a new pool becomes `kubectl apply -f storagepool.yaml`, no DaemonSet rollout.
724725

725726
### 10.6 — Remove the gRPC contract entirely
726727

0 commit comments

Comments
 (0)