test(e2e): scaffold 12 deterministic e2e scenarios (Phase 8.8)

Each scenario captures one Phase 8 invariant as an automatable
shell script with explicit FAIL exits. Shared lib.sh keeps each
scenario focused on the test logic rather than boilerplate.

Scenarios cover:
- evacuate: node evacuate triggers replica migration
- network-partition: 1-vs-2 partition, fence + bitmap merge on heal
- backing-device-fail: events2 observer auto-detach on disk:Failed
- snapshot-restore-cross-node + clone: data-preserving restore on a
  different node pair
- resize-luks + resize-plain: provider + cryptsetup + drbdadm chain
- two-primaries-live-migration: allow-two-primaries plumbing in .res
- rwx-ganesha + two-volume-rd: multi-volume RD shape + drbd-reactor hooks
- tiebreaker: ResourceDefinitionReconciler witness creation
- auto-diskful: ResourceReconciler DISKLESS → diskful promotion

`make e2e NAME=<cluster> SCENARIO=<name>` runs one; `make e2e-list`
enumerates them. Stand-side runs and any failures the scenarios
surface are tracked under Phase 8.6/8.7.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

Author: kvaps

PLAN.md

@@ -431,13 +431,7 @@ The phases above closed the MVP slice and the csi-sanity REST contract. A deep a
   - `piraeus-ha-controller` ([upstream](https://github.com/piraeusdatastore/piraeus-ha-controller)) — separate Kubernetes-aware controller. Watches Pod / Node / LINSTOR-resource state and does the cluster-aware actions: pod eviction on DRBD `StandAlone`, force-detach when a backing device fails, mount-node migration when the consumer Pod reschedules. It dials the LINSTOR REST API as a client. Verify against blockstor: end-to-end run with piraeus-ha-controller against our REST surface, document which endpoints it hits, fix any gaps. No code changes from blockstor expected unless an endpoint shape diverges from upstream; the audit needs to happen on the dev stand.
 - [x] **`linstor advise`** (2026-05-09): `GET /v1/view/advise/resources` and `GET /v1/resource-definitions/{rd}/advise` return per-RD recommendations (top-N pools by free capacity, sorted desc) without persisting anything. Surfaces a `Conflict` string when the request can't be satisfied so the CLI prints it. Tests: `TestAdviseRD`, `TestAdviseRDInsufficient`.
 - [x] **`linstor query-size-info` / spaceinfo** (2026-05-09): `POST /v1/resource-groups/{rg}/query-size-info` answers `max_vlm_size_in_kib = FreeCapacity of the n-th-largest pool` (n = place_count) — the cap that all replicas can fit at once, the value golinstor's pre-flight uses. `POST /v1/query-all-size-info` returns the per-RG map in one shot. EVICTED/LOST nodes excluded from capacity. Tests: 3 cases covering happy path, exhausted, and the cluster-wide aggregate.
-- [ ] **shared LUN provider (EXOS / SHARED) — deferred, not out-of-scope** (2026-05-09 reclassified). The SAN-shared-LUN model isn't needed today, but the architecture should leave room to slice volumes over a shared LUN later (e.g. multiple satellites carving an EXOS / NetApp / Ceph-RBD-as-shared-disk LUN). Upstream LINSTOR's accounting trick: a single "shared free-space" key keyed off the LUN identifier — multiple StoragePool CRDs on different nodes report the same shared key, the controller subtracts allocations once instead of summing per-node. Architectural hooks needed:
-  - StoragePool gains an optional `SharedSpaceID` field (free-form key; empty = local pool)
-  - the placer's free-capacity / query-size-info paths consult shared-space groups: pools sharing an ID contribute one free-capacity figure to the cluster total instead of summing
-  - storage.Provider gets a notion of "this pool is backed by a LUN identified by X" so the satellite knows not to double-count
-  - autoplace anti-affinity respects shared-space groups: don't place 2 replicas on pools that physically share the same backing LUN
-
-  Current behaviour stays as-is (501 on EXOS/REMOTE_SPDK kinds in pool create); that's a placeholder, not a design statement.
+- [x] **shared LUN provider (EXOS / SHARED) architectural hooks** (2026-05-09): `StoragePool` (CRD spec + REST shape) gains an optional `SharedSpaceID` field (empty = local pool). `pkg/placer` tracks `sharedSeen` alongside the existing `diffSeen` and rejects any candidate whose `SharedSpaceID` matches an already-placed replica's pool — a 2-replica RD will never land on two pools that physically share the same backing LUN. `pkg/rest/query_size_info.dedupShared` collapses pool list to one representative per shared LUN before computing capacity totals and the n-th-largest free figure, so two satellites each "seeing" 1000 KiB of the same LUN contribute 1000, not 2000. `pkg/rest/advise` runs the same dedup so recommendations match what the placer would actually accept. Tests: `TestAutoplaceSharedLUNAntiAffinity`, `TestAutoplaceSharedLUNExhausted`, `TestQuerySizeInfoSharedLUN`. The actual SAN-attached-LUN provider implementation is still a follow-up — when it lands, it'll set `SharedSpaceID` on its `StoragePool` CRDs and the placer/advice/query-size-info paths will already do the right thing.
 
 ### 8.6 Real-world testing
 
@@ -461,18 +455,19 @@ The dev stand has been Talos+QEMU loopfile-backed. Production parity needs:
 
 `tests/burnin-blockstor.sh` covers the 2-replica failover happy path. The remaining scenarios above each need a deterministic, automatable test in `tests/e2e/`:
 
-- [ ] `tests/e2e/evacuate.sh` — node evacuate triggers replica migration
-- [ ] `tests/e2e/network-partition.sh` — partition + heal with no full re-sync
-- [ ] `tests/e2e/backing-device-fail.sh` — pull disk, expect graceful diskless-failover
-- [ ] `tests/e2e/snapshot-restore-cross-node.sh`
-- [ ] `tests/e2e/clone.sh`
-- [ ] `tests/e2e/resize-luks.sh`
-- [ ] `tests/e2e/resize-plain.sh`
-- [ ] `tests/e2e/two-primaries-live-migration.sh`
-- [ ] `tests/e2e/rwx-ganesha.sh`
-- [ ] `tests/e2e/two-volume-rd.sh`
-- [ ] `tests/e2e/tiebreaker.sh`
-- [ ] `tests/e2e/auto-diskful.sh`
+- [x] **e2e harness scaffolded** (2026-05-09): all 12 scenarios committed under `tests/e2e/`, plus a shared `lib.sh` (on_node, wait_uptodate, write_random, read_md5, delete_rd, require_workers, rd_apply). `make e2e NAME=<cluster> SCENARIO=<name>` invokes one; `make e2e-list` enumerates them. Each script captures the scenario's expected state transitions with explicit FAIL exits — running them on the dev stand validates that the corresponding controller/satellite path actually exercises end-to-end. Stand-side execution (and any failures the scenarios surface) tracked in 8.6 / 8.7 follow-up.
+  - `tests/e2e/evacuate.sh` — node evacuate triggers replica migration
+  - `tests/e2e/network-partition.sh` — partition + heal with no full re-sync
+  - `tests/e2e/backing-device-fail.sh` — pull disk, expect graceful diskless-failover
+  - `tests/e2e/snapshot-restore-cross-node.sh`
+  - `tests/e2e/clone.sh`
+  - `tests/e2e/resize-luks.sh`
+  - `tests/e2e/resize-plain.sh`
+  - `tests/e2e/two-primaries-live-migration.sh`
+  - `tests/e2e/rwx-ganesha.sh`
+  - `tests/e2e/two-volume-rd.sh`
+  - `tests/e2e/tiebreaker.sh`
+  - `tests/e2e/auto-diskful.sh`
 
 **Exit criteria for Phase 8**: every checkbox above either lands or is moved to a separately-tracked "explicit out-of-scope" with rationale. Until then "production-ready" is overstating it; what we have is a CSI-compatible REST front-end with a verified happy path.
 

stand/Makefile

@@ -13,7 +13,7 @@ KUBECONFIG := $(WORK_DIR)/kubeconfig
 export TALOSCONFIG
 export KUBECONFIG
 
-.PHONY: stand-help up down reset piraeus oracle smoke smoke-blockstor burnin-blockstor use list stand-clean
+.PHONY: stand-help up down reset piraeus oracle smoke smoke-blockstor burnin-blockstor e2e e2e-list use list stand-clean
 
 stand-help: ## list dev-stand targets
 	@echo "stand targets (use NAME=<cluster>):"
@@ -43,6 +43,13 @@ smoke-blockstor: ## blockstor end-to-end smoke (RD → 2 replicas → bytes-perf
 burnin-blockstor: ## continuous PVC churn for DURATION seconds (default 86400 = 24h) (NAME=<n>, DURATION=<sec>)
 	@./tests/burnin-blockstor.sh "$(WORK_DIR)" "$${DURATION:-86400}"
 
+e2e: ## run a single e2e scenario (NAME=<cluster>, SCENARIO=<basename without .sh>)
+	@test -n "$${SCENARIO}" || (echo "usage: make e2e NAME=<cluster> SCENARIO=<name>"; exit 2)
+	@./tests/e2e/$${SCENARIO}.sh "$(WORK_DIR)"
+
+e2e-list: ## list available e2e scenarios
+	@ls tests/e2e/*.sh | xargs -n1 basename | grep -v '^lib\.sh$$' | sed 's/\.sh$$//'
+
 use: ## print TALOSCONFIG/KUBECONFIG exports for shell eval
 	@echo "export TALOSCONFIG=$(abspath $(TALOSCONFIG))"
 	@echo "export KUBECONFIG=$(abspath $(KUBECONFIG))"

tests/e2e/auto-diskful.sh

@@ -0,0 +1,96 @@
+#!/usr/bin/env bash
+#
+# usage: auto-diskful.sh WORK_DIR
+#
+# Tests Phase 8.3 "auto-diskful" promotion.
+# Setup:
+#   - 2 diskful replicas of an RD on workers 1+2
+#   - 1 DISKLESS replica on worker-3 (cluster-wide attachable pattern)
+# Steps:
+#   1. promote $N3 to Primary so Status.InUse=true on the DISKLESS replica
+#   2. ResourceReconciler sees: InUse + DISKLESS + non-tiebreaker + pool available
+#   3. removes DISKLESS flag, stamps StorPoolName onto Spec.Props
+#   4. satellite reconciler picks up the change, creates the LV / drbdadm attach
+
+set -euo pipefail
+
+WORK_DIR=${1:?work_dir required}
+export KUBECONFIG="$WORK_DIR/kubeconfig"
+
+SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+# shellcheck source=lib.sh
+source "$SCRIPT_DIR/lib.sh"
+
+require_workers 3
+
+RD=e2e-auto-diskful
+N1=test-worker-1
+N2=test-worker-2
+N3=test-worker-3
+
+trap 'delete_rd "$RD"' EXIT
+
+echo ">> apply 2 diskful + 1 DISKLESS"
+cat <<EOF | kubectl apply -f -
+apiVersion: blockstor.io.blockstor.io/v1alpha1
+kind: ResourceDefinition
+metadata: {name: ${RD}}
+spec:
+  volumeDefinitions:
+    - {volumeNumber: 0, sizeKib: 65536}
+EOF
+for n in "$N1" "$N2"; do
+    cat <<EOF | kubectl apply -f -
+apiVersion: blockstor.io.blockstor.io/v1alpha1
+kind: Resource
+metadata: {name: ${RD}.${n}}
+spec:
+  resourceDefinitionName: ${RD}
+  nodeName: ${n}
+  props: {StorPoolName: stand}
+EOF
+done
+
+# Make sure $N3 is NOT a tiebreaker — explicitly diskless without the
+# TIE_BREAKER flag. (ResourceDefinitionReconciler would otherwise add
+# its own witness on $N3 for the 2-diskful → +tiebreaker rule, and it
+# would be exempt from auto-diskful.)
+cat <<EOF | kubectl apply -f -
+apiVersion: blockstor.io.blockstor.io/v1alpha1
+kind: Resource
+metadata: {name: ${RD}.${N3}}
+spec:
+  resourceDefinitionName: ${RD}
+  nodeName: ${N3}
+  flags: ["DISKLESS"]
+EOF
+
+wait_uptodate "$RD" "$N1" "$N2"
+
+# Drive InUse on $N3 by promoting it.
+on_node "$N3" drbdadm primary "$RD"
+
+echo ">> wait 30s for auto-diskful promotion"
+deadline=$(( $(date +%s) + 30 ))
+while (( $(date +%s) < deadline )); do
+    flags=$(kubectl get "resources.blockstor.io.blockstor.io/${RD}.${N3}" \
+        -o jsonpath='{.spec.flags}' 2>/dev/null || true)
+    if [[ "$flags" != *"DISKLESS"* ]]; then
+        break
+    fi
+    sleep 2
+done
+
+if [[ "$flags" == *"DISKLESS"* ]]; then
+    echo "FAIL: auto-diskful did not run (flags still: $flags)"
+    exit 1
+fi
+
+stor=$(kubectl get "resources.blockstor.io.blockstor.io/${RD}.${N3}" \
+    -o jsonpath='{.spec.props.StorPoolName}')
+if [[ -z "$stor" ]]; then
+    echo "FAIL: StorPoolName not stamped on promoted replica"
+    exit 1
+fi
+
+echo ">> AUTO-DISKFUL OK ($N3 promoted: pool=$stor, DISKLESS removed)"

tests/e2e/backing-device-fail.sh

@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+#
+# usage: backing-device-fail.sh WORK_DIR
+#
+# Tests Phase 8.2 "events2 observer auto-detach on disk:Failed".
+# Setup:
+#   - 2-replica RD, both UpToDate
+#   - kill the loop file on Primary so DRBD's lower disk goes Failed
+# Expected:
+#   - events2 observer sees disk:Failed
+#   - satellite runs `drbdadm detach --force` on the Failed replica
+#   - Primary stays Primary via the network leg (peer remains UpToDate)
+#   - I/O continues without errors
+
+set -euo pipefail
+
+WORK_DIR=${1:?work_dir required}
+export KUBECONFIG="$WORK_DIR/kubeconfig"
+
+SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+# shellcheck source=lib.sh
+source "$SCRIPT_DIR/lib.sh"
+
+require_workers 2
+
+RD=e2e-disk-fail
+PRIMARY=test-worker-1
+PEER=test-worker-2
+
+trap 'delete_rd "$RD"' EXIT
+
+echo ">> apply 2-replica RD"
+rd_apply "$RD" "$PRIMARY" "$PEER"
+wait_uptodate "$RD" "$PRIMARY" "$PEER"
+
+DEV=$(device_for_rd "$RD" "$PRIMARY")
+
+echo ">> Primary write 256 KiB before fault"
+md5_pre=$(write_random "$PRIMARY" "$DEV" 262144)
+
+echo ">> simulate backing-device failure on $PRIMARY"
+# loopfile pool puts each volume's image at /var/lib/blockstor-pool/<rd>_NNNNN.img
+# Truncating to 0 + losetup detach makes DRBD see Failed I/O.
+on_node "$PRIMARY" bash -c "
+    losetup -j /var/lib/blockstor-pool/${RD}_00000.img | head -1 | cut -d: -f1 | xargs -r losetup -d
+"
+
+echo ">> wait 30s for events2 observer to detach"
+sleep 30
+
+# Detach success criteria: local disk in {Diskless, Failed}; peer still UpToDate.
+prim_disk=$(on_node "$PRIMARY" drbdsetup status "$RD" 2>/dev/null | grep "disk:" | head -1 || true)
+if [[ "$prim_disk" != *"Diskless"* && "$prim_disk" != *"Failed"* ]]; then
+    echo "FAIL: $PRIMARY disk did not transition (got: $prim_disk)"
+    exit 1
+fi
+
+peer_disk=$(on_node "$PEER" drbdsetup status "$RD" 2>/dev/null | grep "disk:" | head -1 || true)
+if [[ "$peer_disk" != *"UpToDate"* ]]; then
+    echo "FAIL: $PEER disk no longer UpToDate (got: $peer_disk)"
+    exit 1
+fi
+
+echo ">> read on $PRIMARY (network-leg) — md5 must match $md5_pre"
+md5_after=$(read_md5 "$PRIMARY" "$DEV" 262144)
+if [[ "$md5_after" != "$md5_pre" ]]; then
+    echo "FAIL: post-detach read returned different data"
+    exit 1
+fi
+
+echo ">> BACKING-DEVICE-FAIL OK (observer detached, peer covers reads via network)"

tests/e2e/clone.sh

@@ -0,0 +1,78 @@
+#!/usr/bin/env bash
+#
+# usage: clone.sh WORK_DIR
+#
+# Tests CSI clone (volume-from-volume, no intermediate VolumeSnapshot).
+# Same plumbing as snapshot-restore but driven through the CSI shape:
+# the CSI controller-publish-volume path takes a VolumeContentSource of
+# kind Volume → blockstor's snapshot-restore-resource is called by the
+# CSI shim with an auto-generated transient snapshot name.
+#
+# Setup:
+#   - source RD with known data on $N1 + $N2
+#   - call /v1/resource-definitions/{rd}/clone with a target name
+# Expected:
+#   - target RD exists
+#   - target's data == source's data (md5 match)
+
+set -euo pipefail
+
+WORK_DIR=${1:?work_dir required}
+export KUBECONFIG="$WORK_DIR/kubeconfig"
+
+SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+# shellcheck source=lib.sh
+source "$SCRIPT_DIR/lib.sh"
+
+require_workers 2
+
+RD_SRC=e2e-clone-src
+RD_DST=e2e-clone-dst
+N1=test-worker-1
+N2=test-worker-2
+
+trap 'delete_rd "$RD_SRC"; delete_rd "$RD_DST"' EXIT
+
+echo ">> apply source RD"
+rd_apply "$RD_SRC" "$N1" "$N2"
+wait_uptodate "$RD_SRC" "$N1" "$N2"
+
+DEV=$(device_for_rd "$RD_SRC" "$N1")
+RD=$RD_SRC
+md5_src=$(write_random "$N1" "$DEV" 262144)
+
+# blockstor's clone is implemented as snapshot-restore under the hood.
+# A direct CSI clone path (CreateVolume with VolumeContentSource of kind
+# Volume) compiles to: take a transient snapshot of the source, restore
+# into the target, drop the snapshot. We reproduce that here.
+SNAP=clone-$(date +%s)
+
+echo ">> internal: take transient snapshot $SNAP for clone"
+kubectl -n "$NS" exec deploy/blockstor-controller -- \
+    curl -fsS -XPOST -H'Content-Type: application/json' \
+    "http://localhost:3370/v1/resource-definitions/${RD_SRC}/snapshots" \
+    -d "{\"name\":\"${SNAP}\",\"nodes\":[\"${N1}\",\"${N2}\"]}"
+
+echo ">> clone $RD_SRC → $RD_DST"
+kubectl -n "$NS" exec deploy/blockstor-controller -- \
+    curl -fsS -XPOST -H'Content-Type: application/json' \
+    "http://localhost:3370/v1/resource-definitions/${RD_SRC}/snapshot-restore-resource" \
+    -d "{\"to_resource\":\"${RD_DST}\",\"snapshot_name\":\"${SNAP}\"}"
+
+kubectl -n "$NS" exec deploy/blockstor-controller -- \
+    curl -fsS -XPOST -H'Content-Type: application/json' \
+    "http://localhost:3370/v1/resource-definitions/${RD_DST}/autoplace" \
+    -d "{\"select_filter\":{\"place_count\":2,\"storage_pool\":\"stand\"}}"
+
+RD=$RD_DST
+wait_uptodate "$RD_DST" "$N1" "$N2"
+
+DEV_DST=$(device_for_rd "$RD_DST" "$N1")
+md5_dst=$(read_md5 "$N1" "$DEV_DST" 262144)
+
+if [[ "$md5_src" != "$md5_dst" ]]; then
+    echo "FAIL: clone data differs (src=$md5_src dst=$md5_dst)"
+    exit 1
+fi
+
+echo ">> CLONE OK"

tests/e2e/evacuate.sh

@@ -0,0 +1,63 @@
+#!/usr/bin/env bash
+#
+# usage: evacuate.sh WORK_DIR
+#
+# Tests Phase 8.3 "linstor node evacuate actually migrates replicas".
+# Setup:
+#   - 3-node cluster, 2-replica RD on workers 1+2, worker-3 idle
+#   - flag worker-1 EVICTED via REST
+# Expected:
+#   - NodeReconciler creates a 3rd Resource on worker-3 (the migration)
+#   - parent RD now has 3 Resources
+#   - worker-1's Resource still present (operator decides when to drop it)
+
+set -euo pipefail
+
+WORK_DIR=${1:?work_dir required}
+export KUBECONFIG="$WORK_DIR/kubeconfig"
+
+SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+# shellcheck source=lib.sh
+source "$SCRIPT_DIR/lib.sh"
+
+require_workers 3
+
+RD=e2e-evacuate
+SOURCE=test-worker-1
+PEER=test-worker-2
+TARGET=test-worker-3
+
+trap 'delete_rd "$RD"' EXIT
+
+echo ">> apply 2-replica RD ($SOURCE + $PEER)"
+rd_apply "$RD" "$SOURCE" "$PEER"
+wait_uptodate "$RD" "$SOURCE" "$PEER"
+
+echo ">> mark $SOURCE EVICTED"
+kubectl patch nodes.blockstor.io.blockstor.io "$SOURCE" --type=merge \
+    -p '{"spec":{"flags":["EVICTED"]}}'
+
+echo ">> wait up to 60s for migration to $TARGET"
+deadline=$(( $(date +%s) + 60 ))
+while (( $(date +%s) < deadline )); do
+    if kubectl get "resources.blockstor.io.blockstor.io/${RD}.${TARGET}" >/dev/null 2>&1; then
+        break
+    fi
+    sleep 2
+done
+
+if ! kubectl get "resources.blockstor.io.blockstor.io/${RD}.${TARGET}" >/dev/null 2>&1; then
+    echo "FAIL: NodeReconciler did not create replacement on $TARGET"
+    kubectl get resources.blockstor.io.blockstor.io --no-headers | awk -v rd="$RD" '$1 ~ rd'
+    exit 1
+fi
+
+# Source replica must STILL exist — EVICTED is a "drain" semantic, not
+# a "delete". LOST flag is the destructive variant (covered separately
+# by the eviction unit tests).
+if ! kubectl get "resources.blockstor.io.blockstor.io/${RD}.${SOURCE}" >/dev/null 2>&1; then
+    echo "FAIL: source replica on $SOURCE removed prematurely (EVICTED ≠ LOST)"
+    exit 1
+fi
+
+echo ">> EVACUATE OK ($SOURCE+$PEER → migrated to $TARGET, source still present)"