Skip to content

Commit c349d79

Browse files
kvapsclaude
andauthored
test(e2e): 2TiB quorum-loss + no-reboot recovery release gate (COV-011) (#146)
* test(e2e): add 2TiB quorum-loss + no-reboot recovery gate (COV-011) Release-gate scenario for the production deadlock fear: quorum loss under active IO wedging a large volume such that only node reboots recover. On a 2 TiB zfs-thick volume (2 diskful + auto-tiebreaker, on-no-quorum=suspend-io) the scenario proves: - bounded-time create (skip-initial-sync contract for thick zvols) - secondary-only outage keeps quorum via the witness, IO continues - losing both the secondary and the witness mid-IO suspends IO (writer blocks with zero errors, no crash) - restoring the links resumes the suspended IO and heals the cluster to all-UpToDate with the seeded marker intact, WITHOUT any node reboot (boot_id + uptime asserted on every worker) Outage mechanism is the per-link iptables DROP of the resource's DRBD mesh port (the drop_pair recipe already proven by quorum-tiebreaker-no-return and the partition scenarios) — the strongest node-outage model reachable from a scenario on this stand, and the only one that breaks kernel replication links while keeping observability up. Stand-"big"-specific: SKIPs unless a zfs-thick (providerKind=ZFS) StoragePool exists on all workers with 2 TiB of headroom. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com> * ci(e2e): allowlist quorum-loss-2tb-recovery SKIP on small stands The COV-011 gate needs the stand-"big" substrate (zfs-thick pool, 2.2T disks) and deliberately SKIPs on the regular CI lanes; without the allowlist entry the runner reclassifies that SKIP as FAIL. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com> --------- Signed-off-by: Andrei Kvapil <kvapss@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
1 parent 210101e commit c349d79

2 files changed

Lines changed: 806 additions & 1 deletion

File tree

stand/run-scenarios-only.sh

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -146,9 +146,12 @@ preflight_check() {
146146
# - resize-pvc: needs a blockstor-side CSI StorageClass, which is not
147147
# shipped yet; it can only exercise piraeus's SC, so it stays an
148148
# explicit xfail until the blockstor CSI provisioner lands.
149+
# - quorum-loss-2tb-recovery: stand-"big"-only (zfs-thick StoragePool
150+
# with providerKind=ZFS + 2 TiB of zpool headroom per diskful node);
151+
# SKIPs by design on the regular CI lanes' small disks.
149152
# EVERY other scenario that skips is a regression (a mandatory test
150153
# silently opting out) and is recorded as FAIL so the lane goes red.
151-
SKIP_ALLOWLIST="backing-device-fail storage-error-injection quorum-loss-recovery recovery-bitmap-drop resize-pvc"
154+
SKIP_ALLOWLIST="backing-device-fail storage-error-injection quorum-loss-recovery recovery-bitmap-drop resize-pvc quorum-loss-2tb-recovery"
152155

153156
scenario_count=${#SCENARIOS[@]}
154157
scenario_idx=0

0 commit comments

Comments
 (0)