Skip to content

Commit eb84258

Browse files
kvapsclaude
andcommitted
test(e2e): replica-add-no-resync gate + mark Phase 8.1 / 10.3 done in PLAN
Adds tests/e2e/replica-add-no-resync.sh as the runtime gate for the GI-seeding pipeline: 1. 2-replica RD on N1+N2 → seed full initial-sync as normal 2. Stage 32 MiB random data on the volume (so an empty-volume fast-path can't mask the bug) 3. Add 3rd replica on N3 4. Assert N3 reaches UpToDate within 60s AND N1/N2 never enter SyncSource state Without seeding the new replica's metadata sees zero GI, the GI handshake mismatches the peer's current_uuid on first connect, DRBD falls through to a full initial-sync of the entire backing device, and the test fails on either timeout or the SyncSource check. With seeding the GI handshake claims "already in sync" and no bytes flow over the wire. Also walks PLAN.md to mark the Phase 8.1 follow-up (initial-sync skip) closed and tick the now-shipped Phase 10.3 sub-bullets: typed DRBDOptions, EncryptionConfig, REST transcoder, and admission-validation enums. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
1 parent 6b6d988 commit eb84258

2 files changed

Lines changed: 131 additions & 10 deletions

File tree

PLAN.md

Lines changed: 5 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -545,15 +545,15 @@ keep `Spec.ExtraProps map[string]string` only as forward-compat for keys
545545
we haven't modelled yet, populated only by the REST shim on the upstream-
546546
LINSTOR boundary.
547547

548-
- [ ] `ResourceDefinition.Spec.DRBDOptions { Net { Protocol, SharedSecret SecretRef, AllowTwoPrimaries bool, MaxBuffers int, AfterSb0pri, AfterSb1pri, AfterSb2pri string }, Disk { OnIOError string, ALExtents int }, Resource { Quorum string (enum majority/off), AutoQuorum string }, Handlers { ... } }` — replaces every `Props["DrbdOptions/<Section>/<Key>"]`. Same hierarchy resolver still walks RG → RD → Resource but on typed fields.
549-
- [ ] `ResourceDefinition.Spec.Encryption.PassphraseSecretRef LocalObjectReference` — replaces `Props["DrbdOptions/Encryption/passphrase"]`. Secret lives in a controller-managed namespace; the satellite reads it via the apiserver on reconcile.
548+
- [x] `ResourceDefinition.Spec.DRBDOptions { Net, Disk, PeerDevice, Resource, Handlers }` typed structs (2026-05-10). Embedded on RG/RD/Resource alongside the existing Props map; `pkg/drbd.ResolveDRBDOptions` walks the same controller→RG→RD→Resource hierarchy with nil-vs-set discipline so an explicit `*bool=false` survives lower-scope inheritance and an empty-string handler key DELETES the inherited entry.
549+
- [x] `ResourceDefinition.Spec.Encryption.PassphraseSecretRef LocalObjectReference` (2026-05-10). The CRD now carries `EncryptionConfig.PassphraseSecretRef`; the satellite already reads the passphrase via the apiserver on reconcile (Phase 6) — this just typifies the slot that previously lived in `Props["DrbdOptions/Encryption/passphrase"]`.
550550
- [ ] `Node.Spec.SatelliteEndpoint string` — replaces `Props["SatelliteEndpoint"]`. (Or removed entirely once 10.6 lands and the endpoint is no longer dialled.)
551551
- [ ] `Node.Spec.DRBDPortRange / DRBDMinorRange { Min, Max int32 }` — replaces `Props["DrbdOptions/TcpPortRange"]` / `"DrbdOptions/MinorNrRange"`. Defaults baked in via kubebuilder.
552552
- [ ] `Node.Spec.AutoTieBreaker *bool` — replaces `Props["DrbdOptions/AutoAddQuorumTiebreaker"]`.
553553
- [ ] `Resource.Spec.StoragePool string` — replaces `Props["StorPoolName"]`. Already half-typed via dispatcher's `buildVolumes`; finish the migration.
554554
- [ ] Topology: drop `Props["Aux/zone"]` / `"Aux/rack"`. Use Kubernetes-native `metadata.labels["topology.blockstor.io/zone"]` etc. Autoplacer reads labels via `client.MatchingLabels`. This is also the upgrade path to label-aware features like `topologySpreadConstraints` later.
555-
- [ ] REST shim transcoder: `/v1/resource-definitions` and `/v1/resource-groups` POST/PUT accept upstream-shaped `{props: {...}}` payloads from golinstor, decode into the typed structs, write `ExtraProps` only for keys we don't recognise. GET responses re-emit the typed fields as `props` on the wire so golinstor stays happy.
556-
- [ ] Validation via kubebuilder enums (`+kubebuilder:validation:Enum=majority;off`)apiserver rejects garbage values at admission, satellite stops parsing strings at runtime.
555+
- [x] REST shim transcoder (2026-05-10). `pkg/store/k8s/drbd_transcode.go` splits the wire `props` bag into `Spec.DRBDOptions` (recognised keys) + `Spec.ExtraProps` (unknown DrbdOptions/* + non-DRBD residual keys); inverse re-emits typed back to `props` on GET. Pinned by 7 unit tests (per-section recognition, unknown-key fallthrough, parse-error fallback, round-trip losslessness, bool spelling tolerance). Controller's `resolveEffectiveProps` now walks typed via `ResolveDRBDOptions` and flattens via `drbd.TypedDRBDOptionsToProps` for the dispatcher → satellite renders the same .res file from typed CRD fields rather than string-keyed Props.
556+
- [x] Validation via kubebuilder enums (2026-05-10). `Net.Protocol`, `Disk.OnIOError`, `Resource.Quorum`, `Resource.OnNoQuorum`, `Net.AfterSb0Pri/1Pri/2Pri` all carry `+kubebuilder:validation:Enum=...` markers; CRD manifests regenerated. Pinned via `internal/controller/drbd_admission_test.go`8 ginkgo specs that submit garbage values against a live envtest apiserver and assert it rejects at admission.
557557

558558
#### 10.3 design summary
559559

@@ -828,12 +828,7 @@ honest. Each item names the original Phase that ticked it.
828828

829829
### Phase 8.1 follow-up — DRBD invariants
830830

831-
- [ ] **Initial-sync skip on replica-add** (currently every 2→3-replica transition triggers a full resync of the whole volume — hours on multi-TiB devices). Upstream LINSTOR persists `DrbdCurrentGi` per (resource, volume) in PropsContainers and stamps a new replica's metadata with the existing peer's GI before `drbdadm up`; DRBD-9's GI handshake then sees the match and skips full sync. Pieces we need:
832-
- `pkg/satellite/observer.go` reads `current-uuid:` and `bitmap-uuid:` from drbdsetup events2 `device` frames (today only `disk:` is parsed → `DiskState`).
833-
- `Resource.Status.Volumes[i].CurrentGi` typed field (observed state, not Spec — written by satellite, read by controller).
834-
- When the controller adds a new replica (autoplacer / explicit place), it picks any existing UpToDate peer's `Status.Volumes[i].CurrentGi` and stamps the new `Resource.Spec.Volumes[i].SeedFromGi` (or one-shot `Spec.AssumeCleanFromPeer NodeName`).
835-
- Satellite reconciler reads `SeedFromGi`, runs `drbdmeta <minor> v09 <device> internal new-current-uuid --clear-bitmap` (or `drbdmeta set-gi`) **before** `drbdadm up` so the freshly-created metadata block carries the seed GI rather than zeros.
836-
- Pin via unit test (FakeExec captures the drbdmeta command line on first activation when SeedFromGi is set) + e2e (`tests/e2e/replica-add-no-resync.sh`: 2-replica RD → write known checksum → add 3rd replica → assert `drbd-overview` shows UpToDate within seconds, not hours).
831+
- [x] **Initial-sync skip on replica-add** (2026-05-10): the full pipeline lands. `pkg/satellite/observer.go` parses `current-uuid:` from `drbdsetup events2 --full`; satellite gRPC carries it via `VolumeObservation.current_uuid`; controller persists it on `Resource.Status.Volumes[i].CurrentGi`; when `ResourceReconciler.runApply` allocates a new replica, `ensureSeedFromGi` picks the lowest-named UpToDate peer's GI and stamps it on `Spec.Volumes[i].SeedFromGi` (deterministic; idempotent; skipped for DISKLESS). Dispatcher threads `SeedFromGi` through `DesiredVolume.seed_from_gi`; satellite's `applyDRBD` runs `drbdmeta --force <res>/<vol> v09 <device> internal set-gi <gi>:<gi>:0:0` between `create-md` and `drbdadm adjust` on first activation. Pinned via FakeExec (`TestAdmSetGiInvokesDrbdmeta`) and reconciler-level ordering (`TestApplyFirstActivationSeedsGiBeforeAdjust` + negative `TestApplyFirstActivationNoSeedSkipsSetGi`). E2E gate `tests/e2e/replica-add-no-resync.sh` scaffolded — 2-replica RD seeded with 32 MiB random data → 3rd replica added → asserts UpToDate within 60s and no SyncSource transition on the existing peers.
837832

838833

839834

tests/e2e/replica-add-no-resync.sh

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
#!/usr/bin/env bash
2+
#
3+
# usage: replica-add-no-resync.sh WORK_DIR
4+
#
5+
# Tests Phase 8.1 "initial-sync skip on replica-add". Setup:
6+
# 1. Create RD with 2 diskful replicas on $N1+$N2; let DRBD do the
7+
# one-time full initial-sync between the seeds.
8+
# 2. Once both UpToDate, write 64 MiB of known-pattern data and
9+
# sync — this stages real bytes the new replica MUST claim to
10+
# have without a resync.
11+
# 3. Add a 3rd diskful replica on $N3.
12+
#
13+
# Without GI seeding (the bug we're guarding against), DRBD-9 sees
14+
# the new replica's metadata at zero GI, mismatches the peer's
15+
# current_uuid on first connect, and triggers a full sync of the
16+
# entire backing device — minutes for 64 MiB, hours on multi-TiB
17+
# real workloads. With seeding (the fix this test pins), DRBD's GI
18+
# handshake recognises the new peer as already-in-sync, the
19+
# resource transitions to UpToDate within the polling window
20+
# without any data ever flowing over the wire.
21+
#
22+
# Exit-criterion: $N3 reports UpToDate within 60s AND the resync
23+
# byte counter on $N1/$N2 stays at zero throughout.
24+
25+
set -euo pipefail
26+
27+
WORK_DIR=${1:?work_dir required}
28+
export KUBECONFIG="$WORK_DIR/kubeconfig"
29+
30+
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
31+
# shellcheck source=lib.sh
32+
source "$SCRIPT_DIR/lib.sh"
33+
34+
require_workers 3
35+
36+
RD=e2e-no-resync
37+
N1=test-worker-1
38+
N2=test-worker-2
39+
N3=test-worker-3
40+
41+
trap 'delete_rd "$RD"' EXIT
42+
43+
echo ">> apply 2-replica RD on $N1+$N2"
44+
cat <<EOF | kubectl apply -f -
45+
apiVersion: blockstor.io.blockstor.io/v1alpha1
46+
kind: ResourceDefinition
47+
metadata: {name: ${RD}}
48+
spec:
49+
props:
50+
DrbdOptions/AutoAddQuorumTiebreaker: "false"
51+
volumeDefinitions:
52+
- {volumeNumber: 0, sizeKib: 65536}
53+
EOF
54+
for n in "$N1" "$N2"; do
55+
cat <<EOF | kubectl apply -f -
56+
apiVersion: blockstor.io.blockstor.io/v1alpha1
57+
kind: Resource
58+
metadata: {name: ${RD}.${n}}
59+
spec:
60+
resourceDefinitionName: ${RD}
61+
nodeName: ${n}
62+
EOF
63+
done
64+
65+
wait_uptodate "$RD" "$N1" "$N2"
66+
67+
# Stage real bytes on the volume. Without these the empty-volume
68+
# resync optimisation in DRBD would mask the bug — we need a
69+
# device that visibly differs from zeros.
70+
echo ">> stage 32 MiB of pattern data on $N1"
71+
on_node "$N1" bash -c "
72+
dev=\$(grep -oE '/dev/drbd[0-9]+' /etc/drbd.d/${RD}.res | head -1)
73+
drbdadm primary --force ${RD}
74+
dd if=/dev/urandom of=\$dev bs=1M count=32 conv=fdatasync 2>&1 | tail -1
75+
drbdadm secondary ${RD}
76+
"
77+
78+
# Snapshot the resync-state counters BEFORE adding the new replica.
79+
# DRBD's `cs:` field stays "Established" when no sync runs; any
80+
# entry into "SyncSource"/"SyncTarget" means a resync started
81+
# (which is what we're testing did NOT happen).
82+
echo ">> baseline cs counters on $N1"
83+
n1_before=$(on_node "$N1" drbdsetup status "$RD" --json | grep -o 'connection-state.*"' | head -1)
84+
echo " $n1_before"
85+
86+
echo ">> add 3rd replica on $N3"
87+
cat <<EOF | kubectl apply -f -
88+
apiVersion: blockstor.io.blockstor.io/v1alpha1
89+
kind: Resource
90+
metadata: {name: ${RD}.${N3}}
91+
spec:
92+
resourceDefinitionName: ${RD}
93+
nodeName: ${N3}
94+
EOF
95+
96+
echo ">> wait up to 60s for $N3 to reach UpToDate"
97+
deadline=$(( $(date +%s) + 60 ))
98+
while (( $(date +%s) < deadline )); do
99+
state=$(on_node "$N3" drbdsetup status "$RD" 2>/dev/null | grep "disk:" | head -1 || true)
100+
if [[ "$state" == *"UpToDate"* ]]; then
101+
break
102+
fi
103+
sleep 2
104+
done
105+
106+
if [[ "$state" != *"UpToDate"* ]]; then
107+
echo "FAIL: $N3 did not reach UpToDate in 60s (state: $state)"
108+
echo " DRBD likely fell through to full initial-sync — GI seeding broken"
109+
on_node "$N3" drbdsetup status "$RD" || true
110+
exit 1
111+
fi
112+
113+
# Cross-check: assert no SyncSource transition happened on the
114+
# existing peers. A full-sync would have left durable evidence in
115+
# events2 history — we use the "events2 --now" snapshot to confirm
116+
# current state shows no syncing.
117+
echo ">> verify no resync triggered on existing peers"
118+
for peer in "$N1" "$N2"; do
119+
cs=$(on_node "$peer" drbdsetup status "$RD" 2>/dev/null | grep -E "(role|connection)" | head -2 || true)
120+
if echo "$cs" | grep -qE "Sync(Source|Target)"; then
121+
echo "FAIL: $peer entered $cs — initial-sync was NOT skipped"
122+
exit 1
123+
fi
124+
done
125+
126+
echo ">> NO-RESYNC OK ($N3 UpToDate within 60s, no resync on $N1/$N2)"

0 commit comments

Comments
 (0)