Flant cloud#614
Draft
dmgtn wants to merge 963 commits into
Draft
Conversation
…rting The datamesh attach/detach cycle is now fully functional end-to-end. Previously, ReplicatedVolumeAttachment conditions were only set during RV deletion — in normal operation the RVA status was essentially blank, and CSI / users had no visibility into why a volume was not attached. This change closes the loop: the attachment pipeline (slot allocation, multiattach, transition creation, confirmation) now feeds structured condition data into every RVA, so each attachment always reports its real state — Attached, Attaching, Detaching, Detached, Pending with a human-readable explanation (e.g. "Waiting for attachment slot", "Quorum not satisfied", "Device in use, detach blocked"). Key changes to make this work: - Wire reconcileRVAConditionsFromAttachmentsSummary into the normal operation path — maps pipeline output to Attached, ReplicaReady, and Ready conditions plus attachment fields (devicePath, ioSuspended, inUse) on each RVA. - Set waiting conditions on RVAs during datamesh formation and RV deletion — previously RVAs had no conditions in these states. - Fix detach transition stuck forever when RVR is deleted or leaves the datamesh — now confirmed when RVR is absent or has revision 0. - Improve quorum check: account for agent readiness and provide diagnostic detail instead of a bare count. - Add VolumeAccess=Local guard in the pipeline so the RVA explains why attach is impossible on a node without a Diskful replica. - Handle nodes with only deleting RVAs: mark as Detached so finalizer removal proceeds correctly. - Remove status.phase from RVA — redundant now that conditions fully describe the state. - Replace legacy condition reasons (SettingPrimary, LocalityNot- Satisfied, etc.) with pipeline-aligned ones (Attaching, Detaching, Pending, VolumeAccessLocalityNotSatisfied, etc.). Signed-off-by: David Magton <david.magton@flant.com>
Add a guard in reconcileCreateAccessReplicas that stops creating new Access RVRs once the total RVR count reaches 32 (RVRMaxID+1). This prevents hitting the DRBD ID range limit (0-31) and the CRD MaxItems=32 validation at patch time. Signed-off-by: David Magton <david.magton@flant.com>
…tead of rv.Status.DesiredAttachTo - Add getIntendedAttachments helper: lists RVAs via UnsafeDisableDeepCopy index, collects unique node names sorted alphabetically - Pass attachToNodes to computeSchedulingContext instead of reading rv.Status.DesiredAttachTo - Remove unused DesiredAttachTo field from ReplicatedVolumeStatus - Remove unused DRBD/DRBDResourceDetails/DRBDResourceConfig types - Update tests: register RVA index, create RVA objects instead of setting rv.Status.DesiredAttachTo - Update README to reflect new data flow Signed-off-by: David Magton <david.magton@flant.com>
… is nil Add a nil-member guard in ensureDatameshDetachTransitions. When a deleting RVA exists on a node that is not a datamesh member (as.member == nil), the node was never attached — skip it with conditionReason=Detached instead of dereferencing nil. - Add guard for as.member == nil (before existing guards) - Simplify next guard (member != nil check no longer needed) - Add two tests covering blocked and non-blocked paths - Update README detach guard table Signed-off-by: David Magton <david.magton@flant.com>
When an RVA is deleted between the cache read and the status patch, patchRVAStatus returned a NotFound error causing reconciler failures. Wrap the patch call with client.IgnoreNotFound to silently skip already-deleted RVAs. Signed-off-by: David Magton <david.magton@flant.com>
- Update controller purpose descriptions in tables - Add normal operation details for rv_controller (Access replicas, attach/detach, multiattach) - Add RVA input to rvr_scheduling_controller in Volume Lifecycle diagram - Fix RSC→RSP edge to creates/updates in Storage Infrastructure diagram - Remove "(deletion)" qualifier from RVA condition management - List all rvr_controller conditions in dependency chain - Add RVA attach-to scoring mention for rvr_scheduling_controller Signed-off-by: David Magton <david.magton@flant.com>
[rv-controller] Implement RVA-driven attach/detach lifecycle
Signed-off-by: Aleksandr Stefurishin <aleksandr.stefurishin@flant.com>
- Add newControlPlane parameter to ModuleConfig (openapi/config-values.yaml) - Wrap old control-plane resources in `if not newControlPlane`: linstor-controller, linstor-node, linstor-affinity-controller, linstor-scheduler-extender, spaas, metadata-backup, webhooks, oldcontroller, agent, drbd NodeGroupConfigurations - Add new control-plane templates: controller, agent, webhooks, CSI - Restructure CSI: move csi-driver to csi, add volume-snapshot-class - Add images/oldcontroller for legacy control-plane implementation Signed-off-by: Pavel Karpov <pavel.karpov@flant.com>
Signed-off-by: Pavel Karpov <pavel.karpov@flant.com>
Switch control-plane v3 Signed-off-by: Aleksandr Zimin <alexandr.zimin@flant.com>
… deletion bugs E2E framework and test suite: - envtesting: test lifecycle framework with options, cleanup, parallel runs - kubetesting: Kubernetes helpers for resource setup, patching, watchers - DRBDResource test suite: R1 single replica (maintenance, state-down), R2/R3/R4 multi-replica peering with promote/demote/remove-peer - Supporting libs (api/list, sync/wait), hack/run-e2e.sh, Cursor skill Agent bug fixes: - Fix LLV finalizer leak: release finalizer before DRBD convergence (phase 3b) while status still carries the attached LLV name - Fix resize comparison: account for DRBD metadata overhead when comparing intended vs actual size; treat backing-not-grown as no-op - Fix NotFound panic: reconcileFinalizer returns Done when the DRBDResource is deleted between read and patch - Remove test-only SharedSecretAlgDummyForTest from API types Signed-off-by: Aleksandr Stefurishin <aleksandr.stefurishin@flant.com>
New cluster-scoped CRD DRBDMapper creates a two-layer dm-linear stack on a given node: an internal layer mapping the specified lower device (e.g. DRBD) and a stable upper layer exposed to users. The two-layer design allows suspending user IO on the upper layer while the internal layer is manipulated (e.g. for DRBD kernel module updates). Key components: - api/v1alpha1: DRBDMapper types, Configured condition, deepcopy - images/agent/internal/controllers/drbdm: controller, reconciler, predicates (node filtering), udev scanner (open count tracking) - images/agent/pkg/dmsetup: create/remove/info wrappers for dmsetup - lib/go/common/blksize: BLKGETSIZE64 ioctl (replaces blockdev binary) - lib/go/common/udev: netlink-based block event monitor (no udevadm) Deletion is refused while the upper device has openers (OpenCount > 0). Signed-off-by: Aleksandr Stefurishin <aleksandr.stefurishin@flant.com>
Replace the volatile /dev/drbd{minor} path in drbdr.status.device with
a stable symlink at /dev/sdsrv/{drbdr.Name}. After a node restart DRBD
resources may receive different minors, but the symlink path stays the
same, keeping downstream references (RVR, RVA, CSI driver) valid.
- Add EnsureDeviceSymlinkAction and RemoveDeviceSymlinkAction
- Insert symlink actions into bring-up and teardown sequences
- Report symlink path in status instead of raw /dev/drbd{minor}
- Clean up symlinks on orphan DRBD resource teardown
- Mount /dev/sdsrv/ as emptyDir in agent DaemonSet
- Add SymlinkFailed condition reason
- Update tests for new command sequences and symlink verification
Signed-off-by: Aleksandr Stefurishin <aleksandr.stefurishin@flant.com>
reconcileLLVFinalizerRelease compared attachedLLVName (from status) with intendedLLVName (from spec.LVMLogicalVolumeName). On deletion the spec doesn't change, so both values stayed equal and the function returned early — the LLV finalizer was never released. Fix: zero intendedLLVName when the DRBDResource is being deleted (DeletionTimestamp set, agent is the last finalizer holder). state=Down intentionally keeps the LLV finalizer since the resource may come back Up. Also: - Extract isBeingDeleted / isBeingDeletedOrDown helpers to unify the deletion+down check across reconcileFinalizer, reconcileLLVFinalizerRelease, and computeIntendedDRBDState. - Add DeleteDiskful e2e test that deletes a diskful DRBDResource directly and asserts the agent releases the LLV finalizer. - Fix pod log watcher for multi-container pods (add container field). Signed-off-by: Aleksandr Stefurishin <aleksandr.stefurishin@flant.com>
Phase 4 (computeIntendedDRBDState) and Phase 6 (reconcileFinalizer) each independently computed isBeingDeletedOrDown from the same drbdr pointer. Phase 5's status patch (patchDRBDRStatus) updates drbdr metadata in-place with the API server response, which may include a DeletionTimestamp and updated finalizers that arrived after the cache read. This caused Phase 4 to take the bring-up path (no DownAction) while Phase 6 removed the agent finalizer, orphaning the DRBD resource on the node. Fix: snapshot isBeingDeleted/isBeingDeletedOrDown once after Phase 1 and thread the values through all subsequent phases, so they all see a consistent view regardless of Phase 5 metadata mutations. Signed-off-by: Aleksandr Stefurishin <aleksandr.stefurishin@flant.com>
…on-drbdr-delete
Fix: drbdsetup error matching used kernel error codes (104, 111, 158, etc.) as process exit codes. These codes only appear in stderr text; the actual exit codes are 10/11/17/20. Error matching now checks both exit code and output substring (e.g., exit 10 + "(111)"). Fix: matched sentinel errors replaced the original error, losing diagnostic context. Now use errors.Join to preserve both. Merge drbdsetup, drbdmeta, and drbdutils into a single drbdutils package. Each command is self-contained in one file with args builder, known errors table, and execute function. Remove obsolete drbdadm and drbdconf packages (superseded by direct drbdsetup/drbdmeta calls). Split Cmd interface: CombinedOutputRunner (CombinedOutput + String) for executeCommand, full Cmd adds StdoutPipe/Start/Wait for streaming. Rewrite DRBDSETUP COMMAND CALLER'S GUIDE with per-command exit codes, error patterns, and option syntax rules. Signed-off-by: Aleksandr Stefurishin <aleksandr.stefurishin@flant.com>
reconcileLLVFinalizerRelease only released the finalizer from the LLV tracked in status.ActiveConfiguration.LVMLogicalVolumeName (the actually attached disk). But reconcileLLVFinalizerAdd (Phase 3) adds the finalizer based on spec.LVMLogicalVolumeName — before the disk is attached. If the DRBDResource is deleted between "finalizer added" and "disk attached and reported in status", the release function never sees the LLV name and the finalizer is stuck. Fix: - When in cleanup, also release the finalizer from spec.LVMLogicalVolumeName (not just from the status-tracked LLV). - Skip Phase 3 (finalizer add) when in cleanup — no point acquiring a finalizer on an LLV that is about to be released. Signed-off-by: Aleksandr Stefurishin <aleksandr.stefurishin@flant.com>
…e-on-attach-failure
- Change drbd-utils source from LINBIT/drbd-utils to deckhouse/3p-drbd-utils in agent and drbd-reactor images - Bump DRBD_UTILS version to 9.31.0-flant.1 (v9.31.0 + Deckhouse patches: quorum-dynamic-voters, non-voting disk) Signed-off-by: David Magton <david.magton@flant.com>
Add plumbing for two new DRBD options from the Flant kernel extension (9.2.16-flant.1): - quorum-dynamic-voters (resource-level, default: yes) — gates the existing "last man standing" voter reduction, making it toggleable. - non-voting (disk-level, default: no) — excludes a diskful node from quorum voting while still replicating data. Changes span seven layers: - pkg/drbdsetup/caps.go: detect Flant extensions at startup by parsing /proc/drbd version string. - pkg/drbdsetup/resource_options.go, disk_options.go: command builders. - pkg/drbdsetup/show.go: JSON parser for drbdsetup show output. - pkg/drbdconf/v9/section_options.go, section_disk_options.go: config file types. - drbdr/actual_drbd_state.go: read options from actual DRBD state. - drbdr/intended_drbd_state.go: intended state interfaces with DRBD defaults (true / false). - drbdr/drbd_actions.go: action structs wired to drbdsetup execution. - drbdr/target_drbd_state.go: target computation with two-tier gating — options are always passed when Flant extensions are available; on non-Flant kernels they are passed only when the intended value differs from the DRBD built-in default, letting drbdsetup surface the error instead of silently ignoring the setting. No reconciler business-logic changes; hardcoded defaults match upstream DRBD behavior. Signed-off-by: David Magton <david.magton@flant.com>
Upstream DRBD unconditionally excludes outdated and quorumless nodes from the voter set when no nodes are in D_UNKNOWN state (the "last man standing" optimization). This can lead to quorum being retained by a single remaining node even when the cluster has genuinely lost majority, which undermines the safety guarantees that quorum is supposed to provide. The Flant kernel fork makes this behavior configurable via the quorum-dynamic-voters resource option (default: yes, matching upstream). When disabled, all known nodes always count as voters, enforcing strict majority-based quorum. Set the intended quorum-dynamic-voters to "no" on Flant kernels so that quorum decisions are always based on strict majority. On non-Flant kernels the option is not passed — the upstream behavior is preserved as a transitional measure until the Flant DRBD module is deployed. Signed-off-by: David Magton <david.magton@flant.com>
Add the non-voting disk option to the DRBDResource API and wire it through the agent so the reconciler applies it via drbdsetup disk-options. A non-voting disk replicates data normally but excludes itself from quorum voting. Combined with allow-remote-read=no on peer connections, it provides full two-sided quorum exclusion for shadow replicas. API changes: - DRBDResourceSpec: add nonVoting bool (default false). - CEL validation: nonVoting must be false when type is Diskless. - DRBDResourceActiveConfiguration: add nonVoting *bool for status reporting. Agent changes: - intended_drbd_state.go: read nonVoting from spec instead of hardcoded false. - actual_drbd_state.go: report nonVoting from drbdsetup show volume disk options into ActiveConfiguration. Signed-off-by: David Magton <david.magton@flant.com>
Add name collision check to the RSC validating webhook: on CREATE, if a StorageClass with the same name already exists with a different provisioner, the request is rejected with a descriptive message. - Add SC name check in RSCValidate (GET by name, reject if foreign) - Add ClusterRole + ClusterRoleBinding for webhooks SA (storageclasses get, nodes list) - Add builder to TestSC (Provisioner, buildObject with stampMetadata) - Add e2e test: rejects RSC when foreign SC with same name exists Signed-off-by: David Magton <david.magton@flant.com>
SatisfyEligibleNodes=False (NodeMismatch, LVMVolumeGroupMismatch, ThinPoolMismatch) no longer bumps a member RVR to PartiallyDegraded. The eligible-node mismatch note is still appended to the status message for all member health phases, including Healthy and Progressing. Signed-off-by: David Magton <david.magton@flant.com>
- .status.phase: RVR, RVA, RSC, RSP, DRBDResourceOperation, DRBDNodeOperation - .spec.nodeName: DRBDResource, DRBDNodeOperation - .spec.drbdResourceName: DRBDResourceOperation Enables server-side field selectors (KEP-4358, GA in K8s 1.32) on frequently queried fields. RVR and RVA already had selectableFields for .spec.nodeName and .spec.replicatedVolumeName; this extends coverage to phase and adds selectors to DRBD-layer objects. Signed-off-by: David Magton <david.magton@flant.com>
Add replicated-volume and replicated-storage-class labels to ReplicatedVolumeAttachment objects, matching the RVR labeling pattern. Rename reconcileRVAFinalizers to reconcileRVAMetadata to reflect the broader metadata management scope. Remove the unused NodeNameLabelKey constant and a dead comment claiming it was managed by rvr_scheduling_controller (it was never set by any controller). Signed-off-by: David Magton <david.magton@flant.com>
The attachment dispatcher checked member.Attached to decide whether a
volume was detached, but applyDetach sets member.Attached=false at
transition creation time — before the transition is confirmed. The
dispatch phase ran after settle and NoDispatch("Detached") overwrote
the correct "Detaching" status that settle had written.
Skip the replica in the dispatcher when an attachment transition is
still active — settle already set the authoritative slot status.
Signed-off-by: David Magton <david.magton@flant.com>
The attachment dispatcher checked member.Attached to decide whether a
volume was detached, but applyDetach sets member.Attached=false at
transition creation time — before the transition is confirmed. The
dispatch phase ran after settle and NoDispatch("Detached") overwrote
the correct "Detaching" status that settle had written.
Skip the replica in the dispatcher when an attachment transition is
still active — settle already set the authoritative slot status.
Signed-off-by: David Magton <david.magton@flant.com>
…DeletionTimestamp Previously, computeRVRPhaseAndMessage returned Terminating as soon as DeletionTimestamp was set, losing all health information while the replica was still an operational datamesh member (serving I/O, participating in quorum). Now the phase logic has three layers: - rvrShouldNotExist (only our finalizer left): Terminating with comprehensive cleanup message combining DRBDConfigured and BackingVolumeReady progress (previously BackingVolumeReady was lost). - Normal phase computation: runs unchanged when DeletionTimestamp is set but the replica is still operational. - Deletion note: appended to the normal phase message when DeletionTimestamp is set but rvrShouldNotExist is false, sourced from the Configured condition (datamesh leave progress) or a generic "Deletion pending" fallback. Signed-off-by: David Magton <david.magton@flant.com>
[agent,controller] Add StorageClass lifecycle management, fix resize/termination/detach bugs, add selectableFields
Diskless peers do not participate in resync, so DRBD's default bitmap tracking (bitmap=yes) causes a spurious out-of-sync counter equal to the full volume size. This adds automated reconciliation of peer-device-options to disable bitmap for Diskless peers, preventing false out-of-sync reports. Key changes: - IntendedPeer: expose Type() from spec.peers[].type - ActualPeer: expose Bitmap() from drbdsetup show peer-device config - New drbdutils.ExecutePeerDeviceOptions wrapper - New PeerDeviceOptionsAction + computePeerDeviceOptionsAction reconciler - Unit tests for the new reconciliation logic Signed-off-by: Aleksandr Zimin <alexandr.zimin@flant.com>
The CSI driver requires the replicated.csi.storage.deckhouse.io/volume-binding-mode parameter in StorageClass to be set, otherwise CreateVolume fails with InvalidArgument. Add it to computeTargetStorageClass so that new StorageClasses get it immediately and existing ones are recreated with the missing parameter on the next reconciliation. Signed-off-by: Aleksandr Zimin <alexandr.zimin@flant.com>
The previous fix (ccbb005) correctly preserved health info for operational datamesh members (datameshRevision > 0) by deferring the Terminating phase until rvrShouldNotExist. However, it introduced a gap for non-member deleting replicas (datameshRevision == 0): they fell through the entire pre-member lifecycle path to the default fallback, producing phase=Configuring with message="Replica is terminating" — a contradictory combination. Add an early Terminating check in computeNormalPhaseAndMessage for replicas with DeletionTimestamp set and datameshRevision == 0. These replicas are not operational (already removed from the datamesh, waiting for finalizer cleanup), so Terminating is the correct phase. Member replicas (datameshRevision > 0) are unaffected and continue through the member health path. Signed-off-by: David Magton <david.magton@flant.com>
Use quantity() function from CEL quantity library for proper numeric comparison of resource.Quantity values. The previous rule used the >= operator directly on an x-kubernetes-int-or-string field, which CEL treats as a lexicographic string comparison (not quantity comparison), because CRDs do not yet support format:quantity mapping to CEL types (kubernetes/kubernetes#130639). This caused "spec.size cannot be decreased" rejections when updating spec.size from e.g. 50Gi to 100Gi, because lexicographically "100Gi" < "50Gi" ('1' < '5'). The permanent validation rejection blocked rvr_controller from updating DRBDResource specs for Diskful replicas, freezing their configuration and cascading into 7 stuck datamesh transitions (resize, multiattach, membership, attachment). Signed-off-by: David Magton <david.magton@flant.com>
Add a guard in computeTargetDRBDRSpec to keep the current spec.size when datamesh.Size is smaller. The DRBD device may be slightly larger than the nominal datamesh size due to sector alignment, and the CRD CEL validation correctly rejects size decreases. This fixes adopt scenarios where a pre-existing DRBDResource was created externally with the aligned (larger) usable size, and the rvr_controller subsequently tried to set the smaller nominal size from datamesh. Also add e2e regression test for the lexicographic Quantity comparison bug (50Mi → 100Mi resize, where "100Mi" < "50Mi" as strings). Signed-off-by: David Magton <david.magton@flant.com>
The rvr_controller may create the LLV (transitioning the RVR to Provisioning phase) before the e2e test observer catches the initial Pending phase. Accept both Pending and Provisioning as the first step in the adopt phase sequence to avoid flaky failures. Signed-off-by: David Magton <david.magton@flant.com>
Redesign require.MinNodes to express node requirements more precisely: MinNodes(diskful) — N nodes with ready LVG (default LVMThin) MinNodes(diskful, extra) — N diskful + M extra nodes (TB/Access) MinNodes(diskful, poolType) — N nodes in a specific pool (e.g. LVM) MinNodes(diskful, extra, pool) — full form The skip check now counts only usable nodes (NodeReady, AgentReady, not Unschedulable) and separately verifies diskful node availability (nodes with ready LVG). Previously MinNodes counted all eligible nodes including unschedulable ones, causing tests to start and then fail inside pickNode when no usable node was available. Add UsableDiskfulNodeCount to PoolScope for the diskful check. Update all ~150 MinNodes call sites with correct diskful/extra counts. Signed-off-by: David Magton <david.magton@flant.com>
When drbdsetup connect returns exit code 10 with error 125 (ERR_NET_CONFIGURED), the connection is already configured — the desired state is achieved. Swallow the error instead of failing with ConnectFailed condition. Rename ErrConnectNeedStandalone → ErrConnectNetConfigured to match the DRBD kernel error name. Signed-off-by: David Magton <david.magton@flant.com>
Hardcode optimized resync controller settings as peer-device-options defaults, applied to all peers on every reconcile: c-plan-ahead = 20 (2.0 s planning horizon) c-delay-target = 1 (100 ms added latency target) c-fill-target = 4096s (2 MB in-flight data) c-max-rate = 625000k (5 Gbps, half of 10G link) c-min-rate = 12500k (100 Mbps minimum forward progress) Tuned for 10 Gbps network with up to 500 DRBD devices per node. - Extend PeerDeviceOptions/PeerDeviceOptionsAction with c-* fields - Add ActualPeer accessors for c-* values from drbdsetup show - Rewrite computePeerDeviceOptionsAction to reconcile resync settings for all peers (not just bitmap for diskless) Signed-off-by: David Magton <david.magton@flant.com>
18 fixes in 3p-drbd since flant.2: drbdsetup down / del-peer / del-connection hangs: - Fix drbdsetup down hang when receiver thread is in connect loop - Fix del_connection hang caused by receiver thread restart race - Fix receiver thread not exiting on _drbd_thread_stop in established connection - Fix del-peer deadlock on blocked-on-al PeerWrite - Fix conn_connect loop not exiting on del_connection - Fall back to force disconnect in drbd_adm_down - Skip sender workqueue flush when sender thread is not running Kernel crashes and deadlocks: - Fix double fput in link_backing_dev error path causing kernel BUG - Reject admin operations on resource being removed - Fix bitmap IO deadlock when IO is suspended due to quorum loss Connectivity: - Fix transport listener race causing rejected connections - Do not force-secondary on transient handshake retry - Skip flush-pending check for resync completion when IO is suspended Resync stability and data integrity: - Fix resync-again triggering on in-flight acks (do_remote guard, skip zero-length resync passes) - Fix was_resync_stable: require L_ESTABLISHED for Primary to be a stable source, cancel stale WFBitMapT states - Fix resync stall after reconnect: clean up stale SENT && !RECEIVED intervals on disconnect - Fix unstable resync completion: re-handshake via UUID_FLAG_RESYNC instead of UUID rotation - Fix data corruption risk when resync finishes with new out-of-sync blocks Signed-off-by: David Magton <david.magton@flant.com>
Fix resize/adopt/termination bugs, enforce resync rate defaults, bump DRBD to flant.3
Resolve merge conflicts in hooks/go/go.mod, hooks/go/go.sum, and images/linstor-drbd-wait/go.mod. Bump go.work to go 1.25.8 and run go mod tidy. Signed-off-by: Aleksandr Zimin <alexandr.zimin@flant.com>
DRBD requires 4Ki-aligned usable sizes. Add AlignTo4Ki() utility and apply it throughout the size pipeline: RV controller (formation, adopt, resize), RVR controller (backing volume size, DRBDR spec.size). Add CEL validation on DRBDResource.spec.size to reject non-aligned values. Key changes: - drbd_size: export AlignTo4Ki(resource.Quantity) with unit tests - drbdr_types: CEL XValidation rule requiring spec.size % 4096 == 0 - rv_controller: align spec.size in formation, adopt, resize dispatch/apply/guard - rvr_controller: align both spec.size and datamesh.size in backing volume and DRBDR spec computation (handles pre-existing unaligned values) - e2e: add resize test with non-4Ki-aligned spec.size - CRD regenerated, controller READMEs updated Signed-off-by: David Magton <david.magton@flant.com>
Signed-off-by: Aleksandr Zimin <alexandr.zimin@flant.com>
Resolve conflicts in 4 go.mod files (hooks/go, images/controller,
images/linstor-drbd-wait, images/webhooks) by taking go 1.25.9 from main.
Unify go directive across the whole workspace to 1.25.9:
- bump remaining go.mod files (api, lib/go/{common,testkit}, e2e/*,
images/{agent,csi-driver,linstor-migrator,megatest,oldcontroller}) and
go.work from 1.25.7/1.25.8 to 1.25.9
- update hack/generate_code.sh and hack/go-mod-tidy to pass -go=1.25.9
so tidy/codegen no longer downgrade the go directive
Signed-off-by: Aleksandr Zimin <alexandr.zimin@flant.com>
Signed-off-by: Vasily Oleynikov <vasily.oleynikov@flant.com> Signed-off-by: Aleksandr Zimin <alexandr.zimin@flant.com>
On rapidly rotating workloads (e.g. GitLab runners) reusing the same PVC
on the same node, ControllerUnpublishVolume returned OK as soon as the
node disappeared from rv.status.actuallyAttachedTo, while the RVA object
itself still carried the rv-controller finalizer. external-attacher then
deleted the VolumeAttachment, and the next ControllerPublishVolume saw a
stale RVA with DeletionTimestamp and failed immediately, surfacing as
"timed out waiting for external-attacher" / "volume attachment is being
deleted".
- Close the race entirely inside the CSI driver:
- Add WaitForRVADeleted helper that polls until the RVA is NotFound,
logging the last observed finalizers/phase for diagnostics.
- EnsureRVA no longer errors on a stale RVA with DeletionTimestamp;
it waits (bounded by ctx) and then recreates a fresh RVA. The
creation path is extracted into a dedicated createRVA helper and
EnsureRVA is rewritten with early returns for clarity.
- ControllerPublishVolume wraps EnsureRVA in a waitActionTimeout
context and maps DeadlineExceeded/Canceled to the proper gRPC codes.
- ControllerUnpublishVolume now waits solely for the RVA object to be
fully gone. DeadlineExceeded bubbles up as codes.DeadlineExceeded so
external-attacher retries with backoff.
- Lower defaultWaitActionTimeout from 5m to 1m so genuinely stuck calls
release the gRPC slot sooner and sidecars retry faster.
- Drop reliance on rv.status.actuallyAttachedTo: the field is flagged
for removal in api/v1alpha1/rv_types.go and is no longer populated by
any controller or agent (verified: zero writers in images/controller
and images/agent; historical writers were removed on refactor branches
vvoyt-scheduler-refactor / vvoyt-move-rsc-controller). Remove the
dead WaitForAttachedToProvided and WaitForAttachedToRemoved helpers
and their unit tests; ControllerUnpublishVolume no longer polls the
field (the RVA-deletion wait replaces it).
Covered by unit tests for WaitForRVADeleted (happy/cancel/deadline),
EnsureRVA with a deferred/stuck stale RVA, and driver-level sanity
tests verifying ControllerUnpublishVolume honors the RVA-deletion wait
and surfaces DeadlineExceeded when the RVA is stuck.
Signed-off-by: Aleksandr Zimin <alexandr.zimin@flant.com>
Signed-off-by: Aleksandr Zimin <alexandr.zimin@flant.com>
CreateVolume used to unconditionally DeleteReplicatedVolume on any error
from WaitForReplicatedVolumeReady. That turns every transient wait failure
(RPC deadline, apiserver hiccup, informer lag) into a destructive action for
already-formed volumes: any CreateVolume retry that lands on a
ReplicatedVolume whose initial Formation has already completed can end up
deleting a healthy RV. Such retries can come from in-flight CSI retries on
a Pending PVC, csi-driver / external-provisioner restarts mid-Provision,
or disaster-recovery flows (etcd / Velero restore) where a PVC is re-queued
in Pending while the backing RV is still past Formation.
Key invariant used: once rv-controller finishes the initial Formation
transition (DatameshRevision > 0 and no Formation transition present),
that state is monotonic - it is only ever reset to zero by a code path that
itself runs while Formation is already active. Export this as a new
predicate utils.IsReplicatedVolumePastFormation and reuse it in
WaitForReplicatedVolumeReady.
Restructure the Create + Wait block in CreateVolume:
- AlreadyExists branch: re-observe the RV once.
* Get error -> codes.Unavailable (provisioner retries).
* DeletionTimestamp != nil -> codes.Aborted.
* IsReplicatedVolumePastFormation -> return Success immediately, no Wait,
no Delete.
* Still forming -> fall through into the common Wait
block.
- Fresh-create branch (Create returned nil): fall through into the common
Wait block.
- Common Wait block: on Wait failure, re-observe the RV once and:
* PastFormation && no DeletionTimestamp -> race win (rv-controller
committed Formation between Wait's last poll and this cleanup); return
Success and keep the early RVA, same as the happy path.
* DeletionTimestamp != nil -> someone else is deleting the
RV; skip Delete to avoid fighting.
* NotFound -> already gone; nothing to do.
* Transient Get error -> skip Delete (cannot decide
safely).
* Otherwise -> uncommitted garbage; delete
the RV so the next CreateVolume retry can recreate it cleanly. The RV
carries no committed data yet (Formation never finished), so deletion
is safe regardless of whether this RPC or a previous one created it.
In all Wait-failure sub-cases that return an error, the WFFC-selected
early RVA is cleaned up; in the race-win success sub-case it is kept,
matching the happy path.
This both stops the destructive behaviour for already-formed RVs and gives
the common Wait block a full waitActionTimeout per RPC regardless of which
branch we arrived via.
Tests:
- Unit tests for IsReplicatedVolumePastFormation (nil, revision=0, Formation
in progress at various revisions, unrelated transitions).
- Driver-level tests covering all AlreadyExists branches (past-formation
fast path, still-forming with Wait success, still-forming with Wait
failure deleting the garbage RV, being-deleted returns Aborted),
fresh-create branches (happy path, Wait-failure garbage collection,
DeletionTimestamp race preserves the RV), and the race-win case where
Formation commits between Wait deadline and post-Wait re-observe
(deterministically simulated via a Get interceptor) and CreateVolume
returns Success without deleting the RV.
Signed-off-by: Aleksandr Zimin <alexandr.zimin@flant.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a pull request for building a temporary version that we are testing in our internal cloud.