Version v4.0.1

Author: mweibel

CHANGELOG.md

@@ -1,5 +1,40 @@
 ## unreleased
 
+## v4.0.1 - 2026.06.04
+
+Fixes a LUKS volume handling bug that could leave a volume stuck attached to a node after pod termination, surfacing later as a `Multi-Attach error`
+when the workload was rescheduled. The node plugin now verifies that an existing LUKS mapping is backed by the expected device before reusing it,
+and rejects a staging path that is already mounted from an unexpected source.
+
+In addition, every gRPC handler now uses a request-scoped logger, so log lines include the volume and request context that triggered them.
+
+### Detecting whether you were affected
+
+The bug surfaces in two places, neither of which is the csi-cloudscale logs:
+
+- `kube-controller-manager` logs (or the pod's events):
+  `Multi-Attach error: volume is already exclusively attached and can't be attached to another node`
+- `kubelet` logs on the node listed in `attachedTo=[...]`: repeated
+  `GetDeviceMountRefs check failed for volume ... is still mounted by other references` entries, typically every ~2 minutes.
+
+### Recovering a stuck volume
+
+Upgrading to v4.0.1 prevents new occurrences but does not clean up volumes that are already stuck. To recover one:
+
+1. From the `Multi-Attach error` message, note the affected node listed in `attachedTo=[...]`. Confirm no pod is actively using the
+   volume on that node (`kubectl get pods --all-namespaces -o wide` and check mount refs under `/var/lib/kubelet/pods/`).
+2. Cordon the node so no new workloads land on it during recovery: `kubectl cordon <node>`.
+3. On that node, inspect the stale state. They are the duplicated `.../globalmount` paths that share a `(major, minor)`:
+   - `findmnt | grep globalmount`
+4. `umount` each stale staging path. Once `/proc/self/mountinfo` is clean, kubelet's next `NodeUnstageVolume` retry (within ~2 minutes) succeeds. 
+   The csi-cloudscale node plugin runs `cryptsetup close` itself as part of that unstage, the `VolumeAttachment` is deleted, 
+   and the cloudscale API detaches the volume. The next pod can then attach it elsewhere.
+5. Uncordon the node: `kubectl uncordon <node>`.
+
+If a leftover `/dev/mapper/pvc-<name>` is still present after several minutes, close it manually with `cryptsetup close pvc-<name>` once
+nothing references it. If you cannot identify or clear the stale state safely, rebooting the node is a valid fallback. 
+Kernel state is discarded and the volume detaches on the next reconcile.
+
 ## v4.0.0 - 2026.03.30
 ⚠️ See the [update instructions](https://github.com/cloudscale-ch/csi-cloudscale#from-csi-cloudscale-v3x-to-v4x).
 

README.md

@@ -64,7 +64,7 @@ secret `my-pvc-luks-key`.
 ## Releases
 
 The cloudscale.ch CSI plugin follows [semantic versioning](https://semver.org/).
-The current version is: **`v4.0.0`**.
+The current version is: **`v4.0.1`**.
 
 * Bug fixes will be released as a `PATCH` update.
 * New features (such as CSI spec bumps) will be released as a `MINOR` update.
@@ -92,15 +92,15 @@ We recommend using the latest cloudscale.ch CSI driver compatible with your Kube
 | 1.25               | v3.3.0                           | v3.5.6                           |
 | 1.26               | v3.3.0                           | v3.5.6                           |
 | 1.27               | v3.3.0                           | v3.5.6                           |
-| 1.28               | v3.3.0                           | v4.0.0                           |
-| 1.29               | v3.3.0                           | v4.0.0                           |
-| 1.30               | v3.3.0                           | v4.0.0                           |
-| 1.31               | v3.3.0                           | v4.0.0                           |
-| 1.32               | v3.3.0                           | v4.0.0                           |
-| 1.33               | v3.3.0                           | v4.0.0                           |
-| 1.34 [1]           | v3.3.0                           | v4.0.0                           |
-| 1.35               | v3.4.1                           | v4.0.0                           |
-| 1.36               | v3.4.1                           | v4.0.0                           |
+| 1.28               | v3.3.0                           | v4.0.1                           |
+| 1.29               | v3.3.0                           | v4.0.1                           |
+| 1.30               | v3.3.0                           | v4.0.1                           |
+| 1.31               | v3.3.0                           | v4.0.1                           |
+| 1.32               | v3.3.0                           | v4.0.1                           |
+| 1.33               | v3.3.0                           | v4.0.1                           |
+| 1.34 [1]           | v3.3.0                           | v4.0.1                           |
+| 1.35               | v3.4.1                           | v4.0.1                           |
+| 1.36               | v3.4.1                           | v4.0.1                           |
 
 [1] Prometheus `kubelet_volume_stats_*` metrics not available in 1.34.0 and 1.34.1 due to a 
     [bug in Kubelet](https://github.com/kubernetes/kubernetes/issues/133847). Fixed in `1.34.2`.
@@ -216,10 +216,10 @@ $ helm install -g -n kube-system --set controller.image.tag=dev --set node.image
 Before you continue, be sure to checkout to a [tagged
 release](https://github.com/cloudscale-ch/csi-cloudscale/releases). 
 Always use the [latest stable version](https://github.com/cloudscale-ch/csi-cloudscale/releases/latest) 
-For example, to use the latest stable version (`v4.0.0`) you can execute the following command:
+For example, to use the latest stable version (`v4.0.1`) you can execute the following command:
 
 ```
-$ kubectl apply -f https://raw.githubusercontent.com/cloudscale-ch/csi-cloudscale/master/deploy/kubernetes/releases/csi-cloudscale-v4.0.0.yaml
+$ kubectl apply -f https://raw.githubusercontent.com/cloudscale-ch/csi-cloudscale/master/deploy/kubernetes/releases/csi-cloudscale-v4.0.1.yaml
 ```
 
 The storage classes `cloudscale-volume-ssd` and `cloudscale-volume-bulk` will be created. The 
@@ -446,15 +446,15 @@ $ git push origin
 
 After it's merged to master, [create a new Github
 release](https://github.com/cloudscale-ch/csi-cloudscale/releases/new) from
-master with the version `v4.0.0` and then publish a new docker build:
+master with the version `v4.0.1` and then publish a new docker build:
 
 ```
 $ git checkout master
 $ make publish
 ```
 
-This will create a binary with version `v4.0.0` and docker image pushed to
-`cloudscalech/cloudscale-csi-plugin:v4.0.0`
+This will create a binary with version `v4.0.1` and docker image pushed to
+`cloudscalech/cloudscale-csi-plugin:v4.0.1`
 
 ### Release a pre-release version
 

VERSION

@@ -1 +1 @@
-v4.0.0
+v4.0.1

charts/csi-cloudscale/Chart.yaml

@@ -2,8 +2,8 @@ apiVersion: v2
 name: csi-cloudscale
 description: A Container Storage Interface Driver for cloudscale.ch volumes.
 type: application
-version: 1.5.0
-appVersion: "4.0.0"
+version: 1.5.1
+appVersion: "4.0.1"
 home: https://github.com/cloudscale-ch/csi-cloudscale
 sources:
   - https://github.com/cloudscale-ch/csi-cloudscale.git

charts/csi-cloudscale/values.yaml

@@ -107,7 +107,7 @@ controller:
   image:
     registry: quay.io
     repository: cloudscalech/cloudscale-csi-plugin
-    tag: v4.0.0
+    tag: v4.0.1
     pullPolicy: IfNotPresent
   serviceAccountName:
   logLevel: info
@@ -123,7 +123,7 @@ node:
   image:
     registry: quay.io
     repository: cloudscalech/cloudscale-csi-plugin
-    tag: v4.0.0
+    tag: v4.0.1
     pullPolicy: IfNotPresent
   nodeSelector: {}
   tolerations: []