Skip to content

Commit c36ee1a

Browse files
bjahagir-OpenShiftbjahagir-OpenShift
committed
LVMS - Introduce StorageClass options
1 parent 6ce4da3 commit c36ee1a

10 files changed

Lines changed: 525 additions & 25 deletions

‎modules/behaviors-not-controlled-by-storage-class-options.adoc‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * storage/persistent_storage_local/persistent-storage-using-lvms.adoc
4+
5+
:_mod-docs-content-type: REFERENCE
6+
[id="behaviors-not-controlled-by-storage-class-options_{context}"]
7+
= Behaviors not controlled by StorageClass options
8+
9+
[role="_abstract"]
10+
Review these behaviors before you delete an LVMCluster. Although these behaviors relate to `storageClassOptions`, the `storageClassOptions` field does not control them.
11+
12+
Volume expansion behavior:: Logical Volume Manager Storage (LVMS) always enables volume expansion by setting `allowVolumeExpansion: true` on generated StorageClasses. You cannot control this setting by using the `storageClassOptions` field. All LVMS volumes support online expansion.
13+
14+
VolumeSnapshotClass management:: The `storageClassOptions` field only affects StorageClasses. When you configure thin provisioning, LVMS generates a `VolumeSnapshotClass` for each device class. This generated class always uses a fixed value `deletionPolicy: Delete`, regardless of the reclaimPolicy that you set in `storageClassOptions`.
15+
+
16+
Additionally, LVMS does not apply the `additionalParameters` and `additionalLabels` fields to `VolumeSnapshotClasses`. If you need to retain snapshot data, you must manage it separately from the StorageClass reclaim policy.
17+
18+
Default StorageClass annotation behavior:: The default field on a device class controls the `storageclass.kubernetes.io/is-default-class` annotation on the generated StorageClass.
19+
+
20+
Setting `default: true` does not guarantee that the LVMS StorageClass becomes the cluster default. If another default StorageClass already exists on the cluster, for example, gp3-csi on AWS-based {product-title} clusters, LVMS sets the annotation to `false` to prevent many cluster-wide defaults. Because the Operator actively manages this annotation, it reverts any manual, out-of-band changes during the next reconciliation loop.

‎modules/deleting-an-lvm-cluster.adoc‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * storage/dynamic-provisioning.adoc
4+
// * microshift_storage/dynamic-provisioning-microshift.adoc
5+
6+
:_mod-docs-content-type: PROCEDURE
7+
[id="deleting-an-lvm-cluster_{context}"]
8+
= Deleting an LVMCluster
9+
10+
[role="_abstract"]
11+
When you delete an `LVMCluster` custom resource (CR), the Operator enforces deletion gates to prevent data loss. The gates that apply depend on the reclaim policy that is configured for the storage class.
12+
13+
.Prerequisites
14+
15+
* You have administrative access to the cluster.
16+
* You have identified the reclaim policy in use: `Delete` or `Retain`.
17+
18+
.Procedure
19+
20+
. Delete all Persistent Volume Claims (PVCs) that reference LVM `StorageClass` resources.
21+
+
22+
If PVCs that reference LVM StorageClasses still exist, the Operator blocks `LVMCluster` deletion and generates a `DeletionPending` event:
23+
+
24+
[source,terminal]
25+
----
26+
found PVCs provisioned by LVMS, waiting 10s for their deletion
27+
----
28+
29+
. Back up any data before deleting PVCs.
30+
31+
.. List the PVCs that use the LVM StorageClass by running the following command:
32+
+
33+
[source,terminal]
34+
----
35+
$ oc get pvc -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,SC:.spec.storageClassName' | grep lvms-vg1
36+
----
37+
38+
.. Delete the PVCs by running the following command:
39+
+
40+
[source,terminal]
41+
----
42+
$ oc delete pvc <pvc_name> -n <namespace>
43+
----
44+
+
45+
With the `Delete` reclaim policy, deleting the PVCs automatically removes the persistent volumes (PVs) and on-disk logical volumes. After all PVCs are removed, `LVMCluster` deletion completes automatically. No further action is required.
46+
47+
. If you use the `Retain` reclaim policy, delete the retained PVs.
48+
+
49+
After you delete PVCs, if the reclaim policy is `Retain`, the Operator blocks `LVMCluster` deletion and generates a `DeletionPending` event:
50+
+
51+
[source,terminal]
52+
----
53+
found PVs with Retain policy from LVMS, waiting 10s for manual cleanup
54+
----
55+
56+
.. List the retained PVs by running the following command:
57+
+
58+
[source,terminal]
59+
----
60+
$ oc get pv -o custom-columns='NAME:.metadata.name,SC:.spec.storageClassName' | grep lvms-vg1
61+
----
62+
63+
.. Delete the PVs by running the following command:
64+
+
65+
[source,terminal]
66+
----
67+
$ oc delete pv <pv_name>
68+
----
69+
70+
. If you are using the `Retain` reclaim policy, delete the TopoLVM `LogicalVolume` custom resources.
71+
+
72+
After you delete PV objects from Kubernetes, the underlying logical volumes remain on disk because the `Retain` policy preserved them. The VG Manager detects this and generates a `ManualCleanupRequired` event:
73+
+
74+
[source,terminal]
75+
----
76+
Warning ManualCleanupRequired volume group vg1 has retained logical volumes [pvc-abc123]; manual cleanup required before deletion can proceed
77+
----
78+
79+
. Deleting the `LogicalVolume` custom resources triggers on-disk logical volume cleanup.
80+
81+
.. List the `LogicalVolume` custom resources by running the following command:
82+
+
83+
[source,terminal]
84+
----
85+
$ oc get logicalvolumes
86+
----
87+
88+
.. Delete the `LogicalVolume` custom resources for your device class by running the following command:
89+
+
90+
[source,terminal]
91+
----
92+
$ oc delete logicalvolume <lv_name>
93+
----
94+
95+
.Verification
96+
97+
* Verify that the `LVMCluster` deletion completed by confirming the resource no longer exists by running the following command:
98+
+
99+
[source,terminal]
100+
----
101+
$ oc get lvmcluster -A
102+
----

‎modules/dynamic-provisioning-change-default-class.adoc‎

Lines changed: 9 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -7,18 +7,15 @@
77
[id="change-default-storage-class_{context}"]
88
= Changing the default storage class
99

10-
Use the following procedure to change the default storage class.
11-
12-
For example, if you have two defined storage classes, `gp3` and `standard`, and you want to change the default storage class from `gp3` to `standard`.
10+
[role="_abstract"]
11+
Learn how to change the cluster's default storage class, for example, reassigning the default status from `gp3` to `standard`.
1312

1413
.Prerequisites
1514

1615
* Access to the cluster with cluster-admin privileges.
1716
1817
.Procedure
1918

20-
To change the default storage class:
21-
2219
. List the storage classes:
2320
+
2421
[source,terminal]
@@ -30,14 +27,15 @@ $ oc get storageclass
3027
[source,terminal]
3128
----
3229
NAME TYPE
33-
gp3 (default) ebs.csi.aws.com <1>
30+
gp3 (default) ebs.csi.aws.com
3431
standard ebs.csi.aws.com
3532
----
36-
<1> `(default)` indicates the default storage class.
33+
+
34+
The `(default)` indicates the default storage class.
3735

38-
. Make the desired storage class the default.
36+
. Make the required storage class the default.
3937
+
40-
For the desired storage class, set the `storageclass.kubernetes.io/is-default-class` annotation to `true` by running the following command:
38+
For the required storage class, set the `storageclass.kubernetes.io/is-default-class` annotation to `true` by running the following command:
4139
+
4240
[source,terminal]
4341
----
@@ -46,11 +44,9 @@ $ oc patch storageclass standard -p '{"metadata": {"annotations": {"storageclass
4644
+
4745
[NOTE]
4846
====
49-
You can have multiple default storage classes for a short time. However, you should ensure that only one default storage class exists eventually.
50-
51-
With multiple default storage classes present, any persistent volume claim (PVC) requesting the default storage class (`pvc.spec.storageClassName`=nil) gets the most recently created default storage class, regardless of the default status of that storage class, and the administrator receives an alert in the alerts dashboard that there are multiple default storage classes, `MultipleDefaultStorageClasses`.
47+
You can have many default storage classes for a short time. However, you must ensure that only one default storage class exists eventually.
5248
53-
// add xref to multi/no default SC module
49+
With many default storage classes present, any persistent volume claim (PVC) requesting the default storage class (`pvc.spec.storageClassName`=nil) gets the most recently created default storage class, regardless of the default status of that storage class. The administrator receives an alert in the alerts dashboard that there are many default storage classes, `MultipleDefaultStorageClasses`.
5450
====
5551

5652
. Remove the default storage class setting from the old default storage class.

‎modules/immutable-fields-of-the-storage-class-options.adoc‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * storage/persistent_storage_local/persistent-storage-using-lvms.adoc
4+
5+
:_mod-docs-content-type: CONCEPT
6+
[id="immutable-fields-of-the-storage-class-options_{context}"]
7+
= Immutable fields of the storage class options
8+
9+
[role="_abstract"]
10+
After you create the `LVMCluster`, you cannot change the value of the some of the `storageClassOptions` fields such as `reclaimPolicy`, `volumeBindingMode`, and `additionalParameters`. This mirrors the behavior of Kubernetes StorageClasses, which do not allow changes to these fields after creation.
11+
12+
If you attempt to modify an immutable field, the API server rejects the request:
13+
14+
[source,terminal]
15+
----
16+
Invalid value: "object": reclaimPolicy is immutable once set
17+
----
18+
19+
There is no way to patch or update immutable fields in place. To change an immutable field, you must delete the `LVMCluster` and recreate it with the new values.
20+
21+
For example, you cannot change the filesystem type through `additionalParameters`. The `csi.storage.k8s.io/fstype` parameter is managed by LVMS and is rejected at admission if set through `additionalParameters`. To use `ext4` instead of the default `xfs`, use the `fstype` field on the device class:
22+
23+
[source,yaml]
24+
----
25+
deviceClasses:
26+
- name: vg1
27+
fstype: ext4
28+
----
29+
However, the `fstype` field is also immutable after creation.
30+
31+
[NOTE]
32+
====
33+
The deletion gates require all PVCs and, for the `Retain` policy, all PVs to be removed before the `LVMCluster` can be deleted. After you recreate the `LVMCluster` with the new values, new PVCs use the updated StorageClass configuration.
34+
====
35+

‎modules/lvms-about-lvmcluster-cr.adoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,7 @@ Wiping the device can lead to inconsistencies in data integrity if any of the fo
9898
* The device is mounted.
9999

100100
If any of these conditions are true, do not force wipe the disk. Instead, you must manually wipe the disk.
101+
| deviceClasses.storageClassOptions | object | Optional. Allows customization of the StorageClass created for this device class, including reclaim policy, volume binding mode, additional parameters, and labels. For more information, see "StorageClass customization for LVMS device classes".
101102

102103
|`deviceClasses.thinPoolConfig`
103104
|`object`

‎modules/sample-lvm-cluster-configuration-with-storage-class-option.adoc‎

Lines changed: 169 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,169 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * storage/persistent_storage_local/persistent-storage-using-lvms.adoc
4+
5+
:_mod-docs-content-type: REFERENCE
6+
[id="sample-lvm-cluster-configuration-with-storage-class-option_{context}"]
7+
= Sample LVM cluster configuration with storage class option
8+
9+
[role="_abstract"]
10+
Use these examples to configure `storageClassOptions` in your `LVMCluster` custom resource (CR) to meet your specific storage requirements.
11+
12+
.Default StorageClass behavior (no options)
13+
[source,yaml]
14+
----
15+
apiVersion: lvm.topolvm.io/v1alpha1
16+
kind: LVMCluster
17+
metadata:
18+
name: my-lvmcluster
19+
namespace: openshift-lvm-storage
20+
spec:
21+
storage:
22+
deviceClasses:
23+
- name: vg1
24+
default: true
25+
thinPoolConfig:
26+
name: thin-pool-1
27+
sizePercent: 90
28+
overprovisionRatio: 10
29+
----
30+
31+
This produces a `StorageClass` with `reclaimPolicy: Delete` and `volumeBindingMode: WaitForFirstConsumer`, which is the same as the behavior before this feature.
32+
33+
.Retain policy for data protection
34+
[source,yaml]
35+
----
36+
apiVersion: lvm.topolvm.io/v1alpha1
37+
kind: LVMCluster
38+
metadata:
39+
name: my-lvmcluster
40+
namespace: openshift-lvm-storage
41+
spec:
42+
storage:
43+
deviceClasses:
44+
- name: vg1
45+
default: true
46+
thinPoolConfig:
47+
name: thin-pool-1
48+
sizePercent: 90
49+
overprovisionRatio: 10
50+
storageClassOptions:
51+
reclaimPolicy: Retain
52+
----
53+
54+
.Immediate binding for pre-provisioning
55+
[source,yaml]
56+
----
57+
apiVersion: lvm.topolvm.io/v1alpha1
58+
kind: LVMCluster
59+
metadata:
60+
name: my-lvmcluster
61+
namespace: openshift-lvm-storage
62+
spec:
63+
storage:
64+
deviceClasses:
65+
- name: vg1
66+
default: true
67+
thinPoolConfig:
68+
name: thin-pool-1
69+
sizePercent: 90
70+
overprovisionRatio: 10
71+
storageClassOptions:
72+
volumeBindingMode: Immediate
73+
----
74+
75+
.All options configured together
76+
[source,yaml]
77+
----
78+
apiVersion: lvm.topolvm.io/v1alpha1
79+
kind: LVMCluster
80+
metadata:
81+
name: my-lvmcluster
82+
namespace: openshift-lvm-storage
83+
spec:
84+
storage:
85+
deviceClasses:
86+
- name: vg1
87+
default: true
88+
thinPoolConfig:
89+
name: thin-pool-1
90+
sizePercent: 90
91+
overprovisionRatio: 10
92+
storageClassOptions:
93+
reclaimPolicy: Retain
94+
volumeBindingMode: WaitForFirstConsumer
95+
additionalParameters:
96+
custom-key: custom-value
97+
additionalLabels:
98+
environment: production
99+
team: storage
100+
----
101+
102+
.Multiple device classes with different options
103+
[source,yaml]
104+
----
105+
apiVersion: lvm.topolvm.io/v1alpha1
106+
kind: LVMCluster
107+
metadata:
108+
name: my-lvmcluster
109+
namespace: openshift-lvm-storage
110+
spec:
111+
storage:
112+
deviceClasses:
113+
- name: vg-fast
114+
default: true
115+
thinPoolConfig:
116+
name: thin-pool-1
117+
sizePercent: 90
118+
overprovisionRatio: 10
119+
deviceSelector:
120+
paths:
121+
- /dev/nvme0n1
122+
storageClassOptions:
123+
reclaimPolicy: Delete
124+
volumeBindingMode: WaitForFirstConsumer
125+
additionalLabels:
126+
tier: fast
127+
- name: vg-archive
128+
thinPoolConfig:
129+
name: thin-pool-1
130+
sizePercent: 90
131+
overprovisionRatio: 10
132+
deviceSelector:
133+
paths:
134+
- /dev/sda
135+
storageClassOptions:
136+
reclaimPolicy: Retain
137+
volumeBindingMode: WaitForFirstConsumer
138+
additionalLabels:
139+
tier: archive
140+
----
141+
142+
For a device class named `vg1` with the full configuration, LVMS generates a `StorageClass` named `lvms-vg1` with the following structure:
143+
144+
[source,yaml]
145+
----
146+
apiVersion: storage.k8s.io/v1
147+
kind: StorageClass
148+
metadata:
149+
name: lvms-vg1
150+
annotations:
151+
description: "Provides RWO and RWOP Filesystem & Block volumes"
152+
storageclass.kubernetes.io/is-default-class: "true"
153+
labels:
154+
environment: production
155+
team: storage
156+
provisioner: topolvm.io
157+
reclaimPolicy: Retain
158+
volumeBindingMode: WaitForFirstConsumer
159+
allowVolumeExpansion: true
160+
parameters:
161+
custom-key: custom-value
162+
topolvm.io/device-class: vg1
163+
csi.storage.k8s.io/fstype: xfs
164+
----
165+
166+
The `StorageClass` name always follows the convention `lvms-<device_class_name>`.
167+
168+
169+

0 commit comments

Comments
 (0)