Skip to content

Commit cb20e5d

Browse files
Pearl1594Pearl1594
committed
Add notes for CLVM storage improvements
1 parent 31baeb8 commit cb20e5d

File tree

2 files changed

+215
-1
lines changed

2 files changed

+215
-1
lines changed

‎source/adminguide/storage.rst‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1490,7 +1490,6 @@ Snapshot data.
14901490
been taken and stored in OVA format will continue to exist in that
14911491
format, and will continue to work as expected.
14921492
1493-
14941493
Linstor Primary Storage
14951494
~~~~~~~~~~~~~~~~~~~~~~~
14961495
@@ -1589,6 +1588,15 @@ Allocated storage is the sum of quota used by all of the buckets.
15891588
Administrators can do this by going to the configure limits tab in accounts, domains and projects
15901589
similar to when enforcing resource limits on volumes, primary storage usage etc.
15911590
1591+
Block Storage
1592+
----------------
1593+
CloudStack offers fully managed block storage to all users. This section gives technical details on how to create and manage Block Storage.
1594+
1595+
Block Storage is supported with CLVM/CLVM_NG (Clustered Logical Volume Manager).
1596+
1597+
.. include:: storage/clvm.rst
1598+
:start-after: under the License.
1599+
15921600
Shared FileSystems
15931601
------------------
15941602

‎source/adminguide/storage/clvm.rst‎

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
.. Licensed to the Apache Software Foundation (ASF) under one
2+
or more contributor license agreements. See the NOTICE file
3+
distributed with this work for additional information#
4+
regarding copyright ownership. The ASF licenses this file
5+
to you under the Apache License, Version 2.0 (the
6+
"License"); you may not use this file except in compliance
7+
with the License. You may obtain a copy of the License at
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
Unless required by applicable law or agreed to in writing,
10+
software distributed under the License is distributed on an
11+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
12+
KIND, either express or implied. See the License for the
13+
specific language governing permissions and limitations
14+
under the License.
15+
16+
From CloudStack 4.23 onwards, CloudStack supports two types for Clustered Logical Volume Manager (CLVM) primary storage:
17+
- CLVM: LVM volumes with RAW format
18+
- CLVM_NG (Next Generation): LVM volumes with QCOW2 format that supports advanced features.
19+
20+
Both use the same underlying infrastructure:
21+
- `lvmlockd` daemon for lock management
22+
- `sanlock` for distributed locking
23+
- Shared volume groups across cluster hosts
24+
25+
The key difference is the disk image format used on top of the LVM volumes which enabled different feature sets.
26+
27+
CLVM (RAW format):
28+
~~~~~~~~~~~~~~~~~~
29+
30+
- Logical volumes are formatted as RAW disk images
31+
- Provides basic block storage functionality
32+
- Suitable for workloads that do not require advanced features
33+
34+
Advantages:
35+
36+
- Sligthly better performance (no QCOW2 overhead)
37+
- Simpler storage stack
38+
39+
Disadvantages:
40+
41+
- No incremental snapshots - always full copy
42+
- Larger snapshot storage requirements
43+
- Slower snapshot operations
44+
- Higher secondary storage bandwidth usage
45+
46+
..note::
47+
The original CLVM implementation used the clvmd (Clustered LVM daemon) along with corosync/pacemaker for cluster coordination. This technology has been deprecated in modern Linux distributions (RHEL 8+, Ubuntu 20.04+). CloudStack's current implementation uses the modern lvmlockd + sanlock stack, which is more reliable. This same modern infrastructure is shared with CLVM_NG - the only difference between CLVM and CLVM_NG is the disk image format (RAW vs QCOW2), not the locking mechanism.
48+
49+
CLVM_NG (QCOW2 format):
50+
~~~~~~~~~~~~~~~~~~~~~~~
51+
52+
- Logical volumes are formatted as QCOW2 disk images
53+
- QCOW2 layer provides advanced features like:
54+
- Incremental snapshots (only changes since last snapshot)
55+
- Linked clones
56+
- Still benefits from LVM volume management
57+
58+
**Advantages:**
59+
60+
- **Incremental snapshots** with dirty bitmaps
61+
- Reduction in snapshot storage
62+
- Faster snapshot operations
63+
- Reduced secondary storage bandwidth
64+
- Persistent bitmap tracking
65+
66+
**Disadvantages:**
67+
68+
- Slight performance overhead from QCOW2 layer
69+
- More complex storage stack
70+
- Bitmaps lost during volume migration - hence the next snapshot will be a full snapshot (but subsequent snapshots will be incremental again)
71+
72+
Key features
73+
~~~~~~~~~~~~
74+
75+
Incremental Snapshots with QCOW2 bitmaps (CLVM_NG only)
76+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
77+
78+
CLVM_NG supports incremental snapshots using QCOW2's bitmap feature. When a snapshot is taken, a bitmap is created to track which blocks have changed since the last snapshot.
79+
80+
CLVM (RAW format) always performs full snapshots, copying all data regardless of changes, which results in larger snapshot sizes and longer snapshot times.
81+
82+
Exclusive volume locking
83+
^^^^^^^^^^^^^^^^^^^^^^^^^
84+
85+
CLVM/CLVM_NG use `lvmlockd` and `sanlock` to provide exclusive locking of logical volumes across cluster hosts. This ensures data integrity and prevents concurrent access issues.
86+
87+
- **Active Volumes**: Locked exclusively on the host running the VM
88+
- **Inactive Volumes**: Can be in shared mode or deactivated
89+
- **Lock Transfer**: Automatic during VM migration
90+
- **Lock Tracking**: CloudStack tracks which host owns each volume lock
91+
92+
Lightweight Migration
93+
^^^^^^^^^^^^^^^^^^^^^
94+
95+
When migrating VMs between pools on the same volume group:
96+
97+
- **No Data Copy**: Volume stays in same VG, only lock is transferred
98+
- **Zero Network Traffic**: No data movement required
99+
- **Same VG Required**: Source and destination pools must use same VG
100+
101+
Physical Extents (PEs) and Size Calculation for CLVM_NG
102+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
103+
104+
LVM divides physical volumes into **Physical Extents (PEs)** - the smallest allocatable unit of storage:
105+
106+
- **Default PE Size**: 4 MiB (4096 KiB)
107+
- **Configurable Range**: 1 KiB to 16 GiB (power of 2)
108+
- **Impact**: All LVM allocations are rounded up to PE boundaries
109+
110+
**Example:**
111+
112+
.. code-block:: text
113+
114+
Requested: 10.5 GB volume
115+
PE Size: 4 MiB
116+
Calculation: ceil(10.5 GB / 4 MiB) = ceil(2688) = 2688 PEs
117+
Actual Size: 2688 × 4 MiB = 10.5 GB (exactly)
118+
119+
QCOW2 metadata overhead
120+
^^^^^^^^^^^^^^^^^^^^^^^
121+
122+
CLVM_NG uses QCOW2 format on LVM volumes, which adds metadata overhead. The actual disk usage may be higher than the requested size due to QCOW2's internal structures, especially when snapshots are involved. This is important to consider when planning storage capacity for CLVM_NG pools.
123+
124+
**QCOW2 Metadata Components:**
125+
126+
1. **QCOW2 Header**: 72 bytes minimum (usually one cluster = 64 KiB)
127+
2. **L1 Table**: References L2 tables (grows with volume size)
128+
3. **L2 Tables**: Map clusters to physical storage
129+
4. **Refcount Tables**: Track cluster reference counts
130+
5. **Bitmaps**: For incremental snapshots (optional)
131+
132+
Size Calculation for CLVM_NG Volumes
133+
"""""""""""""""""""""""""""""""""""""
134+
135+
When creating CLVM_NG volumes, CloudStack accounts for:
136+
137+
1. **Requested Virtual Size**: The size visible to the VM
138+
2. **QCOW2 Metadata**: Overhead for QCOW2 structures
139+
3. **PE Alignment**: Round up to PE boundaries
140+
141+
Checking Volume Sizes
142+
""""""""""""""""""""""
143+
144+
.. code-block:: bash
145+
146+
# Check LVM volume size (physical)
147+
lvs -o lv_name,lv_size /dev/vgname/volumename
148+
149+
# Check QCOW2 virtual size
150+
qemu-img info /dev/vgname/volumename
151+
152+
Backing File Support in CLVM_NG
153+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
154+
155+
CLVM_NG supports **QCOW2 backing files** for template-based volume creation, enabling efficient template deployment and linked clones:
156+
157+
**How It Works:**
158+
1. **Template Volume**: Created as QCOW2 on CLVM_NG pool
159+
2. **Backing File**: Template volume becomes backing file (base image)
160+
3. **Child Volume**: New VM volumes reference template, store only differences
161+
4. **Copy-on-Write (CoW)**: Modified blocks stored in child volume, reads fall through to backing file
162+
163+
**Benefits:**
164+
- **Space Savings**: Multiple VMs share single template image
165+
- **Fast Deployment**: VM creation in seconds (no full copy needed)
166+
- **Efficient Storage**: Only differences (deltas) consume space
167+
168+
When deploying a VM from a template on CLVM_NG:
169+
170+
.. code-block:: text
171+
172+
1. Template registered in CloudStack
173+
↓
174+
2. Template stored as QCOW2 in CLVM_NG pool
175+
/dev/vgname/template-uuid
176+
↓
177+
3. User deploys VM from template
178+
↓
179+
4. CloudStack creates child volume with backing file
180+
lvcreate -L <size> -n vm-volume-uuid vgname
181+
qemu-img create -f qcow2 -F qcow2 -b /dev/vgname/template-uuid /dev/vgname/vm-volume-uuid
182+
↓
183+
5. VM starts using child volume
184+
(reads from template, writes to child)
185+
186+
Global Settings
187+
~~~~~~~~~~~~~~~
188+
189+
clvm.secure.zero.fill
190+
^^^^^^^^^^^^^^^^^^^^^
191+
192+
- **Type:** Boolean
193+
- **Default:** `false`
194+
- **Scope:** Storage Pool
195+
- **Category:** Advanced
196+
- **Description:** When enabled, CLVM volumes are zero-filled at deletion time to prevent data recovery by VMs reusing the space, as thick LVM volumes write data linearly
197+
- **Impact:**
198+
- Enabled: Volumes are securely wiped on deletion (slower deletion, more secure)
199+
- Disabled: Fast deletion, but data may be recoverable
200+
- **Propagation:** Setting is propagated to hosts when they connect to the storage pool. Changing requires disconnecting/reconnecting hosts or restarting KVM agent
201+
- **Recommendation:** Enable for environments with strict security/compliance requirements (PCI-DSS, HIPAA). Disable for performance-critical environments where deletion speed matters more than data security
202+
- **Use Case:** Production environments handling sensitive data, multi-tenant environments with strict data isolation requirements
203+
204+
For incremental snapshots to work with CLVM_NG enabled the `kvm.incremental.snapshot` global setting must also be set to `true`.
205+
206+
VM snapshots are currently not supported on CLVM/CLVM_NG volumes.

0 commit comments

Comments
 (0)