Skip to content

Commit 3eb3fc8

Browse files
jojochuangjojochuang
authored
HDDS-15323. Add storage capacity planning administrator guide. (#436)
1 parent 273f576 commit 3eb3fc8

2 files changed

Lines changed: 149 additions & 0 deletions

File tree

‎docs/03-core-concepts/01-architecture/05-datanodes.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,8 @@ the property `hdds.datanode.volume.choosing.policy`. There are two main policies
7373

7474
### Volume-Related Configuration Properties
7575

76+
For capacity planning, reserved space, and when volumes are treated as full, see [Storage Capacity Planning](../../administrator-guide/configuration/performance/storage-capacity-planning).
77+
7678
| Property Name | Default Value | Description |
7779
|---------------|---------------|-------------|
7880
| `hdds.datanode.volume.choosing.policy` | `CapacityVolumeChoosingPolicy` | The policy used to select a volume for new containers. |

‎docs/05-administrator-guide/02-configuration/04-performance/11-storage-capacity-planning.md‎

Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
---
2+
sidebar_label: Storage Capacity Planning
3+
---
4+
5+
# Storage Capacity Planning
6+
7+
In Apache Ozone, managing disk capacity is critical for maintaining high availability. Ozone uses a multi-layered approach to ensure that Datanodes do not run out of physical space, which could lead to data corruption or service hangs.
8+
9+
For background on Datanode volumes and related settings, see [Datanodes](../../../core-concepts/architecture/datanodes). Full property descriptions are in the [configuration appendix](../appendix).
10+
11+
## When is a Disk Considered "Full"?
12+
13+
A disk (or "Volume" in Ozone terminology) is considered **Full** for new data allocations and replication targets when the available physical space falls below a specific threshold.
14+
15+
This threshold is the sum of two "Reserved Space" configurations plus a safety buffer for the operation itself:
16+
17+
`Failure Threshold = (Volume Min Free Space) + (DU Reserved Space) + (Operation Buffer)`
18+
19+
If a volume's free space is less than this threshold, SCM will stop selecting this volume for new containers or as a destination for re-replication (including decommission and maintenance moves).
20+
21+
## Reserved Space Concepts
22+
23+
Ozone provides two types of reserved space to protect the system:
24+
25+
### 1. Volume Min Free Space
26+
27+
This is a safety margin managed by the Datanode to prevent the disk from ever reaching 100% capacity. It is the most important setting for capacity planning.
28+
29+
- **Purpose**: Protects the Datanode from OS-level disk-full errors and gives SCM headroom for placement decisions.
30+
- **Configuration**:
31+
- `hdds.datanode.volume.min.free.space`: A fixed byte value (e.g., `20GB`) used together with the percent settings below. For each tier, the effective value is the **maximum** of this bytes setting and `capacity × ratio`.
32+
- **Soft limit** (`hdds.datanode.volume.min.free.space.percent`, default `0.02` or 2%): Reported to SCM in heartbeats as `freeSpaceToSpare`. SCM uses this for placement—new containers and re-replication targets avoid volumes below this headroom.
33+
- **Hard limit** (`hdds.datanode.volume.min.free.space.hard.limit.percent`, default `0.015` or 1.5%): Enforced locally on the Datanode. Writes are rejected when available space would drop below `max(capacity × hard limit percent, min.free.space bytes)`. Set the hard limit to be **less than or equal to** the soft limit so SCM plans with more headroom than the Datanode enforces for writes.
34+
- **Soft band**: The gap between soft and hard limits (for example, on a 2 TB disk: 2% ≈ 40 GB reported to SCM vs 1.5% ≈ 30 GB hard limit → ~10 GB) is where the Datanode may close containers while writes can still succeed.
35+
36+
### 2. DU Reserved Space
37+
38+
This reserves space for non-Ozone files on the same disk (e.g., OS logs, system tools, or other services).
39+
40+
- **Purpose**: Prevents Ozone from consuming space that is intended for other uses on the host.
41+
- **Configuration**:
42+
- `hdds.datanode.dir.du.reserved`: Fixed bytes per volume (e.g., `/data1:500MB`).
43+
- `hdds.datanode.dir.du.reserved.percent`: Percentage per volume (default is `0.0001` or 0.01%).
44+
45+
### 3. Operation Buffer (2x Container Size)
46+
47+
When Ozone replicates a container (e.g., during decommissioning), it requires space to hold the incoming data in a temporary directory before importing it.
48+
49+
- **Requirement**: Ozone requires free space equal to **2x the Container Size**.
50+
- **Example**: For a default 5 GB container, this is **10 GB**.
51+
52+
## Configuration Parameters
53+
54+
| Property | Default | Description |
55+
| :--- | :--- | :--- |
56+
| `hdds.datanode.volume.min.free.space` | `-1` (often set to `20GB`) | Fixed minimum free space (bytes) combined with the percent settings; effective value is `max(bytes, capacity × ratio)` per tier. |
57+
| `hdds.datanode.volume.min.free.space.percent` | `0.02` (2%) | Soft limit: minimum free-space fraction reported to SCM as `freeSpaceToSpare` for placement decisions. |
58+
| `hdds.datanode.volume.min.free.space.hard.limit.percent` | `0.015` (1.5%) | Hard limit: minimum free-space fraction enforced locally for write rejection. Should be ≤ `min.free.space.percent`. |
59+
| `hdds.datanode.dir.du.reserved` | (unset) | Fixed bytes reserved for non-Ozone use. |
60+
| `ozone.scm.container.size` | `5GB` | The target size for containers. |
61+
| `hdds.datanode.storage.utilization.warning.threshold` | `0.75` (75%) | Datanode-wide utilization above which SCM logs a warning while processing the node report. |
62+
| `hdds.datanode.storage.utilization.critical.threshold` | `0.95` (95%) | Datanode-wide utilization above which SCM marks the Datanode out of space. |
63+
64+
## Inspecting Storage Usage
65+
66+
Administrators can inspect the actual and "SCM-usable" space using the Ozone CLI or the Recon Web UI.
67+
68+
### 1. Recon Web UI
69+
70+
The [Recon cluster capacity guide](../../operations/observability/recon/recon-capacity-distribution) describes how to monitor cluster-wide storage:
71+
72+
- **Dashboards**: View aggregated capacity, used space, and remaining space for the entire cluster.
73+
- **Datanodes Page**: Sort Datanodes by utilization to quickly identify nodes that are approaching their "Full" threshold.
74+
75+
### 2. Node Usage Summary (CLI)
76+
77+
To see a high-level view of how much space is used across the cluster:
78+
79+
```bash
80+
ozone admin datanode usageinfo
81+
```
82+
83+
This command shows the capacity, SCM used, and remaining space for each Datanode.
84+
85+
### 3. Per-Datanode Usage Summary
86+
87+
To see the aggregated capacity, usage, and remaining space for a specific Datanode:
88+
89+
```bash
90+
ozone admin datanode usageinfo --node-id <datanode-uuid>
91+
```
92+
93+
For per-volume breakdown on a node, use the **Datanodes** view in the Recon Web UI (see above).
94+
95+
## Concrete Examples
96+
97+
### Example 1: 1 TB Disk (Standard)
98+
99+
- **Capacity**: 1,024 GB
100+
- **Min Free Space**: Max(20GB, 2% of 1TB) = **20.48 GB**
101+
- **DU Reserved**: ~0.1 GB
102+
- **Replication Buffer**: 10 GB
103+
- **Full Threshold**: ~30.6 GB
104+
- **Usable Capacity**: ~993 GB (**97% Full**)
105+
106+
### Example 2: 10.7 TB Disk (Large)
107+
108+
- **Capacity**: 10,956 GB
109+
- **Min Free Space**: Max(20GB, 2% of 10.7TB) = **219.1 GB**
110+
- **DU Reserved**: ~1.1 GB
111+
- **Replication Buffer**: 10 GB
112+
- **Full Threshold**: **~230.2 GB**
113+
- **Usable Capacity**: ~10,725 GB (**97.9% Full**)
114+
115+
## What Happens When a Disk is Full?
116+
117+
From a user or client perspective, Ozone is designed to handle "Disk Full" conditions gracefully without data loss or system hangs.
118+
119+
### 1. Automatic Write Rejection
120+
121+
If a write request is sent to a Datanode that has reached its "Full" threshold, the Datanode will proactively reject the request. The client will receive a `DISK_OUT_OF_SPACE` error.
122+
123+
### 2. Client-Side Retries
124+
125+
The Ozone client is intelligent. When it encounters a disk-full error on one node, it will:
126+
127+
- **Failover**: Attempt to write the data to a different pipeline or Datanode that has available space.
128+
- **Transparency**: In most cases, this retry happens automatically in the background, and the application (like Spark or Hive) continues to run without interruption.
129+
- **Performance Impact**: However, if many Datanodes in the cluster are full, writes may feel significantly **slower**. This is because the client must sequentially experience rejections and perform retries until it successfully finds a Datanode with available space.
130+
131+
### 3. Automatic Container Sealing
132+
133+
When a Datanode detects that its volume is full, it informs the Storage Container Manager (SCM). The SCM then:
134+
135+
- **Closes the Containers**: Marks all "Open" containers on that full disk as "Closed."
136+
- **Prevents Future Writes**: Stops directing any new write traffic to those specific containers.
137+
- **Maintains Readability**: The data already stored on a full disk remains fully accessible for **Read** operations.
138+
139+
### 4. Preservation of System Stability
140+
141+
Because Ozone stops writing *before* the physical disk is 100% full (thanks to the **Reserved Space**), the Datanode remains healthy. It can still perform background tasks, report metrics, and serve read requests even when it can no longer accept new data.
142+
143+
---
144+
145+
**Tip for Users**: If you see `DISK_OUT_OF_SPACE` errors in your application logs, it is a signal that your cluster is reaching its usable capacity. While Ozone will try to find other nodes, you should contact your administrator to add more storage or delete unnecessary data.
146+
147+
When disks are unevenly utilized, use [Container Balancer](../../operations/data-balancing/container-balancer) and [Disk Balancer](../../operations/data-balancing/disk-balancer) to redistribute data. For hardware sizing guidance, see [Hardware and Sizing](../../installation/hardware-and-sizing).

0 commit comments

Comments
 (0)