Skip to content

Commit 1159d9b

Browse files
authored
HDDS-13369. [Docs] Maintenance mode. (#8728)
Generated-by: Google Gemini Cli, Gemini 2.5 Flash
1 parent 1a2ab3d commit 1159d9b

1 file changed

Lines changed: 78 additions & 0 deletions

File tree

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
---
2+
title: "Maintenance Mode"
3+
menu:
4+
main:
5+
parent: Features
6+
summary: Maintenance mode for Datanodes.
7+
---
8+
<!---
9+
Licensed to the Apache Software Foundation (ASF) under one or more
10+
contributor license agreements. See the NOTICE file distributed with
11+
this work for additional information regarding copyright ownership.
12+
The ASF licenses this file to You under the Apache License, Version 2.0
13+
(the "License"); you may not use this file except in compliance with
14+
the License. You may obtain a copy of the License at
15+
16+
http://www.apache.org/licenses/LICENSE-2.0
17+
18+
Unless required by applicable law or agreed to in writing, software
19+
distributed under the License is distributed on an "AS IS" BASIS,
20+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
21+
See the License for the specific language governing permissions and
22+
limitations under the License.
23+
-->
24+
25+
# DataNode Maintenance Mode
26+
27+
Maintenance mode is a feature in Apache Ozone that allows you to temporarily take a DataNode offline for maintenance operations (e.g., hardware upgrades, software updates) without triggering immediate data replication. Unlike decommissioning, which aims to permanently remove a DataNode and its data from the cluster, maintenance mode is designed for temporary outages.
28+
29+
While in maintenance mode, a DataNode does not accept new writes but may still serve reads, assuming containers are healthy and online. Existing data on the DataNode will remain in place, and replication of its data will only be triggered if the DataNode remains in maintenance mode beyond a configurable timeout period. This allows for planned downtime without unnecessary data movement, reducing network overhead and cluster load.
30+
31+
The DataNode transitions through the following operational states during maintenance:
32+
33+
1. **IN_SERVICE**: The DataNode is fully operational and participating in data writes and reads.
34+
2. **ENTERING_MAINTENANCE**: The DataNode is transitioning into maintenance mode. New writes will be avoided.
35+
3. **IN_MAINTENANCE**: The DataNode is in maintenance mode. Data will not be written to it. If the DataNode remains in this state beyond the configured maintenance window, its data will start to be replicated to other DataNodes to ensure data durability.
36+
37+
## Command Line Usage
38+
39+
To place a DataNode into maintenance mode, use the `ozone admin datanode maintenance` command. You can specify a duration for the maintenance period. If no duration is specified, a default duration will be used (this can be configured).
40+
41+
To check the current state of the datanodes, including their operational state, you can execute the following command:
42+
43+
```shell
44+
ozone admin datanode list
45+
```
46+
47+
To start maintenance mode for one or more DataNodes:
48+
49+
```shell
50+
ozone admin datanode maintenance [-hV] [-id=<scmServiceId>] [--scm=<scm>] [--end=<hours>] [--force] [<hosts>...]
51+
```
52+
- `<hosts>`: A space-separated list of hostnames or IP addresses of the DataNodes to put into maintenance mode.
53+
- `--end=<hours>`: Optional. Automatically end maintenance after the given hours. By default, maintenance must be ended manually.
54+
- `--force`: Optional. Forcefully try to put the datanode(s) into maintenance mode.
55+
56+
To take a DataNode out of maintenance mode and return it to `IN_SERVICE` state, you can use the `recommission` command:
57+
58+
```shell
59+
ozone admin datanode recommission [-hV] [-id=<scmServiceId>] [--scm=<scm>] [<hosts>...]
60+
```
61+
62+
## Configuration Properties
63+
64+
The following properties, typically set in `ozone-site.xml`, are relevant to maintenance mode:
65+
66+
- `hdds.scm.replication.maintenance.replica.minimum`: The minimum number of container replicas which must be available for a node to enter maintenance. Default value is `2`. If putting a node into maintenance reduces the available replicas for any container below this level, the node will remain in the `ENTERING_MAINTENANCE` state until a new replica is created.
67+
- `hdds.scm.replication.maintenance.remaining.redundancy`: The number of redundant containers in a group which must be available for a node to enter maintenance. Default value is `1`. If putting a node into maintenance reduces the redundancy below this value, the node will remain in the `ENTERING_MAINTENANCE` state until a new replica is created. For Ratis containers, the default value of 1 ensures at least two replicas are online, meaning 1 more can be lost without data becoming unavailable. For any EC container it will have at least dataNum + 1 online, allowing the loss of 1 more replica before data becomes unavailable. Currently only EC containers use this setting. Ratis containers use `hdds.scm.replication.maintenance.replica.minimum`. For EC, if nodes are in maintenance, it is likely reconstruction reads will be required if some of the data replicas are offline. This is seamless to the client, but will affect read performance.
68+
69+
## Metrics
70+
71+
The following SCM metrics are relevant to DataNode maintenance mode:
72+
73+
- `DecommissioningMaintenanceNodesTotal`: This metric reports the total number of DataNodes that are currently in either decommissioning or maintenance mode.
74+
- `RecommissionNodesTotal`: This metric reports the total number of DataNodes that are currently being recommissioned (i.e., returning to `IN_SERVICE` state from either decommissioning or maintenance mode).
75+
76+
## Reference
77+
78+
For information on permanently removing a DataNode from the cluster, refer to the [Decommissioning](decommission.html) user documentation.

0 commit comments

Comments
 (0)