Skip to content

Commit b9370ea

Browse files
siegfriedwebersiegfriedweber
committed
docs: Describe scaling of OpenSearch clusters
1 parent 2f4fb8a commit b9370ea

3 files changed

Lines changed: 257 additions & 1 deletion

File tree

‎docs/modules/opensearch/pages/usage-guide/scaling.adoc‎

Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,255 @@
1+
= Scaling OpenSearch clusters
2+
:description: OpenSearch clusters can be scaled after provisioning but manual steps are required.
3+
4+
OpenSearch clusters can be scaled after provisioning.
5+
CPU and memory settings can be easily adapted as described in xref:opensearch:usage-guide/storage-resource-configuration.adoc#_resource_requests[Resource Requests].
6+
But for changing the number of nodes or resizing the volumes, the following points must be noted.
7+
8+
Horizontal scaling, i.e. changing the replica count of role-groups, can be easily done for non-data nodes by adapting the OpenSearchCluster specification.
9+
The number of data nodes can also be increased.
10+
But decreasing the number of data nodes requires manual steps because if a pod which manages data is just shut down then its data is not reachable anymore.
11+
Manual steps are required to drain the data from the nodes before removing them.
12+
13+
Vertical scaling, i.e. changing the volume size of nodes, is not supported by the operator.
14+
If the size of a volume can be changed depends on its CSI driver.
15+
OpenSearch supports multiple data paths in one data node, but adding volumes in additional data paths usually does not solve the problem of low disk space because the data is not rebalanced across multiple data paths.
16+
17+
[NOTE]
18+
====
19+
The OpenSearch operator is still in an early stage and as development progresses, smart scaling (adapting resources without data loss) and auto scaling (scaling the cluster according to the load) will be eventually supported by the operator.
20+
====
21+
22+
== Manually scaling
23+
24+
As mentioned above, scaling is quite a demanding task but there exists an easy workaround which will be presented here.
25+
26+
For instance, the following OpenSearchCluster with three custer-manager nodes and five small data nodes is already deployed:
27+
28+
[source,yaml]
29+
----
30+
spec:
31+
nodes:
32+
roleGroups:
33+
cluster-manager:
34+
config:
35+
nodeRoles:
36+
- cluster_manager
37+
replicas: 3
38+
data-small:
39+
config:
40+
nodeRoles:
41+
- data
42+
- ingest
43+
- remote_cluster_client
44+
resources:
45+
storage:
46+
data:
47+
capacity: 10Gi
48+
replicas: 5
49+
----
50+
51+
Now, you decide that instead of five small data nodes, three large data nodes are better suited.
52+
To achieve this, you can replace the role-group `data-small` with a desired one.
53+
54+
First, add the new role-group `data-large` with three replicas and a capacity of 100Gi per node:
55+
56+
[source,yaml]
57+
----
58+
spec:
59+
nodes:
60+
roleGroups:
61+
cluster-manager:
62+
config:
63+
nodeRoles:
64+
- cluster_manager
65+
replicas: 3
66+
data-small:
67+
config:
68+
nodeRoles:
69+
- data
70+
- ingest
71+
- remote_cluster_client
72+
resources:
73+
storage:
74+
data:
75+
capacity: 10Gi
76+
replicas: 5
77+
data-large:
78+
config:
79+
nodeRoles:
80+
- data
81+
- ingest
82+
- remote_cluster_client
83+
resources:
84+
storage:
85+
data:
86+
capacity: 100Gi
87+
replicas: 3
88+
----
89+
90+
The data must now be moved from `data-small` to `data-large`.
91+
With the cluster setting `cluster.routing.allocation.exclude`, nodes can be excluded from shard allocation.
92+
If you did not disable the rebalancing, then existing data will be moved from the specified nodes to the allowed ones, in the example case from `data-small` to `data-large`.
93+
94+
[TIP]
95+
====
96+
The OpenSearch operator adds a role-group attribute to every OpenSearch node, so that it is easier to reference all nodes belonging to a role-group.
97+
====
98+
99+
The following REST call excludes the role-group `data-small` from the shard allocation:
100+
101+
[source,http]
102+
----
103+
PUT _cluster/settings
104+
{
105+
"persistent": {
106+
"cluster": {
107+
"routing": {
108+
"allocation.exclude": {
109+
"role-group": "data-small"
110+
}
111+
}
112+
}
113+
}
114+
}
115+
----
116+
117+
You have to wait now until all the data has been moved from `data-small` to `data-large`.
118+
The current shard allocation can be requested at the `_cat/shards` endpoint, e.g.:
119+
120+
[source,http]
121+
----
122+
GET _cat/shards?v
123+
index shard prirep state docs store ip node
124+
logs 0 r STARTED 14074 6.9mb 10.244.0.60 opensearch-nodes-data-large-2
125+
logs 0 p RELOCATING 14074 8.5mb 10.244.0.52 opensearch-nodes-data-small-4
126+
-> 10.244.0.59 NFjQBBmWSm-pijXcxrXnvQ opensearch-nodes-data-large-1
127+
...
128+
129+
GET _cat/shards?v
130+
index shard prirep state docs store ip node
131+
logs 0 r STARTED 14074 6.9mb 10.244.0.60 opensearch-nodes-data-large-2
132+
logs 0 p STARTED 14074 6.9mb 10.244.0.59 opensearch-nodes-data-large-1
133+
...
134+
----
135+
136+
The statistics, especially the document count, can be retrieved at the `_nodes/role-group:data-small/stats` endpoint, e.g.:
137+
138+
[source,http]
139+
----
140+
GET _nodes/role-group:data-small/stats/indices/docs
141+
{
142+
"_nodes": {
143+
"total": 5,
144+
"successful": 5,
145+
"failed": 0
146+
},
147+
"cluster_name": "opensearch",
148+
"nodes": {
149+
"wjaeQJUXQX6eNWYUeiScgQ": {
150+
"timestamp": 1761992580239,
151+
"name": "opensearch-nodes-data-small-4",
152+
"transport_address": "10.244.0.52:9300",
153+
"host": "10.244.0.52",
154+
"ip": "10.244.0.52:9300",
155+
"roles": [
156+
"data",
157+
"ingest",
158+
"remote_cluster_client"
159+
],
160+
"attributes": {
161+
"role-group": "data-small",
162+
"shard_indexing_pressure_enabled": "true"
163+
},
164+
"indices": {
165+
"docs": {
166+
"count": 14686,
167+
"deleted": 0
168+
}
169+
}
170+
},
171+
...
172+
}
173+
}
174+
175+
GET _nodes/role-group:data-small/stats/indices/docs
176+
{
177+
"_nodes": {
178+
"total": 5,
179+
"successful": 5,
180+
"failed": 0
181+
},
182+
"cluster_name": "opensearch",
183+
"nodes": {
184+
"wjaeQJUXQX6eNWYUeiScgQ": {
185+
"timestamp": 1761992817422,
186+
"name": "opensearch-nodes-data-small-4",
187+
"transport_address": "10.244.0.52:9300",
188+
"host": "10.244.0.52",
189+
"ip": "10.244.0.52:9300",
190+
"roles": [
191+
"data",
192+
"ingest",
193+
"remote_cluster_client"
194+
],
195+
"attributes": {
196+
"role-group": "data-small",
197+
"shard_indexing_pressure_enabled": "true"
198+
},
199+
"indices": {
200+
"docs": {
201+
"count": 0,
202+
"deleted": 0
203+
}
204+
}
205+
},
206+
...
207+
}
208+
}
209+
210+
----
211+
212+
When all shards were transferred, the role-group `data-small` can just be removed from the OpenSearchCluster specification:
213+
214+
[source,yaml]
215+
----
216+
spec:
217+
nodes:
218+
roleGroups:
219+
cluster-manager:
220+
config:
221+
nodeRoles:
222+
- cluster_manager
223+
replicas: 3
224+
data-large:
225+
config:
226+
nodeRoles:
227+
- data
228+
- ingest
229+
- remote_cluster_client
230+
resources:
231+
storage:
232+
data:
233+
capacity: 100Gi
234+
replicas: 3
235+
----
236+
237+
Finally, the shard exclusion should be removed again from the cluster settings:
238+
239+
[source,http]
240+
----
241+
PUT _cluster/settings
242+
{
243+
"persistent": {
244+
"cluster": {
245+
"routing": {
246+
"allocation.exclude": {
247+
"role-group": null
248+
}
249+
}
250+
}
251+
}
252+
}
253+
----
254+
255+
If your OpenSearch clients only used the service of the cluster-manager nodes to connect to the cluster, the switch from one to another data role-group should have been transparent for them.

‎docs/modules/opensearch/pages/usage-guide/storage-resource-configuration.adoc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ nodes:
1313
config:
1414
resources:
1515
storage:
16-
logDirs:
16+
data:
1717
capacity: 50Gi
1818
----
1919

‎docs/modules/opensearch/partials/nav.adoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
** xref:opensearch:usage-guide/monitoring.adoc[]
1010
** xref:opensearch:usage-guide/logging.adoc[]
1111
** xref:opensearch:usage-guide/opensearch-dashboards.adoc[]
12+
** xref:opensearch:usage-guide/scaling.adoc[]
1213
** xref:opensearch:usage-guide/operations/index.adoc[]
1314
*** xref:opensearch:usage-guide/operations/cluster-operations.adoc[]
1415
*** xref:opensearch:usage-guide/operations/pod-placement.adoc[]

0 commit comments

Comments
 (0)