You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs_user/assemblies/assembly_migrating-ceph-cluster.adoc
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -31,6 +31,7 @@ already part of the {CephCluster} cluster.
31
31
32
32
Before you begin the migration, complete the tasks in your {rhos_prev_long} {rhos_prev_ver} environment. For more information, see xref:red-hat-ceph-storage-prerequisites_configuring-network[{Ceph} prerequisites].
= {Ceph} migration for Distributed Compute Node deployments
5
+
6
+
[role="_abstract"]
7
+
Before you adopt your Distributed Compute Node (DCN) deployments that host {CephCluster} clusters on Compute nodes at edge sites so that your architecture runs on {rhos_long} (RHOSO), be aware of important considerations.
8
+
9
+
Supported edge storage topologies:: DCN deployments support the following storage topologies at edge sites:
10
+
+
11
+
* Hyperconverged Infrastructure (HCI): {Ceph} daemons run on Compute nodes at each edge site.
12
+
* {OpenStackPreviousInstaller}-deployed dedicated storage: {Ceph} runs on separate storage nodes deployed by {OpenStackPreviousInstaller}.
13
+
* External {Ceph} cluster: Edge sites connect to pre-existing {CephCluster} clusters not managed by {OpenStackPreviousInstaller}.
14
+
15
+
Central site {Ceph} migration:: For the central site, migrate {Ceph} daemons from the {OpenStackShort} Controller nodes by using the same process as a non-DCN deployment. For more information, see xref:ceph-daemon-cardinality_migrating-ceph[{Ceph} daemon cardinality].
16
+
17
+
Edge site {Ceph} migration:: For edge sites that use HCI or {OpenStackPreviousInstaller}-deployed dedicated storage, the {Ceph} daemons can continue to run on their current nodes without migration. The Compute nodes or dedicated storage nodes at edge sites are not decommissioned during adoption, so the {Ceph} daemons remain operational.
18
+
+
19
+
For edge sites that use external {Ceph} clusters, no migration is required because the {CephCluster} cluster is not managed by {OpenStackPreviousInstaller}.
20
+
21
+
{Ceph} back-end configuration and key distribution:: In a DCN deployment, each site has its own {CephCluster} cluster with its own configuration file and {Ceph} keyring. These must be stored in Kubernetes secrets and mounted into the appropriate {rhos_long} service pods.
22
+
+
23
+
Rather than storing all {Ceph} keys in a single secret accessible to every pod, the recommended approach is to create one secret per site containing only the keys that site actually needs. This limits the security impact if a site is compromised: a pod at an edge site can authenticate only to its local {CephCluster} cluster and the central cluster, not to the {Ceph} keyrings of other edge sites.
24
+
+
25
+
The key distribution rule for N sites is:
26
+
+
27
+
* The central site (site 0) receives the {Ceph} keys and configuration for all clusters, because central services such as {image_service} use the `split` back end and must be able to copy images to and from any site.
28
+
* Each edge site (site 1 through N) receives only the keys for the central cluster and its own local cluster.
29
+
+
30
+
For example, in a three-site deployment with a central site and two edge sites:
The per-site secrets are created and then mounted into the appropriate pods using `extraMounts` propagation labels. The procedure in xref:configuring-a-ceph-backend_migrating-databases[Configuring a {Ceph} back end] covers both creating the secrets and applying the propagation labels so that each pod receives only its site-specific keys.
Copy file name to clipboardExpand all lines: docs_user/modules/proc_adopting-compute-services-to-the-data-plane.adoc
+6Lines changed: 6 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -264,6 +264,12 @@ EOF
264
264
[NOTE]
265
265
For {Ceph} environments with multi-cell configurations, you must name configuration maps and {rhos_prev_long} data plane services similar to the following examples: `nova-custom-ceph-cellX` and `nova-compute-extraconfig-cellX`.
266
266
267
+
+
268
+
[NOTE]
269
+
====
270
+
For Distributed Compute Node (DCN) deployments, do not use the single `nova-cells-global-config` ConfigMap. Create a per-site `ConfigMap` and per-site `OpenStackDataPlaneService` for each site in your DCN deployment. Each site's {compute_service} nodes require a different {Ceph} configuration and a different {image_service} endpoint. For more information, see xref:adopting-compute-services-with-dcn-backend_{context}[Adopting Compute services with multiple Ceph back ends (DCN)].
271
+
====
272
+
267
273
. Create the data plane services for {compute_service} cells to enable pre-upgrade workarounds, and to configure the Compute services for your chosen storage back end:
You do not boot a {compute_service_first_ref} instance by using the new `volume from` image or try to detach the previous volume because the {compute_service} and the {block_storage} are still not connected.
362
+
363
+
.Additional resources
364
+
365
+
* xref:adopting-block-storage-service-with-dcn-backend_{context}[Adopting the {block_storage} with multiple {Ceph} back ends (DCN)].
`mon_host`:: specifies the addresses of the cluster's monitors. If you use IPv6, use brackets for the `mon_host`. For example: `mon_host = [v2:[fd00:cccc::100]:3300/0,v1:[fd00:cccc::100]:6789/0]`
80
+
+
81
+
[NOTE]
82
+
====
83
+
For Distributed Compute Node (DCN) deployments with multiple {Ceph} clusters, create one secret per site. Each secret contains only the keys that the respective site requires. For more information on the rationale and key distribution pattern, see xref:ceph-migration-dcn_{context}[{Ceph} migration for Distributed Compute Node deployments].
84
+
85
+
The {Ceph} configuration files for all clusters are available on the {OpenStackShort} controller at either `/var/lib/tripleo-config/ceph/`, or `/etc/ceph`. Copy them locally and create the per-site secrets:
. In your `OpenStackControlPlane` CR, inject `ceph.conf` and `ceph.client.openstack.keyring` to the {OpenStackShort} services that are defined in the propagation list. For example:
128
+
Repeat for each additional edge site. Each edge site secret must include the central cluster files and only the files for that edge site's local cluster.
129
+
130
+
When configuring `extraMounts` on the `OpenStackControlPlane`, use propagation labels matching the service instance names (for example, `central`, `dcn1`, `dcn2`) so that each pod mounts only its site-specific secret.
131
+
====
132
+
133
+
. In your `OpenStackControlPlane` CR, inject the {Ceph} configuration into the {OpenStackShort} service pods using `extraMounts`. For a single-cluster deployment, propagate one secret to all services:
For a DCN deployment with per-site secrets, use propagation labels matching each service instance name so that each pod receives only the keys for its site:
The propagation label `central` matches the {image_service} and {block_storage} pod instances named `central`. The `CinderBackup` and `ManilaShare` labels are service-type propagation and apply to all {block_storage} backup and {rhos_component_storage_file} pods, which run only at the central site. Replace `central`, `dcn1`, and `dcn2` with the instance names used in your deployment.
Copy file name to clipboardExpand all lines: docs_user/modules/proc_deploying-file-systems-service-control-plane.adoc
+6Lines changed: 6 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,6 +6,11 @@
6
6
[role="_abstract"]
7
7
Copy the {rhos_component_storage_file_first_ref} configuration from the {rhos_prev_long} ({OpenStackShort}) {rhos_prev_ver} deployment, and then deploy the {rhos_component_storage_file} on the control plane.
8
8
9
+
[NOTE]
10
+
====
11
+
In a {rhos_prev_long} DCN deployment, the {rhos_component_storage_file} runs only at the central site and is not deployed on edge site Compute nodes. After adoption, {rhos_long} supports deploying the {rhos_component_storage_file} at edge sites if required.
12
+
====
13
+
9
14
.Prerequisites
10
15
11
16
* The {rhos_component_storage_file} systemd services such as `api`, `cron`, and `scheduler` are stopped. For more information, see xref:stopping-openstack-services_migrating-databases[Stopping {rhos_prev_long} services].
@@ -114,6 +119,7 @@ __EOF__
114
119
* `share_backend_name` specifies the names of the back ends to use in {rhos_long}. Ensure that the names of the back ends are the same as they were in {OpenStackShort} {rhos_prev_ver}.
115
120
* `networkAttachments` specifies the appropriate storage management network. For example, the `manilaShares` instance with the CephFS back-end driver is connected to the `storage` network.
116
121
* `extraMounts` specifies additional files to add to any of the services. For example, when using {Ceph}, you can add the {rhos_component_storage_file} Ceph user's keyring file as well as the `ceph.conf` configuration file.
122
+
* In a DCN deployment where {Ceph} configuration files are named by cluster (for example, `central.conf` instead of `ceph.conf`), set `cephfs_conf_path` to the path of the configuration file for the cluster hosting the CephFS back end, and set `cephfs_cluster_name` to the corresponding cluster name. For example, `cephfs_conf_path=/etc/ceph/central.conf` and `cephfs_cluster_name=central`. Using `/etc/ceph/ceph.conf` when that file does not exist in the mounted secret will cause the `manila-share` service to fail with a `conf_read_file` error.
117
123
+
118
124
The following example patch file uses CephFS through NFS:
Copy file name to clipboardExpand all lines: docs_user/modules/proc_stopping-openstack-services.adoc
+151Lines changed: 151 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -235,3 +235,154 @@ endif::[]
235
235
----
236
236
+
237
237
If the status of each service is `OK`, then the services stopped successfully.
238
+
239
+
. For Distributed Compute Node (DCN) deployments where {image_service}, {block_storage}, and {Ceph} services run on edge Compute nodes, stop the {image_service}, {block_storage}, and etcd services on all edge Compute nodes with the `DistributedComputeHCI` role:
240
+
+
241
+
[NOTE]
242
+
====
243
+
The `DistributedComputeHCI` role runs `GlanceApiEdge`, `CinderVolumeEdge`, and `Etcd` services. A minimum of three nodes per site use this role. Skip this step if your DCN deployment does not run these services on edge Compute nodes. The examples in this procedure use hyper-converged (HCI) roles. If your deployment does not use HCI, the same services apply to the `DistributedCompute` role, which runs the same `GlanceApiEdge`, `CinderVolumeEdge`, and `Etcd` services but without Ceph OSD, Ceph Monitor, or Ceph Manager.
244
+
====
245
+
+
246
+
.. Define shell variables for your `DistributedComputeHCI` edge Compute nodes:
247
+
+
248
+
[subs=+quotes]
249
+
----
250
+
# DCN1 edge site DistributedComputeHCI nodes
251
+
DCN1_HCI0_SSH="ssh -i *<path to SSH key>* root@*<dcn1-hci-0 IP>*"
252
+
DCN1_HCI1_SSH="ssh -i *<path to SSH key>* root@*<dcn1-hci-1 IP>*"
253
+
DCN1_HCI2_SSH="ssh -i *<path to SSH key>* root@*<dcn1-hci-2 IP>*"
254
+
255
+
# DCN2 edge site DistributedComputeHCI nodes
256
+
DCN2_HCI0_SSH="ssh -i *<path to SSH key>* root@*<dcn2-hci-0 IP>*"
257
+
DCN2_HCI1_SSH="ssh -i *<path to SSH key>* root@*<dcn2-hci-1 IP>*"
258
+
DCN2_HCI2_SSH="ssh -i *<path to SSH key>* root@*<dcn2-hci-2 IP>*"
259
+
----
260
+
+
261
+
where:
262
+
263
+
`<path to SSH key>`::
264
+
Specifies the path to your SSH key for each `DistributedComputeHCI` edge Compute node on each DCN edge site.
# List of all DistributedComputeHCI node SSH commands
281
+
DCN_HCI_NODES=("$DCN1_HCI0_SSH"
282
+
"$DCN1_HCI1_SSH"
283
+
"$DCN1_HCI2_SSH"
284
+
"$DCN2_HCI0_SSH"
285
+
"$DCN2_HCI1_SSH"
286
+
"$DCN2_HCI2_SSH")
287
+
288
+
echo "Stopping storage services on DistributedComputeHCI nodes"
289
+
for node_ssh in "${DCN_HCI_NODES[@]}"; do
290
+
[ -z "$node_ssh" ] && continue
291
+
echo "Processing node: $node_ssh"
292
+
for service in "${DCN_HCI_SERVICES[@]}"; do
293
+
if $node_ssh sudo systemctl is-active $service 2>/dev/null; then
294
+
echo "Stopping $service"
295
+
$node_ssh sudo systemctl stop $service
296
+
fi
297
+
done
298
+
done
299
+
300
+
echo "Checking storage services on DistributedComputeHCI nodes"
301
+
for node_ssh in "${DCN_HCI_NODES[@]}"; do
302
+
[ -z "$node_ssh" ] && continue
303
+
for service in "${DCN_HCI_SERVICES[@]}"; do
304
+
if ! $node_ssh systemctl show $service 2>/dev/null | grep ActiveState=inactive >/dev/null; then
305
+
echo "ERROR: Service $service still running on $node_ssh"
306
+
else
307
+
echo "OK: Service $service is not running on $node_ssh"
308
+
fi
309
+
done
310
+
done
311
+
----
312
+
+
313
+
[NOTE]
314
+
====
315
+
* On edge sites, the {image_service} runs with the service name `tripleo_glance_api_internal.service`, which is different from the `tripleo_glance_api.service` on the central controller.
316
+
* The {block_storage} volume service (`tripleo_cinder_volume.service`) uses the same service name on both edge sites and the central controller.
317
+
* The etcd service (`tripleo_etcd.service`) is used as a distributed lock manager (DLM) for the {block_storage} volume service running in active/active mode on edge sites.
318
+
====
319
+
320
+
. If your DCN deployment includes `DistributedComputeHCIScaleOut` nodes, stop the HAProxy service on those nodes:
321
+
+
322
+
[NOTE]
323
+
====
324
+
The `DistributedComputeHCIScaleOut` role is used to scale compute and storage capacity beyond the initial three `DistributedComputeHCI` nodes at each site. These nodes run `HAProxyEdge`, which proxies {image_service} requests to the `GlanceApiEdge` instances on `DistributedComputeHCI` nodes. Skip this step if your DCN deployment does not include `DistributedComputeHCIScaleOut` nodes. For non-HCI deployments, the equivalent role is `DistributedComputeScaleOut`, which runs the same `HAProxyEdge` service.
325
+
====
326
+
+
327
+
.. Define shell variables for your `DistributedComputeHCIScaleOut` edge Compute nodes:
328
+
+
329
+
[subs=+quotes]
330
+
----
331
+
# DCN1 edge site DistributedComputeHCIScaleOut nodes
332
+
DCN1_SCALEOUT0_SSH="ssh -i *<path to SSH key>* root@*<dcn1-scaleout-0 IP>*"
333
+
DCN1_SCALEOUT1_SSH="ssh -i *<path to SSH key>* root@*<dcn1-scaleout-1 IP>*"
334
+
335
+
# DCN2 edge site DistributedComputeHCIScaleOut nodes
336
+
DCN2_SCALEOUT0_SSH="ssh -i *<path to SSH key>* root@*<dcn2-scaleout-0 IP>*"
337
+
DCN2_SCALEOUT1_SSH="ssh -i *<path to SSH key>* root@*<dcn2-scaleout-1 IP>*"
338
+
----
339
+
+
340
+
where:
341
+
342
+
`<dcn1-scaleout-0 IP>`, `<dcn1-scaleout-1 IP>`::
343
+
Specifies the IP address for each `DistributedComputeHCIScaleOut` node within the DCN1 edge site.
344
+
345
+
`<dcn2-scaleout-0 IP>`, `<dcn2-scaleout-1 IP>`::
346
+
Specifies the IP address for each `DistributedComputeHCIScaleOut` node within the DCN2 edge site.
347
+
+
348
+
.. Stop the services on all `DistributedComputeHCIScaleOut` nodes:
349
+
+
350
+
----
351
+
# Services to stop on DistributedComputeHCIScaleOut edge compute nodes
# List of all DistributedComputeHCIScaleOut node SSH commands
355
+
DCN_SCALEOUT_NODES=("$DCN1_SCALEOUT0_SSH"
356
+
"$DCN1_SCALEOUT1_SSH"
357
+
"$DCN2_SCALEOUT0_SSH"
358
+
"$DCN2_SCALEOUT1_SSH")
359
+
360
+
echo "Stopping services on DistributedComputeHCIScaleOut nodes"
361
+
for node_ssh in "${DCN_SCALEOUT_NODES[@]}"; do
362
+
[ -z "$node_ssh" ] && continue
363
+
echo "Processing node: $node_ssh"
364
+
for service in "${DCN_SCALEOUT_SERVICES[@]}"; do
365
+
if $node_ssh sudo systemctl is-active $service 2>/dev/null; then
366
+
echo "Stopping $service"
367
+
$node_ssh sudo systemctl stop $service
368
+
fi
369
+
done
370
+
done
371
+
372
+
echo "Checking services on DistributedComputeHCIScaleOut nodes"
373
+
for node_ssh in "${DCN_SCALEOUT_NODES[@]}"; do
374
+
[ -z "$node_ssh" ] && continue
375
+
for service in "${DCN_SCALEOUT_SERVICES[@]}"; do
376
+
if ! $node_ssh systemctl show $service 2>/dev/null | grep ActiveState=inactive >/dev/null; then
377
+
echo "ERROR: Service $service still running on $node_ssh"
378
+
else
379
+
echo "OK: Service $service is not running on $node_ssh"
380
+
fi
381
+
done
382
+
done
383
+
----
384
+
+
385
+
[NOTE]
386
+
====
387
+
* The HAProxy edge service (`tripleo_haproxy_edge.service`) provided a local {image_service} endpoint on `DistributedComputeHCIScaleOut` nodes, proxying requests to the `GlanceApiEdge` instances on `DistributedComputeHCI` nodes. During adoption, {rhocp_long} Kubernetes service endpoints backed by MetalLB replace HAProxy.
![openshift-merge-bot[bot]](https://avatars.githubusercontent.com/in/412865?v=4&size=40){kind=avatar}
0 commit comments