Add dcn_storage scenario for DCN adoption with Ceph HCI

Add a new scenario that deploys 3 Ceph clusters across DCN sites:
- Central: 3 ComputeHCI nodes with Ceph
- DCN1: 3 ComputeHCIDcn1 nodes with Ceph, Cinder, Glance
- DCN2: 3 ComputeHCIDcn2 nodes with Ceph, Cinder, Glance

Key Ceph changes:
- New dcn_storage.yaml scenario with pre_oc_run hooks to deploy
  Ceph via adoption_deploy_ceph.yml for each stack
- HCI compute roles include CephOSD service and StorageMgmt network
- Update ceph_backend_configuration role to handle >1 ceph cluster
- Document adoption for multiple ceph clusters

Key Glance changes:
- Update role stop_openstack_services to stop glance at DCN sites
- Update role glance_adoption to adopt multibackend glance
- Document adoption of DCN multibackend Glance

Key Cinder changes:
- Update role stop_openstack_services to stop cinder-volume
  and etcd at DCN sites
- Update role cinder_adoption to add cinder_volume_backend: dcn
- Adopt cinder-backup
- Document adoption of DCN multisite Cinder

Key Manila changes:
- Add manila_cephfs_conf_path and manila_cephfs_cluster_name variables
  to manila_adoption role, replacing hardcoded ceph.conf/ceph values.
- Override those variables to central in vars.dcn_storage.yaml
- Updated proc_deploying-file-systems-service-control-plane.adoc with
  a NOTE that Manila was central-only in TripleO DCN but can be deployed
  at edge sites in RHOSO, and bullet explaining the cluster-specific
  conf path

Key EDPM changes:
- Update data-plane-adoption role to handle many nodesets
  each containing differnt ceph and glance configurations
- Update proc_adopting-compute-services-to-the-data-plane.adoc
  to document how to adopt DCN compute nodes which ran
  storage services so that they access the APIs for those
  storage services on central OpenShift

This extends dcn_nostorage to support storage adoption with
independent Ceph clusters at each DCN site.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: John Fulton <fulton@redhat.com>

Author: fultonj

docs_user/assemblies/assembly_adopting-openstack-control-plane-services.adoc

@@ -34,6 +34,8 @@ include::../modules/proc_adopting-the-compute-service.adoc[leveloffset=+1]
 
 include::../modules/proc_adopting-the-block-storage-service.adoc[leveloffset=+1]
 
+include::../modules/proc_adopting-block-storage-service-with-dcn-backend.adoc[leveloffset=+1]
+
 include::../modules/proc_adopting-the-openstack-dashboard.adoc[leveloffset=+1]
 
 include::../assemblies/assembly_adopting-the-shared-file-systems-service.adoc[leveloffset=+1]

docs_user/assemblies/assembly_adopting-the-data-plane.adoc

@@ -26,6 +26,8 @@ include::../modules/proc_adopting-compute-services-to-the-data-plane.adoc[levelo
 
 include::../modules/proc_configuring-dcn-data-plane-nodesets.adoc[leveloffset=+1]
 
+include::../modules/proc_adopting-compute-services-with-dcn-backend.adoc[leveloffset=+1]
+
 include::../modules/proc_performing-a-fast-forward-upgrade-on-compute-services.adoc[leveloffset=+1]
 
 include::../modules/proc_adopting-networker-services-to-the-data-plane.adoc[leveloffset=+1]

docs_user/assemblies/assembly_adopting-the-image-service.adoc

@@ -31,6 +31,8 @@ include::../modules/proc_adopting-image-service-with-nfs-backend.adoc[leveloffse
 
 include::../modules/proc_adopting-image-service-with-ceph-backend.adoc[leveloffset=+1]
 
+include::../modules/proc_adopting-image-service-with-dcn-backend.adoc[leveloffset=+1]
+
 include::../modules/proc_verifying-the-image-service-adoption.adoc[leveloffset=+1]
 
 ifdef::parent-context[:context: {parent-context}]

docs_user/assemblies/assembly_migrating-ceph-cluster.adoc

@@ -31,6 +31,7 @@ already part of the {CephCluster} cluster.
 
 Before you begin the migration, complete the tasks in your {rhos_prev_long} {rhos_prev_ver} environment. For more information, see xref:red-hat-ceph-storage-prerequisites_configuring-network[{Ceph} prerequisites].
 
+include::../modules/con_ceph-migration-dcn.adoc[leveloffset=+1]
 
 include::../modules/con_ceph-daemon-cardinality.adoc[leveloffset=+1]
 

docs_user/modules/con_ceph-migration-dcn.adoc

@@ -0,0 +1,44 @@
+:_mod-docs-content-type: CONCEPT
+[id="ceph-migration-dcn_{context}"]
+
+= {Ceph} migration for Distributed Compute Node deployments
+
+[role="_abstract"]
+Before you adopt your Distributed Compute Node (DCN) deployments that host {CephCluster} clusters on Compute nodes at edge sites so that your architecture runs on {rhos_long} (RHOSO), be aware of important considerations.
+
+Supported edge storage topologies:: DCN deployments support the following storage topologies at edge sites:
++
+* Hyperconverged Infrastructure (HCI): {Ceph} daemons run on Compute nodes at each edge site.
+* {OpenStackPreviousInstaller}-deployed dedicated storage: {Ceph} runs on separate storage nodes deployed by {OpenStackPreviousInstaller}.
+* External {Ceph} cluster: Edge sites connect to pre-existing {CephCluster} clusters not managed by {OpenStackPreviousInstaller}.
+
+Central site {Ceph} migration:: For the central site, migrate {Ceph} daemons from the {OpenStackShort} Controller nodes by using the same process as a non-DCN deployment. For more information, see xref:ceph-daemon-cardinality_migrating-ceph[{Ceph} daemon cardinality].
+
+Edge site {Ceph} migration:: For edge sites that use HCI or {OpenStackPreviousInstaller}-deployed dedicated storage, the {Ceph} daemons can continue to run on their current nodes without migration. The Compute nodes or dedicated storage nodes at edge sites are not decommissioned during adoption, so the {Ceph} daemons remain operational.
++
+For edge sites that use external {Ceph} clusters, no migration is required because the {CephCluster} cluster is not managed by {OpenStackPreviousInstaller}.
+
+{Ceph} back-end configuration and key distribution:: In a DCN deployment, each site has its own {CephCluster} cluster with its own configuration file and {Ceph} keyring. These must be stored in Kubernetes secrets and mounted into the appropriate {rhos_long} service pods.
++
+Rather than storing all {Ceph} keys in a single secret accessible to every pod, the recommended approach is to create one secret per site containing only the keys that site actually needs. This limits the security impact if a site is compromised: a pod at an edge site can authenticate only to its local {CephCluster} cluster and the central cluster, not to the {Ceph} keyrings of other edge sites.
++
+The key distribution rule for N sites is:
++
+* The central site (site 0) receives the {Ceph} keys and configuration for all clusters, because central services such as {image_service} use the `split` back end and must be able to copy images to and from any site.
+* Each edge site (site 1 through N) receives only the keys for the central cluster and its own local cluster.
++
+For example, in a three-site deployment with a central site and two edge sites:
++
+----
+ceph-conf-central  ->  central.conf + central.keyring
+                       dcn1.conf    + dcn1.keyring
+                       dcn2.conf    + dcn2.keyring
+
+ceph-conf-dcn1     ->  central.conf + central.keyring
+                       dcn1.conf    + dcn1.keyring
+
+ceph-conf-dcn2     ->  central.conf + central.keyring
+                       dcn2.conf    + dcn2.keyring
+----
++
+The per-site secrets are created and then mounted into the appropriate pods using `extraMounts` propagation labels. The procedure in xref:configuring-a-ceph-backend_migrating-databases[Configuring a {Ceph} back end] covers both creating the secrets and applying the propagation labels so that each pod receives only its site-specific keys.

docs_user/modules/proc_adopting-block-storage-service-with-dcn-backend.adoc

@@ -0,0 +1,216 @@
+:_mod-docs-content-type: PROCEDURE
+[id="adopting-block-storage-service-with-dcn-backend_{context}"]
+
+= Adopt the {block_storage} with multiple {Ceph} back ends (DCN)
+
+[role="_abstract"]
+Adopt the {block_storage_first_ref} in a Distributed Compute Node (DCN) deployment where multiple {CephCluster} clusters provide storage at different sites. You can deploy multiple `CinderVolume` instances, one for each availability zone, with each volume service configured to use its local {Ceph} cluster.
+
+During adoption, the {block_storage} volume services that ran on edge site Compute nodes are migrated to run on {rhocp_long} at the central site. Although the control path for API requests now traverses the WAN to reach the {block_storage} running on {rhocp_long}, the data path remains local. Volume data continues to be stored in the {Ceph} cluster at each edge site. When you create a volume or clone a volume from a snapshot, the operation occurs entirely within the local {Ceph} cluster. This preserves data locality.
+
+.Prerequisites
+
+* You have completed the previous adoption steps.
+* The per-site {Ceph} secrets (`ceph-conf-central`, `ceph-conf-dcn1`, `ceph-conf-dcn2`) exist and contain the configuration and keyrings for each site's {Ceph} cluster. For more information, see xref:configuring-a-ceph-backend_migrating-databases[Configuring a {Ceph} back end].
+* The `extraMounts` property of the `OpenStackControlPlane` custom resource (CR) is configured to mount the {Ceph} configuration to all {block_storage} instances.
+* You have stopped the {block_storage} on all DCN nodes. For more information, see xref:stopping-openstack-services_{context}[Stopping {rhos_prev_long} services]. On edge sites, the {block_storage} volume service runs on Compute nodes with the service name `tripleo_cinder_volume.service`.
+
+.Procedure
+
+. Retrieve the `fsid` for each {Ceph} cluster in your DCN deployment. The `fsid` is used as the `rbd_secret_uuid` for libvirt integration:
++
+----
+$ oc get secret ceph-conf-central -o json | jq -r '.data | to_entries[] | select(.key | endswith(".conf")) | "\(.key): \(.value | @base64d)"' | grep fsid
+----
+
+. Create a patch file for the {block_storage} with multiple {Ceph} back ends. The following example shows a DCN deployment with a central site and two edge sites:
++
+[subs="+quotes"]
+----
+$ cat << EOF > cinder_dcn_patch.yaml
+spec:
+  cinder:
+    enabled: true
+    template:
+      cinderAPI:
+        customServiceConfig: |
+          [DEFAULT]
+          default_availability_zone = az-central
+      cinderScheduler:
+        replicas: 1
+      cinderVolumes:
+        central:
+          networkAttachments:
+          - storage
+          replicas: 1
+          customServiceConfig: |
+            [DEFAULT]
+            enabled_backends = central
+            glance_api_servers = http://glance-central-internal.openstack.svc:9292
+            [central]
+            backend_host = hostgroup
+            volume_backend_name = central
+            volume_driver = cinder.volume.drivers.rbd.RBDDriver
+            rbd_ceph_conf = /etc/ceph/central.conf
+            rbd_user = openstack
+            rbd_pool = volumes
+            rbd_flatten_volume_from_snapshot = False
+            report_discard_supported = True
+            rbd_secret_uuid = *<central_fsid>*
+            rbd_cluster_name = central
+            backend_availability_zone = az-central
+        dcn1:
+          networkAttachments:
+          - storage
+          replicas: 1
+          customServiceConfig: |
+            [DEFAULT]
+            enabled_backends = dcn1
+            glance_api_servers = http://glance-dcn1-internal.openstack.svc:9292
+            [dcn1]
+            backend_host = hostgroup
+            volume_backend_name = dcn1
+            volume_driver = cinder.volume.drivers.rbd.RBDDriver
+            rbd_ceph_conf = /etc/ceph/dcn1.conf
+            rbd_user = openstack
+            rbd_pool = volumes
+            rbd_flatten_volume_from_snapshot = False
+            report_discard_supported = True
+            rbd_secret_uuid = *<dcn1_fsid>*
+            rbd_cluster_name = dcn1
+            backend_availability_zone = az-dcn1
+        dcn2:
+          networkAttachments:
+          - storage
+          replicas: 1
+          customServiceConfig: |
+            [DEFAULT]
+            enabled_backends = dcn2
+            glance_api_servers = http://glance-dcn2-internal.openstack.svc:9292
+            [dcn2]
+            backend_host = hostgroup
+            volume_backend_name = dcn2
+            volume_driver = cinder.volume.drivers.rbd.RBDDriver
+            rbd_ceph_conf = /etc/ceph/dcn2.conf
+            rbd_user = openstack
+            rbd_pool = volumes
+            rbd_flatten_volume_from_snapshot = False
+            report_discard_supported = True
+            rbd_secret_uuid = *<dcn2_fsid>*
+            rbd_cluster_name = dcn2
+            backend_availability_zone = az-dcn2
+EOF
+----
++
+where:
+
+`<central_fsid>`::
+Specifies the `fsid` of the central {Ceph} cluster, used as the libvirt secret UUID.
+
+`<dcn1_fsid>`::
+Specifies the `fsid` of the DCN1 edge {Ceph} cluster.
+
+`<dcn2_fsid>`::
+Specifies the `fsid` of the DCN2 edge {Ceph} cluster.
+
++
+[NOTE]
+====
+* You must configure each `CinderVolume` with the `backend_availability_zone` value that matches your {compute_service} availability zone for that site, because `cross_az_attach = False` is set in the {compute_service} configuration. If the names do not match, instances cannot attach volumes. Replace the examples (`az-central`, `az-dcn1`, `az-dcn2`) with the names used in your {rhos_prev_long} deployment.
+* Each `CinderVolume` points to its local {image_service} API endpoint through `glance_api_servers`. This ensures that volume creation from images uses the local {image_service} and {Ceph} cluster. The examples use `http://` for the {image_service} endpoints. If your {rhos_prev_long} deployment uses TLS for internal endpoints, use `https://` instead, and ensure that you have completed the TLS migration. For more information, see xref:migrating-tls-everywhere_{context}[Migrating TLS-e to the RHOSO deployment].
+* The `rbd_cluster_name` setting identifies which {Ceph} cluster configuration to use from the mounted secrets.
+* Adjust the number of edge sites and their names to match your DCN deployment.
+====
+
+. Patch the `OpenStackControlPlane` CR to deploy the {block_storage} with multiple {Ceph} back ends:
++
+----
+$ oc patch openstackcontrolplane openstack --type=merge --patch-file cinder_dcn_patch.yaml
+----
+
+. Configure the {block_storage} backup service. In this example DCN deployment the backup service runs at the central site and uses the central {Ceph} cluster. Add the `cinderBackups` section to your patch file and re-apply it:
++
+[subs="+quotes"]
+----
+$ cat << EOF >> cinder_dcn_patch.yaml
+      cinderBackups:
+        central:
+          networkAttachments:
+          - storage
+          replicas: 1
+          customServiceConfig: |
+            [DEFAULT]
+            backup_driver=cinder.backup.drivers.ceph.CephBackupDriver
+            backup_ceph_conf=/etc/ceph/central.conf
+            backup_ceph_user=openstack
+            backup_ceph_pool=backups
+            storage_availability_zone=az-central
+EOF
+$ oc patch openstackcontrolplane openstack --type=merge --patch-file cinder_dcn_patch.yaml
+----
++
+[NOTE]
+====
+Unlike a single-site {Ceph} deployment where the backup config references `/etc/ceph/ceph.conf`, in a DCN deployment the {Ceph} configuration files in the `ceph-conf-files` secret are named by cluster. Set `backup_ceph_conf` to the path of the {Ceph} configuration file for whichever cluster hosts your `backups` pool. In this example the file is named `central.conf`, so the path is `/etc/ceph/central.conf`. Using a path that does not match a file in the secret will cause the backup service to fail with a `conf_read_file` error.
+
+Set `storage_availability_zone` to match the availability zone of the volumes you want to back up. The backup scheduler uses this to route backup requests to a service in the correct zone. If the backup service zone does not match the volume zone, backup creation fails with `Service not found for creating backup`.
+====
+
+. Verify that the {block_storage} volume services are running for each availability zone:
++
+----
+$ openstack volume service list --service cinder-volume
+
++------------------+---------------------+------------+---------+-------+----------------------------+
+| Binary           | Host                | Zone       | Status  | State | Updated At                 |
++------------------+---------------------+------------+---------+-------+----------------------------+
+| cinder-volume    | hostgroup@central   | az-central | enabled | up    | 2024-01-01T00:00:00.000000 |
+| cinder-volume    | hostgroup@dcn1      | az-dcn1    | enabled | up    | 2024-01-01T00:00:00.000000 |
+| cinder-volume    | hostgroup@dcn2      | az-dcn2    | enabled | up    | 2024-01-01T00:00:00.000000 |
++------------------+---------------------+------------+---------+-------+----------------------------+
+----
+
+. Verify that the {block_storage} backup service is running and in the correct availability zone:
++
+----
+$ openstack volume service list --service cinder-backup
+
++---------------+-------------------------+------------+---------+-------+----------------------------+
+| Binary        | Host                    | Zone       | Status  | State | Updated At                 |
++---------------+-------------------------+------------+---------+-------+----------------------------+
+| cinder-backup | cinder-backup-central-0 | az-central | enabled | up    | 2024-01-01T00:00:00.000000 |
++---------------+-------------------------+------------+---------+-------+----------------------------+
+----
+
+. Test the backup service by creating a volume, backing it up, and restoring the backup:
++
+----
+$ openstack volume create --size 1 backup-test-vol
+
+$ openstack volume backup create --name backup-test-backup backup-test-vol
+
+$ openstack volume backup show backup-test-backup
++-----------------------+--------------------------------------+
+| Field                 | Value                                |
++-----------------------+--------------------------------------+
+| container             | backups                              |
+| fail_reason           | None                                 |
+| name                  | backup-test-backup                   |
+| size                  | 1                                    |
+| status                | available                            |
++-----------------------+--------------------------------------+
+
+$ openstack volume backup restore backup-test-backup backup-test-restore
+----
++
+[NOTE]
+====
+Some versions of the {rhocp_long} client display a `cannot unpack non-iterable VolumeBackupsRestore object` error after the restore command. This is a known issue in the client, the restore operation might not have failed. Verify by checking the restored volume status directly.
+====
++
+----
+$ openstack volume show backup-test-restore -c status -c availability_zone -c os-vol-host-attr:host -f value
+available
+az-central
+hostgroup@central#central
+----

docs_user/modules/proc_adopting-compute-services-to-the-data-plane.adoc

@@ -264,6 +264,12 @@ EOF
 [NOTE]
 For {Ceph} environments with multi-cell configurations, you must name configuration maps and {rhos_prev_long} data plane services similar to the following examples: `nova-custom-ceph-cellX` and `nova-compute-extraconfig-cellX`.
 
++
+[NOTE]
+====
+For Distributed Compute Node (DCN) deployments, do not use the single `nova-cells-global-config` ConfigMap. Create a per-site `ConfigMap` and per-site `OpenStackDataPlaneService` for each site in your DCN deployment. Each site's {compute_service} nodes require a different {Ceph} configuration and a different {image_service} endpoint. For more information, see xref:adopting-compute-services-with-dcn-backend_{context}[Adopting Compute services with multiple Ceph back ends (DCN)].
+====
+
 . Create the data plane services for {compute_service} cells to enable pre-upgrade workarounds, and to configure the Compute services for your chosen storage back end:
 +
 [subs="+quotes"]