Add dcn_storage scenario for DCN adoption with Ceph HCI

Add a new scenario that deploys 3 Ceph clusters across DCN sites:
- Central: 3 ComputeHCI nodes with Ceph
- DCN1: 3 ComputeHCIDcn1 nodes with Ceph, Cinder, Glance
- DCN2: 3 ComputeHCIDcn2 nodes with Ceph, Cinder, Glance

Key Ceph changes:
- New dcn_storage.yaml scenario with pre_oc_run hooks to deploy
  Ceph via adoption_deploy_ceph.yml for each stack
- HCI compute roles include CephOSD service and StorageMgmt network
- Update ceph_backend_configuration role to handle >1 ceph cluster
- Document adoption for multiple ceph clusters

Key Glance changes:
- Update role stop_openstack_services to stop glance at DCN sites
- Update role glance_adoption to adopt multibackend glance
- Document adoption of DCN multibackend Glance

Key Cinder changes:
- Update role stop_openstack_services to stop cinder-volume
  and etcd at DCN sites
- Update role cinder_adoption to add cinder_volume_backend: dcn
- Adopt cinder-backup
- Document adoption of DCN multisite Cinder

Key Manila changes:
- Add manila_cephfs_conf_path and manila_cephfs_cluster_name variables
  to manila_adoption role, replacing hardcoded ceph.conf/ceph values.
- Override those variables to central in vars.dcn_storage.yaml
- Updated proc_deploying-file-systems-service-control-plane.adoc with
  a NOTE that Manila was central-only in TripleO DCN but can be deployed
  at edge sites in RHOSO, and bullet explaining the cluster-specific
  conf path

Key EDPM changes:
- Update data-plane-adoption role to handle many nodesets
  each containing differnt ceph and glance configurations
- Update proc_adopting-compute-services-to-the-data-plane.adoc
  to document how to adopt DCN compute nodes which ran
  storage services so that they access the APIs for those
  storage services on central OpenShift

This extends dcn_nostorage to support storage adoption with
independent Ceph clusters at each DCN site.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: John Fulton <fulton@redhat.com>

Author: fultonj

docs_user/assemblies/assembly_adopting-openstack-control-plane-services.adoc

@@ -32,6 +32,8 @@ include::../modules/proc_adopting-the-compute-service.adoc[leveloffset=+1]
 
 include::../modules/proc_adopting-the-block-storage-service.adoc[leveloffset=+1]
 
+include::../modules/proc_adopting-block-storage-service-with-dcn-backend.adoc[leveloffset=+1]
+
 include::../modules/proc_adopting-the-openstack-dashboard.adoc[leveloffset=+1]
 
 include::../assemblies/assembly_adopting-the-shared-file-systems-service.adoc[leveloffset=+1]

docs_user/assemblies/assembly_adopting-the-data-plane.adoc

@@ -24,6 +24,8 @@ include::../modules/proc_stopping-infrastructure-management-and-compute-services
 
 include::../modules/proc_adopting-compute-services-to-the-data-plane.adoc[leveloffset=+1]
 
+include::../modules/proc_adopting-compute-services-with-dcn-backend.adoc[leveloffset=+1]
+
 include::../modules/proc_performing-a-fast-forward-upgrade-on-compute-services.adoc[leveloffset=+1]
 
 include::../modules/proc_adopting-networker-services-to-the-data-plane.adoc[leveloffset=+1]

docs_user/assemblies/assembly_adopting-the-image-service.adoc

@@ -31,6 +31,8 @@ include::../modules/proc_adopting-image-service-with-nfs-backend.adoc[leveloffse
 
 include::../modules/proc_adopting-image-service-with-ceph-backend.adoc[leveloffset=+1]
 
+include::../modules/proc_adopting-image-service-with-dcn-backend.adoc[leveloffset=+1]
+
 include::../modules/proc_verifying-the-image-service-adoption.adoc[leveloffset=+1]
 
 ifdef::parent-context[:context: {parent-context}]

docs_user/assemblies/assembly_migrating-ceph-cluster.adoc

@@ -31,6 +31,7 @@ already part of the {CephCluster} cluster.
 
 Before you begin the migration, complete the tasks in your {rhos_prev_long} {rhos_prev_ver} environment. For more information, see xref:red-hat-ceph-storage-prerequisites_configuring-network[{Ceph} prerequisites].
 
+include::../modules/con_ceph-migration-dcn.adoc[leveloffset=+1]
 
 include::../modules/con_ceph-daemon-cardinality.adoc[leveloffset=+1]
 

docs_user/modules/con_ceph-migration-dcn.adoc

@@ -0,0 +1,52 @@
+:_mod-docs-content-type: CONCEPT
+[id="ceph-migration-dcn_{context}"]
+
+= {Ceph} migration for Distributed Compute Node deployments
+
+[role="_abstract"]
+Distributed Compute Node (DCN) deployments that host {CephCluster} clusters on compute nodes at edge sites have additional {Ceph} migration considerations because multiple {CephCluster} clusters exist across sites. This section applies only to DCN deployments where {Ceph}, {image_service}, and {block_storage} services are deployed on edge compute nodes.
+
+.Supported edge storage topologies
+
+DCN deployments support the following storage topologies at edge sites:
+
+* Hyperconverged Infrastructure (HCI): {Ceph} daemons run on Compute nodes at each edge site.
+* {OpenStackPreviousInstaller}-deployed dedicated storage: {Ceph} runs on separate storage nodes deployed by {OpenStackPreviousInstaller}.
+* External {Ceph} cluster: Edge sites connect to pre-existing {CephCluster} clusters not managed by {OpenStackPreviousInstaller}.
+
+.Central site {Ceph} migration
+
+For the central site, migrate {Ceph} daemons from the {OpenStackShort} Controller nodes by using the same process as a non-DCN deployment. For more information, see xref:ceph-daemon-cardinality_migrating-ceph[{Ceph} daemon cardinality].
+
+.Edge site {Ceph} migration
+
+For edge sites that use HCI or {OpenStackPreviousInstaller}-deployed dedicated storage, the {Ceph} daemons can continue to run on their current nodes without migration. The Compute nodes or dedicated storage nodes at edge sites are not decommissioned during adoption, so the {Ceph} daemons remain operational.
+
+For edge sites that use external {Ceph} clusters, no migration is required because the {CephCluster} cluster is not managed by {OpenStackPreviousInstaller}.
+
+.{Ceph} backend configuration and key distribution
+
+In a DCN deployment, each site has its own {CephCluster} cluster with its own configuration file and {CephX} keyring. These must be stored in Kubernetes secrets and mounted into the appropriate {rhos_long} service pods.
+
+Rather than storing all {CephX} keys in a single secret accessible to every pod, the recommended approach is to create one secret per site containing only the keys that site actually needs. This limits the security impact if a site is compromised: a pod at an edge site can authenticate only to its local {CephCluster} cluster and the central cluster, not to the {CephX} keyrings of other edge sites.
+
+The key distribution rule for N sites is:
+
+* The central site (site 0) receives the {CephX} keys and configuration for all clusters, because central services such as {image_service} use the `split` back end and must be able to copy images to and from any site.
+* Each edge site (site 1 through N) receives only the keys for the central cluster and its own local cluster.
+
+For example, in a three-site deployment with a central site and two edge sites:
+
+----
+ceph-conf-central  ->  central.conf + central.keyring
+                       dcn1.conf    + dcn1.keyring
+                       dcn2.conf    + dcn2.keyring
+
+ceph-conf-dcn1     ->  central.conf + central.keyring
+                       dcn1.conf    + dcn1.keyring
+
+ceph-conf-dcn2     ->  central.conf + central.keyring
+                       dcn2.conf    + dcn2.keyring
+----
+
+The per-site secrets are created and then mounted into the appropriate pods using `extraMounts` propagation labels. The procedure in xref:configuring-a-ceph-backend_migrating-databases[Configuring a {Ceph} back end] covers both creating the secrets and applying the propagation labels so that each pod receives only its site-specific keys.

docs_user/modules/proc_adopting-block-storage-service-with-dcn-backend.adoc

@@ -0,0 +1,217 @@
+:_mod-docs-content-type: PROCEDURE
+[id="adopting-block-storage-service-with-dcn-backend_{context}"]
+
+= Adopting the {block_storage} with multiple {Ceph} back ends (DCN)
+
+[role="_abstract"]
+Adopt the {block_storage_first_ref} in a Distributed Compute Node (DCN) deployment where multiple {CephCluster} clusters provide storage at different sites. This configuration deploys multiple `CinderVolume` instances, one for each availability zone, with each volume service configured to use its local {Ceph} cluster.
+
+.Architecture change during adoption
+
+During adoption, the {block_storage} volume services that ran on edge site compute nodes are migrated to run on {rhocp_long} at the central site. Although the control path for API requests now traverses the WAN to reach the {block_storage} running on {rhocp_long}, the data path remains local. Volume data continues to be stored in the {Ceph} cluster at each edge site. When you create a volume or clone a volume from a snapshot, the operation occurs entirely within the local {Ceph} cluster. This architecture preserves data locality and avoids transferring volume data across the WAN.
+
+.Prerequisites
+
+* You have completed the previous adoption steps.
+* The per-site {Ceph} secrets (`ceph-conf-central`, `ceph-conf-dcn1`, `ceph-conf-dcn2`) exist and contain the configuration and keyrings for each site's {Ceph} cluster. For more information, see xref:configuring-a-ceph-backend_migrating-databases[Configuring a {Ceph} back end].
+* The `extraMounts` property of the `OpenStackControlPlane` CR is configured to mount the {Ceph} configuration to all {block_storage} instances.
+* You have stopped the {block_storage} on all DCN nodes. For more information, see xref:stopping-openstack-services_{context}[Stopping {rhos_prev_long} services]. On edge sites, the {block_storage} volume service runs on compute nodes with the service name `tripleo_cinder_volume.service`.
+
+.Procedure
+
+. Retrieve the `fsid` for each {Ceph} cluster in your DCN deployment. The `fsid` is used as the `rbd_secret_uuid` for libvirt integration:
++
+----
+$ oc get secret ceph-conf-central -o json | jq -r '.data | to_entries[] | select(.key | endswith(".conf")) | "\(.key): \(.value | @base64d)"' | grep fsid
+----
+
+. Create a patch file for the {block_storage} with multiple {Ceph} back ends. The following example shows a DCN deployment with a central site and two edge sites:
++
+[subs="+quotes"]
+----
+$ cat << EOF > cinder_dcn_patch.yaml
+spec:
+  cinder:
+    enabled: true
+    template:
+      cinderAPI:
+        customServiceConfig: |
+          [DEFAULT]
+          default_availability_zone = az-central
+      cinderScheduler:
+        replicas: 1
+      cinderVolumes:
+        central:
+          networkAttachments:
+          - storage
+          replicas: 1
+          customServiceConfig: |
+            [DEFAULT]
+            enabled_backends = central
+            glance_api_servers = http://glance-central-internal.openstack.svc:9292
+            [central]
+            backend_host = hostgroup
+            volume_backend_name = central
+            volume_driver = cinder.volume.drivers.rbd.RBDDriver
+            rbd_ceph_conf = /etc/ceph/central.conf
+            rbd_user = openstack
+            rbd_pool = volumes
+            rbd_flatten_volume_from_snapshot = False
+            report_discard_supported = True
+            rbd_secret_uuid = *<central_fsid>*
+            rbd_cluster_name = central
+            backend_availability_zone = az-central
+        dcn1:
+          networkAttachments:
+          - storage
+          replicas: 1
+          customServiceConfig: |
+            [DEFAULT]
+            enabled_backends = dcn1
+            glance_api_servers = http://glance-dcn1-internal.openstack.svc:9292
+            [dcn1]
+            backend_host = hostgroup
+            volume_backend_name = dcn1
+            volume_driver = cinder.volume.drivers.rbd.RBDDriver
+            rbd_ceph_conf = /etc/ceph/dcn1.conf
+            rbd_user = openstack
+            rbd_pool = volumes
+            rbd_flatten_volume_from_snapshot = False
+            report_discard_supported = True
+            rbd_secret_uuid = *<dcn1_fsid>*
+            rbd_cluster_name = dcn1
+            backend_availability_zone = az-dcn1
+        dcn2:
+          networkAttachments:
+          - storage
+          replicas: 1
+          customServiceConfig: |
+            [DEFAULT]
+            enabled_backends = dcn2
+            glance_api_servers = http://glance-dcn2-internal.openstack.svc:9292
+            [dcn2]
+            backend_host = hostgroup
+            volume_backend_name = dcn2
+            volume_driver = cinder.volume.drivers.rbd.RBDDriver
+            rbd_ceph_conf = /etc/ceph/dcn2.conf
+            rbd_user = openstack
+            rbd_pool = volumes
+            rbd_flatten_volume_from_snapshot = False
+            report_discard_supported = True
+            rbd_secret_uuid = *<dcn2_fsid>*
+            rbd_cluster_name = dcn2
+            backend_availability_zone = az-dcn2
+EOF
+----
++
+where:
+
+<central_fsid>::
+Specifies the `fsid` of the central {Ceph} cluster, used as the libvirt secret UUID.
+
+<dcn1_fsid>::
+Specifies the `fsid` of the DCN1 edge {Ceph} cluster.
+
+<dcn2_fsid>::
+Specifies the `fsid` of the DCN2 edge {Ceph} cluster.
+
++
+[NOTE]
+====
+* Each `CinderVolume` is configured with `backend_availability_zone` matching the {compute_service} availability zone for that site. The {block_storage} availability zone names must match the {compute_service} availability zone names exactly, because `cross_az_attach = False` is set in the {compute_service} configuration. If the names do not match, instances cannot attach volumes. Replace the example availability zone names (`az-central`, `az-dcn1`, `az-dcn2`) with the names used in your {rhos_prev_long} deployment.
+* Each `CinderVolume` points to its local {image_service} API endpoint via `glance_api_servers`. This ensures that volume creation from images uses the local {image_service} and {Ceph} cluster. The examples use `http://` for the {image_service} endpoints. If your {rhos_prev_long} deployment uses TLS for internal endpoints, use `https://` instead, and ensure that you have completed the TLS migration. For more information, see xref:migrating-tls-everywhere_{context}[Migrating TLS-e to the RHOSO deployment].
+* The `rbd_cluster_name` setting identifies which {Ceph} cluster configuration to use from the mounted secrets.
+* Adjust the number of edge sites and their names to match your DCN deployment.
+====
+
+. Patch the `OpenStackControlPlane` CR to deploy the {block_storage} with multiple {Ceph} back ends:
++
+----
+$ oc patch openstackcontrolplane openstack --type=merge --patch-file cinder_dcn_patch.yaml
+----
+
+. Configure the {block_storage} backup service. In this example DCN deployment the backup service runs at the central site and uses the central {Ceph} cluster. Add the `cinderBackup` section to your patch file and re-apply it:
++
+[subs="+quotes"]
+----
+$ cat << EOF >> cinder_dcn_patch.yaml
+      cinderBackup:
+        networkAttachments:
+        - storage
+        replicas: 1
+        customServiceConfig: |
+          [DEFAULT]
+          backup_driver=cinder.backup.drivers.ceph.CephBackupDriver
+          backup_ceph_conf=/etc/ceph/central.conf
+          backup_ceph_user=openstack
+          backup_ceph_pool=backups
+          storage_availability_zone=az-central
+EOF
+$ oc patch openstackcontrolplane openstack --type=merge --patch-file cinder_dcn_patch.yaml
+----
++
+[NOTE]
+====
+Unlike a single-site {Ceph} deployment where the backup config references `/etc/ceph/ceph.conf`, in a DCN deployment the {Ceph} configuration files in the `ceph-conf-files` secret are named by cluster. Set `backup_ceph_conf` to the path of the {Ceph} configuration file for whichever cluster hosts your `backups` pool. In this example the file is named `central.conf`, so the path is `/etc/ceph/central.conf`. Using a path that does not match a file in the secret will cause the backup service to fail with a `conf_read_file` error.
+
+Set `storage_availability_zone` to match the availability zone of the volumes you want to back up. The backup scheduler uses this to route backup requests to a service in the correct zone. If the backup service zone does not match the volume zone, backup creation fails with `Service not found for creating backup`.
+====
+
+. Verify that the {block_storage} volume services are running for each availability zone:
++
+----
+$ openstack volume service list --service cinder-volume
+
++------------------+---------------------+------------+---------+-------+----------------------------+
+| Binary           | Host                | Zone       | Status  | State | Updated At                 |
++------------------+---------------------+------------+---------+-------+----------------------------+
+| cinder-volume    | hostgroup@central   | az-central | enabled | up    | 2024-01-01T00:00:00.000000 |
+| cinder-volume    | hostgroup@dcn1      | az-dcn1    | enabled | up    | 2024-01-01T00:00:00.000000 |
+| cinder-volume    | hostgroup@dcn2      | az-dcn2    | enabled | up    | 2024-01-01T00:00:00.000000 |
++------------------+---------------------+------------+---------+-------+----------------------------+
+----
+
+. Verify that the {block_storage} backup service is running and in the correct availability zone:
++
+----
+$ openstack volume service list --service cinder-backup
+
++---------------+-----------------+------------+---------+-------+----------------------------+
+| Binary        | Host            | Zone       | Status  | State | Updated At                 |
++---------------+-----------------+------------+---------+-------+----------------------------+
+| cinder-backup | cinder-backup-0 | az-central | enabled | up    | 2024-01-01T00:00:00.000000 |
++---------------+-----------------+------------+---------+-------+----------------------------+
+----
+
+. Test the backup service by creating a volume, backing it up, and restoring the backup:
++
+----
+$ openstack volume create --size 1 backup-test-vol
+
+$ openstack volume backup create --name backup-test-backup backup-test-vol
+
+$ openstack volume backup show backup-test-backup
++-----------------------+--------------------------------------+
+| Field                 | Value                                |
++-----------------------+--------------------------------------+
+| container             | backups                              |
+| fail_reason           | None                                 |
+| name                  | backup-test-backup                   |
+| size                  | 1                                    |
+| status                | available                            |
++-----------------------+--------------------------------------+
+
+$ openstack volume backup restore backup-test-backup backup-test-restore
+----
++
+[NOTE]
+====
+Some versions of the {rhocp_long} client display a `cannot unpack non-iterable VolumeBackupsRestore object` error after the restore command. This is a display bug in the client so it does not necessarily mean that the restore operation failed. Verify by checking the restored volume status directly.
+====
++
+----
+$ openstack volume show backup-test-restore -c status -c availability_zone -c os-vol-host-attr:host -f value
+available
+az-central
+hostgroup@central#central
+----

docs_user/modules/proc_adopting-compute-services-to-the-data-plane.adoc

@@ -264,6 +264,12 @@ EOF
 [NOTE]
 For {Ceph} environments with multi-cell configurations, you must name configuration maps and {rhos_prev_long} data plane services similar to the following examples: `nova-custom-ceph-cellX` and `nova-compute-extraconfig-cellX`.
 
++
+[NOTE]
+====
+For Distributed Compute Node (DCN) deployments where {Ceph}, {image_service}, and {block_storage} services run on edge compute nodes, do not use the single `nova-cells-global-config` ConfigMap shown above. Instead, create a per-site `ConfigMap` and per-site `OpenStackDataPlaneService` for each site in your DCN deployment. Each site's {compute_service} nodes require a different {Ceph} configuration and a different {image_service} endpoint. For more information, see xref:adopting-compute-services-with-dcn-backend_{context}[Adopting Compute services with multiple Ceph back ends (DCN)].
+====
+
 . Create the data plane services for {compute_service} cells to enable pre-upgrade workarounds, and to configure the Compute services for your chosen storage back end:
 +
 [subs="+quotes"]