[ovn] Update backup/restore procedure

With current backup/restore uuids for the records are being
changed and that's result into geneve tunnels to be
disabled with the compute nodes and thus centralized
SNAT traffic is impacted when ovn_controller are
started post adoption on the networker nodes. The
traffic remains impacted until ovn_controllers are started
also on the compute nodes.

So to fix this switch to use full db files for restoration
after converting the DBs to standalone. With this also
cleaned the unused resources.

Resolves: #OSPRH-29914
Signed-off-by: Yatin Karel <ykarel@redhat.com>

Author: karelyatin

docs_user/modules/proc_migrating-ovn-data.adoc

@@ -11,8 +11,8 @@ Migrate the data in the OVN databases from the original {rhos_prev_long} deploym
 * The `OpenStackControlPlane` resource is created.
 * `NetworkAttachmentDefinition` custom resources (CRs) for the original cluster are defined. Specifically, the `internalapi` network is defined.
 * The original {networking_first_ref} and OVN `northd` are not running.
-* There is network routability between the control plane services and the adopted cluster.
 * The cloud is migrated to the Modular Layer 2 plug-in with Open Virtual Networking (ML2/OVN) mechanism driver.
+* SSH access is available to the original {rhos_prev_long} Controller nodes for direct database file access.
 * Define the following shell variables. Replace the example values with values that are correct for your environment:
 +
 ----
@@ -24,28 +24,15 @@ ifeval::["{build}" == "downstream"]
 STORAGE_CLASS=local-storage
 OVSDB_IMAGE=registry.redhat.io/rhoso/openstack-ovn-base-rhel9:18.0
 endif::[]
-SOURCE_OVSDB_IP=172.17.0.100 # For IPv4
-SOURCE_OVSDB_IP=[fd00:bbbb::100] # For IPv6
 ----
 +
 ifeval::["{build_variant}" == "ospdo"]
-Update the IP address value for `SOURCE_OVSDB_IP` to the internalApi IP address that is associated with the remaining RHOSP 17 controller VM. The IP address can be retrieved by using the following command:
-----
-$ $CONTROLLER_SSH ip a s enp4s0
-** Select the non /32 IP address
-----
 [NOTE]
 If you use a disconnected environment director Operator deployment, use
 `OVSDB_IMAGE=registry.redhat.io/rhoso/openstack-ovn-base-rhel9@sha256:967046c6bdb8f55c236085b5c5f9667f0dbb9f3ac52a6560dd36a6bfac051e1f`.
 For more information, see link:https://docs.redhat.com/en/documentation/red_hat_openstack_platform/17.1/html-single/deploying_an_overcloud_in_a_red_hat_openshift_container_platform_cluster_with_director_operator/index#proc_configuring-an-airgapped-environment_air-gapped-environment[Configuring an airgapped environment] in _Deploying an overcloud in a Red Hat OpenShift Container Platform cluster with director Operator_.
 endif::[]
-ifeval::["{build_variant}" != "ospdo"]
-To get the value to set `SOURCE_OVSDB_IP`, query the puppet-generated configurations in a Controller node:
-+
-----
-$ grep -rI 'ovn_[ns]b_conn' /var/lib/config-data/puppet-generated/
-----
-endif::[]
+* Ensure you have SSH access to the original {rhos_prev_long} Controller nodes. This procedure extracts database files directly from the controller filesystem.
 
 .Procedure
 ifeval::["{build_variant}" == "ospdo"]
@@ -57,67 +44,97 @@ $ oc get vmi -n $<ospdo_namespace> -o jsonpath='{.items[0].metadata.labels.kubev
 +
 * Replace `<ospdo_namespace>` with your OSPdO namespace.
 endif::[]
-. Prepare a temporary `PersistentVolume` claim and the helper pod for the OVN backup. Adjust the storage requests for a large database, if needed:
+. Convert the OVN cluster databases to standalone format on the Controller node and copy them to the local system:
++
+----
+# Set up variables for the controller SSH connection
+CONTROLLER1_SSH="ssh -i ~/.ssh/id_rsa heat-admin@$CONTROLLER_IP"
+
+# Create working directory on controller and convert databases
+$CONTROLLER1_SSH sudo mkdir -p /tmp/ovn_standalone_conversion
+$CONTROLLER1_SSH sudo ovsdb-tool cluster-to-standalone /tmp/ovn_standalone_conversion/ovnnb_db.db /var/lib/openvswitch/ovn/ovnnb_db.db
+$CONTROLLER1_SSH sudo ovsdb-tool cluster-to-standalone /tmp/ovn_standalone_conversion/ovnsb_db.db /var/lib/openvswitch/ovn/ovnsb_db.db
+$CONTROLLER1_SSH sudo chmod 644 /tmp/ovn_standalone_conversion/ovn*.db
+
+# Create local directory and copy converted databases
+mkdir -p /tmp/ovn_adoption_dbs
+CONTROLLER1_SCP=$(echo "$CONTROLLER1_SSH" | sed 's/^ssh/scp/')
+${CONTROLLER1_SCP}:/tmp/ovn_standalone_conversion/ovnnb_db.db /tmp/ovn_adoption_dbs/
+${CONTROLLER1_SCP}:/tmp/ovn_standalone_conversion/ovnsb_db.db /tmp/ovn_adoption_dbs/
+
+# Cleanup on controller
+$CONTROLLER1_SSH sudo rm -rf /tmp/ovn_standalone_conversion
+----
++
+
+. Create the PersistentVolumeClaim resources that will be used by the OVN database pods:
 +
 ----
+ifeval::["{build_variant}" != "ospdo"]
 $ oc apply -f - <<EOF
+endif::[]
+ifeval::["{build_variant}" == "ospdo"]
+$ oc apply -n $<ospdo_namespace> -f - <<EOF
+endif::[]
 ---
-apiVersion: cert-manager.io/v1
-kind: Certificate
+apiVersion: v1
+kind: PersistentVolumeClaim
 metadata:
-  name: ovn-data-cert
+  name: ovndbcluster-nb-etc-ovn-ovsdbserver-nb-0
+  labels:
+    service: ovsdbserver-nb
 spec:
-  commonName: ovn-data-cert
-  secretName: ovn-data-cert
-  issuerRef:
-    name: rootca-internal
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: $STORAGE_CLASS
+  resources:
+    requests:
+      storage: 10G
 ---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
-ifeval::["{build_variant}" == "ospdo"]
-  namespace: $OSPDO_NAMESPACE
-endif::[]
-  name: ovn-data
+  name: ovndbcluster-sb-etc-ovn-ovsdbserver-sb-0
+  labels:
+    service: ovsdbserver-sb
 spec:
-  storageClassName: $STORAGE_CLASS
   accessModes:
     - ReadWriteOnce
+  storageClassName: $STORAGE_CLASS
   resources:
     requests:
-      storage: 10Gi
+      storage: 10G
+EOF
+----
+
+. Create a helper pod to populate the PVCs with the converted databases:
++
+----
+ifeval::["{build_variant}" != "ospdo"]
+$ oc apply -f - <<EOF
+endif::[]
+ifeval::["{build_variant}" == "ospdo"]
+$ oc apply -n $<ospdo_namespace> -f - <<EOF
+endif::[]
 ---
 apiVersion: v1
 kind: Pod
 metadata:
   name: ovn-copy-data
   annotations:
     openshift.io/scc: anyuid
-ifeval::["{build_variant}" != "ospdo"]
-    k8s.v1.cni.cncf.io/networks: internalapi
-endif::[]
-ifeval::["{build_variant}" == "ospdo"]
-  '[{"name": "internalapi-static", "namespace": $<ospdo_namespace>, "ips": ["<internalapi-static-ips>"]}]'
-endif::[]
   labels:
     app: adoption
-ifeval::["{build_variant}" == "ospdo"]
-  namespace: $OSPDO_NAMESPACE
-endif::[]
 spec:
-ifeval::["{build_variant}" == "ospdo"]
-  nodeName: '{{ <ocp_node_holding_controller> }}'
-endif::[]
   containers:
   - image: $OVSDB_IMAGE
     command: [ "sh", "-c", "sleep infinity"]
     name: adoption
     volumeMounts:
-    - mountPath: /backup
-      name: ovn-data
-    - mountPath: /etc/pki/tls/misc
-      name: ovn-data-cert
-      readOnly: true
+    - mountPath: /nb-data
+      name: nb-data
+    - mountPath: /sb-data
+      name: sb-data
   securityContext:
     allowPrivilegeEscalation: false
     capabilities:
@@ -126,62 +143,58 @@ endif::[]
     seccompProfile:
       type: RuntimeDefault
   volumes:
-  - name: ovn-data
+  - name: nb-data
     persistentVolumeClaim:
-      claimName: ovn-data
-  - name: ovn-data-cert
-    secret:
-      secretName: ovn-data-cert
+      claimName: ovndbcluster-nb-etc-ovn-ovsdbserver-nb-0
+  - name: sb-data
+    persistentVolumeClaim:
+      claimName: ovndbcluster-sb-etc-ovn-ovsdbserver-sb-0
 EOF
 ----
-+
-ifeval::["{build_variant}" == "ospdo"]
-where:
 
-<ocp_node_holding_controller>::
-Specifies the {OpenStackShort} node that contains the Controller node.
-endif::[]
-
-. Wait for the pod to be ready:
+. Wait for the helper pod to be ready:
 +
 ----
 ifeval::["{build_variant}" != "ospdo"]
 $ oc wait --for=condition=Ready pod/ovn-copy-data --timeout=30s
 endif::[]
 ifeval::["{build_variant}" == "ospdo"]
-$ oc wait --for=condition=Ready -n $OSPDO_NAMESPACE pod/ovn-copy-data --timeout=30s
+$ oc wait --for=condition=Ready pod/ovn-copy-data -n $<ospdo_namespace> --timeout=30s
 endif::[]
 ----
 
-. If the podified internalapi cidr is different than the source internalapi cidr, add an iptables accept rule on the Controller nodes:
+. Copy the converted standalone databases to the PVCs:
 +
 ----
-$ $CONTROLLER1_SSH sudo iptables -I INPUT -s {PODIFIED_INTERNALAPI_NETWORK} -p tcp -m tcp --dport 6641 -m conntrack --ctstate NEW -j ACCEPT
-$ $CONTROLLER1_SSH sudo iptables -I INPUT -s {PODIFIED_INTERNALAPI_NETWORK} -p tcp -m tcp --dport 6642 -m conntrack --ctstate NEW -j ACCEPT
+ifeval::["{build_variant}" != "ospdo"]
+$ oc cp /tmp/ovn_adoption_dbs/ovnnb_db.db ovn-copy-data:/nb-data/ovnnb_db.db
+$ oc cp /tmp/ovn_adoption_dbs/ovnsb_db.db ovn-copy-data:/sb-data/ovnsb_db.db
+endif::[]
+ifeval::["{build_variant}" == "ospdo"]
+$ oc cp /tmp/ovn_adoption_dbs/ovnnb_db.db ovn-copy-data:/nb-data/ovnnb_db.db -n $<ospdo_namespace>
+$ oc cp /tmp/ovn_adoption_dbs/ovnsb_db.db ovn-copy-data:/sb-data/ovnsb_db.db -n $<ospdo_namespace>
+endif::[]
 ----
 
-. Back up your OVN databases:
-* If you did not enable TLS everywhere, run the following command:
+. Remove the helper pod now that the data is in the PVCs:
 +
 ----
 ifeval::["{build_variant}" != "ospdo"]
-$ oc exec ovn-copy-data -- bash -c "ovsdb-client backup tcp:$SOURCE_OVSDB_IP:6641 > /backup/ovs-nb.db"
-$ oc exec ovn-copy-data -- bash -c "ovsdb-client backup tcp:$SOURCE_OVSDB_IP:6642 > /backup/ovs-sb.db"
+$ oc delete pod ovn-copy-data
 endif::[]
 ifeval::["{build_variant}" == "ospdo"]
-$ oc exec -n $OSPDO_NAMESPACE ovn-copy-data -- bash -c "ovsdb-client backup tcp:$SOURCE_OVSDB_IP:6641 > /backup/ovs-nb.db"
-$ oc exec -n $OSPDO_NAMESPACE ovn-copy-data -- bash -c "ovsdb-client backup tcp:$SOURCE_OVSDB_IP:6642 > /backup/ovs-sb.db"
+$ oc delete pod ovn-copy-data -n $<ospdo_namespace>
 endif::[]
 ----
-+
-* If you enabled TLS everywhere, run the following command:
+
+. Clean up the local temporary files:
 +
 ----
-$ oc exec ovn-copy-data -- bash -c "ovsdb-client backup --ca-cert=/etc/pki/tls/misc/ca.crt --private-key=/etc/pki/tls/misc/tls.key --certificate=/etc/pki/tls/misc/tls.crt ssl:$SOURCE_OVSDB_IP:6641 > /backup/ovs-nb.db"
-$ oc exec ovn-copy-data -- bash -c "ovsdb-client backup --ca-cert=/etc/pki/tls/misc/ca.crt --private-key=/etc/pki/tls/misc/tls.key --certificate=/etc/pki/tls/misc/tls.crt ssl:$SOURCE_OVSDB_IP:6642 > /backup/ovs-sb.db"
+$ rm -f /tmp/ovn_adoption_dbs/ovn*.db
+$ rmdir /tmp/ovn_adoption_dbs
 ----
 
-. Start the control plane OVN database services prior to import, with `northd` disabled:
+. Start the control plane OVN database services. The databases will automatically load the converted data that was populated in the PVCs:
 +
 ----
 $ oc patch openstackcontrolplane openstack --type=merge --patch '
@@ -200,70 +213,27 @@ spec:
           dbType: SB
           storageRequest: 10G
           networkAttachment: internalapi
-      ovnNorthd:
-        replicas: 0
 '
 ----
-
-. Wait for the OVN database services to reach the `Running` phase:
-+
-----
-$ oc wait --for=jsonpath='{.status.phase}'=Running pod --selector=service=ovsdbserver-nb
-$ oc wait --for=jsonpath='{.status.phase}'=Running pod --selector=service=ovsdbserver-sb
-----
-
-. Fetch the OVN database IP addresses on the `clusterIP` service network:
-+
-----
-PODIFIED_OVSDB_NB_IP=$(oc get svc --selector "statefulset.kubernetes.io/pod-name=ovsdbserver-nb-0" -ojsonpath='{.items[0].spec.clusterIP}')
-PODIFIED_OVSDB_SB_IP=$(oc get svc --selector "statefulset.kubernetes.io/pod-name=ovsdbserver-sb-0" -ojsonpath='{.items[0].spec.clusterIP}')
-----
-
-. If you are  using IPv6, adjust the address to the format expected by `ovsdb-*` tools:
-+
-----
-PODIFIED_OVSDB_NB_IP=[$PODIFIED_OVSDB_NB_IP]
-PODIFIED_OVSDB_SB_IP=[$PODIFIED_OVSDB_SB_IP]
-----
-
-. Upgrade the database schema for the backup files:
-.. If you did not enable TLS everywhere, use the following command:
-+
-----
-$ oc exec ovn-copy-data -- bash -c "ovsdb-client get-schema tcp:$PODIFIED_OVSDB_NB_IP:6641 > /backup/ovs-nb.ovsschema && ovsdb-tool convert /backup/ovs-nb.db /backup/ovs-nb.ovsschema"
-$ oc exec ovn-copy-data -- bash -c "ovsdb-client get-schema tcp:$PODIFIED_OVSDB_SB_IP:6642 > /backup/ovs-sb.ovsschema && ovsdb-tool convert /backup/ovs-sb.db /backup/ovs-sb.ovsschema"
-----
-
-.. If you enabled TLS everywhere, use the following command:
 +
-----
-$ oc exec ovn-copy-data -- bash -c "ovsdb-client get-schema --ca-cert=/etc/pki/tls/misc/ca.crt --private-key=/etc/pki/tls/misc/tls.key --certificate=/etc/pki/tls/misc/tls.crt ssl:$PODIFIED_OVSDB_NB_IP:6641 > /backup/ovs-nb.ovsschema && ovsdb-tool convert /backup/ovs-nb.db /backup/ovs-nb.ovsschema"
-$ oc exec ovn-copy-data -- bash -c "ovsdb-client get-schema --ca-cert=/etc/pki/tls/misc/ca.crt --private-key=/etc/pki/tls/misc/tls.key --certificate=/etc/pki/tls/misc/tls.crt ssl:$PODIFIED_OVSDB_SB_IP:6642 > /backup/ovs-sb.ovsschema && ovsdb-tool convert /backup/ovs-sb.db /backup/ovs-sb.ovsschema"
-----
-
-. Restore the database backup to the new OVN database servers:
-.. If you did not enable TLS everywhere, use the following command:
-+
-----
-ifeval::["{build_variant}" != "ospdo"]
-$ oc exec ovn-copy-data -- bash -c "ovsdb-client restore tcp:$PODIFIED_OVSDB_NB_IP:6641 < /backup/ovs-nb.db"
-$ oc exec ovn-copy-data -- bash -c "ovsdb-client restore tcp:$PODIFIED_OVSDB_SB_IP:6642 < /backup/ovs-sb.db"
-endif::[]
-----
+[NOTE]
+The OVN database pods will automatically handle schema conversion during startup and form a proper 3-replica cluster with the preserved data.
 
-.. If you enabled TLS everywhere, use the following command:
+. Wait for the OVN database services to start up:
 +
 ----
-$ oc exec ovn-copy-data -- bash -c "ovsdb-client restore --ca-cert=/etc/pki/tls/misc/ca.crt --private-key=/etc/pki/tls/misc/tls.key --certificate=/etc/pki/tls/misc/tls.crt ssl:$PODIFIED_OVSDB_NB_IP:6641 < /backup/ovs-nb.db"
-$ oc exec ovn-copy-data -- bash -c "ovsdb-client restore --ca-cert=/etc/pki/tls/misc/ca.crt --private-key=/etc/pki/tls/misc/tls.key --certificate=/etc/pki/tls/misc/tls.crt ssl:$PODIFIED_OVSDB_SB_IP:6642 < /backup/ovs-sb.db"
+$ oc wait pod --for condition=Ready --selector=service=ovsdbserver-nb --timeout=300s
+$ oc wait pod --for condition=Ready --selector=service=ovsdbserver-sb --timeout=300s
 ----
 
-. Check that the data was successfully migrated by running the following commands against the new database servers, for example:
+. Verify that the data was successfully migrated
 +
 ----
 $ oc exec -it ovsdbserver-nb-0 -- ovn-nbctl show
+$ oc exec -it ovsdbserver-sb-0 -- ovn-sbctl show
 $ oc exec -it ovsdbserver-sb-0 -- ovn-sbctl list Chassis
 ----
++
 
 . Start the control plane `ovn-northd` service to keep both OVN databases in sync:
 +
@@ -297,15 +267,12 @@ spec:
 [NOTE]
 Running OVN gateways on {OpenShiftShort} nodes might be prone to data plane downtime during Open vSwitch upgrades. Consider running OVN gateways on dedicated `Networker` data plane nodes for production deployments instead.
 
-. Delete the `ovn-data` helper pod and the temporary `PersistentVolumeClaim` that is used to store OVN database backup files:
-+
-----
-$ oc delete --ignore-not-found=true pod ovn-copy-data
-$ oc delete --ignore-not-found=true pvc ovn-data
-----
+. The migration is now complete.
+
+Consider creating a backup of the current state for rollback purposes:
 +
 [NOTE]
-Consider taking a snapshot of the `ovn-data` helper pod and the temporary `PersistentVolumeClaim` before deleting them. For more information, see link:{defaultOCPURL}/storage/index#lvms-about-volume-snapsot_logical-volume-manager-storage[About volume snapshots] in _OpenShift Container Platform storage overview_.
+The OVN databases now contain the migrated data with preserved UUIDs. Consider taking a snapshot of the database PVCs for backup purposes. For more information, see link:{defaultOCPURL}/storage/index#lvms-about-volume-snapsot_logical-volume-manager-storage[About volume snapshots] in _OpenShift Container Platform storage overview_.
 
 . Stop the adopted OVN database servers:
 +

tests/roles/ovn_adoption/handlers/main.yaml

@@ -1,8 +1 @@
-- name: delete adoption helper pod and pvc
-  ansible.builtin.shell: |
-    {{ shell_header }}
-    {{ oc_header }}
-    oc delete --ignore-not-found=true pod ovn-copy-data
-    oc delete --ignore-not-found=true certificate ovn-data-cert
-    oc delete --ignore-not-found=true secret ovn-data-cert
-    {% if storage_reclaim_policy.lower() == "delete" %}oc delete pvc --ignore-not-found=true ovn-data{% endif %}
+---