Fix swift migration ring push and race condition in data check

Updated rings were only pushed to the first controller node, resulting
in a data movement loop: controller 1 pushed data to OpenShift and
removed it locally, while controller 2 and 3 were detecting missing data
on controller 1 (because they were still using old rings) and
replicating back to controller 1.

Update both the docs and the test role to distribute rings to all three
controller nodes and restart Swift services on each.

Also add the -ignore_readdir_race option to the find command that checks
for remaining .data/.db files on the standalone node. The replicator may
remove empty directories while find is iterating, causing spurious
errors.

JIRA: OSPCIX-1318

Signed-off-by: Christian Schwede <cschwede@redhat.com>

Author: cschwede

docs_user/modules/proc_migrating-object-storage-data-to-rhoso-nodes.adoc

@@ -98,6 +98,10 @@ ssh commands for your existing nodes that store the {object_storage} data:
 $ oc extract --confirm cm/swift-ring-files
 $ $CONTROLLER1_SSH "tar -C /var/lib/config-data/puppet-generated/swift/etc/swift/ -xzf -" < swiftrings.tar.gz
 $ $CONTROLLER1_SSH "systemctl restart tripleo_swift_*"
+$ $CONTROLLER2_SSH "tar -C /var/lib/config-data/puppet-generated/swift/etc/swift/ -xzf -" < swiftrings.tar.gz
+$ $CONTROLLER2_SSH "systemctl restart tripleo_swift_*"
+$ $CONTROLLER3_SSH "tar -C /var/lib/config-data/puppet-generated/swift/etc/swift/ -xzf -" < swiftrings.tar.gz
+$ $CONTROLLER3_SSH "systemctl restart tripleo_swift_*"
 ----
 
 . Track the replication progress by using the `swift-dispersion-report` tool:

tests/roles/swift_migration/tasks/main.yaml

@@ -57,7 +57,7 @@
   ansible.builtin.shell: |
     {{ shell_header }}
     CONTROLLER1_SSH="{{ controller1_ssh }}"
-    $CONTROLLER1_SSH "timeout 900s bash -c 'while \$(find /srv/node/ -type f -name \"*.db\" -o -name \"*.data\" | grep -q \".\"); do sleep 5; done'"
+    $CONTROLLER1_SSH "timeout 900s bash -c 'while \$(find /srv/node/ -ignore_readdir_race -type f -name \"*.db\" -o -name \"*.data\" | grep -q \".\"); do sleep 5; done'"
 
 - name: remove standalone node from rings
   ansible.builtin.shell: |

tests/roles/swift_migration/tasks/rebalance_and_wait.yaml

@@ -6,14 +6,23 @@
     swift-ring-tool forced_rebalance
     swift-ring-tool push'
 
-- name: push rings to standalone and restart swift services
+- name: push rings to controller nodes and restart swift services
   ansible.builtin.shell: |
     {{ shell_header }}
     {{ oc_header }}
     CONTROLLER1_SSH="{{ controller1_ssh }}"
+    CONTROLLER2_SSH="{{ controller2_ssh }}"
+    CONTROLLER3_SSH="{{ controller3_ssh }}"
+
     oc extract --confirm cm/swift-ring-files
-    $CONTROLLER1_SSH "tar -C /var/lib/config-data/puppet-generated/swift/etc/swift/ -xzf -" < swiftrings.tar.gz
-    $CONTROLLER1_SSH "systemctl restart tripleo_swift_*"
+
+    for i in {1..3}; do
+        SSH_CMD="CONTROLLER${i}_SSH"
+        if [ ! -z "${!SSH_CMD}" ]; then
+            ${!SSH_CMD} "tar -C /var/lib/config-data/puppet-generated/swift/etc/swift/ -xzf -" < swiftrings.tar.gz
+            ${!SSH_CMD} "systemctl restart tripleo_swift_*"
+        fi
+    done
 
 - name: wait until all replicas are 100% available after rebalance
   ansible.builtin.shell: |