mariadb_copy: wait for connectivity to TripleO Galera before sync check

[eduolivares]

Commit 824d9ed replaced sleep-based BGP workarounds with a retry loop that checks connectivity to the podified MariaDB, but did not add an equivalent check for the source TripleO Galera nodes. With BGP, the route to the mariadb-copy-data pod may take a few seconds to be advertised, causing the subsequent WSREP sync check to fail with "ERROR 2002 (HY000): Can't connect to MySQL server" (EINPROGRESS).

Add a retry-loop task that verifies TCP connectivity to every TripleO Galera member via 'select 1;' before running the non-retrying WSREP sync check. Also update the documentation procedure to include this new connectivity check step and fix the BGP note wording to correctly describe route advertisement direction.

Assisted-By: Claude (claude-opus-4.6)

[jistr]

/lgtm

This issue is what i was wondering about in my comment here but i didn't express myself clearly 😀 : #1361 (comment)

[eduolivares]

/lgtm

This issue is what i was wondering about in my comment here but i didn't express myself clearly 😀 : #1361 (comment)

Yes, you were right. My fault I didn't realize about that.

So BGP routes are required for the traffic that is routed through the mariadb-copy-data pod's internalapi interface (net1), but not for the traffic routed through ovn-k8s (eth0):

[zuul@controller-0 ~]$ oc rsh mariadb-copy-data ip r
default via 172.17.0.1 dev net1 
100.64.0.0/16 via 192.168.52.1 dev eth0 
169.254.0.5 via 192.168.52.1 dev eth0 
172.17.0.0/24 dev net1 proto kernel scope link src 172.17.0.30 
172.30.0.0/16 via 192.168.52.1 dev eth0 
192.168.32.0/19 via 192.168.52.1 dev eth0 
192.168.52.0/23 dev eth0 proto kernel scope link src 192.168.52.14 

Hence, without this PR, the pod can't connect to tripleo databases:

TASK [mariadb_copy : wait until podified MariaDB is reachable] *****************
ok: [localhost] => {"attempts": 1, "changed": false, "cmd": "
set -euxo pipefail
...
+ PODIFIED_MARIADB_IP[$CELL]=172.30.21.98
+ oc rsh mariadb-copy-data mysql -rsh 172.30.116.108 -uroot -p0bc0eabdaf235b184061b86a5607e3df -e 'select 1;'
...
TASK [mariadb_copy : check that the Galera database cluster(s) members are online and synced, for all cells] ***
fatal: [localhost]: FAILED! => {"changed": true, "cmd": "
set -euxo pipefail
...
+ oc rsh mariadb-copy-data mysql -h 99.99.0.9 -uroot -pojLPfiNyTh -e 'show global status like '\\''wsrep_local_state_comment'\\'''
+ grep -qE '\\bSynced\\b'
ERROR 2002 (HY000): Can't connect to MySQL server on '99.99.0.9' (115)
command terminated with exit code 1

More accurately, the problem is that it takes some time to advertise via BGP the mariadb-copy-data pod IP.

[jistr]

Fixes a CI job, approving.
/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jistr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [jistr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment