Skip to content

Commit e4d96e0

Browse files
gthiemongegthiemonge
committed
Fix Octavia adoption
- add octavia to the list of services to remove from the old control plane - remove SSL certificates adoption, keep the migration of the passphrase - remove SSH key migration adoption - add an option (prelaunch_octavia_workload) that creates a load balancer on the source cloud and check its availability before and after the adoption - trigger failover of the existing load balancer after the adoption of the dataplane (use the new VM image) - move clean up tasks after the load balancer migrated - update documentation JIRA: OSPRH-25502
1 parent 032df1c commit e4d96e0

14 files changed

Lines changed: 247 additions & 326 deletions

File tree

‎docs_user/modules/proc_adopting-the-loadbalancer-service.adoc‎

Lines changed: 39 additions & 115 deletions
Original file line numberDiff line numberDiff line change
@@ -4,51 +4,14 @@
44
= Adopting the {loadbalancer_service}
55

66
[role="_abstract"]
7-
To adopt the {loadbalancer_first_ref}, you patch an existing `OpenStackControlPlane` custom resource (CR) where the {loadbalancer_service} is disabled. The patch starts the service with the configuration parameters that are provided by the {rhos_prev_long} ({OpenStackShort}) environment. Existing load balancers fail over after the adoption process to upgrade their image, and set up a network connection with the new control plane.
7+
To adopt the {loadbalancer_first_ref}, you patch an existing `OpenStackControlPlane` custom resource (CR) where the {loadbalancer_service} is disabled. The patch starts the service with the configuration parameters that are provided by the {rhos_prev_long} ({OpenStackShort}) environment. After completing the data plane adoption, you must trigger a failover of existing load balancers to upgrade their amphora VMs to use the new image and establish connectivity with the new control plane.
88

99
.Procedure
1010

11-
. Create an alias for the `openstack` command:
11+
. Migrate the server certificate authority (CA) passphrase from the previous deployment:
1212
+
1313
----
14-
$ alias openstack="oc exec -t openstackclient -- openstack"
15-
----
16-
. Use the `CONTROLLER1_SCP` shell variable to set the value of the existing
17-
`CONTROLLER1_SSH` variable:
18-
+
19-
----
20-
$ CONTROLLER1_SCP=$(echo "$CONTROLLER1_SSH" | sed 's/^ssh/scp/g')
21-
----
22-
23-
. Run the following set of commands to regenerate the keys and certificates and install the data in {rhocp_long}. Convert the existing single certificate authority (CA) configuration into a dual CA configuration. You can use these commands to regenerate the keys and certificates and insert them into {OpenShiftShort}:
24-
+
25-
----
26-
include::../../tests/roles/octavia_adoption/tasks/octavia_certs.yaml[lines="7..83",indent=0]
27-
----
28-
29-
. Optional: Copy the existing public SSH key that you can use for connecting to the amphorae and install it into {OpenShiftShort}:
30-
+
31-
----
32-
${CONTROLLER1_SCP}:<octavia_ssh_pubkey_path> $HOME/octavia_sshkey.pub
33-
34-
# Install new data in k8s
35-
oc apply -f - <<EOF
36-
apiVersion: v1
37-
kind: ConfigMap
38-
metadata:
39-
name: octavia-ssh-pubkey
40-
data:
41-
key: $(cat $HOME/octavia_sshkey.pub)
42-
EOF
43-
44-
rm -f $HOME/octavia_sshkey.pub
45-
----
46-
+
47-
* Replace `<octavia_ssh_pubkey_path>` with the path that you configured by using the `OctaviaAmphoraSshKeyFile` parameter in the previous deployment. The default path is `/etc/octavia/ssh/octavia_id_rsa`. For example:
48-
+
49-
----
50-
parameter_defaults:
51-
OctaviaAmphoraSshKeyFile: /etc/octavia/ssh/octavia_id_rsa
14+
include::../../tests/roles/octavia_adoption/tasks/octavia_certs.yaml[lines="6..16",indent=0]
5215
----
5316

5417
. To isolate the management network, add the network interface for the VLAN base interface:
@@ -104,7 +67,7 @@ Specifies the `mtu` value in your environment.
10467
. To connect pods that manage load balancer virtual machines (amphorae) and the OpenvSwitch pods that are managed by the OVN operator, configure the {loadbalancer_service} network attachment definition:
10568
+
10669
----
107-
$ cat >> octavia-nad.yaml << EOF_CAT
70+
$ cat octavia-nad.yaml << EOF_CAT
10871
apiVersion: k8s.cni.cncf.io/v1
10972
kind: NetworkAttachmentDefinition
11073
metadata:
@@ -149,9 +112,7 @@ spec:
149112
octavia:
150113
enabled: true
151114
template:
152-
apacheContainerImage: registry.redhat.io/rhel8/httpd-24:latest
153115
amphoraImageContainerImage: quay.io/gthiemonge/octavia-amphora-image
154-
databaseInstance: openstack
155116
octaviaHousekeeping:
156117
networkAttachments:
157118
- octavia
@@ -173,99 +134,62 @@ $ oc wait --for condition=Ready --timeout=600s octavia.octavia.openstack.org/oct
173134
. Ensure that the {loadbalancer_service} is registered in the {identity_service}:
174135
+
175136
----
137+
$ alias openstack="oc exec -t openstackclient -- openstack"
176138
$ openstack service list | grep load-balancer
177139
| bd078ca6f90c4b86a48801f45eb6f0d7 | octavia | load-balancer |
178-
$ openstack endpoint list | grep load-balancer
179-
| f1ae7756b6164baf9cb82a1a670067a2 | regionOne | octavia | load-balancer | True | public | https://octavia-public-openstack.apps-crc.testing |
180-
| ff3222b4621843669e89843395213049 | regionOne | octavia | load-balancer | True | internal | http://octavia-internal.openstack.svc:9876 |
140+
$ openstack endpoint list --service load-balancer
141+
+----------------------------------+-----------+--------------+---------------+---------+-----------+---------------------------------------------------+
142+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
143+
+----------------------------------+-----------+--------------+---------------+---------+-----------+---------------------------------------------------+
144+
| f1ae7756b6164baf9cb82a1a670067a2 | regionOne | octavia | load-balancer | True | public | https://octavia-public-openstack.apps-crc.testing |
145+
| ff3222b4621843669e89843395213049 | regionOne | octavia | load-balancer | True | internal | http://octavia-internal.openstack.svc:9876 |
146+
+----------------------------------+-----------+--------------+---------------+---------+-----------+---------------------------------------------------+
181147
----
182148

183-
. Change the `ONBOOT` option in the network script for the management interface to `no` to ensure that the interface is disabled on reboot:
149+
.Post-adoption cleanup
184150

185-
.. Connect to each of the Controller nodes on the old control plane, for example, `overcloud-controller-0`.
151+
Before running the post-adoption cleanup, you can ensure that the connectivty between the new control plane and the adopted compute nodes is functional by creating a new load balancer and checking that its `provisioning_status` becomes `ACTIVE`.
186152

187-
.. Open the management interface configuration file in a text editor such as `vi`:
188-
+
189153
----
190-
$ sudo vi /etc/sysconfig/network-scripts/ifcfg-o-hm0
191-
----
192-
+
193-
The script name is based on the previous `OctaviaMgmtPortDevName` setting and might differ in your environment.
194-
195-
. Delete old flavors that are migrated to the new control plane:
196-
+
197-
----
198-
$ openstack flavor list --all | grep octavia
199-
| 484c351a-57ca-4a4b-8e6e-93d31596fec5 | octavia-amphora-4vcpus | 4096 | 3 | 0 | 4 | False |
200-
| 65 | octavia_65 | 1024 | 3 | 0 | 1 | False |
201-
| amphora-mvcpu-ha | octavia_amphora-mvcpu-ha | 4096 | 3 | 0 | 4 | False |
202-
| cf9d1d80-5680-4ed8-a051-e8ec4c5871e0 | octavia-amphora | 1024 | 3 | 0 | 1 | False |
203-
$ openstack flavor delete octavia_65
204-
$ openstack flavor delete octavia_amphora-mvcpu-ha
205-
$ openstack flavor list --all | grep octavia
206-
| 484c351a-57ca-4a4b-8e6e-93d31596fec5 | octavia-amphora-4vcpus | 4096 | 3 | 0 | 4 | False |
207-
| cf9d1d80-5680-4ed8-a051-e8ec4c5871e0 | octavia-amphora | 1024 | 3 | 0 | 1 | False |
154+
$ alias openstack="oc exec -t openstackclient -- openstack"
155+
$ openstack loadbalancer create --vip-subnet-id public-subnet --name lb-post-adoption --wait
208156
----
209157

210-
. Delete old {loadbalancer_service} flavors that are migrated to the new control plane:
211-
+
212-
----
213-
$ openstack loadbalancer flavor list
214-
+--------------------------------------+--------------------------+--------------------------------------+---------+
215-
| id | name | flavor_profile_id | enabled |
216-
+--------------------------------------+--------------------------+--------------------------------------+---------+
217-
| 5db54d9b-ba08-4b51-a859-0a81533604aa | octavia_amphora-mvcpu-ha | 4fa6a793-4c20-4480-be4f-806912840511 | True |
218-
| 6d649fd5-6322-4265-b5f3-c3277fc29ec8 | amphora-4vcpus | d9764a80-99f5-4f22-bbe0-3ddbdc5c485c | True |
219-
| 93f34308-24a7-42de-9065-959a3b36e7f6 | amphora | e75e50c8-7786-4623-abcf-bccbea59d213 | True |
220-
+--------------------------------------+--------------------------+--------------------------------------+---------+
221-
$ openstack loadbalancer flavor delete octavia_amphora-mvcpu-ha
222-
$ openstack loadbalancer flavor list
223-
+--------------------------------------+----------------+--------------------------------------+---------+
224-
| id | name | flavor_profile_id | enabled |
225-
+--------------------------------------+----------------+--------------------------------------+---------+
226-
| 6d649fd5-6322-4265-b5f3-c3277fc29ec8 | amphora-4vcpus | d9764a80-99f5-4f22-bbe0-3ddbdc5c485c | True |
227-
| 93f34308-24a7-42de-9065-959a3b36e7f6 | amphora | e75e50c8-7786-4623-abcf-bccbea59d213 | True |
228-
+--------------------------------------+----------------+--------------------------------------+---------+
229-
----
158+
After you complete the data plane adoption, perform the following cleanup steps to upgrade existing load balancers and remove old resources.
230159

231-
. Delete old flavor profiles that are migrated to the new control plane:
160+
. Trigger a failover for all existing load balancers to upgrade the amphorae virtual machines to use the new image and establish connectivity with the new control plane:
232161
+
233162
----
234-
$ openstack loadbalancer flavorprofile list
235-
+--------------------------------------+----------------------------------+---------------+
236-
| id | name | provider_name |
237-
+--------------------------------------+----------------------------------+---------------+
238-
| 4fa6a793-4c20-4480-be4f-806912840511 | octavia_amphora-mvcpu-ha_profile | amphora |
239-
| d9764a80-99f5-4f22-bbe0-3ddbdc5c485c | amphora-4vcpus | amphora |
240-
| e75e50c8-7786-4623-abcf-bccbea59d213 | amphora | amphora |
241-
+--------------------------------------+----------------------------------+---------------+
242-
$ openstack loadbalancer flavorprofile delete octavia_amphora-mvcpu-ha_profile
243-
$ openstack loadbalancer flavorprofile list
244-
+--------------------------------------+----------------+---------------+
245-
| id | name | provider_name |
246-
+--------------------------------------+----------------+---------------+
247-
| d9764a80-99f5-4f22-bbe0-3ddbdc5c485c | amphora-4vcpus | amphora |
248-
| e75e50c8-7786-4623-abcf-bccbea59d213 | amphora | amphora |
249-
+--------------------------------------+----------------+---------------+
163+
$ openstack loadbalancer list -f value -c id | \
164+
xargs -r -n1 -P4 ${BASH_ALIASES[openstack]} loadbalancer failover --wait
250165
----
251166

252-
. Delete the old management network ports. Store the network ID of the old management network in the variable `WALLABY_LB_MGMT_NET_ID` to use later:
167+
. Delete old flavors that were migrated to the new control plane:
253168
+
254169
----
255-
$ for net_id in $(openstack network list -f value -c ID --name lb-mgmt-net); do desc=$(openstack network show "$net_id" -f value -c description); [ -z "$desc" ] && WALLABY_LB_MGMT_NET_ID="$net_id" ; done
256-
$ echo $WALLABY_LB_MGMT_NET_ID
257-
1e21f9c1-7485-4104-a2f3-eed098ab9cad
170+
$ openstack flavor delete octavia_65
171+
# The following flavors might not exist in OSP 17.1 deployments
172+
$ openstack flavor show octavia_amphora-mvcpu-ha && \
173+
openstack flavor delete octavia_amphora-mvcpu-ha
174+
$ openstack loadbalancer flavor show octavia_amphora-mvcpu-ha && \
175+
openstack loadbalancer flavor delete octavia_amphora-mvcpu-ha
176+
$ openstack loadbalancer flavorprofile show octavia_amphora-mvcpu-ha_profile && \
177+
openstack loadbalancer flavorprofile delete octavia_amphora-mvcpu-ha_profile
258178
----
259179

260-
. Delete all ports that are used in this network:
261-
+
262-
----
263-
$ for id in $(openstack port list --network "$WALLABY_LB_MGMT_NET_ID" -f value -c ID) ; do openstack port delete "$id" ; done
264-
----
180+
[NOTE]
181+
Some flavors may still be in-use by load balancers and cannot be deleted.
265182

266-
. Delete the old management network:
183+
. Delete the old management network and its ports:
267184
+
268185
----
186+
$ for net_id in $(openstack network list -f value -c ID --name lb-mgmt-net); do \
187+
desc=$(openstack network show "$net_id" -f value -c description); \
188+
[ -z "$desc" ] && WALLABY_LB_MGMT_NET_ID="$net_id" ; \
189+
done
190+
$ for id in $(openstack port list --network "$WALLABY_LB_MGMT_NET_ID" -f value -c ID); do \
191+
openstack port delete "$id" ; \
192+
done
269193
$ openstack network delete "$WALLABY_LB_MGMT_NET_ID"
270194
----
271195

‎tests/roles/dataplane_adoption/defaults/main.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -234,6 +234,7 @@ skip_patching_ansibleee_csv: false
234234
os_diff_dir: tmp/os-diff
235235
os_diff_data_dir: tmp/os-diff
236236
prelaunch_test_instance: true
237+
prelaunch_octavia_workload: false
237238
telemetry_adoption: true
238239

239240
# nodes data will be templated in as a separate

‎tests/roles/dataplane_adoption/tasks/main.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -902,7 +902,7 @@
902902
- name: Adopted Octavia post-checks
903903
when: octavia_adoption|bool
904904
ansible.builtin.include_tasks:
905-
file: octavia_verify.yaml
905+
file: octavia_post.yaml
906906

907907
- name: Adopted Cinder post-checks
908908
ansible.builtin.include_tasks:

‎tests/roles/dataplane_adoption/tasks/octavia_post.yaml‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
---
2+
- name: set Octavia services shell vars
3+
no_log: "{{ use_no_log }}"
4+
ansible.builtin.set_fact:
5+
octavia_header: |
6+
alias openstack="oc exec -t openstackclient -- openstack"
7+
8+
# Creating a new LB requires resources setup by the prelaunch scripts
9+
- name: create a load balancer with the new control plane
10+
when:
11+
- prelaunch_octavia_workload|bool
12+
- prelaunch_test_instance|bool
13+
- "'pre_launch.bash' in prelaunch_test_instance_scripts"
14+
ansible.builtin.shell: |
15+
{{ shell_header }}
16+
{{ octavia_header }}
17+
${BASH_ALIASES[openstack]} loadbalancer create --vip-subnet-id public_subnet --name lb-post-adoption --tag new --wait
18+
${BASH_ALIASES[openstack]} loadbalancer listener create --protocol http --protocol-port 80 --name listener2 lb-post-adoption --wait
19+
20+
- name: get new load balancer VIP address
21+
when:
22+
- prelaunch_octavia_workload|bool
23+
- prelaunch_test_instance|bool
24+
- "'pre_launch.bash' in prelaunch_test_instance_scripts"
25+
ansible.builtin.shell: |
26+
{{ shell_header }}
27+
{{ octavia_header }}
28+
${BASH_ALIASES[openstack]} loadbalancer show -f value -c vip_address lb-post-adoption
29+
register: octavia_new_vip_address
30+
ignore_errors: true
31+
32+
- name: test the load balancer VIP address
33+
when:
34+
- prelaunch_octavia_workload|bool
35+
- prelaunch_test_instance|bool
36+
- "'pre_launch.bash' in prelaunch_test_instance_scripts"
37+
- octavia_new_vip_address.rc == 0
38+
ansible.builtin.shell: |
39+
{{ shell_header }}
40+
curl {{ octavia_new_vip_address.stdout }}
41+
42+
- name: perform failover of existing load balancers
43+
ansible.builtin.shell: |
44+
{{ shell_header }}
45+
{{ octavia_header }}
46+
${BASH_ALIASES[openstack]} loadbalancer list -f value -c id --not-tag new | \
47+
xargs -r -n1 -P4 ${BASH_ALIASES[openstack]} loadbalancer failover --wait
48+
49+
- name: check for existing load balancer
50+
ansible.builtin.shell: |
51+
{{ shell_header }}
52+
{{ octavia_header }}
53+
${BASH_ALIASES[openstack]} loadbalancer show -f value -c vip_address lb1
54+
register: octavia_vip_address
55+
ignore_errors: true
56+
57+
- name: test the load balancer VIP address
58+
when:
59+
- octavia_vip_address.rc == 0
60+
ansible.builtin.shell: |
61+
{{ shell_header }}
62+
curl {{ octavia_vip_address.stdout }}
63+
64+
- name: delete old flavors that have been migrated
65+
ansible.builtin.shell: |
66+
{{ shell_header }}
67+
{{ octavia_header }}
68+
${BASH_ALIASES[openstack]} flavor delete octavia_65
69+
# these might not be in OSP 17.1
70+
${BASH_ALIASES[openstack]} flavor show octavia_amphora-mvcpu-ha && \
71+
${BASH_ALIASES[openstack]} flavor delete octavia_amphora-mvcpu-ha
72+
${BASH_ALIASES[openstack]} loadbalancer flavor show octavia_amphora-mvcpu-ha && \
73+
${BASH_ALIASES[openstack]} loadbalancer flavor delete octavia_amphora-mvcpu-ha
74+
${BASH_ALIASES[openstack]} loadbalancer flavorprofile show octavia_amphora-mvcpu-ha_profile && \
75+
${BASH_ALIASES[openstack]} loadbalancer flavorprofile delete octavia_amphora-mvcpu-ha_profile
76+
77+
- name: delete old mangement network ports and network
78+
ansible.builtin.shell: |
79+
{{ shell_header }}
80+
{{ octavia_header }}
81+
for net_id in $(${BASH_ALIASES[openstack]} network list -f value -c ID --name lb-mgmt-net); do desc=$(${BASH_ALIASES[openstack]} network show "$net_id" -f value -c description); [ -z "$desc" ] && WALLABY_LB_MGMT_NET_ID="$net_id" ; done
82+
for id in $(${BASH_ALIASES[openstack]} port list --network "$WALLABY_LB_MGMT_NET_ID" -f value -c ID) ; do ${BASH_ALIASES[openstack]} port delete "$id" ; done
83+
84+
${BASH_ALIASES[openstack]} network delete "$WALLABY_LB_MGMT_NET_ID"
85+
86+
- name: get nova flavors for octavia
87+
ansible.builtin.shell: |
88+
{{ shell_header }}
89+
{{ octavia_header }}
90+
output=$(${BASH_ALIASES[openstack]} flavor list --all | grep octavia)
91+
[ "$(echo "$output" | wc -l)" -eq 2 ] && echo PASS
92+
register: octavia_novaflavor_result
93+
94+
- name: verify nova flavors for octavia
95+
when: octavia_adoption|bool
96+
ansible.builtin.assert:
97+
that:
98+
- octavia_novaflavor_result.stdout == 'PASS'
99+
fail_msg: "verification of nova flavors for octavia failed"
100+
success_msg: "verification of nova flavors for octavia succeeded"
101+
102+
# TODO(tweining): This fail upstream because the quay.io image contains only
103+
# one amp image
104+
# - name: get octavia flavors for octavia
105+
# ansible.builtin.shell: |
106+
# {{ shell_header }}
107+
# {{ octavia_header }}
108+
# output=$(${BASH_ALIASES[openstack]} loadbalancer flavor list -f value)
109+
# [ "$(echo "$output" | wc -l)" -eq 2 ] && echo PASS
110+
# register: octavia_octaviaflavor_result
111+
112+
# - name: verify octavia flavors for octavia
113+
# when: octavia_adoption|bool
114+
# ansible.builtin.assert:
115+
# that:
116+
# - octavia_octaviaflavor_result.stdout == 'PASS'
117+
# fail_msg: "verification of octavia flavors for octavia failed"
118+
# success_msg: "verification of octavia flavors for octavia succeeded"
119+
120+
# - name: get flavorprofiles for octavia
121+
# ansible.builtin.shell: |
122+
# {{ shell_header }}
123+
# {{ octavia_header }}
124+
# output=$(${BASH_ALIASES[openstack]} loadbalancer flavorprofile list -f value)
125+
# [ "$(echo "$output" | wc -l)" -eq 2 ] && echo PASS
126+
# register: octavia_flavorprofiles_result
127+
128+
# - name: verify flavorprofiles for octavia
129+
# when: octavia_adoption|bool
130+
# ansible.builtin.assert:
131+
# that:
132+
# - octavia_flavorprofiles_result.stdout == 'PASS'
133+
# fail_msg: "verification of flavorprofiles for octavia failed"
134+
# success_msg: "verification of flavorprofiles for octavia succeeded"
135+
136+
- name: get networks named lb-mgmt-net
137+
ansible.builtin.shell: |
138+
{{ shell_header }}
139+
{{ octavia_header }}
140+
output=$(${BASH_ALIASES[openstack]} network list | grep lb-mgmt-net)
141+
[ "$(echo "$output" | wc -l)" -eq 1 ] && echo PASS
142+
register: octavia_mgmt_network_result
143+
144+
- name: verify that one network lb-mgmt-net exists
145+
when: octavia_adoption|bool
146+
ansible.builtin.assert:
147+
that:
148+
- octavia_mgmt_network_result.stdout == 'PASS'
149+
fail_msg: "verification of lb-mgmt-net network failed"
150+
success_msg: "verification of lb-mgmt-net network succeeded"
151+
152+
- name: get subnets named lb-mgmt-subnet
153+
ansible.builtin.shell: |
154+
{{ shell_header }}
155+
{{ octavia_header }}
156+
output=$(${BASH_ALIASES[openstack]} subnet list | grep lb-mgmt-subnet)
157+
[ "$(echo "$output" | wc -l)" -eq 1 ] && echo PASS
158+
register: octavia_mgmt_subnet_result
159+
160+
- name: verify that one subnet lb-mgmt-subnet exists
161+
when: octavia_adoption|bool
162+
ansible.builtin.assert:
163+
that:
164+
- octavia_mgmt_subnet_result.stdout == 'PASS'
165+
fail_msg: "verification of subnet lb-mgmt-subnet failed"
166+
success_msg: "verification of subnet lb-mgmt-subnet succeeded"

0 commit comments

Comments
 (0)