Skip to content

Commit 2071ac8

Browse files
eduolivareseduolivares
committed
Add compute1 node to BGP multinode tempest job
Add a second compute node (compute1) to the neutron-ovn-bgp-tempest-multinode job so that VMs can be scheduled on a node that does not run the OpenStack control plane. Compute1 is dual-homed to both leaf switches, runs FRR with BFD, and the OVN agent with the ovn-bgp extension, matching the controller's BGP setup. Changes: - Nodeset: add compute1 to nodes, compute, subnode, and new zbgp_compute group; rename to neutron-five-node-noble-bgp. The zbgp_compute group name is chosen to sort after "subnode" alphabetically so its group-vars override the subnode group-vars from the devstack base job. - Job: move spine config from subnode to tempest group-vars; add zbgp_compute group-vars for compute1. Both tempest and zbgp_compute group-vars include the base subnode devstack_localrc settings (passwords, host addresses) since their alphabetical precedence replaces the subnode group-vars entirely. - Job: add zuul_copy_output for FRR, OVN, and OVS logs. - Playbook: merge controller and compute1 tunnel plays into a single play targeting the compute group with a per-host config map for VXLAN key/VLAN prefixes. - Playbook: fix FRR start guard to use a last_compute_node variable (last compute subnode) instead of relying on Ansible group ordering. Generate SSH keys on all compute nodes so the last one can start FRR on leaf/spine. - Leaf FRR: add compute1-port as a downlink neighbor. - Docs: update topology diagram, node descriptions, and VXLAN wiring table. - Test test_reuse_ip_address_with_other_fip_on_other_router is skipped because of a core ovn issue. Depends-On: https://review.opendev.org/c/openstack/neutron/+/991433 Assisted-By: Claude Opus 4.6 Change-Id: I5f981f1838c3b51b6a7d2c9032192fc6c5c88f15 Signed-off-by: Eduardo Olivares <eolivare@redhat.com>
1 parent 570efdb commit 2071ac8

11 files changed

Lines changed: 338 additions & 61 deletions

File tree

‎devstack/etc/frr/frr.conf‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ service integrated-vtysh-config
77
line vty
88

99
router bgp 64999
10-
bgp router-id 172.24.4.1
10+
! BGP_ROUTER_ID is replaced by local.sh with a per-node value
11+
bgp router-id BGP_ROUTER_ID
1112
bgp log-neighbor-changes
1213
bgp graceful-shutdown
1314
no bgp default ipv4-unicast
@@ -37,7 +38,8 @@ ip prefix-list only-host-prefixes permit 0.0.0.0/0 ge 32
3738

3839
route-map rm-only-default permit 10
3940
match ip address prefix-list only-default
40-
set src 172.24.4.1
41+
! BGP_ROUTER_ID is replaced by local.sh with a per-node value
42+
set src BGP_ROUTER_ID
4143

4244
ip protocol bgp route-map rm-only-default
4345

@@ -46,7 +48,8 @@ ipv6 prefix-list only-host-prefixes permit ::/0 ge 128
4648

4749
route-map rm-only-default permit 11
4850
match ipv6 address prefix-list only-default
49-
set src 2001:db8::2
51+
! BGP_IPV6_SRC is replaced by local.sh with a per-node value
52+
set src BGP_IPV6_SRC
5053

5154
ipv6 protocol bgp route-map rm-only-default
5255

‎devstack/etc/frr_with_bfd/frr.conf‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,8 @@ router bgp 64999 vrf ovnvrf42
3434
exit-address-family
3535

3636
router bgp 64999
37-
bgp router-id 172.24.4.1
37+
! BGP_ROUTER_ID is replaced by local.sh with a per-node value
38+
bgp router-id BGP_ROUTER_ID
3839
bgp log-neighbor-changes
3940
bgp graceful-shutdown
4041
no bgp default ipv4-unicast
@@ -68,7 +69,8 @@ ip prefix-list only-host-prefixes permit 0.0.0.0/0 ge 32
6869

6970
route-map rm-only-default permit 10
7071
match ip address prefix-list only-default
71-
set src 172.24.4.1
72+
! BGP_ROUTER_ID is replaced by local.sh with a per-node value
73+
set src BGP_ROUTER_ID
7274

7375
ip protocol bgp route-map rm-only-default
7476

@@ -77,7 +79,8 @@ ipv6 prefix-list only-host-prefixes permit ::/0 ge 128
7779

7880
route-map rm-only-default permit 11
7981
match ipv6 address prefix-list only-default
80-
set src 2001:db8::2
82+
! BGP_IPV6_SRC is replaced by local.sh with a per-node value
83+
set src BGP_IPV6_SRC
8184

8285
route-map rm-install-only-default permit 10
8386
match ip address prefix-list only-default

‎devstack/lib/frr‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,14 @@ function configure_frr {
3333
sudo install -o root -g root -m 644 $FRR_ETC_SRC_DIR/frr/* $FRR_CONF_DIR/
3434
fi
3535

36+
# Replace per-node placeholders in frr.conf
37+
if [[ -n "$FRR_BGP_ROUTER_ID" ]]; then
38+
sudo sed -i "s/BGP_ROUTER_ID/$FRR_BGP_ROUTER_ID/g" $FRR_CONF_DIR/frr.conf
39+
fi
40+
if [[ -n "$FRR_BGP_IPV6_SRC" ]]; then
41+
sudo sed -i "s/BGP_IPV6_SRC/$FRR_BGP_IPV6_SRC/g" $FRR_CONF_DIR/frr.conf
42+
fi
43+
3644
}
3745

3846
function init_frr {

‎doc/source/ovn/bgp_tempest_job.rst‎

Lines changed: 58 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ BGP Multinode Tempest Job
66

77
The ``neutron-ovn-bgp-tempest-multinode`` Zuul job is a CI job that validates
88
Neutron with Native OVN BGP in a realistic leaf-spine network topology. It
9-
deploys a four-node environment where VXLAN tunnels simulate the physical links
9+
deploys a five-node environment where VXLAN tunnels simulate the physical links
1010
of a leaf-spine fabric, and FRR (Free Range Routing) provides BGP and BFD on
1111
every network node. The job then runs the Neutron Tempest scenario tests to
1212
verify that tenant workloads are reachable through dynamically advertised BGP
@@ -17,7 +17,7 @@ The job is defined in ``zuul.d/tempest-multinode-bgp.yaml``.
1717
Network Topology
1818
================
1919

20-
The job provisions four nodes arranged in a leaf-spine topology.
20+
The job provisions five nodes arranged in a leaf-spine topology.
2121
Because CI nodes only have IP-level connectivity to each other, the playbook
2222
``playbooks/configure_bgp_networking.yaml`` builds the fabric links on top of
2323
VXLAN tunnels carried over the nodepool underlay.
@@ -35,19 +35,24 @@ VXLAN tunnels carried over the nodepool underlay.
3535
| |
3636
+--------+-+ +-+--------+
3737
| Leaf-1 | | Leaf-2 |
38-
| (AS 64999)| |(AS 64999)|
38+
|(AS 64999)| |(AS 64999)|
3939
| FRR/BGP | | FRR/BGP |
40-
+-----+----+ +----+-----+
41-
| |
42-
VXLAN | | VXLAN
43-
| |
44-
+-----+----------------+-----+
45-
| Controller |
46-
| OpenStack control plane |
47-
| OVN / Neutron / Nova |
48-
| OVN agent (ovn-bgp ext.) |
49-
| (br-bgp-0, br-bgp-1) |
50-
+----------------------------+
40+
+---+--+---+ +---+--+---+
41+
| | | |
42+
VXLAN | | VXLAN VXLAN | | VXLAN
43+
| | | |
44+
+-------------------+ | | +-------------------+
45+
| Controller | | | | Compute1 |
46+
| OpenStack | | | | OpenStack |
47+
| control + compute+--|--------------+ | compute |
48+
| OVN agent | | | OVN agent |
49+
| (ovn-bgp ext.) | +-----------------+ (ovn-bgp ext.) |
50+
| (br-bgp-0/1) | | (br-bgp-0/1) |
51+
+-------------------+ +-------------------+
52+
53+
Both the controller and compute1 are dual-homed: each connects to **both**
54+
leaf-1 and leaf-2 through independent VXLAN tunnels (see the wiring table
55+
below for details).
5156

5257
Every link in the diagram is a VXLAN tunnel terminated on an OVS bridge
5358
(``br-infra``) at each end. VLAN tags on ``br-infra`` isolate the individual
@@ -79,16 +84,37 @@ Key characteristics:
7984
* The gateway chassis is **not** assigned to the public bridge
8085
(``Q_ASSIGN_GATEWAY_TO_PUBLIC_BRIDGE: false``), since BGP handles
8186
external reachability instead of the traditional ``br-ex`` path.
87+
* The OVN BGP agent (``q-ovn-bgp``) manages OpenFlow rules on the BGP
88+
bridges. FRR redistributes routes via BGP.
89+
90+
Compute1
91+
--------
92+
93+
Compute1 runs as a second compute node. No OpenStack control plane services
94+
run on this node, but Nova instances can be scheduled onto it. Like the
95+
controller, it runs the **OVN agent** with the ``ovn-bgp`` and ``metadata``
96+
extensions, and FRR with BFD for BGP route advertisement.
97+
98+
Key characteristics:
99+
100+
* OVN is built from source (same pinned commits as the controller).
101+
* Two OVS bridges, ``br-bgp-0`` and ``br-bgp-1``, connect to the leaf
102+
switches through VXLAN tunnels on ``br-infra``, using the same veth-pair
103+
wiring as the controller.
104+
* Distributed floating IPs are enabled
105+
(``enable_distributed_floating_ip: True``).
106+
* The OVN BGP agent (``q-ovn-bgp``) manages OpenFlow rules on the BGP
107+
bridges. FRR redistributes routes via BGP.
82108

83109
Leaf-1 and Leaf-2
84110
-----------------
85111

86112
The two leaf nodes act as Top-of-Rack (ToR) switches in the simulated
87-
leaf-spine fabric. Each leaf has two BGP sessions:
113+
leaf-spine fabric. Each leaf has three BGP sessions:
88114

89-
* **Downlink** (to the controller) -- iBGP within AS 64999. The leaf is
90-
configured as a route-reflector client toward the controller and
91-
originates a default route.
115+
* **Downlinks** (to the controller and compute1) -- iBGP within AS 64999.
116+
The leaf is configured as a route-reflector client toward both the
117+
controller and compute1, and originates a default route on each downlink.
92118
* **Uplink** (to the spine) -- eBGP peering between AS 64999 (leaf) and
93119
AS 65000 (spine).
94120

@@ -153,6 +179,14 @@ The pre-run playbook creates the following tunnels:
153179
- 10001
154180
- 1001
155181
- Controller (br-bgp-1) <-> Leaf-2 (controller-port)
182+
* - Compute1 <-> Leaf-1
183+
- 10020
184+
- 3000
185+
- Compute1 (br-bgp-0) <-> Leaf-1 (compute1-port)
186+
* - Compute1 <-> Leaf-2
187+
- 10021
188+
- 3001
189+
- Compute1 (br-bgp-1) <-> Leaf-2 (compute1-port)
156190
* - Spine <-> Leaf-1
157191
- 10010
158192
- 2000
@@ -165,19 +199,20 @@ The pre-run playbook creates the following tunnels:
165199
BGP Route Flow
166200
==============
167201

168-
When the ``ovn-bgp`` extension on the controller detects a new tenant VM or
169-
floating IP, the following sequence propagates the route to the spine:
202+
When the ``ovn-bgp`` extension on a compute node (the controller or compute1)
203+
detects a new tenant VM or floating IP, the following sequence propagates the
204+
route to the spine:
170205

171206
#. The ``ovn-bgp`` extension adds a ``/32`` (or ``/128``) route to the
172-
controller's kernel routing table and programs OpenFlow rules on
207+
node's kernel routing table and programs OpenFlow rules on
173208
``br-bgp-0`` and ``br-bgp-1``.
174-
#. FRR on the controller (via zebra) picks up the connected route and
209+
#. FRR on the node (via zebra) picks up the connected route and
175210
advertises it over the iBGP sessions to the leaf switches.
176211
#. Each leaf, acting as a route-reflector, propagates the route over eBGP
177212
to the spine.
178213
#. The spine installs the route into its FIB. Traffic from the Tempest runner
179214
on the spine can now reach the tenant VM by following the advertised path
180-
back through the leaf to the controller.
215+
back through the leaf to the originating compute node.
181216

182217
Test Scope
183218
==========

‎playbooks/collect_bgp_diagnostics.yaml‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
- name: Collect BGP and OVN diagnostic data before teardown
2+
hosts: compute
3+
become: true
4+
ignore_unreachable: true
5+
tasks:
6+
- name: Ensure /var/log/openvswitch directory exists
7+
ansible.builtin.file:
8+
path: /var/log/openvswitch
9+
state: directory
10+
mode: '0755'
11+
ignore_errors: true
12+
13+
- name: Dump OVS configuration
14+
ansible.builtin.command: ovs-vsctl show
15+
register: ovs_vsctl
16+
ignore_errors: true
17+
18+
- name: Save OVS configuration
19+
ansible.builtin.copy:
20+
content: "{{ ovs_vsctl.stdout | default(ovs_vsctl.stderr | default('command failed')) }}"
21+
dest: /var/log/openvswitch/ovs-vsctl-show-dump.txt
22+
mode: '0644'
23+
when: ovs_vsctl is not skipped
24+
ignore_errors: true
25+
26+
- name: Dump FRR BGP state
27+
ansible.builtin.shell: |
28+
vtysh -c "show bgp summary" 2>&1
29+
vtysh -c "show bfd peers" 2>&1
30+
register: frr_state
31+
ignore_errors: true
32+
33+
- name: Save FRR BGP state
34+
ansible.builtin.copy:
35+
content: "{{ frr_state.stdout | default(frr_state.stderr | default('command failed')) }}"
36+
dest: /var/log/frr/bgp_state.txt
37+
mode: '0644'
38+
when: frr_state is not skipped
39+
ignore_errors: true
40+
41+
- name: Collect diagnostics from spine and leafs
42+
hosts: spine,leafs
43+
become: true
44+
ignore_unreachable: true
45+
tasks:
46+
- name: Dump FRR BGP state
47+
ansible.builtin.shell: |
48+
vtysh -c "show bgp summary" 2>&1
49+
vtysh -c "show bfd peers" 2>&1
50+
register: frr_state
51+
ignore_errors: true
52+
53+
- name: Save FRR BGP state
54+
ansible.builtin.copy:
55+
content: "{{ frr_state.stdout | default(frr_state.stderr | default('command failed')) }}"
56+
dest: /var/log/frr/bgp_state.txt
57+
mode: '0644'
58+
when: frr_state is not skipped
59+
ignore_errors: true

0 commit comments

Comments
 (0)