OVS role follow up minor issues

[dann1]

Description
After some more testing we found out some minor problems we would like to have addresses in the OVS role.

Interfaces in UP

By default, with NetworkManager, all of the interfaces are set in UP state (unless the link is physically down). After the opennebula-ovs unit switches over networking from NetworkManager only the used interfaces will be in UP.

We need the openvswitch role to handle this. Before the one-deploy implementation we had this other systemd unit that did this task

[Unit]
Description=Enable Physical Network Interfaces
Before=openvswitch.service network-online.target
After=network-pre.target
DefaultDependencies=no

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/bash -c 'for iface in /sys/class/net/*; do dev=$(basename $iface); if [ -d $iface/device ] && [ "$dev" != "lo" ]; then ip link set dev $dev up; fi; done'

[Install]
WantedBy=multi-user.target

perhaps it is useful as a reference for the official implementation. It also works with the current OVS role.

Without this, the next power cycle will have a lot of interfaces in DOWN state.

This issue sort of ties to #218 as blacklisting the OVS interfaces from NetworkManager also solves the problem at hand, which is downed links.

VLAN Tagged Management interface

If the management IP used for the SSH connection exists in a VLAN interface, the script generated by the opennebula-ovs.sh script will only try to flush the address from the real interface

14: ens3f0.741@vmnic6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 6c:fe:54:99:6a:90 brd ff:ff:ff:ff:ff:ff
    inet 172.23.16.23/24 brd 172.23.16.255 scope global noprefixroute ens3f0.741
       valid_lft forever preferred_lft forever
    inet6 fe80::4f0f:fbf0:fdb0:8b80/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
[root@nfhhvbcnlb03 ~]# ip addr show ovsbr0
1044: ovsbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 6c:fe:54:99:65:b0 brd ff:ff:ff:ff:ff:ff
    inet 172.23.16.23/24 metric 400 scope global ovsbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::6efe:54ff:fe99:65b0/64 scope link
       valid_lft forever preferred_lft forever

In the next power cycle it will be lost, and is actually harmless, but it is a leftover. Ideally the logic should automatically detect if this is the case. Or, for example, see if a tag is being set in the internal bridge port, which is usually what is stated in the inventory when accessing a host with a vlan tag in the management interface

14: ens3f0.741@vmnic6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 6c:fe:54:99:6a:90 brd ff:ff:ff:ff:ff:ff
    inet 172.23.16.23/24 brd 172.23.16.255 scope global noprefixroute ens3f0.741
       valid_lft forever preferred_lft forever
    inet6 fe80::4f0f:fbf0:fdb0:8b80/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
[root@nfhhvbcnlb03 ~]# ip addr show ovsbr0
1044: ovsbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 6c:fe:54:99:65:b0 brd ff:ff:ff:ff:ff:ff
    inet 172.23.16.23/24 metric 400 scope global ovsbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::6efe:54ff:fe99:65b0/64 scope link
       valid_lft forever preferred_lft forever

The openvswitch role targets the frontend group

The condition to install the selinux changes is to check whether the openvswitch user exists or not. When declaring the same host in frontend group and node group, we get a lock, similar to what happens when package installation tasks occur

TASK [opennebula.deploy.openvswitch : Create custom SELinux policy for DPDK / Libvirt shared sockets] ****************************************************************************
Wednesday 03 June 2026  18:47:42 +0200 (0:00:00.642)       0:04:31.957 ********
fatal: [kvm1]: FAILED! =>
    changed: false
    cmd: |-
        set -o errexit
        TDIR="$(mktemp -d)" && trap "rm -rf $TDIR" EXIT ERR
        tee "$TDIR/dpdk_virt.te"
        if (cd "$TDIR" && semodule -E dpdk_virt); then
          BEFORE="$(cksum $TDIR/dpdk_virt.pp)"
        else
          BEFORE=""
        fi
        checkmodule -M -m -o "$TDIR/dpdk_virt.mod" "$TDIR/dpdk_virt.te"
        semodule_package -o "$TDIR/dpdk_virt.pp" -m "$TDIR/dpdk_virt.mod"
        AFTER="$(cksum $TDIR/dpdk_virt.pp)"
        if [[ "$AFTER" == "$BEFORE" ]]; then
          exit 0
        else
          semodule -i "$TDIR/dpdk_virt.pp"
          exit 78 # EREMCHG
        fi
    delta: '0:00:05.031798'
    end: '2026-06-03 12:47:47.326165'
    failed_when_result: true
    msg: non-zero return code
    rc: 1
    start: '2026-06-03 12:47:42.294367'
    stderr: |-
        semodule:  Failed on dpdk_virt!
        libsemanage.semanage_get_lock: Could not get direct transaction lock at /var/lib/selinux/targeted/semanage.trans.LOCK. (Resource temporarily unavailable).
        semodule:  Failed on /tmp/tmp.afmYSlfX8Z/dpdk_virt.pp!
    stderr_lines: <omitted>
    stdout: |-
        module dpdk_virt 1.0;
        require {
            type openvswitch_t;
            type virt_var_run_t;
            class dir { search getattr read open };
            class sock_file { write getattr append };
        }
        allow openvswitch_t virt_var_run_t:dir { search getattr read open };
        allow openvswitch_t virt_var_run_t:sock_file { write getattr append };
    stdout_lines: <omitted></omitted>

In this case I had kvm1 and fe1 be the same host. So, the task ran successfully in fe1 (because openvswitch is installed) and then failed in kvm1 because it was already running in fe1 which is kvm1.

When running it again with a different frontend it went well

skipping: [fe1] => (item={'path': '/var/run/one/', 'entity': 'openvswitch', 'permissions': 'rx'})  =>
    ansible_loop_var: item
    changed: false
    false_condition: _has_openvswitch_user
    item:
        entity: openvswitch
        path: /var/run/one/
        permissions: rx
    skip_reason: Conditional result was False
skipping: [fe1] => (item={'path': '/var/run/one/vhost-socks/', 'entity': 'openvswitch', 'permissions': 'rx'})  =>
    ansible_loop_var: item
    changed: false
    false_condition: _has_openvswitch_user
    item:
        entity: openvswitch
        path: /var/run/one/vhost-socks/
        permissions: rx
    skip_reason: Conditional result was False
skipping: [fe1] => (item={'path': '/var/run/one/vhost-socks/', 'entity': 'openvswitch', 'permissions': 'rwX', 'default': True})  =>
    ansible_loop_var: item
    changed: false
    false_condition: _has_openvswitch_user
    item:
        default: true
        entity: openvswitch
        path: /var/run/one/vhost-socks/
        permissions: rwX
    skip_reason: Conditional result was False
skipping: [fe1] =>
    changed: false
    msg: All items skipped

however, if for some reason openvswitch is installed in the frontend (like when doing fe + kvm in the same host) the role will try to do selinux stuff in the frontend.

So, maybe better if we skip running openvswitch role tasks in the frontend group ?

SRIOV renaming with 128*4 VF

I got an error about a list index out of range when trying to rename 4 sets of 128 VF.

This was the inventory

        # PF
        - address: "0000:16:00.0"
          set_name: vmnic7
          set_numvfs: max # activate all VF on this PF
          unlisted: true # skip PF monitoring to avoid accidental PT
          set_switchdev: true # switchdev mode, requires hw and driver support
        - address: "0000:16:00.1"
          set_name: vmnic8
          unlisted: true # skip PF monitoring to avoid accidental PT
          set_numvfs: max # activate all VF on this PF
          set_switchdev: true # switchdev mode, requires hw and driver support
        - address: "0000:40:00.0"
          set_name: vmnic1
          set_numvfs: max # activate all VF on this PF
          unlisted: true # skip PF monitoring to avoid accidental PT
          set_switchdev: true # switchdev mode, requires hw and driver support
        - address: "0000:40:00.1"
          set_name: vmnic4
          unlisted: true # skip PF monitoring to avoid accidental PT
          set_numvfs: max # activate all VF on this PF
          set_switchdev: true # switchdev mode, requires hw and driver support

        # VF
        - vendor: "8086" # intel
          device: "159b" # 810 vf device
          class: "0200"
          set_driver: vfio-pci
          set_name: "pf{1[1]}{1[3]}vf{2}"
          virtual: true
          unlisted: false

I can't show the output right as I didn't copy it and I can't run this server again, but the #VF section renaming failed. This is another server with 810 instead of mellanox. The mellanox server had 2 PF with 8 VF each. This one has 4 PF with 128 VF each. Is the set_name logic or declaration wrong in there ?

This is not OVS related though, but found within the same context.

Let me know if it's better to open an issue for each of these or per role.

[sk4zuzu]

@dann1 Hi!

Interfaces in UP

Yes let's make it a part of the "218 campaign".

VLAN Tagged Management interface

The two snippets you added are identical, by mistake? 🤔

The openvswitch role targets the frontend group

That's a bug 😞. I forgot that OVS is run also in FEs, and running it in FEs was my intention for a complete OVS-management (if user desires). So yes we should exclude vhost.yml code from being run on FEs.

SRIOV renaming with 128*4 VF

Could you paste sysfs structure after sriov_numvfs is set to max in that specific env? In particular if in the PF can you see all symlinks to all VFs (maybe it's driver dependent problem) ?

Thank you, these are always very insightful. 🤗

[dann1]

perfect, I accidentally double pasted the vlan tag, only the first one is needed.

Next time I'll run this env (don't really have the availability to run whenever there) I'll test specifically this mega-renaming. The SRIOV interactions are indeed super driver dependent, sometimes there is no offload support which makes the swtichdev usage suboptimal, sometimes there is just legacy mode.

Also I remembered another minor problem

Renaming VFs pattern

So, the patterns we have available can make use of this

        # NOTE: 0[0] -> PCI Domain
        #       0[1] -> PCI Bus
        #       0[2] -> PCI Device
        #       0[3] -> PCI Function
        # NOTE: 1[0] -> PCI Domain (SR-IOV PF)
        #       1[1] -> PCI Bus (SR-IOV PF)
        #       1[2] -> PCI Device (SR-IOV PF)
        #       1[3] -> PCI Function (SR-IOV PF)
        # NOTE: 2    -> "virtfn" index (SR-IOV VF)
        # NOTE: 3    -> index derived from set_counter
        set_name: "pf{1[1]}{1[2]}vf{2}"

However, one of the things we need is to, when doing passthrough of a VF interface in opennebula, have the ability to tell which is its parent PF. Normally you can tell this by the default naming scheme, but if we were to rename the PFs (by targeting specific device addresses), this suggestiveness breaks. The patterning makes use of pci addressing, but ideally we would have a

        # NOTE: 4    -> "virtfn" parent PF (SR-IOV VF)

to name the VFs by prefixing them with their respective parent PF. Alternatively, if this is possible right now, please let me know.

We are working on making this more resilient by adding the parent PF to PCI monitoring and exposing the parenthood of each VF on SRIOV mode when using sunstone OpenNebula/one#7671 but in the meantime we could possible use suggestive naming patterns (which is the default behavior, but alas) matching the renaming done.

Ty very much for the quick feedback

[sk4zuzu]

So inspecting this code

one-deploy/roles/helper/pci/tasks/udev.yml

Lines 33 to 57 in 75f92da

	_vf_to_pf: >-
	{{ shell_pf_vf_fn.stdout_lines | d([])
	| map('split', ';')
	| items2dict(key_name=1, value_name=0) }}
	_vf_to_fn: >-
	{{ shell_pf_vf_fn.stdout_lines | d([])
	| map('split', ';')
	| items2dict(key_name=1, value_name=2) }}
	block:
	- name: Scan /sys/bus/pci/devices/*/virtfn* (SR-IOV)
	ansible.builtin.shell:
	cmd: |
	set -o errexit -o pipefail; shopt -s nullglob
	{% for v in _sriov_devices %}
	for VF_PATH in '/sys/bus/pci/devices/{{ v.Slot }}/virtfn'*; do
	VF="$(basename "$(realpath "$VF_PATH")")"
	FN="$(basename "$VF_PATH")"
	echo "{{ v.Slot }};$VF;${FN#virtfn}"
	done
	{% endfor %}
	executable: /bin/bash
	register: shell_pf_vf_fn
	changed_when: false
	when: _sriov_devices | count > 0

It's possible you can "reconstruct" PFs name, I think this example illustrates it

one-deploy/roles/helper/pci/README.md

Lines 121 to 134 in 75f92da

	pci_devices:
	# Enable all available VFs for all existing Mellanox PFs + rename PFs.
	- vendor: "15b3"
	device: "1015"
	class: "0200"
	set_numvfs: max
	set_name: "pf{0[1]}{0[2]}{0[3]}"
	# Rename all existing Mellanox VFs.
	- vendor: "15b3"
	device: "1016"
	class: "0200"
	virtual: true
	set_name: "pf{1[1]}{1[2]}{1[3]}vf{2}"

no? 🤔 But we can introduce 4 as a direct "borrow" from the lspci_devices structure for user convenience if needed.

[dann1]

So, in that case it works very well. But it might be that the user targets the PF renaming specifically by address, without a pattern, for example

        # PF
        - address: "0000:16:00.0"
          set_name: vmnic7
          set_numvfs: max # activate all VF on this PF
          unlisted: true # skip PF monitoring to avoid accidental PT
          set_switchdev: true # switchdev mode, requires hw and driver support

        # VF
        - vendor: "8086" # intel
          device: "159b" # 810 vf device
          class: "0200"
          set_driver: vfio-pci
          set_name: "pf{1[1]}{1[3]}vf{2}"
          virtual: true
          unlisted: false

in this case, we want to rename all VFs after PF renaming has been individually targeted. Since there is no PCI based pattern for PF set_name then the pattern is off.

The idea is to be able to tell which device is the parent pf of a vf, something like

        - vendor: "8086" # intel
          device: "159b" # 810 vf device
          class: "0200"
          set_driver: vfio-pci
          set_name: "pf{4}vf{2}"
          virtual: true
          unlisted: false

[sk4zuzu]

OK it makes sense to me now. We'll have to introduce another "preproccessor" step of lspci_devices to resolve names then merge them back into the structure before we generate UDEV rules. 👍 😍

[dann1]

DNS

Some potentially dangerous behaviour about DNS config. By default the package list for ovs requirements includes systemd-resolved. In a fresh install, DNS is managed by NetworkManager, which gets deactivated by opennebula-ovs. So prior to running one-deploy we already have a working generated resolv.conf

[root@nfhhvbcnlb03 ~]# cat /etc/resolv.conf
# Generated by NetworkManager
search maqueta
nameserver 192.168.142.111
nameserver 192.168.115.15
[root@nfhhvbcnlb03 ~]# ls -l /etc/resolv.conf
-rw-r--r--. 1 root root 98 Jun 12 10:32 /etc/resolv.conf

When switching over to OVS network configuration management, even if we don't declare a dns configuration within the bridge, for example

        ovsbr0:
          set:
            - datapath_type: netdev
          ports: [bond0]
          addrs:
            - cidr: "{{ ansible_default_ipv4.address ~ '/' ~ ansible_default_ipv4.prefix }}"
              metric: 400
          gw: "{{ ansible_default_ipv4.gateway }}"

we get that link reworked because systemd-resolved is pre-installed behind the scenes and this logic checks its existence to trigger resolv.conf management.If this logic doesn't exist, the previous resolv.conf works perfectly post opennebula-ovs. And if ever reverting back to NetworkManager is necessary, then it's also simpler.

It is possible to avoid the if block changes by setting the packages in the inventory without systemd-resolved

    ovs_packages_dpdk:
      RedHat:
        - dpdk-tools
        - ethtool
        - iproute
        - iputils
        - openvswitch3.6
    ovs_packages:
      RedHat:
        - ethtool
        - iproute
        - iputils
        - openvswitch3.6

Besides that, if dns is declared, then we would also need search domain. As of now it only supports servers.

So, summarizing

• do we really need to migrate the previous resolv.conf to be managed by systemd-resolved ?
  • and consequentially do we need to install systemd-resolved ?
• does it make sense to avoid any dns related changes if we don't state dns within the ovs bridge dict ?
  • as of now resolv.conf is always relinked if the systemd-resolved systemd unit exists
• support for search domain when migrating to systemd-resolved

[tinova]

pending review of #230