Add OVN RBAC support with per-node ovn-controller certificates

Enable OVN role-based access control (RBAC) on the Southbound database
so that ovn-controller nodes can only modify their own chassis rows.

When the openstack-operator provides an RBAC issuer name (from a
dedicated rootca-ovn-rbac CA, see patch [1]), this patch:

* Creates per-node cert-manager Certificate CRs for each ovn-controller
  pod, with CN set to a deterministic UUID5 system-id derived from the
  node name (ComputeSystemID). This CN must match the chassis
  system-id for RBAC to authorize operations.

* Copies the RBAC client cert/key into /etc/openvswitch/ on each node
  via the config job, and switches ovn-controller to use these dedicated
  paths instead of the shared OVN DB cert.

* Mounts the RBAC CA certificate into ovsdbserver-sb pods and builds a
  combined CA bundle (regular CA + RBAC CA) so the SB database can
  verify ovn-controller client certificates.

* Sets role=ovn-controller on the SB DB connection (port 6642) to enforce RBAC.

* Creates a second SB DB listener on port 16642 with full (unrestricted)
  access, used by ovn-northd.

* Updates inactivity probe handling in setup.sh and runtime-config.sh to
  iterate over all connections, since SB now has two listeners.

* ovn-controller POD now waits for the Northd to be ready before start,
  it is done to avoid race condition when ovn-controller POD could be
  started before Northd would populate RBAC rules in the SB DB and that
  could cause issues with connection of the ovn-controller to the SB DB.

* remove OVS DaemonSet readiness gate in the ovncontroller controller -
  it was there to make sure that local ovsdb is up so that config job
  would be able to store config values in it. But init scripts are
  already waiting actively for the ovsdb to become active before it
  anything else will be done. This check is also causing deadlock with
  deploying ovs and ovn-controller PODs now with RBAC enabled as
  ovn-controller needs to have certificates ready to start and create
  br-int brigde. That can't be done if the config job is not started and
  config job couldn't be started because ovncontroller controller was
  waiting for the OVS DaemonSet to be ready.

* Remove OVS DaemonSet readiness gate from the ovncontroller controller -
  the gate ensured that the local ovsdb was running before the config job
  attempted to store configuration values. However, the init scripts
  already poll for ovsdb availability before doing anything else, making
  the gate redundant.
  With RBAC enabled, this gate also causes a deadlock: ovn-controller
  needs its RBAC certificates to start and create the br-int bridge, but
  those certificates are deployed by the config job, which cannot run
  until the OVS DaemonSet is ready — and the OVS DaemonSet cannot become
  ready without br-int.

[1] openstack-k8s-operators/openstack-operator#1906

Related: #OSPRH-1921
Closes: #OSPRH-1922

Signed-off-by: Slawek Kaplonski <skaplons@redhat.com>

Author: slawqo

api/bases/ovn.openstack.org_ovncontrollers.yaml

@@ -178,6 +178,13 @@ spec:
                 - info
                 - dbg
                 type: string
+              rbacIssuerName:
+                description: |-
+                  RbacIssuerName - The name of the cert-manager Issuer used to sign
+                  per-node ovn-controller RBAC certificates. When set, the controller
+                  creates cert-manager Certificate resources for each node instead of
+                  signing certificates locally with the CA key.
+                type: string
               resources:
                 description: |-
                   Resources - Compute Resources required by this service (Limits/Requests).

api/bases/ovn.openstack.org_ovndbclusters.yaml

@@ -274,6 +274,12 @@ spec:
                   Active probe interval from standby to active ovsdb-server remote
                 format: int32
                 type: integer
+              rbacCACertSecretName:
+                description: |-
+                  RbacCACertSecretName - The name of the K8s Secret containing the RBAC
+                  PKI CA certificate (tls.crt). Used by the SB database to verify
+                  ovn-controller client certificates when RBAC is enabled.
+                type: string
               replicas:
                 default: 1
                 description: Replicas of OVN DBCluster to run
@@ -438,6 +444,11 @@ spec:
                 description: InternalDBAddress - DB IP address used by other Pods
                   in the cluster
                 type: string
+              internalDbAddressRbacFullAccess:
+                description: |-
+                  InternalDBAddressRbacFullAccess - DB IP address for full-access (non-RBAC)
+                  connections, used by ovn-northd when RBAC is enabled on SB DB
+                type: string
               lastAppliedTopology:
                 description: LastAppliedTopology - the last applied Topology
                 properties:

api/v1beta1/client.go

@@ -72,6 +72,27 @@ func GetOVNController(
 	return nil, nil
 }
 
+// GetOVNNorthd - return OVNNorthd instance in the given namespace
+func GetOVNNorthd(
+	ctx context.Context,
+	h *helper.Helper,
+	namespace string,
+) (*OVNNorthd, error) {
+	ovnNorthdList := &OVNNorthdList{}
+	listOpts := []client.ListOption{
+		client.InNamespace(namespace),
+	}
+	err := h.GetClient().List(ctx, ovnNorthdList, listOpts...)
+	if err != nil {
+		return nil, err
+	}
+	if len(ovnNorthdList.Items) > 0 {
+		return &ovnNorthdList.Items[0], nil
+	}
+
+	return nil, nil
+}
+
 // GetDBClusterByType - return OVNDBCluster for the given dbType
 func GetDBClusterByType(
 	ctx context.Context,

api/v1beta1/ovncontroller_types.go

@@ -124,6 +124,13 @@ type OVNControllerSpecCore struct {
 	// +kubebuilder:validation:Optional
 	// MetricsTLS - Parameters related to TLS for metrics daemonset
 	MetricsTLS tls.SimpleService `json:"metricsTLS,omitempty"`
+
+	// +kubebuilder:validation:Optional
+	// RbacIssuerName - The name of the cert-manager Issuer used to sign
+	// per-node ovn-controller RBAC certificates. When set, the controller
+	// creates cert-manager Certificate resources for each node instead of
+	// signing certificates locally with the CA key.
+	RbacIssuerName string `json:"rbacIssuerName,omitempty"`
 }
 
 // OVNControllerStatus defines the observed state of OVNController

api/v1beta1/ovndbcluster_types.go

@@ -146,6 +146,12 @@ type OVNDBClusterSpecCore struct {
 	// +kubebuilder:validation:Optional
 	// MetricsTLS - Parameters related to TLS for metrics sidecar
 	MetricsTLS tls.SimpleService `json:"metricsTLS,omitempty"`
+
+	// +kubebuilder:validation:Optional
+	// RbacCACertSecretName - The name of the K8s Secret containing the RBAC
+	// PKI CA certificate (tls.crt). Used by the SB database to verify
+	// ovn-controller client certificates when RBAC is enabled.
+	RbacCACertSecretName string `json:"rbacCACertSecretName,omitempty"`
 }
 
 // OVNDBClusterOverrideSpec to override the generated manifest of several child resources.
@@ -171,6 +177,10 @@ type OVNDBClusterStatus struct {
 	// InternalDBAddress - DB IP address used by other Pods in the cluster
 	InternalDBAddress string `json:"internalDbAddress,omitempty"`
 
+	// InternalDBAddressRbacFullAccess - DB IP address for full-access (non-RBAC)
+	// connections, used by ovn-northd when RBAC is enabled on SB DB
+	InternalDBAddressRbacFullAccess string `json:"internalDbAddressRbacFullAccess,omitempty"`
+
 	// NetworkAttachments status of the deployment pods
 	NetworkAttachments map[string][]string `json:"networkAttachments,omitempty"`
 
@@ -239,6 +249,18 @@ func (instance OVNDBCluster) GetInternalEndpoint() (string, error) {
 	return instance.Status.InternalDBAddress, nil
 }
 
+// GetInternalEndpointRbacFullAccess - return the full-access (non-RBAC) internal endpoint for SB DB.
+// Falls back to GetInternalEndpoint if the DB is not SB or TLS is not enabled.
+func (instance OVNDBCluster) GetInternalEndpointRbacFullAccess() (string, error) {
+	if instance.Spec.DBType != SBDBType || !instance.Spec.TLS.Enabled() {
+		return instance.GetInternalEndpoint()
+	}
+	if instance.Status.InternalDBAddressRbacFullAccess == "" {
+		return "", fmt.Errorf("internal RBAC full-access DBEndpoint not ready yet for %s", instance.Spec.DBType)
+	}
+	return instance.Status.InternalDBAddressRbacFullAccess, nil
+}
+
 // GetExternalEndpoint - return the DNS that openstack dnsmasq can resolve
 func (instance OVNDBCluster) GetExternalEndpoint() (string, error) {
 	if (instance.Spec.NetworkAttachment != "" || instance.Spec.Override.Service != nil) && instance.Status.DBAddress == "" {

cmd/main.go

@@ -46,6 +46,7 @@ import (
 
 	// +kubebuilder:scaffold:imports
 
+	certmgrv1 "github.com/cert-manager/cert-manager/pkg/apis/certmanager/v1"
 	networkv1 "github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/apis/k8s.cni.cncf.io/v1"
 	infranetworkv1 "github.com/openstack-k8s-operators/infra-operator/apis/network/v1beta1"
 	topologyv1 "github.com/openstack-k8s-operators/infra-operator/apis/topology/v1beta1"
@@ -66,6 +67,7 @@ func init() {
 	utilruntime.Must(networkv1.AddToScheme(scheme))
 	utilruntime.Must(infranetworkv1.AddToScheme(scheme))
 	utilruntime.Must(topologyv1.AddToScheme(scheme))
+	utilruntime.Must(certmgrv1.AddToScheme(scheme))
 	//+kubebuilder:scaffold:scheme
 }
 

config/crd/bases/ovn.openstack.org_ovncontrollers.yaml

@@ -178,6 +178,13 @@ spec:
                 - info
                 - dbg
                 type: string
+              rbacIssuerName:
+                description: |-
+                  RbacIssuerName - The name of the cert-manager Issuer used to sign
+                  per-node ovn-controller RBAC certificates. When set, the controller
+                  creates cert-manager Certificate resources for each node instead of
+                  signing certificates locally with the CA key.
+                type: string
               resources:
                 description: |-
                   Resources - Compute Resources required by this service (Limits/Requests).

config/crd/bases/ovn.openstack.org_ovndbclusters.yaml

@@ -274,6 +274,12 @@ spec:
                   Active probe interval from standby to active ovsdb-server remote
                 format: int32
                 type: integer
+              rbacCACertSecretName:
+                description: |-
+                  RbacCACertSecretName - The name of the K8s Secret containing the RBAC
+                  PKI CA certificate (tls.crt). Used by the SB database to verify
+                  ovn-controller client certificates when RBAC is enabled.
+                type: string
               replicas:
                 default: 1
                 description: Replicas of OVN DBCluster to run
@@ -438,6 +444,11 @@ spec:
                 description: InternalDBAddress - DB IP address used by other Pods
                   in the cluster
                 type: string
+              internalDbAddressRbacFullAccess:
+                description: |-
+                  InternalDBAddressRbacFullAccess - DB IP address for full-access (non-RBAC)
+                  connections, used by ovn-northd when RBAC is enabled on SB DB
+                type: string
               lastAppliedTopology:
                 description: LastAppliedTopology - the last applied Topology
                 properties:

config/rbac/role.yaml

@@ -65,6 +65,26 @@ rules:
   - patch
   - update
   - watch
+- apiGroups:
+  - cert-manager.io
+  resources:
+  - certificates
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - cert-manager.io
+  resources:
+  - issuers
+  verbs:
+  - get
+  - list
+  - watch
 - apiGroups:
   - k8s.cni.cncf.io
   resources:

go.mod

@@ -3,6 +3,7 @@ module github.com/openstack-k8s-operators/ovn-operator
 go 1.24.4
 
 require (
+	github.com/cert-manager/cert-manager v1.16.5
 	github.com/go-logr/logr v1.4.3
 	github.com/google/uuid v1.6.0
 	github.com/k8snetworkplumbingwg/network-attachment-definition-client v1.7.7
@@ -65,7 +66,7 @@ require (
 	github.com/prometheus/procfs v0.16.1 // indirect
 	github.com/spf13/cobra v1.9.1 // indirect
 	github.com/spf13/pflag v1.0.7 // indirect
-	github.com/stoewer/go-strcase v1.2.0 // indirect
+	github.com/stoewer/go-strcase v1.3.0 // indirect
 	github.com/x448/float16 v0.8.4 // indirect
 	go.opentelemetry.io/auto/sdk v1.1.0 // indirect
 	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.58.0 // indirect
@@ -103,6 +104,7 @@ require (
 	k8s.io/klog/v2 v2.130.1 // indirect
 	k8s.io/kube-openapi v0.0.0-20250902184714-7fc278399c7f // indirect
 	sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.30.3 // indirect
+	sigs.k8s.io/gateway-api v1.1.0 // indirect
 	sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 // indirect
 	sigs.k8s.io/randfill v1.0.0 // indirect
 	sigs.k8s.io/structured-merge-diff/v4 v4.7.0 // indirect