Add metrics Service and TLS support for InstanceHA

- Add a Kubernetes Service exposing the InstanceHA Prometheus metrics
  endpoint, with labels for automatic discovery by the telemetry
  operator's ScrapeConfig.
- Add MetricsTLS field (tls.SimpleService) to the InstanceHa API,
  allowing TLS certificate configuration for the metrics endpoint.
- Mount TLS certificate secret into the deployment and pass cert/key
  paths via environment variables when MetricsTLS is enabled.
- Validate the MetricsTLS secret in the controller with hash tracking
  for automatic pod rollout on certificate rotation.
- Add field indexer for the metrics TLS secret so the controller
  reconciles on secret changes.
- Update the Python health/metrics server to wrap the HTTP socket with
  TLS when certificate environment variables are present.
- Add RBAC annotation for Services to the InstanceHA controller.
- Add functional tests for the metrics Service creation.
- Update documentation for Prometheus metrics integration.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Author: lmiccini

apis/bases/instanceha.openstack.org_instancehas.yaml

@@ -100,6 +100,18 @@ spec:
                 default: 7410
                 format: int32
                 type: integer
+              metricsTLS:
+                description: MetricsTLS - Parameters related to TLS for the metrics
+                  endpoint
+                properties:
+                  caBundleSecretName:
+                    description: CaBundleSecretName - holding the CA certs in a pre-created
+                      bundle file
+                    type: string
+                  secretName:
+                    description: SecretName - holding the cert, key for the service
+                    type: string
+                type: object
               networkAttachments:
                 description: |-
                   NetworkAttachments is a list of NetworkAttachment resource names to expose

apis/instanceha/v1beta1/instanceha_types.go

@@ -115,6 +115,11 @@ type InstanceHaSpec struct {
 	// +kubebuilder:validation:Optional
 	// Auth - Parameters related to authentication
 	Auth AuthSpec `json:"auth,omitempty"`
+
+	// +kubebuilder:validation:Optional
+	//+operator-sdk:csv:customresourcedefinitions:type=spec
+	// MetricsTLS - Parameters related to TLS for the metrics endpoint
+	MetricsTLS tls.SimpleService `json:"metricsTLS,omitempty"`
 }
 
 // InstanceHaStatus defines the observed state of InstanceHa

apis/instanceha/v1beta1/zz_generated.deepcopy.go

@@ -100,6 +100,18 @@ spec:
                 default: 7410
                 format: int32
                 type: integer
+              metricsTLS:
+                description: MetricsTLS - Parameters related to TLS for the metrics
+                  endpoint
+                properties:
+                  caBundleSecretName:
+                    description: CaBundleSecretName - holding the CA certs in a pre-created
+                      bundle file
+                    type: string
+                  secretName:
+                    description: SecretName - holding the cert, key for the service
+                    type: string
+                type: object
               networkAttachments:
                 description: |-
                   NetworkAttachments is a list of NetworkAttachment resource names to expose

config/crd/bases/instanceha.openstack.org_instancehas.yaml

@@ -190,7 +190,9 @@ groups:
 
 #### Scraping Configuration
 
-The InstanceHA pod exposes metrics on TCP port 8080. To scrape with Prometheus, create a `PodMonitor` or `ServiceMonitor`:
+The InstanceHA pod exposes metrics on TCP port 8080. The infra-operator automatically creates a Kubernetes Service (`<instance-name>-metrics`) with the labels `metrics: enabled` and `service: instanceha`, which the telemetry-operator discovers and scrapes via the COO Prometheus. **No manual configuration is needed when the telemetry-operator is deployed.**
+
+For environments using OpenShift user workload monitoring instead of (or in addition to) the telemetry-operator, create a `PodMonitor`:
 
 ```yaml
 apiVersion: monitoring.coreos.com/v1

docs/instanceha_guide.md

@@ -482,37 +482,23 @@ When the [telemetry-operator](https://github.com/openstack-k8s-operators/telemet
 | OpenShift user workload monitoring | `prometheus-user-workload` in `openshift-user-workload-monitoring` | `thanos-querier` route in `openshift-monitoring` |
 | telemetry-operator (COO) | `prometheus-metric-storage` in `openstack` | `metric-storage-prometheus.openstack.svc:9090` |
 
-The PodMonitor approach described above places InstanceHA metrics in the OpenShift user workload Prometheus. If you want InstanceHA metrics alongside other OpenStack metrics (Ceilometer, RabbitMQ, node-exporter, OVN) in the COO Prometheus, create a `ScrapeConfig` CR instead.
+### Automatic Discovery (default)
 
-### Creating a ScrapeConfig for COO Prometheus
+The telemetry-operator **automatically discovers and scrapes InstanceHA metrics** — no manual configuration is required. The infra-operator creates a Kubernetes Service (`<instance-name>-metrics`) with the labels `metrics: enabled` and `service: instanceha`. The telemetry-operator's `MetricStorage` controller watches for Services with these labels and automatically generates a `ScrapeConfig` CR named `telemetry-instanceha` targeting port 8080.
 
-The COO Prometheus only picks up CRs with the label `service: metricStorage`. Create a `ScrapeConfig` targeting the InstanceHA pod:
+This works the same way as the OVN metrics integration. When a `MetricStorage` CR exists in the namespace:
 
-```yaml
-apiVersion: monitoring.rhobs/v1alpha1
-kind: ScrapeConfig
-metadata:
-  name: instanceha-metrics
-  namespace: openstack
-  labels:
-    service: metricStorage
-spec:
-  scrapeInterval: 30s
-  metricsPath: /metrics
-  staticConfigs:
-    - targets:
-        - "<instanceha-pod-ip>:8080"
-```
+1. The telemetry-operator discovers the InstanceHA metrics Service via label selectors
+2. A `ScrapeConfig` CR is created with the target `<service-name>.<namespace>.svc:8080`
+3. The COO Prometheus picks up the `ScrapeConfig` and begins scraping
+4. If the InstanceHA Service is deleted or recreated, the `ScrapeConfig` is automatically reconciled
 
-To discover the pod IP dynamically:
+To verify the automatic scrapeconfig was created:
 
 ```bash
-POD_IP=$(oc get pod -n openstack -l service=instanceha -o jsonpath='{.items[0].status.podIP}')
-echo "Target: ${POD_IP}:8080"
+oc get scrapeconfig -n openstack telemetry-instanceha -o yaml
 ```
 
-> **Note**: The COO `ScrapeConfig` uses static targets (IP:port), not label-based pod discovery like a `PodMonitor`. If the InstanceHA pod is rescheduled and gets a new IP, the `ScrapeConfig` must be updated. For automatic discovery, consider requesting native InstanceHA support in the telemetry-operator — the OVN metrics integration uses a label-based service discovery pattern that could be extended to InstanceHA.
-
 ### Alert Rules for COO Prometheus
 
 The alert rules from the [Alert Rules](#alert-rules) section use the `monitoring.coreos.com/v1` API group, which is picked up by OpenShift's built-in Prometheus Operator. To use these alerts with the COO Prometheus instead, change the API group and add the `service: metricStorage` label:
@@ -532,7 +518,7 @@ spec:
 ### Which Approach to Use
 
 - **OpenShift user workload monitoring only** (no telemetry-operator): Use the PodMonitor approach from [Enabling Scraping](#enabling-scraping). This is simpler and uses automatic pod discovery.
-- **telemetry-operator deployed**: Use the ScrapeConfig approach if you want all OpenStack metrics in a single Prometheus. You can also use both approaches simultaneously — the PodMonitor and ScrapeConfig target different Prometheus instances and do not conflict.
+- **telemetry-operator deployed** (default): InstanceHA metrics are automatically scraped by the COO Prometheus alongside other OpenStack metrics (Ceilometer, RabbitMQ, node-exporter, OVN). No manual configuration needed. You can also deploy the PodMonitor simultaneously — it targets the OpenShift user workload Prometheus and does not conflict with the COO scrapeconfig.
 - **Querying across both**: OpenShift's `thanos-querier` route aggregates the cluster and user workload Prometheus instances. The COO Prometheus is separate and must be queried directly at `metric-storage-prometheus.openstack.svc:9090`.
 
 ---

docs/instanceha_prometheus.md

@@ -55,6 +55,7 @@ import (
 
 	commondeployment "github.com/openstack-k8s-operators/lib-common/modules/common/deployment"
 	"github.com/openstack-k8s-operators/lib-common/modules/common/secret"
+	commonservice "github.com/openstack-k8s-operators/lib-common/modules/common/service"
 	"github.com/openstack-k8s-operators/lib-common/modules/common/util"
 
 	networkv1 "github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/apis/k8s.cni.cncf.io/v1"
@@ -80,6 +81,7 @@ func (r *Reconciler) GetLogger(ctx context.Context) logr.Logger {
 // +kubebuilder:rbac:groups=instanceha.openstack.org,resources=instancehas/finalizers,verbs=update;patch
 // +kubebuilder:rbac:groups=core,resources=configmaps,verbs=get;list;watch;
 // +kubebuilder:rbac:groups=core,resources=secrets,verbs=get;list;watch;
+// +kubebuilder:rbac:groups=core,resources=services,verbs=get;list;watch;create;update;patch;delete
 // +kubebuilder:rbac:groups=k8s.cni.cncf.io,resources=network-attachment-definitions,verbs=get;list;watch
 // service account, role, rolebinding
 // +kubebuilder:rbac:groups="",resources=serviceaccounts,verbs=get;list;watch;create;update;patch
@@ -164,6 +166,7 @@ func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (result ct
 		condition.UnknownCondition(condition.RoleReadyCondition, condition.InitReason, condition.RoleReadyInitMessage),
 		condition.UnknownCondition(condition.RoleBindingReadyCondition, condition.InitReason, condition.RoleBindingReadyInitMessage),
 		condition.UnknownCondition(condition.NetworkAttachmentsReadyCondition, condition.InitReason, condition.NetworkAttachmentsReadyInitMessage),
+		condition.UnknownCondition(condition.CreateServiceReadyCondition, condition.InitReason, condition.CreateServiceReadyInitMessage),
 	)
 	instance.Status.Conditions.Init(&cl)
 	instance.Status.ObservedGeneration = instance.Generation
@@ -369,8 +372,6 @@ func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (result ct
 		)
 		if err != nil {
 			if k8s_errors.IsNotFound(err) {
-				// Since the CA cert secret should have been manually created by the user and provided in the spec,
-				// we treat this as a warning because it means that the service will not be able to start.
 				instance.Status.Conditions.Set(condition.FalseCondition(
 					condition.TLSInputReadyCondition,
 					condition.ErrorReason,
@@ -390,6 +391,28 @@ func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (result ct
 		configVars[instance.Spec.CaBundleSecretName] = env.SetValue(secretHash)
 	}
 
+	if instance.Spec.MetricsTLS.Enabled() {
+		hash, err := instance.Spec.MetricsTLS.ValidateCertSecret(ctx, helper, instance.Namespace)
+		if err != nil {
+			if k8s_errors.IsNotFound(err) {
+				instance.Status.Conditions.Set(condition.FalseCondition(
+					condition.TLSInputReadyCondition,
+					condition.RequestedReason,
+					condition.SeverityInfo,
+					condition.TLSInputReadyWaitingMessage, err.Error()))
+				return ctrl.Result{}, nil
+			}
+			instance.Status.Conditions.Set(condition.FalseCondition(
+				condition.TLSInputReadyCondition,
+				condition.ErrorReason,
+				condition.SeverityWarning,
+				condition.TLSInputErrorMessage,
+				err.Error()))
+			return ctrl.Result{}, err
+		}
+		configVars[tls.TLSHashName+"_metrics"] = env.SetValue(hash)
+	}
+
 	// all cert input checks out so report InputReady
 	instance.Status.Conditions.MarkTrue(condition.TLSInputReadyCondition, condition.InputReadyMessage)
 
@@ -505,6 +528,28 @@ func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (result ct
 		// remove LastAppliedTopology from the .Status
 		instance.Status.LastAppliedTopology = nil
 	}
+	commonsvc, err := commonservice.NewService(instanceha.MetricsService(instance), time.Duration(5)*time.Second, nil)
+	if err != nil {
+		instance.Status.Conditions.Set(condition.FalseCondition(
+			condition.CreateServiceReadyCondition,
+			condition.ErrorReason,
+			condition.SeverityWarning,
+			condition.CreateServiceReadyErrorMessage,
+			err.Error()))
+		return ctrl.Result{}, err
+	}
+	sres, serr := commonsvc.CreateOrPatch(ctx, helper)
+	if serr != nil {
+		instance.Status.Conditions.Set(condition.FalseCondition(
+			condition.CreateServiceReadyCondition,
+			condition.ErrorReason,
+			condition.SeverityWarning,
+			condition.CreateServiceReadyErrorMessage,
+			serr.Error()))
+		return sres, serr
+	}
+	instance.Status.Conditions.MarkTrue(condition.CreateServiceReadyCondition, condition.CreateServiceReadyMessage)
+
 	deployment := commondeployment.NewDeployment(instanceha.Deployment(instance, deploymentLabels, serviceAnnotations, cloud, configVarsHash, containerImage, topology, acSecretName), time.Duration(5)*time.Second)
 	sfres, sferr := deployment.CreateOrPatch(ctx, helper)
 	if sferr != nil {
@@ -558,6 +603,7 @@ const (
 	instanceHaConfigMapField   = ".spec.instanceHaConfigMap"
 	topologyField              = ".spec.topologyRef.Name"
 	acSecretField              = ".spec.auth.applicationCredentialSecret" // #nosec G101
+	metricsTLSField            = ".spec.metricsTLS.secretName"            // #nosec G101
 )
 
 var allWatchFields = []string{
@@ -568,6 +614,7 @@ var allWatchFields = []string{
 	instanceHaConfigMapField,
 	topologyField,
 	acSecretField,
+	metricsTLSField,
 }
 
 // SetupWithManager sets up the controller with the Manager.
@@ -649,9 +696,21 @@ func (r *Reconciler) SetupWithManager(mgr ctrl.Manager) error {
 		return err
 	}
 
+	// index metricsTLSField
+	if err := mgr.GetFieldIndexer().IndexField(context.Background(), &instancehav1.InstanceHa{}, metricsTLSField, func(rawObj client.Object) []string {
+		cr := rawObj.(*instancehav1.InstanceHa)
+		if cr.Spec.MetricsTLS.SecretName == nil {
+			return nil
+		}
+		return []string{*cr.Spec.MetricsTLS.SecretName}
+	}); err != nil {
+		return err
+	}
+
 	return ctrl.NewControllerManagedBy(mgr).
 		For(&instancehav1.InstanceHa{}).
 		Owns(&appsv1.Deployment{}).
+		Owns(&corev1.Service{}).
 		Owns(&corev1.ServiceAccount{}).
 		Owns(&rbacv1.Role{}).
 		Owns(&rbacv1.RoleBinding{}).

internal/controller/instanceha/instanceha_controller.go

@@ -0,0 +1,10 @@
+package instanceha
+
+const (
+	// MetricsCertPath is the path to the metrics certificate file
+	MetricsCertPath = "/etc/pki/tls/certs/metrics.crt"
+	// MetricsKeyPath is the path to the metrics private key file
+	MetricsKeyPath = "/etc/pki/tls/private/metrics.key"
+	// DefaultMetricsCertSecret is the default secret name for the metrics TLS certificate
+	DefaultMetricsCertSecret = "cert-instanceha-metrics" //nolint:gosec
+)

internal/instanceha/const.go

@@ -17,6 +17,7 @@ import (
 	instancehav1 "github.com/openstack-k8s-operators/infra-operator/apis/instanceha/v1beta1"
 	topologyv1 "github.com/openstack-k8s-operators/infra-operator/apis/topology/v1beta1"
 	env "github.com/openstack-k8s-operators/lib-common/modules/common/env"
+	"github.com/openstack-k8s-operators/lib-common/modules/common/tls"
 
 	"fmt"
 	appsv1 "k8s.io/api/apps/v1"
@@ -103,6 +104,27 @@ func Deployment(
 		volumeMounts = append(volumeMounts, instance.Spec.CreateVolumeMounts(nil)...)
 	}
 
+	// add metrics TLS cert if defined
+	if instance.Spec.MetricsTLS.Enabled() {
+		certSecretName := DefaultMetricsCertSecret
+		if instance.Spec.MetricsTLS.SecretName != nil && *instance.Spec.MetricsTLS.SecretName != "" {
+			certSecretName = *instance.Spec.MetricsTLS.SecretName
+		}
+		metricsSvc := tls.Service{
+			SecretName: certSecretName,
+			CertMount:  ptr.To(MetricsCertPath),
+			KeyMount:   ptr.To(MetricsKeyPath),
+		}
+		volumes = append(volumes, metricsSvc.CreateVolume("metrics-certs"))
+		volumeMounts = append(volumeMounts, metricsSvc.CreateVolumeMounts("metrics-certs")...)
+
+		envVars["METRICS_TLS_CERT"] = env.SetValue(MetricsCertPath)
+		envVars["METRICS_TLS_KEY"] = env.SetValue(MetricsKeyPath)
+
+		livenessProbe.HTTPGet.Scheme = corev1.URISchemeHTTPS
+		readinessProbe.HTTPGet.Scheme = corev1.URISchemeHTTPS
+	}
+
 	dep := &appsv1.Deployment{
 		ObjectMeta: metav1.ObjectMeta{
 			Name:      instance.Name,

internal/instanceha/funcs.go

@@ -0,0 +1,45 @@
+/*
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+    http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package instanceha
+
+import (
+	instancehav1 "github.com/openstack-k8s-operators/infra-operator/apis/instanceha/v1beta1"
+	common "github.com/openstack-k8s-operators/lib-common/modules/common"
+	labels "github.com/openstack-k8s-operators/lib-common/modules/common/labels"
+	service "github.com/openstack-k8s-operators/lib-common/modules/common/service"
+	corev1 "k8s.io/api/core/v1"
+)
+
+// MetricsService exposes the InstanceHA metrics endpoint for Prometheus scraping
+func MetricsService(instance *instancehav1.InstanceHa) *corev1.Service {
+	svcLabels := labels.GetLabels(instance, labels.GetGroupLabel("instanceha"), map[string]string{
+		common.AppSelector: "instanceha",
+		"metrics":          "enabled",
+	})
+
+	details := &service.GenericServiceDetails{
+		Name:      instance.GetName() + "-metrics",
+		Namespace: instance.GetNamespace(),
+		Labels:    svcLabels,
+		Selector: map[string]string{
+			common.AppSelector: "instanceha",
+		},
+		Port: service.GenericServicePort{
+			Name:     "metrics",
+			Port:     8080,
+			Protocol: "TCP",
+		},
+	}
+
+	return service.GenericService(details)
+}