Skip to content

Commit 7df857c

Browse files
Fortune-NdlovuFortune-Ndlovu
authored
chore: add monitoring docs (#220)
* add monitoring docs Signed-off-by: Fortune-Ndlovu <fndlovu@redhat.com> * clean up links Signed-off-by: Fortune-Ndlovu <fndlovu@redhat.com> --------- Signed-off-by: Fortune-Ndlovu <fndlovu@redhat.com>
1 parent 083121a commit 7df857c

2 files changed

Lines changed: 175 additions & 0 deletions

File tree

‎docs/images/openshift-metrics.png‎

213 KB
Loading

‎docs/monitoring.md‎

Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
# Metrics Monitoring for RHDH Helm Chart
2+
3+
The RHDH provides a `/metrics` endpoint on port `9464` that provides OpenTelemetry metrics about your Backstage application. This endpoint can be used to monitor your Backstage instance using OpenTelemetry and Grafana.
4+
5+
When deploying RHDH using the [RHDH Helm chart](https://github.com/redhat-developer/rhdh-chart), monitoring for your RHDH instance can be configured using the following steps.
6+
7+
## Prerequisites
8+
9+
- Kubernetes 1.27+ ([OpenShift 4.14+](https://docs.redhat.com/en/documentation/openshift_container_platform/4.14/html-single/release_notes/index#ocp-4-14-about-this-release))
10+
- Helm 3.10+ or [latest release](https://github.com/helm/helm/releases)
11+
- PV provisioner support in the underlying infrastructure
12+
- The [RHDH Helm chart repositories](https://github.com/redhat-developer/rhdh-chart#installing-from-the-chart-repository)
13+
14+
## Metrics Monitoring
15+
16+
### Enabling Metrics Monitoring on OpenShift
17+
18+
To enable metrics monitoring on OpenShift, we need to create a `ServiceMonitor` resource in the OpenShift cluster that will be used by Prometheus to scrape metrics from your Backstage instance. For the metrics to be ingested by the built-in Prometheus instances in OpenShift, please ensure you enabled [monitoring for user-defined projects](https://docs.redhat.com/en/documentation/openshift_container_platform/latest/html/monitoring/configuring-user-workload-monitoring#preparing-to-configure-the-monitoring-stack-uwm).
19+
20+
#### Helm deployment
21+
22+
To enable metrics on OpenShift when deploying with the RHDH Helm chart, you will need to modify the [`values.yaml`](https://github.com/redhat-developer/rhdh-chart/blob/main/charts/backstage/values.yaml) of the Chart.
23+
24+
To obtain the `values.yaml`, you can run the following command:
25+
26+
```bash
27+
helm show values redhat-developer/backstage > values.yaml
28+
```
29+
30+
Then, you will need to modify the `values.yaml` to enable metrics monitoring by setting `upstream.metrics.serviceMonitor.enabled` to true:
31+
32+
```yaml title="values.yaml"
33+
upstream:
34+
# Other Configurations Above
35+
metrics:
36+
serviceMonitor:
37+
enabled: true
38+
path: /metrics
39+
port: http-metrics
40+
```
41+
42+
Then you can deploy the RHDH Helm chart with the modified `values.yaml`:
43+
44+
```bash
45+
helm upgrade -i <release_name> redhat-developer/backstage -f values.yaml
46+
```
47+
48+
You can then verify metrics are being captured by navigating to the OpenShift Console. Go to `Developer` Mode, change to the namespace the showcase is deployed on, selecting `Observe` and navigating to the `Metrics` tab. Here you can create PromQL queries to query the metrics being captured by OpenTelemetry.
49+
50+
![OpenShift Metrics](./images/openshift-metrics.png)
51+
52+
### Enabling Metrics Monitoring on Azure Kubernetes Service (AKS)
53+
54+
To enable metrics monitoring for RHDH on Azure Kubernetes Service (AKS), you can use the [Azure Monitor managed service for Prometheus](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/prometheus-metrics-overview). The AKS cluster will need to have an associated [Azure Monitor workspace](https://learn.microsoft.com/en-us/azure/azure-monitor/containers/prometheus-metrics-enable?tabs=azure-portal).
55+
56+
One method is to configure the metrics scraping of your AKS cluster using the [Azure Monitor _metrics_ add-on](https://learn.microsoft.com/en-us/azure/azure-monitor/containers/prometheus-metrics-scrape-configuration).
57+
58+
The other method is to configure the Azure Monitor _monitoring_ add-on which also allows you to [send Prometheus metrics to the Log Analytics workspace](https://learn.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-prometheus-logs). These metrics can then be queried using [Log Analytics queries](https://learn.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-log-query#prometheus-metrics) as well as be visible in a Grafana instance.
59+
60+
In both methods, we can configure the metrics scraping to scrape from pods based on pod annotations. Follow the steps below for Helm deployment.
61+
62+
#### Helm Deployment for AKS
63+
64+
To add annotations to the backstage pod, add the following to the RHDH Helm chart `values.yaml`:
65+
66+
```yaml title="values.yaml"
67+
upstream:
68+
backstage:
69+
# Other configurations above
70+
podAnnotations:
71+
# Other annotations above
72+
prometheus.io/scrape: 'true'
73+
prometheus.io/path: '/metrics'
74+
prometheus.io/port: '9464'
75+
prometheus.io/scheme: 'http'
76+
```
77+
78+
#### Metrics Add-on
79+
80+
For the _metrics_ add-on, we can modify the [`ama-metrics-settings-configmap`](https://github.com/Azure/prometheus-collector/blob/main/otelcollector/configmaps/ama-metrics-settings-configmap.yaml) Config Map and enable pod annotations based scraping for the namespace your showcase instance is in. In your example Config Map, you can change the regex for the `podannotationnamespaceregex` option to match the namespaces you want to scrape from. For more information on how to configure this refer to the [official Azure docs](https://learn.microsoft.com/en-us/azure/azure-monitor/containers/prometheus-metrics-scrape-configuration#customize-metrics-collected-by-default-targets).
81+
82+
To view the metrics, you can create a Grafana instance, [configure an Azure Monitor data source plug-in](https://learn.microsoft.com/en-us/azure/azure-monitor/visualize/grafana-plugin#configure-an-azure-monitor-data-source-plug-in) and view the metrics using PromQL queries. In terms of Monitoring Add-on, refer to the [official Azure docs](https://learn.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-prometheus-logs?tabs=cluster-wide).
83+
84+
To view the metrics, you can create a Grafana instance, [configure an Azure Monitor data source plug-in](https://learn.microsoft.com/en-us/azure/azure-monitor/visualize/grafana-plugin#configure-an-azure-monitor-data-source-plug-in) and view the metrics using PromQL queries.
85+
86+
Alternatively, you can use [Log Analytics](https://learn.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-log-query#prometheus-metrics) to query the metrics using KQL. The following is an example query to get a custom metric for the Backstage instance:
87+
88+
```kql
89+
let custom_metrics = "custom-metric-name";
90+
InsightsMetrics
91+
| where Namespace contains "prometheus"
92+
| where Name == custom_metrics
93+
| extend tags = parse_json(Tags)
94+
| where tostring(tags['app.kubernetes.io/component']) == "backstage"
95+
```
96+
97+
## Configuration Examples
98+
99+
### Complete Monitoring Configuration
100+
101+
Here's a complete example of a `values.yaml` configuration with monitoring enabled:
102+
103+
```yaml title="values.yaml"
104+
upstream:
105+
backstage:
106+
# Add pod annotations for AKS monitoring (if deploying on AKS)
107+
podAnnotations:
108+
prometheus.io/scrape: 'true'
109+
prometheus.io/path: '/metrics'
110+
prometheus.io/port: '9464'
111+
prometheus.io/scheme: 'http'
112+
113+
# Enable ServiceMonitor for OpenShift monitoring
114+
metrics:
115+
serviceMonitor:
116+
enabled: true
117+
path: /metrics
118+
port: http-metrics
119+
```
120+
121+
### OpenShift-specific Configuration
122+
123+
For OpenShift deployments, focus on the ServiceMonitor configuration:
124+
125+
```yaml title="values.yaml"
126+
upstream:
127+
# Enable ServiceMonitor for OpenShift Prometheus
128+
metrics:
129+
serviceMonitor:
130+
enabled: true
131+
path: /metrics
132+
port: http-metrics
133+
134+
backstage:
135+
# Other backstage configurations as needed
136+
```
137+
138+
### AKS-specific Configuration
139+
140+
For AKS deployments, focus on pod annotations:
141+
142+
```yaml title="values.yaml"
143+
upstream:
144+
backstage:
145+
# Add annotations for Azure Monitor
146+
podAnnotations:
147+
prometheus.io/scrape: 'true'
148+
prometheus.io/path: '/metrics'
149+
prometheus.io/port: '9464'
150+
prometheus.io/scheme: 'http'
151+
152+
```
153+
154+
## Troubleshooting
155+
156+
### Metrics Not Appearing
157+
158+
1. **Verify the ServiceMonitor is created** (OpenShift):
159+
160+
```bash
161+
oc get servicemonitor -n <namespace>
162+
```
163+
164+
2. **Check if metrics endpoint is accessible**:
165+
166+
```bash
167+
oc port-forward pod/<backstage-pod> 9464:9464
168+
curl http://localhost:9464/metrics
169+
```
170+
171+
3. **Verify pod annotations** (AKS):
172+
173+
```bash
174+
kubectl get pod <backstage-pod> -o yaml | grep -A 5 annotations
175+
```

0 commit comments

Comments
 (0)