Skip to content

Commit 70ef84d

Browse files
YoEightYoEight
authored
feat(k8s): add grafana dashboard (#40)
1 parent 54077d9 commit 70ef84d

6 files changed

Lines changed: 801 additions & 19 deletions

File tree

‎METRICS.md‎

Lines changed: 183 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,183 @@
1+
# Geth Engine Metrics Documentation
2+
3+
This document describes the metrics exposed by the Geth engine and monitored in the Grafana dashboard.
4+
5+
## Overview
6+
7+
The Geth engine exposes Prometheus-format metrics that provide insights into system performance, I/O operations, and resource utilization. These metrics are organized into several categories for comprehensive monitoring.
8+
9+
## System Resource Metrics
10+
11+
### CPU Metrics
12+
13+
| Metric Name | Type | Description |
14+
|-------------|------|-------------|
15+
| `geth_sys_cpu_usage_percent` | Gauge | Current CPU usage as a percentage (0-100). Indicates how much CPU resources the Geth engine is consuming. |
16+
17+
**Usage**: Monitor for high CPU usage that might indicate performance bottlenecks or heavy computational load.
18+
19+
### Memory Metrics
20+
21+
| Metric Name | Type | Description |
22+
|-------------|------|-------------|
23+
| `geth_sys_memory_total_bytes` | Gauge | Total system memory available in bytes. This represents the total RAM capacity of the system. |
24+
| `geth_sys_memory_used_bytes` | Gauge | Currently used system memory in bytes. Shows how much RAM is being utilized. |
25+
26+
**Usage**: Calculate memory usage percentage and monitor for memory pressure or potential out-of-memory conditions.
27+
28+
### Swap Metrics
29+
30+
| Metric Name | Type | Description |
31+
|-------------|------|-------------|
32+
| `geth_sys_swap_total_bytes` | Gauge | Total swap space available in bytes. Represents the disk space allocated for virtual memory. |
33+
| `geth_sys_swap_used_bytes` | Gauge | Currently used swap space in bytes. Shows how much swap is being utilized. |
34+
35+
**Usage**: Monitor swap usage to detect memory pressure. High swap usage indicates the system is running low on physical memory.
36+
37+
## I/O Operation Metrics
38+
39+
### Read Operations
40+
41+
| Metric Name | Type | Description |
42+
|-------------|------|-------------|
43+
| `geth_read_entry_entries_total` | Counter | Total number of read entries processed since startup. Continuously increments with each read operation. |
44+
45+
**Usage**: Track read operation volume and calculate read rates using `rate()` function.
46+
47+
### Write Operations
48+
49+
| Metric Name | Type | Description |
50+
|-------------|------|-------------|
51+
| `geth_write_propose_event_events_total` | Counter | Total number of write propose events processed since startup. Tracks write operation frequency. |
52+
53+
**Usage**: Monitor write operation volume and calculate write rates to understand database activity.
54+
55+
### Cache Performance
56+
57+
| Metric Name | Type | Description |
58+
|-------------|------|-------------|
59+
| `geth_index_cache_miss_misses_total` | Counter | Total number of index cache misses since startup. High values indicate poor cache performance. |
60+
61+
**Usage**: Monitor cache efficiency. High miss rates may indicate need for cache tuning or insufficient cache size.
62+
63+
## I/O Size Distribution Metrics
64+
65+
### Read Size Histograms
66+
67+
| Metric Name | Type | Description |
68+
|-------------|------|-------------|
69+
| `geth_read_size_bytes_bucket` | Histogram | Distribution of read operation sizes across different byte ranges (buckets). |
70+
| `geth_read_size_bytes_sum` | Histogram | Total bytes read across all operations. |
71+
| `geth_read_size_bytes_count` | Histogram | Total number of read operations. |
72+
73+
**Buckets**: 0, 5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000, +Inf
74+
75+
### Write Size Histograms
76+
77+
| Metric Name | Type | Description |
78+
|-------------|------|-------------|
79+
| `geth_write_size_bytes_bucket` | Histogram | Distribution of write operation sizes across different byte ranges (buckets). |
80+
| `geth_write_size_bytes_sum` | Histogram | Total bytes written across all operations. |
81+
| `geth_write_size_bytes_count` | Histogram | Total number of write operations. |
82+
83+
**Buckets**: 0, 5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000, +Inf
84+
85+
**Usage**: Analyze I/O patterns, calculate percentiles, and understand the distribution of operation sizes to optimize performance.
86+
87+
## Common Metric Calculations
88+
89+
### Rate Calculations
90+
```promql
91+
# Read operations per second
92+
rate(geth_read_entry_entries_total[5m])
93+
94+
# Write operations per second
95+
rate(geth_write_propose_event_events_total[5m])
96+
97+
# Cache misses per second
98+
rate(geth_index_cache_miss_misses_total[5m])
99+
```
100+
101+
### Memory Usage Percentage
102+
```promql
103+
# Memory usage as percentage
104+
geth_sys_memory_used_bytes / geth_sys_memory_total_bytes * 100
105+
```
106+
107+
### Average I/O Sizes
108+
```promql
109+
# Average read size
110+
rate(geth_read_size_bytes_sum[5m]) / rate(geth_read_size_bytes_count[5m])
111+
112+
# Average write size
113+
rate(geth_write_size_bytes_sum[5m]) / rate(geth_write_size_bytes_count[5m])
114+
```
115+
116+
### I/O Throughput
117+
```promql
118+
# Read throughput (bytes per second)
119+
rate(geth_read_size_bytes_sum[5m])
120+
121+
# Write throughput (bytes per second)
122+
rate(geth_write_size_bytes_sum[5m])
123+
```
124+
125+
### Percentile Analysis
126+
```promql
127+
# 90th percentile read size
128+
histogram_quantile(0.90, rate(geth_read_size_bytes_bucket[5m]))
129+
130+
# 99th percentile write size
131+
histogram_quantile(0.99, rate(geth_write_size_bytes_bucket[5m]))
132+
```
133+
134+
## Alerting Recommendations
135+
136+
### Critical Alerts
137+
- **High CPU Usage**: `geth_sys_cpu_usage_percent > 90`
138+
- **High Memory Usage**: `(geth_sys_memory_used_bytes / geth_sys_memory_total_bytes) * 100 > 90`
139+
- **Swap Usage**: `geth_sys_swap_used_bytes > 0` (any swap usage may indicate memory pressure)
140+
141+
### Warning Alerts
142+
- **Moderate CPU Usage**: `geth_sys_cpu_usage_percent > 70`
143+
- **Moderate Memory Usage**: `(geth_sys_memory_used_bytes / geth_sys_memory_total_bytes) * 100 > 70`
144+
- **High Cache Miss Rate**: `rate(geth_index_cache_miss_misses_total[5m]) > threshold`
145+
146+
## Dashboard Panels
147+
148+
The Grafana dashboard includes the following visualization panels:
149+
150+
1. **System Overview**: CPU usage, memory usage, memory details, swap usage
151+
2. **I/O Operations**: Read/write operation rates, cache miss rate
152+
3. **I/O Size Analysis**: Read/write size distributions with percentiles
153+
4. **Summary Stats**: Total operation counters, system resource timeline
154+
155+
## Troubleshooting
156+
157+
### No Data in Panels
158+
- Verify Prometheus is scraping the Geth metrics endpoint
159+
- Check that metric names match exactly (case-sensitive)
160+
- Ensure time range covers period when metrics were generated
161+
- Confirm Prometheus datasource is configured correctly
162+
163+
### High Resource Usage
164+
- Check CPU and memory trends over time
165+
- Analyze I/O patterns for unusual spikes
166+
- Review cache performance metrics
167+
- Consider scaling resources if sustained high usage
168+
169+
### Performance Issues
170+
- Monitor I/O operation rates and sizes
171+
- Check for large operations that might cause bottlenecks
172+
- Analyze cache miss patterns
173+
- Review histogram data for operation size distribution
174+
175+
## Metric Labels
176+
177+
All metrics include the following labels:
178+
- `job`: "geth-engine"
179+
- `otel_scope_name`: "geth-engine"
180+
- `otel_scope_schema_url`: ""
181+
- `otel_scope_version`: ""
182+
183+
These labels can be used for filtering and aggregation in Prometheus queries.

‎README.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,16 @@ kubectl port-forward -n geth-app service/seq-service 5341:80
6969
```
7070
The Seq UI will be available at http://localhost:5341
7171

72+
The configuration also includes an OpenTelemetry collector for metrics ingestion, a Prometheus aggregator for data storage and querying, and a Grafana instance pre-configured with a dashboard dedicated to GethDB metrics visualization.
73+
74+
## Metrics and Monitoring
75+
76+
GethDB exposes comprehensive Prometheus-format metrics that provide detailed insights into system performance, I/O operations, resource utilization, and cache efficiency. These metrics include system-level measurements such as CPU and memory usage, detailed I/O operation tracking with histogram distributions for read/write sizes, and performance indicators like cache miss rates. The metrics are designed to give operators full visibility into GethDB's operational health and performance characteristics.
77+
78+
For complete metric definitions, usage examples, and alerting recommendations, see [METRICS.md](./METRICS.md).
79+
80+
A pre-configured Grafana dashboard is available at [`./k8s/manifests/dashboards/geth-engine.json`](./k8s/manifests/dashboards/geth-engine.json) that visualizes all key metrics with organized panels for system overview, I/O operations analysis, and performance monitoring.
81+
7282
## Examples
7383
To quickly see how to interact with GethDB, use the `geth-quickstart` project, which is configured to work with any GethDB node running on `localhost:2113`. Currently, only a Rust client is available, but more clients will be added later.
7484

‎geth-quickstart/src/main.rs‎

Lines changed: 28 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
use std::time::Duration;
2+
13
use geth_client::{
24
Client, ContentType, Direction, EndPoint, ExpectedRevision, GrpcClient, Propose, Revision,
35
};
@@ -17,29 +19,36 @@ async fn main() -> eyre::Result<()> {
1719
})
1820
.await?;
1921

20-
let mut proposes = Vec::new();
22+
let count = 1_000;
2123

22-
for i in 0..10 {
23-
proposes.push(Propose {
24-
id: Uuid::new_v4(),
25-
content_type: ContentType::Json,
26-
class: "foobar".to_string(),
27-
data: serde_json::to_vec(&Foobar { value: 10 * i })?.into(),
28-
});
29-
}
24+
for i in 0..count {
25+
let mut proposes = Vec::new();
26+
27+
for i in 0..10 {
28+
proposes.push(Propose {
29+
id: Uuid::new_v4(),
30+
content_type: ContentType::Json,
31+
class: "foobar".to_string(),
32+
data: serde_json::to_vec(&Foobar { value: 10 * i })?.into(),
33+
});
34+
}
35+
36+
client
37+
.append_stream("baz", ExpectedRevision::Any, proposes)
38+
.await?
39+
.success()?;
3040

31-
client
32-
.append_stream("baz", ExpectedRevision::Any, proposes)
33-
.await?
34-
.success()?;
41+
let mut stream = client
42+
.read_stream("baz", Direction::Forward, Revision::Start, u64::MAX)
43+
.await?
44+
.success()?;
3545

36-
let mut stream = client
37-
.read_stream("baz", Direction::Forward, Revision::Start, u64::MAX)
38-
.await?
39-
.success()?;
46+
while let Some(event) = stream.next().await? {
47+
println!("{event:?}");
48+
}
4049

41-
while let Some(event) = stream.next().await? {
42-
println!("{event:?}");
50+
println!("loop {i}/{count} done");
51+
tokio::time::sleep(Duration::from_secs(1)).await;
4352
}
4453

4554
Ok(())

‎k8s/manifests/06-grafana-deployment.yaml‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
apiVersion: v1
2+
kind: ConfigMap
3+
metadata:
4+
name: grafana-provisioning-datasources # <-- This one!
5+
namespace: geth-app
6+
data:
7+
datasource.yaml: |
8+
apiVersion: 1
9+
datasources:
10+
- name: Prometheus
11+
type: prometheus
12+
url: http://prometheus-service:9090
13+
access: proxy
14+
isDefault: true
15+
editable: true
16+
17+
---
18+
19+
apiVersion: v1
20+
kind: ConfigMap
21+
metadata:
22+
name: grafana-provisioning-dashboards
23+
namespace: geth-app
24+
data:
25+
dashboard.yaml: |
26+
apiVersion: 1
27+
providers:
28+
- name: 'default'
29+
orgId: 1
30+
folder: ''
31+
type: file
32+
disableDeletion: false
33+
updateIntervalSeconds: 10
34+
allowUiUpdates: true
35+
options:
36+
path: /var/lib/grafana/dashboards
37+
38+
---
39+
40+
apiVersion: apps/v1
41+
kind: Deployment
42+
metadata:
43+
name: grafana
44+
namespace: geth-app
45+
spec:
46+
replicas: 1
47+
selector:
48+
matchLabels:
49+
app: grafana
50+
template:
51+
metadata:
52+
labels:
53+
app: grafana
54+
spec:
55+
containers:
56+
- name: grafana
57+
image: grafana/grafana:latest
58+
ports:
59+
- containerPort: 3000
60+
env:
61+
- name: GF_SECURITY_ADMIN_PASSWORD
62+
value: "changeit"
63+
volumeMounts:
64+
- name: grafana-storage
65+
mountPath: /var/lib/grafana
66+
- name: geth-engine-dashboard
67+
mountPath: /var/lib/grafana/dashboards
68+
- name: grafana-provisioning-dashboards
69+
mountPath: /etc/grafana/provisioning/dashboards
70+
- name: grafana-provisioning-datasources
71+
mountPath: /etc/grafana/provisioning/datasources
72+
volumes:
73+
- name: grafana-storage
74+
emptyDir: {}
75+
- name: geth-engine-dashboard
76+
configMap:
77+
name: geth-engine-dashboard
78+
- name: grafana-provisioning-dashboards
79+
configMap:
80+
name: grafana-provisioning-dashboards
81+
- name: grafana-provisioning-datasources
82+
configMap:
83+
name: grafana-provisioning-datasources
84+
---
85+
apiVersion: v1
86+
kind: Service
87+
metadata:
88+
name: grafana-service
89+
namespace: geth-app
90+
spec:
91+
selector:
92+
app: grafana
93+
ports:
94+
- port: 3000
95+
targetPort: 3000
96+
type: NodePort

0 commit comments

Comments
 (0)