Skip to content

Commit c9f816a

Browse files
wu-shengwu-sheng
authored
SWIP-15: implement BanyanDB self-observability (cluster / container / group model) (#13903)
### Implement SWIP-15 BanyanDB self-observability around the cluster / container / group model Rebuilds the `otel-rules/banyandb/` self-observability feature to match how BanyanDB is actually built and operated (**requires BanyanDB 0.11+**): - **Entity model**: `Service` = one BanyanDB **cluster** (`service(['cluster'])`); `ServiceInstance` = one **container** keyed on `pod_name` + `container_name` (with `node_role` / `node_type` / `container_name` / `pod_name` as instance attributes); `Endpoint` = one storage **group**. - **Rules**: new `banyandb-endpoint.yaml`; `banyandb-service.yaml` (7 rules) and `banyandb-instance.yaml` (43 rules) redesigned. Source expressions mirror the upstream BanyanDB FODC-proxy Grafana boards (`grafana-fodc-nodes.json` / `grafana-fodc-workload.json`) so the SkyWalking dashboards stay in lockstep. The stale single-node `host_name` model and the removed `etcd_operation_rate` / `up`-derived `active_instance` / `queue_sub_total_msg_sent_err` metrics are dropped. - **Docs**: `docs/en/banyandb/dashboards-banyandb.md` rewritten to the cluster/container/group model. - **E2E**: rewritten to a no-FODC, file-discovery BanyanDB 0.11 cluster (1 liaison + 1 hot data node, no etcd). The OTel collector scrapes each node's `:2121` directly and injects the FODC-equivalent identity labels as static per-scrape-job labels, so the full rule set is exercised without a FODC proxy.
1 parent a144e53 commit c9f816a

25 files changed

Lines changed: 2316 additions & 923 deletions

File tree

‎docs/en/banyandb/dashboards-banyandb.md‎

Lines changed: 170 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -1,49 +1,177 @@
1-
# BanyanDB self observability dashboard
1+
# BanyanDB self-observability dashboard
22

3-
[BanyanDB](https://skywalking.apache.org/docs/skywalking-banyandb/next/readme/), as an observability database, aims to ingest, analyze and store Metrics, Tracing, and Logging data. It's designed to handle observability data generated by **Apache SkyWalking**，it also provides a dashboard to visualize the self-observability metrics.
3+
[Apache SkyWalking BanyanDB](https://skywalking.apache.org/docs/skywalking-banyandb/next/readme/) is the
4+
native storage for SkyWalking. A production deployment is one **cluster** made of many **nodes**, each
5+
running one or more **containers** with a role (`liaison` front door, `data` backend, and the `lifecycle`
6+
tier-migration sidecar), and data is organized into **groups**. SkyWalking models that reality directly
7+
and renders it on the `Layer: BANYANDB` dashboards in the Horizon UI:
8+
9+
| SkyWalking entity | BanyanDB concept | Identity |
10+
| ----------------- | ---------------- | -------- |
11+
| `Service` | one BanyanDB **cluster** | the `cluster` label |
12+
| `ServiceInstance` | one **container** on a node | `pod_name` + `container_name` (joined by `@`) |
13+
|   ↳ attributes | role / tier | `container_name` (`liaison`/`data`/`lifecycle`), `node_type` (`hot`/`warm`/`cold`), `node_role`, `pod_name` |
14+
| `Endpoint` | one **group** (storage partition) | the `group` label (e.g. `sw_metricsMinute`) |
15+
16+
> **Requires BanyanDB 0.11+.** This feature reads the FODC-proxy cluster-observability metric families
17+
> and the queue / lifecycle metric families that BanyanDB introduced after 0.10. Run a 0.11+ cluster
18+
> with the FODC proxy and the Prometheus metrics provider enabled.
419
520
## Data flow
6-
1. [BanyanDB](https://skywalking.apache.org/docs/skywalking-banyandb/next/readme/) collects metrics data internally and exposes a Prometheus http endpoint to retrieve the metrics.
7-
2. OpenTelemetry Collector fetches metrics from BanyanDB and pushes metrics to SkyWalking OAP Server via OpenTelemetry gRPC exporter.
8-
3. The SkyWalking OAP Server parses the expression with [MAL](../concepts-and-designs/mal.md) to filter/calculate/aggregate and store the results.
21+
22+
1. Each BanyanDB container exposes its metrics; in a cluster the
23+
[FODC proxy](https://skywalking.apache.org/docs/skywalking-banyandb/next/operation/fodc/overview/)
24+
aggregates every container's Prometheus metrics onto a single `/metrics` endpoint (default `:17913`)
25+
and stamps each sample with per-container identity labels (`pod_name`, `container_name`, `node_role`,
26+
and `node_type` on data containers).
27+
2. An OpenTelemetry Collector scrapes the FODC proxy `/metrics` as the single Prometheus target, adds a
28+
static `cluster: <name>` label (the only label SkyWalking must inject), and pushes via the
29+
OpenTelemetry gRPC exporter to the SkyWalking OAP Server.
30+
3. The OAP Server parses the [MAL](../concepts-and-designs/mal.md) rules under `otel-rules/banyandb/` to
31+
filter / calculate / aggregate and store the cluster, instance and group metrics.
932

1033
## Set up
11-
1. Start [BanyanDB](https://skywalking.apache.org/docs/skywalking-banyandb/next/readme/),supporting both [Standalone Mode](https://skywalking.apache.org/docs/skywalking-banyandb/next/installation/standalone/) and [Cluster Mode](https://skywalking.apache.org/docs/skywalking-banyandb/next/installation/cluster/).
12-
2. Set up [OpenTelemetry Collector ](https://opentelemetry.io/docs/collector/getting-started/#docker). For details on Prometheus Receiver in OpenTelemetry Collector, refer to [here](../../../test/e2e-v2/cases/banyandb/otel-collector-config.yaml).
13-
3. Config SkyWalking [OpenTelemetry receiver](https://skywalking.apache.org/docs/main/next/en/setup/backend/opentelemetry-receiver/).
14-
15-
## BanyanDB monitoring
16-
Self observability monitoring provides monitoring of the status and resources of the [BanyanDB](https://skywalking.apache.org/docs/skywalking-banyandb/next/readme/) server itself. `banyandb-server` is a `Service` in BanyanDB, and land on the `Layer: BANYANDB`.
17-
18-
### Self observability metrics
19-
20-
| Unit | Metric Name | Description | Data Source |
21-
|------|---------------------------------------------------|-------------|-------------|
22-
| o/s | meter_banyandb_write_rate | Write Rate (Operations per Second) | BanyanDB |
23-
| GiB | meter_banyandb_total_memory | Total Memory | BanyanDB |
24-
| GiB | meter_banyandb_disk_usage | Disk Usage | BanyanDB |
25-
| r/s | meter_banyandb_query_rate | Query Rate (Requests per Second) | BanyanDB |
26-
| Count | meter_banyandb_total_cpu | Total CPU Cores | BanyanDB |
27-
| c/m | meter_banyandb_write_and_query_errors_rate | Write and Query Errors Rate（Counts per Minute） | BanyanDB |
28-
| c/s | meter_banyandb_etcd_operation_rate | Etcd Operation Rate（Counts per Second） | BanyanDB |
29-
| Count | meter_banyandb_active_instance | Active Instances | BanyanDB |
30-
| % | meter_banyandb_cpu_usage | CPU Usage Percentage | BanyanDB |
31-
| % | meter_banyandb_rss_memory_usage | RSS Memory Usage Percentage | BanyanDB |
32-
| % | meter_banyandb_disk_usage_all | Disk Usage Percentage | BanyanDB |
33-
| KiB/s | meter_banyandb_network_usage_recv | Network Receive Rate | BanyanDB |
34-
| KiB/s | meter_banyandb_network_usage_sent | Network Send Rate | BanyanDB |
35-
| o/s | meter_banyandb_storage_write_rate | Storage Write Rate (Operations per Second) | BanyanDB |
36-
| s | meter_banyandb_query_latency | Query Latency (s) | BanyanDB |
37-
| Count | meter_banyandb_total_data | Total Data Elements | BanyanDB |
38-
| r/m | meter_banyandb_merge_file_data | Merge File Data Rate(Revolutions per Minute) | BanyanDB |
39-
| s | meter_banyandb_merge_file_latency | Merge File Latency(s) | BanyanDB |
40-
| Count | meter_banyandb_merge_file_partitions | Merge File Partitions | BanyanDB |
41-
| o/s | meter_banyandb_series_write_rate | Series Write Rate (Operations per Second) | BanyanDB |
42-
| o/s | meter_banyandb_series_term_search_rate | Series Term Search Rate (Operations per Second) | BanyanDB |
43-
| Count | meter_banyandb_total_series | Total Series Count | BanyanDB |
44-
| ops | meter_banyandb_stream_write_rate | Stream Write Rate (Operations per Second) | BanyanDB |
45-
| ops | meter_banyandb_term_search_rate | Term Search Rate (Operations per Second) | BanyanDB |
46-
| Count | meter_banyandb_total_document | Total Document Count | BanyanDB |
34+
35+
1. Run a BanyanDB **0.11+** cluster (liaison + data nodes; data nodes may be tiered hot/warm/cold) with
36+
the **FODC proxy** enabled and the Prometheus metrics provider on (default). Standalone mode is the
37+
degenerate case — one cluster, one node, one `container_name=standalone`.
38+
2. Run an **OpenTelemetry Collector** whose `prometheus` receiver scrapes the FODC proxy `/metrics`
39+
(`:17913`) as the single target and adds a static `cluster: <name>` label, exporting OTLP to OAP.
40+
The FODC proxy already stamps the per-node identity labels (`pod_name` / `container_name` /
41+
`node_role` / `node_type`), so `cluster` is the only label the collector must inject.
42+
43+
> **The scrape `job_name` MUST be `banyandb-monitoring`.** Every rule file filters on
44+
> `{ tags -> tags.job_name == 'banyandb-monitoring' }` (the OTel receiver maps the Prometheus `job`
45+
> to the `job_name` tag), so a differently-named job produces no metrics.
46+
47+
```yaml
48+
receivers:
49+
prometheus:
50+
config:
51+
scrape_configs:
52+
- job_name: "banyandb-monitoring" # REQUIRED — the rules filter on it
53+
scrape_interval: 15s
54+
static_configs:
55+
- targets: ["<fodc-proxy-host>:17913"] # the FODC proxy aggregates every node's metrics
56+
labels:
57+
cluster: <your-cluster-name> # the only label SkyWalking must inject
58+
exporters:
59+
otlp:
60+
endpoint: <oap-host>:11800
61+
tls:
62+
insecure: true
63+
service:
64+
pipelines:
65+
metrics:
66+
receivers: [prometheus]
67+
processors: [batch]
68+
exporters: [otlp]
69+
```
70+
71+
Scrape the **FODC proxy**, not the individual nodes. The proxy resolves each node's identity from
72+
the cluster and stamps `pod_name` / `container_name` / `node_role` and the data-node tier
73+
(`node_type`) onto every sample — context the raw per-node `:2121` endpoints do not carry. Direct
74+
per-node scraping is not recommended: it would have to hand-inject all of those identity labels for
75+
every node. (The e2e does exactly that only because it runs no FODC proxy — see the
76+
[test collector config](../../../test/e2e-v2/cases/banyandb/otel-collector-config.yaml) — and is not
77+
a production pattern.)
78+
3. Enable SkyWalking's
79+
[OpenTelemetry receiver](https://skywalking.apache.org/docs/main/next/en/setup/backend/opentelemetry-receiver/).
80+
The `banyandb/*` rules are enabled by default in `enabledOtelMetricsRules`.
81+
4. Open the **Horizon UI** → `BanyanDB` layer.
82+
83+
## Metrics
84+
85+
The metric source expressions mirror the upstream BanyanDB Grafana boards, so the SkyWalking dashboards
86+
stay in lockstep with the BanyanDB catalog. The rule files are
87+
`otel-rules/banyandb/banyandb-service.yaml`, `banyandb-instance.yaml` and `banyandb-endpoint.yaml`.
88+
89+
### Service scope — cluster summary (`meter_banyandb_*`)
90+
91+
| Unit | Metric | Description |
92+
| ---- | ------ | ----------- |
93+
| w/s | `meter_banyandb_cluster_write_rate` | Cluster write rate across measure/stream/trace |
94+
| r/s | `meter_banyandb_cluster_query_rate` | Cluster query rate |
95+
| c/m | `meter_banyandb_cluster_error_rate` | Cluster error rate (counts/min) |
96+
| Count | `meter_banyandb_reporting_instances` | Live container count by role |
97+
| Count | `meter_banyandb_total_cpu_cores` | Cluster CPU capacity |
98+
| Bytes | `meter_banyandb_total_memory_used` | Cluster memory used |
99+
| Bytes | `meter_banyandb_total_disk_used` | Cluster disk used |
100+
101+
### Instance scope — per container (`meter_banyandb_instance_*`)
102+
103+
**All roles** (every container emits these):
104+
105+
| Unit | Metric | Description |
106+
| ---- | ------ | ----------- |
107+
| s | `node_uptime` | Node uptime |
108+
| Cores | `cpu_usage` | CPU usage |
109+
| Bytes | `rss_memory` | Resident memory |
110+
| percentunit | `system_memory_percent` | System memory used fraction |
111+
| percentunit | `disk_usage_percent` | Disk used fraction (Σused/Σtotal) |
112+
| Bytes | `disk_used_by_path` / `disk_total_by_path` | Disk used / total by mount path |
113+
| percentunit | `disk_used_percent_by_path` | Disk used fraction by mount path |
114+
| Bytes/s | `network_recv` / `network_sent` | Network throughput by interface |
115+
| Count | `goroutines` | Go goroutines |
116+
| s | `gc_pause_avg` | Average GC pause |
117+
| Bytes | `heap_inuse` / `heap_next_gc` | Go heap in-use / next-GC threshold |
118+
| Bytes/s | `alloc_rate` | Go allocation rate |
119+
120+
**Liaison** (front door; the dashboard gates these on `container_name == 'liaison'`):
121+
122+
| Unit | Metric | Description |
123+
| ---- | ------ | ----------- |
124+
| r/s | `query_rate_by_service` | Query rate by data-model service |
125+
| c/m | `grpc_error_rate` | gRPC error rate |
126+
| r/s | `non_query_op_rate` | Registry / non-query operation rate |
127+
| w/s | `write_rate` | Write rate seen at the front door |
128+
| ops | `publish_throughput` | Tier-2 publish throughput by operation |
129+
| Bytes/s | `publish_bytes` | Publish bytes |
130+
| s | `publish_latency_p99` | Publish send latency p99 |
131+
| Count | `wqueue_pending` / `wqueue_file_parts` / `wqueue_mem_part` | Write-queue depth |
132+
133+
**Data** (backend; the dashboard gates these on `container_name == 'data'`):
134+
135+
| Unit | Metric | Description |
136+
| ---- | ------ | ----------- |
137+
| Count | `total_data` | Total stored data elements |
138+
| o/s | `merge_file_rate` | Merge-loop rate |
139+
| Count | `merge_file_partitions` | Avg parts merged per loop |
140+
| s | `merge_file_latency` | Avg file-merge latency |
141+
| o/s | `series_write_rate` / `series_term_search_rate` | Inverted-index write / term-search rate |
142+
| Count | `total_series` | Inverted-index documents |
143+
| o/s | `stream_tst_write_rate` / `stream_tst_term_search_rate` | Stream tst index write / term-search rate |
144+
| Count | `stream_tst_total_docs` | Stream tst index documents |
145+
| ops | `queue_sub_throughput` | Subscribe-queue throughput by operation |
146+
| s | `queue_sub_latency_p99` | Subscribe-queue latency p99 |
147+
| percent | `retention_measure_disk_usage_percent` / `retention_stream_disk_usage_percent` / `retention_trace_disk_usage_percent` | Retention disk-usage % per scope |
148+
149+
**Lifecycle** (the tier-migration sidecar on hot/warm data pods; `container_name == 'lifecycle'`):
150+
151+
| Unit | Metric | Description |
152+
| ---- | ------ | ----------- |
153+
| Count | `lifecycle_cycles` | Cumulative migration cycles |
154+
| s | `lifecycle_last_run` | Seconds since the last migration cycle started |
155+
| Status | `lifecycle_last_run_success` | Last cycle status (1 = OK, 0 = failed) |
156+
157+
### Endpoint scope — per group (`meter_banyandb_endpoint_*`)
158+
159+
| Unit | Metric | Description |
160+
| ---- | ------ | ----------- |
161+
| w/s | `write_rate` | Write rate for the group |
162+
| s | `query_latency` | Mean query latency for the group |
163+
| Count | `total_data` | Total stored data elements for the group |
164+
| o/s | `merge_file_rate` | Merge-loop rate for the group |
165+
| s | `merge_file_latency` | Avg file-merge latency for the group |
166+
| Count | `merge_file_partitions` | Avg parts merged per loop for the group |
167+
| o/s | `series_write_rate` | Inverted-index write rate for the group |
168+
| Count | `total_series` | Inverted-index documents for the group |
169+
| ops | `queue_throughput` | Subscribe-queue throughput by operation for the group |
170+
| s | `queue_latency_p99` | Publish-queue latency p99 for the group |
171+
| Bytes/s | `publish_bytes` | Publish bytes for the group |
47172

48173
## Customizations
49-
You can customize your own metrics/expression/dashboard panel.The metrics definition and expression rules are found in `/config/otel-rules/banyandb`.The [BanyanDB](https://skywalking.apache.org/docs/skywalking-banyandb/next/readme/) dashboard panel configurations ship from the SkyWalking Horizon UI bundle (apache/skywalking-horizon-ui); the OAP backend no longer hosts UI dashboard JSONs.
174+
175+
You can customize your own metrics / expressions. The metric definitions and expression rules are in
176+
`/config/otel-rules/banyandb`. The dashboard panel configurations ship from the SkyWalking Horizon UI
177+
bundle (apache/skywalking-horizon-ui); the OAP backend does not host UI dashboard JSONs.

‎docs/en/changes/changes.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -242,6 +242,14 @@
242242
admin-host only" entry above for the public REST retirement.
243243

244244
#### OAP Server
245+
* SWIP-15: rebuild BanyanDB self-observability around the cluster / container / group model
246+
(requires BanyanDB 0.11+). `otel-rules/banyandb/` now models a BanyanDB cluster as one `Service`
247+
(`service(['cluster'])`), each container as a `ServiceInstance` keyed on `pod_name` + `container_name`
248+
(with `node_role` / `node_type` / `container_name` / `pod_name` as instance attributes), and each
249+
storage group as an `Endpoint`. New `banyandb-endpoint.yaml`; `banyandb-service.yaml` and
250+
`banyandb-instance.yaml` redesigned to mirror the upstream FODC-proxy Grafana boards. The stale
251+
single-node `host_name` model and the removed `etcd_operation_rate` / `up`-derived `active_instance`
252+
metrics are gone.
245253
* Runtime MAL/LAL hot-update rules can declare `layerDefinitions:` to introduce new
246254
layers. Ordinals are operator-pinned in the `100_000+` tier; the layer is
247255
refcount-tracked and unregistered when the last declaring rule is removed. See

‎docs/en/setup/backend/opentelemetry-receiver.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,5 +83,6 @@ the natural dot-to-underscore conversion (e.g., `host.name` → `host_name` in M
8383
| Metrics of Flink | otel-rules/flink/flink-jobManager.yaml | flink jobManager -> OpenTelemetry Collector -- OTLP exporter --> SkyWalking OAP Server |
8484
| Metrics of Flink | otel-rules/flink/flink-taskManager.yaml | flink taskManager -> OpenTelemetry Collector -- OTLP exporter --> SkyWalking OAP Server |
8585
| Metrics of Flink | otel-rules/flink/flink-job.yaml | flink jobManager & flink taskManager-> OpenTelemetry Collector -- OTLP exporter --> SkyWalking OAP Server |
86+
| Metrics of BanyanDB | otel-rules/banyandb/banyandb-service.yaml | BanyanDB(embedded prometheus endpoint) -> OpenTelemetry Collector – OTLP exporter –> SkyWalking OAP Server |
8687
| Metrics of BanyanDB | otel-rules/banyandb/banyandb-instance.yaml | BanyanDB(embedded prometheus endpoint) -> OpenTelemetry Collector – OTLP exporter –> SkyWalking OAP Server |
87-
| Metrics of BanyanDB | otel-rules/banyandb/banyandb-service.yaml | BanyanDB(embedded prometheus endpoint) -> OpenTelemetry Collector – OTLP exporter –> SkyWalking OAP Server |
88+
| Metrics of BanyanDB | otel-rules/banyandb/banyandb-endpoint.yaml | BanyanDB(embedded prometheus endpoint) -> OpenTelemetry Collector – OTLP exporter –> SkyWalking OAP Server |

0 commit comments

Comments
 (0)