Skip to content

Commit 75d9433

Browse files
wu-shengwu-sheng
authored
SWIP-15: add BanyanDB queue batch/message metrics (apache/skywalking-banyandb#1169) (#13904)
1 parent c9f816a commit 75d9433

8 files changed

Lines changed: 170 additions & 1 deletion

File tree

‎docs/en/banyandb/dashboards-banyandb.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -126,8 +126,10 @@ stay in lockstep with the BanyanDB catalog. The rule files are
126126
| r/s | `non_query_op_rate` | Registry / non-query operation rate |
127127
| w/s | `write_rate` | Write rate seen at the front door |
128128
| ops | `publish_throughput` | Tier-2 publish throughput by operation |
129+
| ops | `publish_batch_throughput` | Tier-2 publish batch throughput by operation |
129130
| Bytes/s | `publish_bytes` | Publish bytes |
130131
| s | `publish_latency_p99` | Publish send latency p99 |
132+
| s | `publish_batch_latency_p99` | Publish batch send latency p99 |
131133
| Count | `wqueue_pending` / `wqueue_file_parts` / `wqueue_mem_part` | Write-queue depth |
132134

133135
**Data** (backend; the dashboard gates these on `container_name == 'data'`):
@@ -143,6 +145,7 @@ stay in lockstep with the BanyanDB catalog. The rule files are
143145
| o/s | `stream_tst_write_rate` / `stream_tst_term_search_rate` | Stream tst index write / term-search rate |
144146
| Count | `stream_tst_total_docs` | Stream tst index documents |
145147
| ops | `queue_sub_throughput` | Subscribe-queue throughput by operation |
148+
| ops | `queue_sub_message_throughput` | Subscribe-queue per-message throughput by operation |
146149
| s | `queue_sub_latency_p99` | Subscribe-queue latency p99 |
147150
| percent | `retention_measure_disk_usage_percent` / `retention_stream_disk_usage_percent` / `retention_trace_disk_usage_percent` | Retention disk-usage % per scope |
148151

@@ -167,6 +170,8 @@ stay in lockstep with the BanyanDB catalog. The rule files are
167170
| o/s | `series_write_rate` | Inverted-index write rate for the group |
168171
| Count | `total_series` | Inverted-index documents for the group |
169172
| ops | `queue_throughput` | Subscribe-queue throughput by operation for the group |
173+
| ops | `queue_batch_throughput` | Subscribe-queue batch throughput by operation for the group |
174+
| ops | `queue_message_throughput` | Subscribe-queue per-message throughput by operation for the group |
170175
| s | `queue_latency_p99` | Publish-queue latency p99 for the group |
171176
| Bytes/s | `publish_bytes` | Publish bytes for the group |
172177

‎docs/en/changes/changes.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -250,6 +250,14 @@
250250
`banyandb-instance.yaml` redesigned to mirror the upstream FODC-proxy Grafana boards. The stale
251251
single-node `host_name` model and the removed `etcd_operation_rate` / `up`-derived `active_instance`
252252
metrics are gone.
253+
* SWIP-15: add BanyanDB queue batch / message granularity metrics (requires BanyanDB 0.11.x with the
254+
`queue_pub_total_batch_*` / `queue_sub_total_message_*` families, apache/skywalking-banyandb#1169). Instance scope gains
255+
`publish_batch_throughput` / `publish_batch_latency_p99` (liaison publish-side batch granularity) and
256+
`queue_sub_message_throughput` (data per-record subscribe rate); endpoint scope gains
257+
`queue_batch_throughput` / `queue_message_throughput` per storage group (subscribe-side counters that
258+
carry a real `group`). The subscribe-side batch counters and the publish-side batch-latency histogram
259+
are not surfaced where they do not populate per their modeled scope (the data node ingests via the
260+
per-message path, and `queue_pub_total_batch_latency` is emitted with an empty `group`).
253261
* Runtime MAL/LAL hot-update rules can declare `layerDefinitions:` to introduce new
254262
layers. Ordinals are operator-pinned in the `100_000+` tier; the layer is
255263
refcount-tracked and unregistered when the last declaring rule is removed. See

‎oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-otel-rules/banyandb/banyandb-endpoint.data.yaml‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,18 @@ input:
167167
remote_role: 'data'
168168
remote_tier: 'hot'
169169
value: 12.0
170+
banyandb_queue_sub_total_batch_finished:
171+
- labels:
172+
cluster: 'test-cluster'
173+
pod_name: 'test-pod'
174+
container_name: 'data'
175+
node_role: 'ROLE_DATA'
176+
node_type: 'hot'
177+
operation: 'batch-write'
178+
group: 'test-group'
179+
remote_node: 'n1'
180+
remote_role: 'data'
181+
value: 100.0
170182
banyandb_queue_sub_total_finished:
171183
- labels:
172184
cluster: 'test-cluster'
@@ -179,6 +191,18 @@ input:
179191
remote_node: 'n1'
180192
remote_role: 'data'
181193
value: 100.0
194+
banyandb_queue_sub_total_message_finished:
195+
- labels:
196+
cluster: 'test-cluster'
197+
pod_name: 'test-pod'
198+
container_name: 'data'
199+
node_role: 'ROLE_DATA'
200+
node_type: 'hot'
201+
operation: 'batch-write'
202+
group: 'test-group'
203+
remote_node: 'n1'
204+
remote_role: 'data'
205+
value: 100.0
182206
banyandb_stream_storage_inverted_index_total_doc_count:
183207
- labels:
184208
cluster: 'test-cluster'
@@ -374,6 +398,18 @@ expected:
374398
service: test-cluster
375399
endpoint: test-group
376400
layer: BANYANDB
401+
meter_banyandb_endpoint_queue_batch_throughput:
402+
entities:
403+
- scope: ENDPOINT
404+
service: test-cluster
405+
endpoint: test-group
406+
layer: BANYANDB
407+
meter_banyandb_endpoint_queue_message_throughput:
408+
entities:
409+
- scope: ENDPOINT
410+
service: test-cluster
411+
endpoint: test-group
412+
layer: BANYANDB
377413
meter_banyandb_endpoint_publish_bytes:
378414
entities:
379415
- scope: ENDPOINT

‎oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-otel-rules/banyandb/banyandb-instance.data.yaml‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -220,6 +220,59 @@ input:
220220
remote_role: 'data'
221221
remote_tier: 'hot'
222222
value: 100.0
223+
banyandb_queue_pub_total_batch_finished:
224+
- labels:
225+
cluster: 'test-cluster'
226+
pod_name: 'test-pod'
227+
container_name: 'data'
228+
node_role: 'ROLE_DATA'
229+
node_type: 'hot'
230+
operation: 'batch-write'
231+
group: 'test-group'
232+
remote_node: 'n1'
233+
remote_role: 'data'
234+
remote_tier: 'hot'
235+
value: 100.0
236+
banyandb_queue_pub_total_batch_latency:
237+
- labels:
238+
cluster: 'test-cluster'
239+
pod_name: 'test-pod'
240+
container_name: 'data'
241+
node_role: 'ROLE_DATA'
242+
node_type: 'hot'
243+
operation: 'batch-write'
244+
group: 'test-group'
245+
le: '50'
246+
remote_node: 'n1'
247+
remote_role: 'data'
248+
remote_tier: 'hot'
249+
value: 5.0
250+
- labels:
251+
cluster: 'test-cluster'
252+
pod_name: 'test-pod'
253+
container_name: 'data'
254+
node_role: 'ROLE_DATA'
255+
node_type: 'hot'
256+
operation: 'batch-write'
257+
group: 'test-group'
258+
le: '100'
259+
remote_node: 'n1'
260+
remote_role: 'data'
261+
remote_tier: 'hot'
262+
value: 10.0
263+
- labels:
264+
cluster: 'test-cluster'
265+
pod_name: 'test-pod'
266+
container_name: 'data'
267+
node_role: 'ROLE_DATA'
268+
node_type: 'hot'
269+
operation: 'batch-write'
270+
group: 'test-group'
271+
le: '500'
272+
remote_node: 'n1'
273+
remote_role: 'data'
274+
remote_tier: 'hot'
275+
value: 12.0
223276
banyandb_queue_pub_total_finished:
224277
- labels:
225278
cluster: 'test-cluster'
@@ -322,6 +375,18 @@ input:
322375
remote_node: 'n1'
323376
remote_role: 'data'
324377
value: 12.0
378+
banyandb_queue_sub_total_message_finished:
379+
- labels:
380+
cluster: 'test-cluster'
381+
pod_name: 'test-pod'
382+
container_name: 'data'
383+
node_role: 'ROLE_DATA'
384+
node_type: 'hot'
385+
operation: 'batch-write'
386+
group: 'test-group'
387+
remote_node: 'n1'
388+
remote_role: 'data'
389+
value: 100.0
325390
banyandb_storage_retention_measure_disk_usage_percent:
326391
- labels:
327392
cluster: 'test-cluster'
@@ -834,6 +899,18 @@ expected:
834899
service: test-cluster
835900
instance: test-pod@data
836901
layer: BANYANDB
902+
meter_banyandb_instance_publish_batch_throughput:
903+
entities:
904+
- scope: SERVICE_INSTANCE
905+
service: test-cluster
906+
instance: test-pod@data
907+
layer: BANYANDB
908+
meter_banyandb_instance_publish_batch_latency_p99:
909+
entities:
910+
- scope: SERVICE_INSTANCE
911+
service: test-cluster
912+
instance: test-pod@data
913+
layer: BANYANDB
837914
meter_banyandb_instance_wqueue_pending:
838915
entities:
839916
- scope: SERVICE_INSTANCE
@@ -924,6 +1001,12 @@ expected:
9241001
service: test-cluster
9251002
instance: test-pod@data
9261003
layer: BANYANDB
1004+
meter_banyandb_instance_queue_sub_message_throughput:
1005+
entities:
1006+
- scope: SERVICE_INSTANCE
1007+
service: test-cluster
1008+
instance: test-pod@data
1009+
layer: BANYANDB
9271010
meter_banyandb_instance_retention_measure_disk_usage_percent:
9281011
entities:
9291012
- scope: SERVICE_INSTANCE

‎oap-server/server-starter/src/main/resources/otel-rules/banyandb/banyandb-endpoint.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,17 @@ metricsRules:
9090
- name: queue_latency_p99
9191
exp: banyandb_queue_pub_total_latency.sum(['le', 'cluster', 'group', 'operation']).histogram().histogram_percentile([99])
9292

93+
# batch + message granularity per group (BanyanDB #1169). Both are subscribe-side counters that
94+
# carry a real `group`, so they roll up per Endpoint: batch throughput is the per-group rate of
95+
# write-batches finished (the liaison's OAP-facing receive hop contributes these), message
96+
# throughput is the per-group per-record rate. (The publish-side batch latency histogram is NOT
97+
# modeled here: queue_pub_total_batch_latency is emitted with group="" so it cannot roll up per
98+
# group -- it is an instance-scope metric only, see banyandb-instance.yaml publish_batch_latency_p99.)
99+
- name: queue_batch_throughput
100+
exp: banyandb_queue_sub_total_batch_finished.sum(['cluster', 'group', 'operation']).rate('PT1M')
101+
- name: queue_message_throughput
102+
exp: banyandb_queue_sub_total_message_finished.sum(['cluster', 'group', 'operation']).rate('PT1M')
103+
93104
# publish bytes/s for the group. Wire family is banyandb_queue_pub_sent_bytes -- NO `total`
94105
# infix (unlike queue_pub_total_started/_finished). Liaison-only; sum collapses
95106
# operation/remote_node/remote_role/remote_tier before the rate.

‎oap-server/server-starter/src/main/resources/otel-rules/banyandb/banyandb-instance.yaml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,12 @@ metricsRules:
100100
exp: banyandb_queue_pub_sent_bytes.sum(['cluster','pod_name','container_name','node_role','node_type']).rate('PT15S')
101101
- name: publish_latency_p99
102102
exp: banyandb_queue_pub_total_latency.sum(['cluster','pod_name','container_name','node_role','node_type','operation','le']).histogram().histogram_percentile([99])
103+
# tier-2 publish, batch granularity (BanyanDB #1169): batches published/s by operation and the batch
104+
# send-latency p99. total_batch_latency uses a coarser bucket set than the per-operation total_latency.
105+
- name: publish_batch_throughput
106+
exp: banyandb_queue_pub_total_batch_finished.sum(['cluster','pod_name','container_name','node_role','node_type','operation']).rate('PT15S')
107+
- name: publish_batch_latency_p99
108+
exp: banyandb_queue_pub_total_batch_latency.sum(['cluster','pod_name','container_name','node_role','node_type','operation','le']).histogram().histogram_percentile([99])
103109
# write-queue (wqueue) depth: pending records, on-disk file parts, in-memory parts. On the liaison
104110
# these reflect the write buffer; the same families on data containers reflect storage parts (the
105111
# dashboard gates on container_name). Gauges, summed to the instance.
@@ -145,6 +151,14 @@ metricsRules:
145151
exp: banyandb_queue_sub_total_finished.sum(['cluster','pod_name','container_name','node_role','node_type','operation']).rate('PT15S')
146152
- name: queue_sub_latency_p99
147153
exp: banyandb_queue_sub_total_latency.sum(['cluster','pod_name','container_name','node_role','node_type','operation','le']).histogram().histogram_percentile([99])
154+
# subscribe-side per-message throughput (BanyanDB #1169). A data node ingests writes via the
155+
# per-message dispatch path, which ticks only the message counter (sub_total_message_finished),
156+
# NOT the batch counter -- the sub-side batch counters fire on the liaison's OAP-facing receive
157+
# hop (handleEOF), not on the data node, so a per-node `queue_sub_batch_*` is empty on data and is
158+
# intentionally not modeled here. Batch-level granularity lives on the liaison's publish side
159+
# (publish_batch_throughput / publish_batch_latency_p99 above).
160+
- name: queue_sub_message_throughput
161+
exp: banyandb_queue_sub_total_message_finished.sum(['cluster','pod_name','container_name','node_role','node_type','operation']).rate('PT15S')
148162
# retention disk-usage % per data-model scope (0-100 gauge). Kept per scope rather than summed (a sum
149163
# of three percentages is meaningless). Not in the upstream Grafana boards; a SkyWalking addition.
150164
- name: retention_measure_disk_usage_percent

‎test/e2e-v2/cases/banyandb/banyandb-cases.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,13 +54,20 @@ cases:
5454
expected: expected/metrics-has-label-value.yml
5555
- query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_instance_queue_sub_throughput --service-name=e2e-banyandb --instance-name=banyandb-data-hot-0@data
5656
expected: expected/metrics-has-label-value.yml
57+
# per-message throughput on the data node (BanyanDB #1169). The data node ingests via the per-message
58+
# dispatch path, so the sub-side message counter ticks (batch counters fire on the liaison hop, not here).
59+
- query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_instance_queue_sub_message_throughput --service-name=e2e-banyandb --instance-name=banyandb-data-hot-0@data
60+
expected: expected/metrics-has-label-value.yml
5761
# liaison node (banyandb-liaison-0@liaison)
5862
- query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_instance_node_uptime --service-name=e2e-banyandb --instance-name=banyandb-liaison-0@liaison
5963
expected: expected/metrics-has-value.yml
6064
- query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_instance_write_rate --service-name=e2e-banyandb --instance-name=banyandb-liaison-0@liaison
6165
expected: expected/metrics-has-label-value.yml
6266
- query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_instance_publish_throughput --service-name=e2e-banyandb --instance-name=banyandb-liaison-0@liaison
6367
expected: expected/metrics-has-label-value.yml
68+
# publish-side batch throughput on the liaison (BanyanDB #1169).
69+
- query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_instance_publish_batch_throughput --service-name=e2e-banyandb --instance-name=banyandb-liaison-0@liaison
70+
expected: expected/metrics-has-label-value.yml
6471

6572
# ---- Endpoint scope (storage group sw_metricsMinute) ----
6673
- query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_endpoint_write_rate --service-name=e2e-banyandb --endpoint-name=sw_metricsMinute
@@ -69,3 +76,8 @@ cases:
6976
expected: expected/metrics-has-value.yml
7077
- query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_endpoint_queue_throughput --service-name=e2e-banyandb --endpoint-name=sw_metricsMinute
7178
expected: expected/metrics-has-label-value.yml
79+
# batch + message granularity per group (BanyanDB #1169).
80+
- query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_endpoint_queue_batch_throughput --service-name=e2e-banyandb --endpoint-name=sw_metricsMinute
81+
expected: expected/metrics-has-label-value.yml
82+
- query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_banyandb_endpoint_queue_message_throughput --service-name=e2e-banyandb --endpoint-name=sw_metricsMinute
83+
expected: expected/metrics-has-label-value.yml

‎test/e2e-v2/script/env‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ SW_AGENT_CLIENT_JS_COMMIT=f08776d909eb1d9bc79c600e493030651b97e491
2323
SW_AGENT_CLIENT_JS_TEST_COMMIT=4f1eb1dcdbde3ec4a38534bf01dded4ab5d2f016
2424
SW_KUBERNETES_COMMIT_SHA=da0e267f877b9b8e5f7728ae4ea7dc7723a2a073
2525
SW_ROVER_COMMIT=79292fe07f17f98f486e0c4471213e1961fb2d1d
26-
SW_BANYANDB_COMMIT=8a1936ce96653e89d3d13250a42abc6e3d42fae7
26+
SW_BANYANDB_COMMIT=c2d925e4eae4d77edda94e1fd438243483960150
2727
SW_AGENT_PHP_COMMIT=d1114e7be5d89881eec76e5b56e69ff844691e35
2828
SW_PREDICTOR_COMMIT=54a0197654a3781a6f73ce35146c712af297c994
2929

0 commit comments

Comments
 (0)