feat(metrics): add block transaction count and SR set change monitoring

[warku123]

What does this PR do?

Add two new metrics enhancements for better network monitoring:

1. Block Transaction Count Histogram

   • Replace tron:block_empty_total counter with tron:block_transaction_count histogram
   • Track transaction count distribution per block with buckets: [0, 10, 50, 100, 200, 500, 1000, 2000, 5000, 10000]
   • Empty blocks can be monitored via le=0.0 bucket

2. SR Set Change Monitoring

   • Add tron:sr_set_change counter to track SR membership changes
   • Detect SR additions and removals at each maintenance time interval
   • Labels: action (add/remove), witness (SR address)

Changes:

• Add overloaded init() method in MetricsHistogram for custom buckets
• Add BLOCK_TRANSACTION_COUNT histogram metric
• Add SR_SET_CHANGE counter with SR_ADD/SR_REMOVE labels
• Implement SR set change detection logic in BlockChainMetricManager
• Update PrometheusApiServiceTest with comprehensive test coverage

Why are these changes required?

• Transaction histogram provides richer insights than simple counter for network analysis while maintaining empty block detection capability
• SR set change monitoring enables tracking of network consensus participant changes for governance analysis

Compared to #6602, this PR provides a cleaner implementation focused solely on metrics enhancement without additional complexity.

This PR has been tested by:

• Unit Tests (updated PrometheusApiServiceTest)
• Manual Testing

Follow up

N/A

Extra details

• Closes [Feature] Add Prometheus metrics for empty blocks and SR set changes #6590
• Related to feat(metrics): add block transaction count histogram for empty block monitoring #6602 (alternative cleaner implementation)

[halibobo1205]

The witness label uses addresses. Although the Active SR pool is relatively limited (27 slots), as SRs rotate in and out over time, label values will accumulate monotonically — they only grow, never shrink. This is a known Prometheus anti-pattern.

Practical impact assessment:

• TRON's SR candidate pool is not infinite, and rotation frequency is low, so it won't explode in the short term

• However, as a long-running node metric, months or years of operation could still accumulate a significant number of time series

Possible approaches:

• If the address dimension is truly needed, keep the current design, but document this characteristic

• Or remove the witness label entirely, keeping only action (add/remove), and output address details via logs instead. Follow the principle that Prometheus should be used for alerting and awareness, while logs are for precise investigation and root-cause analysis.

[warku123]

Thanks for raising the high-cardinality concern — it's a valid Prometheus best practice to watch for.

However, I believe the witness label is acceptable here given TRON's specific constraints:

1. Bounded candidate pool: The SR candidate pool is finite. Even over years of operation, the total number of distinct addresses that have ever held an SR slot has a practical ceiling — far from the
   unbounded growth typical of high-cardinality anti-patterns.
2. Low rotation frequency: SR set changes only occur at maintenance intervals (every 6 hours), so new label values are introduced very infrequently.
3. Diagnostic value: The primary purpose of this metric is to identify which SR was added or removed. Dropping the witness label would reduce the metric to just "something changed", requiring operators to
   cross-reference logs — losing the at-a-glance visibility in Grafana dashboards that makes this metric useful in the first place.

In summary, the worst-case time series count is roughly candidate_pool_size × 2 (add/remove), which stays well within Prometheus's comfortable operating range.

That said, I agree it's worth documenting this characteristic. Could you suggest what form you'd prefer — a code comment at the metric registration site, a note in the PR description, or something else?

[halibobo1205]

prefer to a code comment at the metric registration site.

[warku123]

@halibobo1205 Added a comment at the SR_SET_CHANGE constant definition in MetricKeys.java (commit 96bf315) explaining the cardinality rationale:

witness label: bounded cardinality -- SR candidate pool is finite, rotation is infrequent (at most once per maintenance interval); kept for at-a-glance SR identification in dashboards rather than requiring log cross-referencing.

[halibobo1205]

TODO

1. Adding a METRICS_CHANGELOG.md to the repo to track metric additions, modifications, and deprecations — future PRs can maintain it as well
2. Including a metrics change entry in release notes
3. Filing a follow-up issue/TODO to update the official metrics documentation and the Grafana dashboard template