feat(cluster): add metrics for endpoint snapshot publication

[mochengqian]

What

Adds observability for endpoint snapshot publication (follow-up to #932). Operators currently have no direct signal for snapshot publish frequency or snapshot endpoint size, which makes registry churn, unexpected cluster size, or excessive health-update publication hard to diagnose.

Metrics

Three OpenTelemetry instruments, emitted on each successful snapshot CompareAndSwap, labeled only by cluster:

Metric	Type	Meaning
pixiu_cluster_snapshot_publish_total	counter	Incremented once per successful snapshot publish
pixiu_cluster_snapshot_endpoint_count	gauge	Total endpoints in the latest published snapshot
pixiu_cluster_snapshot_healthy_endpoint_count	gauge	Healthy endpoints in the latest published snapshot

Instruments follow the existing pixiu_/snake_case convention and bind lazily to otel.GetMeterProvider() (same pattern as pkg/filter/metric and the LLM tokenizer), so they wire into the existing Prometheus exporter automatically.

Where it emits

recordSnapshotPublish is called inside the successful-CAS branch of the three publish paths in pkg/cluster/cluster.go:

• RefreshEndpointsFrom — membership/address changes
• UpdateEndpointHealth — endpoint-keyed health flips
• UpdateEndpointAddressHealth — address-keyed health flips

The counter increments only on a real swap: no-op health updates return before CAS and are not counted, which is what makes "excessive health update publication" diagnosable.

Design notes / trade-offs

• Synchronous emission, not an ObservableGauge. A callback gauge would require a global live-cluster registry to enumerate clusters at scrape time; this PR avoids introducing new global mutable state and emits at the publish site instead.
• Stale series on cluster deletion. Because emission is synchronous, a deleted cluster's gauge retains its last value until the process restarts. Clusters are long-lived configuration objects with rare name churn, so this is an accepted trade-off rather than a leak.
• One attribute set allocated per publish (WithAttributes). Publish is a low-frequency path (not the pick hot path), so this is not cached; the read/pick path is untouched.
• Scope held deliberately tight: no duration histogram, generation gauge, or per-state breakdown — those can follow as separate issues.

Out of scope (per #944)

No new metrics backend, no endpoint-level (high-cardinality) labels, and no change to snapshot publication behavior.

Tests

pkg/cluster/metrics_test.go uses a ManualReader to assert:

• publish counter and endpoint/healthy gauges after an endpoint-health flip
• no-op health update does not increment the counter
• address-keyed health flip publishes once and marks all endpoints sharing the address

pkg/server/cluster_manager_test.go includes TestStaticClusterSnapshotMetricsRecordedAtStartup to guard against startup ordering regression (see fix below).

go test ./pkg/cluster/... ./pkg/server/...
go vet ./pkg/cluster/

All pass; gofmt clean.

---

🔧 Fix: Startup ordering for static clusters

Problem found during review: Static clusters' initial snapshot publish was landing on the no-op delegating provider, causing counter/gauges to remain empty in steady state — exactly the "unexpected cluster size" scenario this PR is meant to diagnose.

Root cause: Start(bs) called server.initialize(bs) (which constructs clusters) before registerOtelMetricMeter(). While otel-go's delegation back-fills instruments when SetMeterProvider is called later, the data from those dropped emissions is lost forever.

Fix applied (commit 7ac917f):

• Moved registerOtelMetricMeter(bs.Metric) from (*Server).Start() to Start(bs), positioned before server.initialize(bs)
• Added TestStaticClusterSnapshotMetricsRecordedAtStartup in pkg/server to guard against ordering regression
• Expanded test helper documentation about global mutation constraints

Verification: All tests pass including the new ordering guard; local CI clean (gofmt, imports, golangci-lint, race detector).

Closes #944

[Copilot]

Pull request overview

Adds OpenTelemetry metrics to improve observability of cluster endpoint snapshot publication, including a startup-ordering fix so initial static-cluster snapshot emissions are not dropped before a real meter provider is installed.

Changes:

• Introduces three pixiu_cluster_snapshot_* instruments (publish counter + endpoint/healthy endpoint gauges) and records them on successful snapshot CompareAndSwap.
• Adds focused metric tests in pkg/cluster plus a pkg/server regression test to ensure startup ordering records static cluster initial publish metrics.
• Moves registerOtelMetricMeter earlier in startup (Start(bs)) so cluster construction happens after the meter provider is installed.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pkg/server/pixiu_start.go	Registers the OTel meter provider before cluster initialization to avoid losing initial snapshot metric emissions.
pkg/server/cluster_manager_test.go	Adds a regression test ensuring static cluster initial snapshot publish metrics are recorded at startup.
pkg/cluster/metrics.go	Adds lazy-initialized OTel instruments and a recordSnapshotPublish helper invoked on successful snapshot CAS.
pkg/cluster/metrics_test.go	Adds ManualReader-based tests validating counter/gauges and confirming no-op health updates don’t increment publish count.
pkg/cluster/cluster.go	Hooks recordSnapshotPublish into the successful CAS branches of the three snapshot publication paths.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 74.07407% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 26.21%. Comparing base (d2e421d) to head (a7f6b5b).
⚠️ Report is 16 commits behind head on develop.

Files with missing lines	Patch %	Lines
pkg/cluster/metrics.go	73.91%	3 Missing and 3 partials ⚠️
pkg/server/pixiu_start.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #962      +/-   ##
===========================================
+ Coverage    26.14%   26.21%   +0.06%     
===========================================
  Files          275      276       +1     
  Lines        21981    22007      +26     
===========================================
+ Hits          5748     5770      +22     
  Misses       15623    15623              
- Partials       610      614       +4     

Flag	Coverage Δ
unittests	26.21% <74.07%> (+0.06%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[AlexStocks]

[P1] initSnapshotMetrics 第一次调用会在 healthMu 持有期间执行：UpdateEndpointHealth/UpdateEndpointAddressHealth 都在 healthMu.Lock() 之后直接走到 recordSnapshotPublish → initSnapshotMetrics → meter.Int64Counter/Int64Gauge × 3，注册失败时还要走 logger.Errorf，整个路径的分配和潜在 IO 全部发生在锁内。这与本函数注释“health-update callers invoke this while holding the cluster s healthMu, so the body must stay allocation-light and must not block”直接冲突，会让首个健康事件、或者每次 registerOtelMeterProvider 之后第一次 publish 抢占 healthMu，阻塞所有并发健康事件。建议在 cluster 包初始化（或 registerOtelMetricMeter 完成后）显式调用一次 initSnapshotMetrics 把仪表预绑定好，让 recordSnapshotPublish 只剩 Add/Record。