feat(metrics): add endpoint server and metrics

[MaurUppi]

Background

This PR supersedes #941 with a clean metrics-only branch based on daeuniverse/dae:main.

The previous PR mixed in many unrelated commits. This one keeps only metrics endpoint work and required dependencies.

dae 一直没有实施 metrics，从 clash/Surge 一路过来，都是有 Dashboard 可以看看，所以不是很习惯老去翻 journal 日志，且 trace/debug 的大量日志中翻找信息还是挺麻烦/费劲的，而且消耗空间还很快。dae 的机器配置不高，查询量大很费劲。哦， 忘了说，试过用 vector 解析日志，调试是在太麻烦且多个香炉多个鬼，维护不易。
因此，假期无事就 vibe coding 完成这个 PR，希望给 v1.1.0 送去有价值功能，实际效果有待 team 审核。
附带我基于 metrics 捣鼓的 Grafana Dashboard，有事没事看看还是挺有趣的。

本 PR 实现 metrics Phase1（不进入热路径）：

• 新增 global.endpoint_* 配置项
• 接入 endpoint server（metrics + pprof）
• 新增 metrics 基础模块：pkg/metrics/{state,registry,server,auth}
• 新增 Phase1 gauge collectors（dialer/dns/connection）
• 更新 example.dae 示例配置 <-- user-facing docs PR

在 Phase1 基础上实现 Phase2：

• 增加 DNS 计数器与延迟直方图快照
• 增加 TCP/UDP 连接总量计数器
• 增加 dialer 健康检查计数器
• 扩展 metrics collectors 输出 Phase2 指标
• 补充对应测试
• 补充 dae Transparent Proxy-Grafana_dashboard.json

Checklist

• [ X ] The Pull Request has been fully tested
• [ X ] There's an entry in the CHANGELOGS
• [ X ] There is a user-facing docs PR against https://github.com/daeuniverse/dae

Full Changelogs

1. Phase 1 Gauges
2. Phase 2 Counters / Histograms

请见 #948 (comment)

---

Test Result

• Local verification:
  • go build ./...
  • go test ./...
  • Runtime check: /metrics, /debug/pprof/
• Note:
  • eBPF/Kernel CI validation has been passed.

Notes

• Branch base: latest daeuniverse/dae:main at split time
• This PR is intended to replace feat(metrics): add endpoint server and metrics #941 for review/merge

Checklist

• Metrics-related changes are isolated from unrelated optimization/history commits
• Required dependency entries for Prometheus are included

---

最终实现效果：

[MaurUppi]

代码审计报告

dae Metrics Endpoint — Audit Report

Date: 2026-02-23
Scope: Phase 1 + Phase 2 metrics implementation (feat/metrics-endpoint-phase1, PR#941)
Live endpoint verified: http://192.168.1.174:5556/metrics
Files reviewed: metrics-related changes in pkg/metrics/, control/, component/outbound/dialer/, cmd/run.go

---

Code Review Summary

Overall assessment: APPROVE (no blocking issue)

Notes:

• Previous dae_dns_concurrency_in_use inversion issue has been fixed in metrics branch (fix(metrics): correct dns concurrency in-use gauge semantics).
• Collector descriptor coverage has been strengthened by test (test(metrics): verify all collector descriptors are exposed).

---

Findings

P0 — Critical

(none)

P1 — High

(none)

P2 — Medium

(none)

P3 — Low

(none)

---

Endpoint Verification

All metrics from both phases are present at http://192.168.1.174:5556/metrics.

Phase 1 Gauges — All present ✅

Metric	Type	Status
dae_dialer_alive	gauge	✅ Values: 0/1 per dialer×network
dae_dialer_latency_last_seconds	gauge	✅ Real latency values (e.g., 0.0497s)
dae_dialer_latency_avg10_seconds	gauge	✅ Real latency averages
dae_dialer_latency_moving_avg_seconds	gauge	✅ EWMA values
dae_group_alive_dialers_total	gauge	✅ Per group×network
dae_dns_cache_entries	gauge	✅ (0 — cache expired at scrape time)
dae_dns_concurrency_in_use	gauge	✅ Current semantics are correct; on dae/main fallback path this is typically 0
dae_dns_concurrency_limit	gauge	✅ On dae/main fallback path this is 0 (legacy DNS controller has no explicit limiter)
dae_dns_forwarder_cache_entries	gauge	✅ (0 — no long-lived forwarders active)
dae_dns_forwarder_in_flight{upstream}	gauge	✅ Implemented; emits only when there are in-flight upstream requests
dae_tcp_connections_active	gauge	✅
dae_udp_endpoints_active	gauge	✅
dae_udp_task_queues_active	gauge	✅

Phase 2 Counters / Histograms — All present ✅

Metric	Type	Status
dae_dns_query_total	counter	✅ 4
dae_dns_cache_hit_total	counter	✅ 1
dae_dns_cache_lazy_hit_total	counter	✅ 0
dae_dns_cache_miss_total	counter	✅ 3
dae_dns_upstream_query_total{upstream}	counter	✅ tcp://192.168.1.8:5553
dae_dns_upstream_err_total{upstream}	counter	✅
dae_dns_rejected_total	counter	✅ 0
dae_dns_refused_total	counter	✅ 0
dae_dns_response_latency_seconds	histogram	✅ 12 buckets + sum + count
dae_dns_upstream_latency_seconds{upstream}	histogram	✅ per upstream
dae_health_check_total{group,dialer,network}	counter	✅
dae_health_check_failure_total{group,dialer,network}	counter	✅
dae_tcp_connections_total{protocol,group}	counter	✅ tcp4/HK = 1
dae_udp_connections_total{protocol,group}	counter	✅

Process / Go Runtime ✅

All standard process_* and go_* metrics are present.

---

Architecture Assessment

Concern	Status	Notes
Dependency direction	✅	pkg/metrics/ depends on control/; no reverse dependency
prometheus import isolation	✅	Metrics dependency remains in metrics package; domain structs expose snapshots/getters
Hot-path safety	✅	Counters use atomic.Uint64; collectors scrape snapshots
Thread safety — gauge reads	✅	Guarded by mutex/sync.Map/atomic in corresponding components
Thread safety — histogram	✅	Atomic bucket/counter/sum update and snapshot
Nil safety in collectors	✅	Collectors guard state == nil, cp == nil, dc == nil
Reload handling	✅	metrics.State swaps ControlPlane atomically
Deterministic output	✅	Key sorting is used for labeled DNS upstream metrics

---

Required Action Before Upstream PR

No blocking change required based on current audited state.

---

Optional (Non-blocking)

• Keep dashboard and changelog text aligned with dae/main fallback semantics for DNS concurrency gauges (0/0 until PR936-style limiter model is present).

metrics 观测开销评估

dae metrics 观测开销评估

基准环境

• 机器：192.168.1.15
• Prometheus scrape_interval：5s
• dae DNS QPS：日常 ~几十，压测峰值 200 QPS
• metrics endpoint：http://192.168.1.15:5556/metrics（响应体 ~224KB / 2338 行）

---

一、实测基线（采集自 /metrics）

process_cpu_seconds_total       386.55
process_resident_memory_bytes   139,923,456  (~133 MB RSS)
go_memstats_heap_alloc_bytes    62,481,112   (~60 MB)
go_memstats_heap_inuse_bytes    73,883,648   (~70 MB)
go_memstats_heap_objects        658,288

---

二、热路径开销（每条 DNS 查询）

代码路径（control/dns_metrics.go）

每次 HandleWithResponseWriter_ 调用触发：

操作	实现	估算耗时
dnsQueryTotal.Add(1)	atomic.Uint64	~5ns
分支计数器 .Add(1)	atomic.Uint64	~5ns
Observe(seconds)	线性扫描 12 个 bucket	~5ns
atomic.Add × 2	count + bucket	~10ns
addAtomicFloat64	CAS 循环（无竞争路径）	~10ns
每条 query 合计		~35–50 ns

upstream miss 时额外触发一次 Observe()，成本相同。

200 QPS 时 CPU 消耗：

200 × 50ns = 10µs/s → 占单核 0.001%

关键设计：全程使用 atomic.Uint64 + 自定义 histogram，热路径无任何 mutex。

---

三、抓取路径开销（Prometheus pull）

Collect() 执行内容

DnsCountersSnapshot()      → 8 × atomic.Load + sync.Map.Range（当前 1 个 upstream）
DnsResponseLatencySnapshot → 14 × atomic.Load + map 分配（12 buckets）
DnsUpstreamSnapshot()      → 14 × atomic.Load + map 分配（per upstream）
ConcurrencyInfo()          → len(channel)，O(1)
CacheSize()                → sync.Map 遍历
ForwarderCacheInfo()       → sync.Map 遍历
HTTP 序列化                → ~224KB 文本输出

5s 抓取间隔下的 CPU 影响

参数	值
单次 scrape 估算耗时	~1–2ms
每分钟 scrape 次数	12 次
每分钟占用 CPU 时间	12–24ms
占单核比例	~0.02–0.04%

与 15s 间隔对比：

scrape_interval	每分钟次数	CPU 占比
15s	4 次	~0.01%
5s	12 次	~0.03%
1s	60 次	~0.15%

5s 间隔将抓取频率提高 3 倍，但绝对值仍可忽略不计。

每次 scrape 的内存分配

DnsHistogramSnapshot.Buckets  map 分配：12 个 entry × 2 = ~400B/次
DnsUpstreamSnapshot           map 分配：per upstream ~200B
合计：< 1KB/次 → GC 一个周期内回收

---

四、常驻内存占用（静态结构）

dnsLatencyHistogram（per histogram）：
  buckets: 13 × atomic.Uint64 = 104B
  count + sumBits              =  16B
  ─────────────────────────────── ~120B

当前实例：
  response latency histogram   = 120B
  upstream latency histogram × 1 = 120B
  计数器 atomic × 6            =  48B
  metrics HTTP server goroutine = ~8KB（goroutine stack）

metrics 模块总静态内存：< 10KB

---

五、结论

开销来源	5s scrape_interval 下的影响	评级
热路径 atomic 计数（per query）	< 0.001% CPU @ 200 QPS	可忽略
Histogram Observe CAS	极低（CAS 无竞争）	可忽略
Scrape 序列化（5s 周期）	~0.03% CPU	可忽略
常驻内存（metrics state）	< 10KB	可忽略
Heap 分配（per scrape）	< 1KB，GC 即回收	可忽略
metrics HTTP goroutine	1 goroutine ~8KB stack	可忽略

整体结论：scrape_interval: 5s 相比 15s 将抓取 CPU 开销提高 3 倍，但绝对值仍在 0.03% 量级，在任何实际场景下均不构成性能瓶颈。当前实现的无锁设计保证了热路径与抓取路径互不阻塞，即使在 200 QPS 压测期间也不会产生可观测的影响。

若要设置更激进的 scrape_interval（如 1s），建议先通过方法 2 确认单次 scrape 耗时，再评估是否产生可感知影响。

[MaurUppi]

@cubercsl

请检查

[MaurUppi]

看来，我这个 PR 需要 rebase 重新解决冲突，且需要将 PR#968 的部分 runtime/latency 数据也接进 Prometheus
这两天解决了，再重新push

[MaurUppi]

太难了。。。。
PR#968 这个其实加了很简单的 runtime/latency 接口
但是 #970 改动太大了，，，恐怕我这个 PR 得重写。。。。而不是仅仅 rebase
而且 #980 的 fix 又改动了不少。。。

算了算了，，，我关闭这个 PR 吧，等有空了重新搞。

现在自己用着，PR#970/980没有并入