Merge pull request #108 from hyp3rd/feat/dist-mem-cache

feat(dist): add structured logging to dist backend (Phase A.1)

Author: hyp3rd

.mdl_style.rb

@@ -5,3 +5,11 @@
 rule 'MD007', :indent => 3
 
 rule "MD029", style => "one"
+
+# Keep-a-Changelog (https://keepachangelog.com) uses repeated `### Added`,
+# `### Fixed`, `### Security` headings under each `## [version]` heading by
+# design. MD024 with the default config flags those as duplicates.
+# allow_different_nesting permits same-text headings as long as they sit
+# under distinct parent headings — which is exactly the Keep-a-Changelog
+# shape, and still catches genuine duplicates within the same section.
+rule "MD024", :allow_different_nesting => true

CHANGELOG.md

@@ -6,7 +6,48 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
-## [2.0.1] — 2026-05-05
+### Added
+
+- **Structured logging on the dist backend.** New `WithDistLogger(*slog.Logger)`
+  option wires a structured logger into the dist backend's background
+  loops (heartbeat, hint replay, rebalance, merkle sync) and operational
+  error surfaces (HTTP listener bind failures, serve-goroutine exits,
+  failed migrations during rebalance, dropped hints, peer state
+  transitions). Library default is silent — `WithDistLogger` not called
+  installs a `slog.DiscardHandler` so the dist backend never writes to
+  stderr unless the caller opts in. Every record is pre-bound with
+  `component=dist_memory` and `node_id=<id>` attributes for grep/filter.
+  Phase A.1 of the production-readiness work.
+- **OpenTelemetry tracing on the dist backend.** New
+  `WithDistTracerProvider(trace.TracerProvider)` option opens spans on
+  every public `Get` / `Set` / `Remove`, with child spans
+  (`dist.replicate.set` / `dist.replicate.remove`) per peer during
+  fan-out. Span attributes include `cache.key.length`,
+  `dist.consistency`, `dist.owners.count`, `dist.acks`, `cache.hit`,
+  and `peer.id`. Cache key *values* are intentionally never recorded
+  on spans — keys can be PII (user IDs, session tokens). Library
+  default is a no-op tracer (`noop.NewTracerProvider`), so spans cost
+  nothing unless the caller opts in. New `ConsistencyLevel.String()`
+  method renders consistency levels human-readably for log/span attrs.
+  Phase A.2 of the production-readiness work.
+- **OpenTelemetry metrics on the dist backend.** New
+  `WithDistMeterProvider(metric.MeterProvider)` option registers an
+  observable instrument for every field on `DistMetrics` — counters
+  for cumulative totals (`dist.write.attempts`, `dist.forward.*`,
+  `dist.hinted.*`, `dist.merkle.syncs`, `dist.rebalance.*`, etc.),
+  gauges for current state (`dist.members.alive`,
+  `dist.tombstones.active`, `dist.hinted.bytes`, last-operation
+  latencies in nanoseconds, etc.). A single registered callback
+  observes all instruments from one `Metrics()` snapshot per
+  collection cycle, so there is no per-operation overhead beyond the
+  existing atomic counters. Names use the `dist.` prefix so a
+  Prometheus exporter renders them under a single subsystem.
+  `Stop` unregisters the callback so the SDK does not invoke it
+  against a stopped backend. Library default is a no-op meter, so
+  metrics cost nothing unless the caller opts in. Phase A.3 of the
+  production-readiness work.
+
+## [0.5.0] — 2026-05-05
 
 ### Security
 
@@ -25,7 +66,7 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
   in via the new `DistHTTPAuth.AllowAnonymousInbound` field. All other
   configurations (`Token`-only, `Token+ServerVerify`, `Token+ClientSign`,
   `ServerVerify`-only) are unaffected. Reported by the post-tag
-  security review; addressed before any v2.0.0 public announcement.
+  security review; addressed before any v0.5.0 public announcement.
 
 ### Added
 
@@ -34,7 +75,7 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 - `sentinel.ErrInsecureAuthConfig` — surfaced from `NewDistMemory` when
   the auth policy would silently disable inbound enforcement.
 
-## [2.0.0] — 2026-05-04
+## [0.4.3] — 2026-05-04
 
 A modernization release. The headline themes:
 
@@ -86,7 +127,6 @@ RFCs that informed the design decisions live under [docs/rfcs/](docs/rfcs/).
 ### Performance
 
 Measurements on Apple M4 Pro, `go test -bench`, `count=5`, benchstat.
-Full release snapshot captured in [bench-v2.0.0.txt](bench-v2.0.0.txt).
 
 - **Per-shard atomic `Count`.** `BenchmarkConcurrentMap_Count`:
   53 → ~10 ns/op. `_CountParallel`: 1181 → ~13 ns/op. Eliminates the
@@ -186,5 +226,5 @@ Worth surfacing for contributors:
   [RFC document](docs/rfcs/0001-backend-owned-eviction.md) preserves
   the measurement and the lessons.
 
-[Unreleased]: https://github.com/hyp3rd/hypercache/compare/v2.0.0...HEAD
-[2.0.0]: https://github.com/hyp3rd/hypercache/releases/tag/v2.0.0
+Unreleased: <https://github.com/hyp3rd/hypercache/compare/v0.5.0...HEAD>
+Released: [0.5.0](https://github.com/hyp3rd/hypercache/releases/tag/v0.5.0)

README.md

@@ -137,7 +137,7 @@ Available algorithm names you can pass to `WithEvictionAlgorithm`:
 
 Note: ARC is experimental and isn’t included in the default registry. If you choose to use it, register it manually or enable it explicitly in your build.
 
-#### Sharded eviction (default since v2.0.0)
+#### Sharded eviction (default since v0.5.0)
 
 The configured algorithm is wrapped by a 32-shard router (`pkg/eviction/sharded.go`) that uses the same key hash as `ConcurrentMap` — so a key's data shard and eviction shard line up. This eliminates the global mutex contention single-instance algorithms (LRU/LFU/Clock/CAWOLFU) suffer from. Total capacity is honored within ±32 (one slot of slack per shard), and items evict per-shard rather than in strict global LRU/LFU order.
 
@@ -263,7 +263,7 @@ Limitations / not yet implemented:
 - Compression on the wire.
 - Persistence / durability (out of scope presently).
 
-#### Transport hardening (since v2.0.0)
+#### Transport hardening (since v0.5.0)
 
 The dist HTTP server and the auto-created HTTP client share a single configuration surface — apply the same option to every node in the cluster.
 
@@ -347,7 +347,7 @@ Test helpers `AddPeer` and `RemovePeer` simulate join / leave events that trigge
 | Advanced versioning (HLC/vector) | Planned |
 | Client SDK (direct routing) | Planned |
 | Tracing spans | Planned |
-| Security (TLS/auth) | Done (since v2.0.0; see "Transport hardening") |
+| Security (TLS/auth) | Done (since v0.5.0; see "Transport hardening") |
 | Compression | Planned |
 | Persistence | Out of scope (current phase) |
 | Chaos / fault injection | Planned |

__examples/observability/observability.go

@@ -34,7 +34,8 @@ func main() {
 	tracer := trace.NewNoopTracerProvider().Tracer("hypercache/examples")
 
 	// Apply OTel tracing and metrics middleware.
-	svc = hypercache.ApplyMiddleware(svc,
+	svc = hypercache.ApplyMiddleware(
+		svc,
 		func(next hypercache.Service) hypercache.Service {
 			return middleware.NewOTelTracingMiddleware(next, tracer, middleware.WithCommonAttributes(
 				attribute.String("component", "hypercache"),

__examples/service/service.go

@@ -37,7 +37,8 @@ func main() {
 	logger := log.Default()
 
 	// apply middleware in the same order as you want to execute them
-	svc = hypercache.ApplyMiddleware(svc,
+	svc = hypercache.ApplyMiddleware(
+		svc,
 		// middleware.YourMiddleware,
 		func(next hypercache.Service) hypercache.Service {
 			return middleware.NewLoggingMiddleware(next, logger)

cspell.config.yaml

@@ -132,6 +132,8 @@ words:
   - longbridgeapp
   - maxmemory
   - memprofile
+  - metricdata
+  - metricnoop
   - Merkle
   - mfinal
   - Mgmt
@@ -146,6 +148,7 @@ words:
   - noctx
   - noinlineerr
   - nolint
+  - nolintlint
   - nonamedreturns
   - nosec
   - NOVENDOR
@@ -162,6 +165,7 @@ words:
   - Repls
   - rerr
   - sarif
+  - sdkmetric
   - sectools
   - securego
   - sess
@@ -180,6 +184,7 @@ words:
   - thelper
   - toplevel
   - tparallel
+  - tracetest
   - traefik
   - ugorji
   - unmarshals

docs/rfcs/0001-backend-owned-eviction.md

@@ -253,7 +253,7 @@ Per the RFC's own discipline (`Reject A and revisit if any criterion fails`):
    "slower on Get, semantically-correct LRU." Default stays legacy.
 1. **Do not pursue Option A2** (co-located locks) — the win Option A
    would have justified A2 isn't there to amortize the bigger refactor.
-1. **The "Get does not touch LRU" semantic gap is a separate concern**
+1n. **The "Get does not touch LRU" semantic gap is a separate concern**
    that could be addressed inside the legacy path (have HyperCache.Get
    call `evictionAlgorithm.Get(key)`) at similar cost to the Item-aware
    Touch — i.e., the cost is fundamental to "real LRU", not specific

go.mod

@@ -13,12 +13,16 @@ require (
 	github.com/ugorji/go/codec v1.3.1
 	go.opentelemetry.io/otel v1.43.0
 	go.opentelemetry.io/otel/metric v1.43.0
+	go.opentelemetry.io/otel/sdk v1.43.0
+	go.opentelemetry.io/otel/sdk/metric v1.43.0
 	go.opentelemetry.io/otel/trace v1.43.0
 )
 
 require (
 	github.com/andybalholm/brotli v1.2.1 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
+	github.com/go-logr/logr v1.4.3 // indirect
+	github.com/go-logr/stdr v1.2.2 // indirect
 	github.com/gofiber/schema v1.7.1 // indirect
 	github.com/gofiber/utils/v2 v2.0.4 // indirect
 	github.com/google/uuid v1.6.0 // indirect
@@ -27,15 +31,14 @@ require (
 	github.com/mattn/go-isatty v0.0.22 // indirect
 	github.com/philhofer/fwd v1.2.0 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
-	github.com/rogpeppe/go-internal v1.14.1 // indirect
 	github.com/tinylib/msgp v1.6.4 // indirect
 	github.com/valyala/bytebufferpool v1.0.0 // indirect
 	github.com/valyala/fasthttp v1.71.0 // indirect
+	go.opentelemetry.io/auto/sdk v1.2.1 // indirect
 	go.uber.org/atomic v1.11.0 // indirect
 	golang.org/x/crypto v0.50.0 // indirect
 	golang.org/x/net v0.53.0 // indirect
 	golang.org/x/sys v0.43.0 // indirect
 	golang.org/x/text v0.36.0 // indirect
-	gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 )

go.sum

@@ -10,6 +10,7 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/fxamacker/cbor/v2 v2.9.1 h1:2rWm8B193Ll4VdjsJY28jxs70IdDsHRWgQYAI80+rMQ=
 github.com/fxamacker/cbor/v2 v2.9.1/go.mod h1:vM4b+DJCtHn+zz7h3FFp/hDAI9WNWCsZj23V5ytsSxQ=
+github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
 github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
 github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
 github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
@@ -34,11 +35,8 @@ github.com/klauspost/compress v1.18.6 h1:2jupLlAwFm95+YDR+NwD2MEfFO9d4z4Prjl1XXD
 github.com/klauspost/compress v1.18.6/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
 github.com/klauspost/cpuid/v2 v2.2.10 h1:tBs3QSyvjDyFTq3uoc/9xFpCuOsJQFNPiAhYdw2skhE=
 github.com/klauspost/cpuid/v2 v2.2.10/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
-github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
 github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
 github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
-github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
-github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
 github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
 github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
 github.com/mattn/go-colorable v0.1.14 h1:9A9LHSqF/7dyVVX6g0U9cwm9pG3kP9gSzcuIPHPsaIE=
@@ -77,10 +75,16 @@ go.opentelemetry.io/otel v1.43.0 h1:mYIM03dnh5zfN7HautFE4ieIig9amkNANT+xcVxAj9I=
 go.opentelemetry.io/otel v1.43.0/go.mod h1:JuG+u74mvjvcm8vj8pI5XiHy1zDeoCS2LB1spIq7Ay0=
 go.opentelemetry.io/otel/metric v1.43.0 h1:d7638QeInOnuwOONPp4JAOGfbCEpYb+K6DVWvdxGzgM=
 go.opentelemetry.io/otel/metric v1.43.0/go.mod h1:RDnPtIxvqlgO8GRW18W6Z/4P462ldprJtfxHxyKd2PY=
+go.opentelemetry.io/otel/sdk v1.43.0 h1:pi5mE86i5rTeLXqoF/hhiBtUNcrAGHLKQdhg4h4V9Dg=
+go.opentelemetry.io/otel/sdk v1.43.0/go.mod h1:P+IkVU3iWukmiit/Yf9AWvpyRDlUeBaRg6Y+C58QHzg=
+go.opentelemetry.io/otel/sdk/metric v1.43.0 h1:S88dyqXjJkuBNLeMcVPRFXpRw2fuwdvfCGLEo89fDkw=
+go.opentelemetry.io/otel/sdk/metric v1.43.0/go.mod h1:C/RJtwSEJ5hzTiUz5pXF1kILHStzb9zFlIEe85bhj6A=
 go.opentelemetry.io/otel/trace v1.43.0 h1:BkNrHpup+4k4w+ZZ86CZoHHEkohws8AY+WTX09nk+3A=
 go.opentelemetry.io/otel/trace v1.43.0/go.mod h1:/QJhyVBUUswCphDVxq+8mld+AvhXZLhe+8WVFxiFff0=
 go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE=
 go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
+go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
+go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
 golang.org/x/crypto v0.50.0 h1:zO47/JPrL6vsNkINmLoo/PH1gcxpls50DNogFvB5ZGI=
 golang.org/x/crypto v0.50.0/go.mod h1:3muZ7vA7PBCE6xgPX7nkzzjiUq87kRItoJQM1Yo8S+Q=
 golang.org/x/net v0.53.0 h1:d+qAbo5L0orcWAr0a9JweQpjXF19LMXJE8Ey7hwOdUA=

pkg/backend/dist_http_server.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"crypto/subtle"
 	"crypto/tls"
+	"log/slog"
 	"net"
 	"net/http"
 	"strconv"
@@ -47,6 +48,12 @@ type distHTTPServer struct {
 	// use, TLS handshake failure on accept) instead of having them
 	// silently swallowed.
 	serveErr atomic.Pointer[error]
+	// logger is the structured logger inherited from the parent
+	// DistMemory. Used to surface serve-goroutine errors that previously
+	// only landed in serveErr (LastServeError accessor) — operators
+	// running with a configured logger now see them in their log stream
+	// at the moment of failure, not just on demand.
+	logger *slog.Logger
 }
 
 // DistHTTPAuth configures authentication for the dist HTTP server
@@ -482,8 +489,18 @@ func (s *distHTTPServer) listen(ctx context.Context) error {
 		if serveErr != nil {
 			// Stash so operators can read it via LastServeError(); a
 			// listener that crashed silently is the worst kind of
-			// production bug.
+			// production bug. Also surface to the structured logger when
+			// configured so the failure shows up in the operator's log
+			// stream at the moment it happens, not just on demand.
 			s.serveErr.Store(&serveErr)
+
+			if s.logger != nil {
+				s.logger.Error(
+					"dist HTTP serve goroutine exited",
+					slog.String("addr", s.addr),
+					slog.Any("err", serveErr),
+				)
+			}
 		}
 	}()