Skip to content

Commit 14593bf

Browse files
hyp3rdhyp3rd
authored
Merge pull request #120 from hyp3rd/feat/dist-mem-cache
feat: Introduce an OIDC-based authentication path
2 parents b394cee + 7b090ea commit 14593bf

18 files changed

Lines changed: 1213 additions & 111 deletions

‎.golangci.yaml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,20 @@ linters:
6363
# (access to unexported helpers without re-exporting). Documented choice.
6464
- testpackage
6565

66+
exclusions:
67+
rules:
68+
# Phase C OIDC: makeOIDCServerVerify returns a closure with the
69+
# signature `func(fiber.Ctx) (httpauth.Identity, error)` per
70+
# the Policy.ServerVerify contract. The closure derives ctx
71+
# from `c.Context()` (fiber's request-scoped ctx) — tying JWT
72+
# verification to the boot-time discovery context would freeze
73+
# cancellation at the wrong scope. contextcheck can't tell the
74+
# closure receives ctx via the framework parameter and false-
75+
# positives. Targeted to this single file.
76+
- path: cmd/hypercache-server/oidc\.go
77+
linters:
78+
- contextcheck
79+
6680
settings:
6781
cyclop:
6882
# The maximal code complexity to report.

‎CHANGELOG.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,34 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
88

99
### Added
1010

11+
- **OIDC verifier on the client API (`Policy.ServerVerify` hook).**
12+
When `HYPERCACHE_OIDC_ISSUER` + `HYPERCACHE_OIDC_AUDIENCE` are
13+
set, the binary fetches the IdP's
14+
`/.well-known/openid-configuration` at boot, builds a
15+
go-oidc-backed JWT verifier, and attaches it to
16+
[`Policy.ServerVerify`](pkg/httpauth/policy.go). JWTs presented
17+
via `Authorization: Bearer <jwt>` are validated for signature
18+
and `iss`/`aud`/`exp`/`iat`/`nbf`, then the configured identity
19+
claim (`HYPERCACHE_OIDC_IDENTITY_CLAIM`, default `sub`; common
20+
override `email`) becomes `Identity.ID` and the configured
21+
scope claim (`HYPERCACHE_OIDC_SCOPE_CLAIM`, default `scope` —
22+
standard OAuth2 space-separated string; arrays also supported)
23+
maps to `Identity.Scopes` (only `read`/`write`/`admin` survive;
24+
unknown OIDC scopes are dropped silently). Coexists with the
25+
static-bearer flow: a JWT that doesn't match any configured
26+
`Tokens` entry falls through `resolveBearer` →
27+
`ServerVerify`, so operators can run hybrid deployments
28+
(machine integrations on static bearers, humans on OIDC).
29+
Fail-fast at boot on an unreachable issuer URL or partial
30+
config (one of issuer/audience set without the other). New
31+
dep: `github.com/coreos/go-oidc/v3` (justified — JWKS
32+
rotation, key id selection, alg validation, claim validation
33+
are non-trivial; rolling our own is a CVE-magnet). 10 unit
34+
tests in [oidc_test.go](cmd/hypercache-server/oidc_test.go)
35+
drive an in-process IdP stub (signs JWTs with a hand-rolled
36+
RSA key, serves discovery + JWKS); 4 integration tests in
37+
[management_http_test.go](tests/management_http_test.go)
38+
verify bearer + OIDC coexistence on the management port.
1139
- **`GET /cluster/events` — SSE stream of topology updates.** New
1240
Server-Sent Events endpoint on the management HTTP port that
1341
pushes `members` (full membership snapshot) and `heartbeat`

‎Makefile‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,27 @@ stop-dev-cluster:
3737
@echo
3838
docker compose -f docker-compose.cluster.yml down -v --rmi local --remove-orphans
3939

40+
# OIDC end-to-end example. The full stack — cache cluster +
41+
# Keycloak + Monitor — is defined and orchestrated from the
42+
# monitor repo's `examples/oidc/`. This target is a thin
43+
# passthrough so cache-repo operators can boot the stack
44+
# without leaving their working tree. The monitor repo must be
45+
# cloned as a sibling: ../hypercache-monitor.
46+
start-oidc:
47+
@if [ ! -f ../hypercache-monitor/examples/oidc/docker-compose.yml ]; then \
48+
echo "expected ../hypercache-monitor/examples/oidc/docker-compose.yml; clone the monitor repo as a sibling"; \
49+
exit 1; \
50+
fi
51+
$(MAKE) -C ../hypercache-monitor start-oidc
52+
53+
stop-oidc:
54+
@if [ ! -f ../hypercache-monitor/examples/oidc/docker-compose.yml ]; then exit 0; fi
55+
$(MAKE) -C ../hypercache-monitor stop-oidc
56+
57+
clean-oidc:
58+
@if [ ! -f ../hypercache-monitor/examples/oidc/docker-compose.yml ]; then exit 0; fi
59+
$(MAKE) -C ../hypercache-monitor clean-oidc
60+
4061
# test-cluster brings up the 5-node docker-compose cluster, waits for
4162
# every node's /healthz to be 200, runs the cross-node smoke test
4263
# (PUT/GET/DELETE asserted on every node), and tears the stack down —
@@ -235,4 +256,5 @@ help:
235256
@echo
236257
@echo "For more information, see the project README."
237258

238-
.PHONY: init prepare-toolchain prepare-base-tools update-toolchain test test-race typecheck build ci bench bench-baseline vet update-deps lint sec help
259+
.PHONY: init prepare-toolchain prepare-base-tools update-toolchain test test-race typecheck build ci bench bench-baseline vet update-deps lint sec help \
260+
start-dev-cluster stop-dev-cluster start-oidc stop-oidc clean-oidc

‎README.md‎

Lines changed: 64 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -29,10 +29,7 @@ It ships with a default [histogram stats collector](./stats/stats.go) and severa
2929

3030
- Thread-safe & lock‑optimized (sharded map + worker pool)
3131
- High-performance (low allocations on hot paths, pooled items, serializer‑aware sizing)
32-
- Multiple backends (extensible):
33-
1. [In-memory](./pkg/backend/inmemory.go)
34-
1. [Redis](./pkg/backend/redis.go)
35-
1. [Redis Cluster](./pkg/backend/redis_cluster.go)
32+
- Multiple backends (extensible): 1. [In-memory](./pkg/backend/inmemory.go) 1. [Redis](./pkg/backend/redis.go) 1. [Redis Cluster](./pkg/backend/redis_cluster.go)
3633
- Item expiration & proactive expiration triggering (debounced/coalesced)
3734
- Background or proactive (interval = 0) eviction using pluggable algorithms
3835
- Manual, non-blocking eviction triggering (`TriggerEviction()`)
@@ -178,32 +175,32 @@ if err != nil {
178175

179176
### Advanced options quick reference
180177

181-
| Option | Purpose |
182-
| -------- | --------- |
183-
| `WithEvictionInterval` | Periodic eviction loop; set to `0` for proactive per-write eviction. |
184-
| `WithExpirationInterval` | Periodic scan for expired items. |
185-
| `WithExpirationTriggerBuffer` | Buffer size for coalesced expiration trigger channel. |
186-
| `WithExpirationTriggerDebounce` | Drop rapid-fire triggers within a window. |
187-
| `WithEvictionAlgorithm` | Select eviction algorithm (lru, lfu, clock, cawolfu, arc*). |
188-
| `WithEvictionShardCount` | Number of eviction-algorithm shards (default 32; 1 disables sharding). |
189-
| `WithMaxEvictionCount` | Cap number of items evicted per cycle. |
190-
| `WithMaxCacheSize` | Max cumulative serialized item size (bytes). |
191-
| `WithStatsCollector` | Choose stats collector implementation. |
192-
| `WithManagementHTTP` | Start optional management HTTP server. |
193-
| `WithDistReplication` | (DistMemory) Set replication factor (owners per key). |
194-
| `WithDistVirtualNodes` | (DistMemory) Virtual nodes per physical node for consistent hashing. |
195-
| `WithDistMerkleChunkSize` | (DistMemory) Keys per Merkle leaf chunk (power-of-two recommended). |
196-
| `WithDistMerkleAutoSync` | (DistMemory) Interval for background Merkle sync (<=0 disables). |
197-
| `WithDistMerkleAutoSyncPeers` | (DistMemory) Limit peers synced per auto-sync tick (0=all). |
198-
| `WithDistListKeysCap` | (DistMemory) Cap number of keys fetched via fallback enumeration. |
199-
| `WithDistNode` | (DistMemory) Explicit node identity (id/address). |
200-
| `WithDistSeeds` | (DistMemory) Static seed addresses to pre-populate membership. |
201-
| `WithDistTombstoneTTL` | (DistMemory) Retain delete tombstones for this duration before compaction (<=0 = infinite). |
202-
| `WithDistTombstoneSweep` | (DistMemory) Interval to run tombstone compaction (<=0 disables). |
203-
| `WithDistHTTPLimits` | (DistMemory) Body / response / timeout / concurrency caps for the dist HTTP server + auto-client. |
204-
| `WithDistHTTPAuth` | (DistMemory) Bearer-token auth (`Token`) plus optional `ServerVerify` / `ClientSign` hooks. |
205-
206-
*ARC is experimental (not registered by default).
178+
| Option | Purpose |
179+
| ------------------------------- | ------------------------------------------------------------------------------------------------- |
180+
| `WithEvictionInterval` | Periodic eviction loop; set to `0` for proactive per-write eviction. |
181+
| `WithExpirationInterval` | Periodic scan for expired items. |
182+
| `WithExpirationTriggerBuffer` | Buffer size for coalesced expiration trigger channel. |
183+
| `WithExpirationTriggerDebounce` | Drop rapid-fire triggers within a window. |
184+
| `WithEvictionAlgorithm` | Select eviction algorithm (lru, lfu, clock, cawolfu, arc\*). |
185+
| `WithEvictionShardCount` | Number of eviction-algorithm shards (default 32; 1 disables sharding). |
186+
| `WithMaxEvictionCount` | Cap number of items evicted per cycle. |
187+
| `WithMaxCacheSize` | Max cumulative serialized item size (bytes). |
188+
| `WithStatsCollector` | Choose stats collector implementation. |
189+
| `WithManagementHTTP` | Start optional management HTTP server. |
190+
| `WithDistReplication` | (DistMemory) Set replication factor (owners per key). |
191+
| `WithDistVirtualNodes` | (DistMemory) Virtual nodes per physical node for consistent hashing. |
192+
| `WithDistMerkleChunkSize` | (DistMemory) Keys per Merkle leaf chunk (power-of-two recommended). |
193+
| `WithDistMerkleAutoSync` | (DistMemory) Interval for background Merkle sync (<=0 disables). |
194+
| `WithDistMerkleAutoSyncPeers` | (DistMemory) Limit peers synced per auto-sync tick (0=all). |
195+
| `WithDistListKeysCap` | (DistMemory) Cap number of keys fetched via fallback enumeration. |
196+
| `WithDistNode` | (DistMemory) Explicit node identity (id/address). |
197+
| `WithDistSeeds` | (DistMemory) Static seed addresses to pre-populate membership. |
198+
| `WithDistTombstoneTTL` | (DistMemory) Retain delete tombstones for this duration before compaction (<=0 = infinite). |
199+
| `WithDistTombstoneSweep` | (DistMemory) Interval to run tombstone compaction (<=0 disables). |
200+
| `WithDistHTTPLimits` | (DistMemory) Body / response / timeout / concurrency caps for the dist HTTP server + auto-client. |
201+
| `WithDistHTTPAuth` | (DistMemory) Bearer-token auth (`Token`) plus optional `ServerVerify` / `ClientSign` hooks. |
202+
203+
\*ARC is experimental (not registered by default).
207204

208205
### Type-safe access (`Typed[V]`)
209206

@@ -280,7 +277,10 @@ limits := backend.DistHTTPLimits{
280277

281278
// 2) Auth: a shared bearer token covers most clusters; ServerVerify /
282279
// ClientSign hooks are escape hatches for JWT, mTLS-derived
283-
// identity, HMAC, etc.
280+
// identity, HMAC, etc. The hypercache-server binary uses the
281+
// ServerVerify hook to plug in an OIDC verifier
282+
// (cmd/hypercache-server/oidc.go) when HYPERCACHE_OIDC_ISSUER is
283+
// set; static-bearer logins coexist with IdP-issued JWTs.
284284
auth := backend.DistHTTPAuth{Token: "shared-cluster-secret"}
285285

286286
bi, _ := backend.NewDistMemory(ctx,
@@ -316,44 +316,44 @@ Replica removal cleanup (actively dropping data from nodes no longer replicas) i
316316

317317
Metrics (via management or `Metrics()`):
318318

319-
| Metric | Description |
320-
| -------- | ------------- |
321-
| RebalancedKeys | Count of all rebalance-related migrations (primary changes + replica diff replications). |
322-
| RebalancedPrimary | Count of primary ownership change migrations (subset of RebalancedKeys). |
323-
| RebalanceBatches | Number of migration batches executed. |
324-
| RebalanceThrottle | Times migration concurrency limiter saturated. |
325-
| RebalanceLastNanos | Duration (ns) of last rebalance scan. |
326-
| RebalancedReplicaDiff | Count of replica-only diff replications (new replicas seeded). |
327-
| RebalanceReplicaDiffThrottle | Times replica-only diff processing hit per-tick cap. |
319+
| Metric | Description |
320+
| ---------------------------- | ---------------------------------------------------------------------------------------- |
321+
| RebalancedKeys | Count of all rebalance-related migrations (primary changes + replica diff replications). |
322+
| RebalancedPrimary | Count of primary ownership change migrations (subset of RebalancedKeys). |
323+
| RebalanceBatches | Number of migration batches executed. |
324+
| RebalanceThrottle | Times migration concurrency limiter saturated. |
325+
| RebalanceLastNanos | Duration (ns) of last rebalance scan. |
326+
| RebalancedReplicaDiff | Count of replica-only diff replications (new replicas seeded). |
327+
| RebalanceReplicaDiffThrottle | Times replica-only diff processing hit per-tick cap. |
328328

329329
Test helpers `AddPeer` and `RemovePeer` simulate join / leave events that trigger redistribution in integration tests (`dist_rebalance_*.go`).
330330

331331
### Roadmap / PRD Progress Snapshot
332332

333-
| Area | Status |
334-
| ------ | -------- |
335-
| Core in-process sharding | Done |
336-
| Replication fan-out | Done |
337-
| Read-repair | Done |
338-
| Quorum consistency (R/W) | Done |
339-
| Hinted handoff (TTL, replay, caps) | Done |
340-
| Tombstones (TTL + compaction) | Done |
341-
| Merkle anti-entropy (pull) | Done |
342-
| Merkle phase metrics | Done |
343-
| Auto Merkle background sync | Done |
333+
| Area | Status |
334+
| ------------------------------------------------------------------ | --------------------------------------------------------- |
335+
| Core in-process sharding | Done |
336+
| Replication fan-out | Done |
337+
| Read-repair | Done |
338+
| Quorum consistency (R/W) | Done |
339+
| Hinted handoff (TTL, replay, caps) | Done |
340+
| Tombstones (TTL + compaction) | Done |
341+
| Merkle anti-entropy (pull) | Done |
342+
| Merkle phase metrics | Done |
343+
| Auto Merkle background sync | Done |
344344
| Rebalancing (primary/lost ownership + replica-add diff + shedding) | Partial (shedding grace delete added; future retry/queue) |
345-
| Failure detection (heartbeat) | Partial (basic) |
346-
| Lightweight gossip snapshot | Partial |
347-
| Replica-only migration diff | Planned |
348-
| Adaptive Merkle scheduling | Planned |
349-
| Advanced versioning (HLC/vector) | Planned |
350-
| Client SDK (direct routing) | Planned |
351-
| Tracing spans | Planned |
352-
| Security (TLS/auth) | Done (since v0.5.0; see "Transport hardening") |
353-
| Compression | Planned |
354-
| Persistence | Out of scope (current phase) |
355-
| Chaos / fault injection | Planned |
356-
| Benchmarks & tests | Ongoing (broad coverage) |
345+
| Failure detection (heartbeat) | Partial (basic) |
346+
| Lightweight gossip snapshot | Partial |
347+
| Replica-only migration diff | Planned |
348+
| Adaptive Merkle scheduling | Planned |
349+
| Advanced versioning (HLC/vector) | Planned |
350+
| Client SDK (direct routing) | Planned |
351+
| Tracing spans | Planned |
352+
| Security (TLS/auth) | Done (since v0.5.0; see "Transport hardening") |
353+
| Compression | Planned |
354+
| Persistence | Out of scope (current phase) |
355+
| Chaos / fault injection | Planned |
356+
| Benchmarks & tests | Ongoing (broad coverage) |
357357

358358
Example minimal setup:
359359

@@ -433,4 +433,4 @@ I'm a surfer, and a software architect with 15 years of experience designing hig
433433
[![LinkedIn](https://img.shields.io/badge/LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white)](https://www.linkedin.com/in/francesco-cosentino/)
434434

435435
[build-link]: https://github.com/hyp3rd/hypercache/actions/workflows/go.yml
436-
[codeql-link]:https://github.com/hyp3rd/hypercache/actions/workflows/codeql.yml
436+
[codeql-link]: https://github.com/hyp3rd/hypercache/actions/workflows/codeql.yml

0 commit comments

Comments
 (0)