security: cache SCRAM password validation#31054
Open
nguyen-andrew wants to merge 4 commits into
Open
Conversation
Measures the per-call cost of the SCRAM password check that HTTP Basic auth runs on every pandaproxy/schema registry/admin request. The salted password is rederived per request (4096 HMAC rounds), which saturated pandaproxy CPU under unbatched HTTP produce in a production incident: ~473us per SHA-256 validation, so a few thousand requests per second exhaust a core before any request work happens. Covers the wrong password (reject) path, which costs the same as an accept, SHA-512 (~3x SHA-256), and a unique-password case that always pays the full derivation.
Exposes the password -> stored_key derivation as a reusable step so a subsequent commit can memoize it.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a per-shard memoization layer for SCRAM password validation to eliminate repeated PBKDF2-style derivations on hot authentication paths (HTTP Basic auth and SASL/PLAIN), controlled via a new cluster configuration kill switch.
Changes:
- Add
security::scram_credential_cacheand integrate it intovalidate_scram_credentialwith athread_localper-shard cache gated byscram_credential_cache_enabled. - Refactor SCRAM algorithm code to expose
derive_stored_key()and reuse it in password validation. - Add unit tests and an rpbench benchmark covering cached vs uncached behavior and correctness.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/v/security/tests/scram_credential_cache_test.cc | New unit tests validating cache hit/miss behavior, key composition, eviction bounds, and cached credential validation. |
| src/v/security/tests/scram_credential_bench.cc | New perf benchmark exercising repeated/unique password validation cases to quantify caching impact. |
| src/v/security/tests/scram_algorithm_test.cc | Adds a test ensuring derive_stored_key() matches stored keys produced by credential creation. |
| src/v/security/tests/BUILD | Wires new test and benchmark targets into the build. |
| src/v/security/scram_credential_cache.h | Declares the per-shard SCRAM derivation cache and its keying/metrics interface. |
| src/v/security/scram_credential_cache.cc | Implements HMAC-keyed cache entries backed by utils::chunked_kv_cache. |
| src/v/security/scram_authenticator.h | Documents cached validation behavior and introduces a detail::validate_scram_credential overload for caller-provided caches. |
| src/v/security/scram_authenticator.cc | Implements cached derivation path and config-gated use of a per-thread cache. |
| src/v/security/scram_algorithm.h | Adds derive_stored_key() and refactors validate_password() to use it. |
| src/v/security/BUILD | Adds the new cache implementation to the security library and links required deps. |
| src/v/config/configuration.h | Declares the new scram_credential_cache_enabled configuration property. |
| src/v/config/configuration.cc | Defines the new configuration property and its description/default. |
Per-shard memoization of SCRAM password derivations, keyed by an HMAC (random per-instance key) of (mechanism, password, salt, iterations) so plaintext passwords are not retained, holding only the derived stored key: the same datum already persisted in the credential store. The mapping is purely functional, so entries never go stale: updating a credential generates a fresh salt, which changes the cache key. Built on the S3-FIFO chunked_kv_cache, whose scan resistance keeps one-shot wrong-password entries from displacing hot legitimate ones.
validate_scram_credential now consults a shard-local scram_credential_cache, skipping the salted password derivation for repeat authentications. HTTP Basic auth (pandaproxy, schema registry, admin API) and SASL/PLAIN pay that derivation on every request; in production, unbatched HTTP produce through pandaproxy at a few hundred requests per second saturated a core on it. Cache hits skip the derivation entirely; the uncached path is unchanged. The scram_credential_cache_enabled cluster config (default true) is an operational kill switch.
Member
Author
|
Force push to address Copilot comments. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
HTTP Basic auth (PandaProxy, Schema Registry, Admin API) and Kafka SASL/PLAIN
rederive the credential's stored SCRAM key on every request: a PBKDF2-style
loop of >=4096 HMAC rounds, run on-reactor before the body is parsed. Under
high-rate authenticated traffic this dominates CPU. A cluster serving a few
hundred HTTP-produce req/s per shard through PandaProxy saturated its reactor
core almost entirely in this path (~45% of on-CPU time in
validate_scram_credential), starving heartbeats.This memoizes the derivation with a bounded per-shard cache, keyed by an HMAC
of (algorithm, password, salt, iterations) and holding only the derived stored
key (already in the credential store). It is purely functional, so a password
change re-salts and re-keys and entries never go stale, and it is gated by the
scram_credential_cache_enabledcluster config (default on) as a kill switch.Fixes CORE-16829
Microbench (
--config=release, single core)Repeat = same credential per call (steady-state auth); unique = fresh password
per call (cache-miss path).
End-to-end (single core, unbatched ~2 KiB HTTP produce)
After the fix, Basic-auth throughput matches the no-auth baseline. Disabling the
cache (
scram_credential_cache_enabled=false) drops it back to 298 req/s,confirming the kill switch reproduces pre-fix behavior.
Backports Required
Release Notes