Add JMH benchmarks (CAE-2930)#4523
Open
ggivo wants to merge 36 commits into
Open
Conversation
🛡️ Jit Security Scan Results
✅ No security findings were detected in this PR
Security scan by Jit
|
Test Results 191 files ± 0 191 suites ±0 9m 25s ⏱️ -26s Results for commit 7b57c57. ± Comparison against base commit 2a2c46c. This pull request removes 14 and adds 2 tests. Note that renamed tests count towards both.This pull request skips 630 tests.♻️ This comment has been updated with latest results. |
ggivo
force-pushed
the
topic/ggivo/CAE-2930-setup-benchmark-nightly
branch
from
df14ad1 to
5a01eab
Compare
…0-setup-benchmark-nightly
migrate PipelinedGetSetBenchmark.java
Workload benchmark: mixed GET/SET at 90% read / 10% write. Provides a single comparable throughput number per client configuration:
…ound-trip throughput
ggivo
changed the title
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
uglide
reviewed
May 14, 2026
Contributor
uglide
left a comment
uglide
left a comment
There was a problem hiding this comment.
Good initiative, but needs some polishing.
| run: | | ||
| docker run --name redis-benchmark-test \ | ||
| -p 6379:6379 \ | ||
| -d redis:8.6.3 \ |
Contributor
There was a problem hiding this comment.
Better to use without a patch :
Suggested change
| -d redis:8.6.3 \ | |
| -d redis:8.6 \ |
Collaborator
Author
There was a problem hiding this comment.
Do you mean 8.6.0?
I think benchmarks should run against pinned version
Contributor
There was a problem hiding this comment.
Then its better to use 8.2, because it's an LTS version
Collaborator
Author
There was a problem hiding this comment.
WDYT about pinned to 8.2.6 (latest 8.2), make sure we don't pick patch releases
| * Recreate streams before each benchmark invocation. This is necessary because streams get | ||
| * consumed during reading. | ||
| */ | ||
| @Setup(Level.Invocation) |
Contributor
There was a problem hiding this comment.
I'm not a big expert on JMH, but the docs say that:
This level is only usable for benchmarks taking more than a millisecond per single Benchmark method invocation. It is a good idea to validate the impact for your case on ad-hoc basis as well.
So, for ProtocolRead, we need some kind of batching.
Contributor
There was a problem hiding this comment.
Implemented the batching approach you described. Summary:
- Removed
@Setup(Level.Invocation) recreateStreams()and the eight pre-allocatedRedisInputStreamfields. - Pre-allocated only the source
byte[]s (the cheap part). A smallstream(byte[])helper wraps a freshRedisInputStreamper op — required becauseRedisInputStreamretains internalbuf/count/limitstate and has no public reset. - Each
@Benchmarkis now a tightBATCH_SIZE-loop with@OperationsPerInvocation(BATCH_SIZE)(BATCH_SIZE = 100). - Applied the same batching to
encodeSetCommandfor consistency.
Trade-off worth flagging: stream construction now lives inside the measured op rather than excluded via Setup. Reported ns/op will be slightly higher than before, but the measurement is jitter-free and JMH no longer emits the sub-microsecond invocation-setup warning. I think that's the right call — Setup-excluded numbers were optimistic anyway since the JMH machinery transitions between setup and measurement aren't free.
Verified locally: compiles under mvn -Pjmh test-compile with JDK 8 (Azul Zulu 1.8.0_422). Formatter already applied.
GitHub can't render this as a one-click suggestion block here because the parent comment is anchored to a single diff position rather than a multi-line range — pasting the full new file body for review:
Full ProtocolReadBenchmark.java (lines 18 → EOF)
/**
* Comprehensive JMH Benchmark for Jedis Protocol operations. This benchmark covers: 1. Baseline
* protocol operations (parsing, encoding) 2. Cache-aware protocol reads (push notification
* overhead) 3. Push invalidation message processing at scale (1, 10, 100 pattern) 4. Realistic
* mixed scenarios (push messages + regular responses) Run with: mvn -Pjmh clean test
* <p>
* Each benchmark wraps a {@code BATCH_SIZE} loop, with {@link OperationsPerInvocation} declaring
* the per-invocation op count. Streams are constructed inside the loop from pre-allocated byte
* arrays — {@link RedisInputStream} retains internal read state, so a fresh wrapper is required per
* op. This pattern replaces {@code @Setup(Level.Invocation)}, which JMH flags as unreliable for
* sub-microsecond operations.
*/
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@State(Scope.Benchmark)
@Fork(1)
@Warmup(iterations = 5, time = 2, timeUnit = TimeUnit.SECONDS)
@Measurement(iterations = 5, time = 2, timeUnit = TimeUnit.SECONDS)
@Threads(1)
public class ProtocolReadBenchmark {
private static final int BATCH_SIZE = 100;
// ========== BASELINE PROTOCOL DATA ==========
private static final byte[] KEY = "123456789".getBytes();
private static final byte[] VAL = "FooBar".getBytes();
private static final byte[] SIMPLE_STRING_BYTES = "+OK\r\n".getBytes();
private static final byte[] BULK_STRING_BYTES = "$5\r\nHello\r\n".getBytes();
private static final byte[] ARRAY_BYTES = "*3\r\n$3\r\nfoo\r\n$3\r\nbar\r\n$3\r\nbaz\r\n"
.getBytes();
private static final byte[] MULTI_BULK_BYTES = "*4\r\n$3\r\nfoo\r\n$13\r\nbarbarbarfooz\r\n$5\r\nHello\r\n$5\r\nWorld\r\n"
.getBytes();
// ========== PUSH INVALIDATION DATA ==========
// >2\r\n$10\r\ninvalidate\r\n*1\r\n$3\r\nkey\r\n
private byte[] singlePushBytes;
// >2\r\n$10\r\ninvalidate\r\n*100\r\n... (100 keys)
private byte[] largePushBytes;
// Mixed: N pushes followed by a regular response
private byte[] mixedSinglePushBytes;
private byte[] mixedTenPushBytes;
private byte[] mixedHundredPushBytes;
// Cache for testing
private Cache cache;
@Setup
public void setup() {
// Single push with 1 key
singlePushBytes = ">2\r\n$10\r\ninvalidate\r\n*1\r\n$3\r\nkey\r\n".getBytes();
// Large push with 100 keys invalidated at once
StringBuilder largePush = new StringBuilder(">2\r\n$10\r\ninvalidate\r\n*100\r\n");
for (int i = 0; i < 100; i++) {
String key = "key:" + i;
largePush.append("$").append(key.length()).append("\r\n").append(key).append("\r\n");
}
largePushBytes = largePush.toString().getBytes();
// 1 push + response
mixedSinglePushBytes = ">2\r\n$10\r\ninvalidate\r\n*1\r\n$3\r\nkey\r\n+OK\r\n".getBytes();
// 10 pushes + response
StringBuilder mixed10 = new StringBuilder();
for (int i = 0; i < 10; i++) {
mixed10.append(">2\r\n$10\r\ninvalidate\r\n*1\r\n$4\r\nkey").append(i).append("\r\n");
}
mixed10.append("+OK\r\n");
mixedTenPushBytes = mixed10.toString().getBytes();
// 100 pushes + response (heavy invalidation scenario)
StringBuilder mixed100 = new StringBuilder();
for (int i = 0; i < 100; i++) {
mixed100.append(">2\r\n$10\r\ninvalidate\r\n*1\r\n$5\r\nkey").append(String.format("%02d", i))
.append("\r\n");
}
mixed100.append("+OK\r\n");
mixedHundredPushBytes = mixed100.toString().getBytes();
cache = CacheFactory.getCache(
CacheConfig.builder().maxSize(10000).cacheable(DefaultCacheable.INSTANCE).build());
}
private static RedisInputStream stream(byte[] bytes) {
return new RedisInputStream(new ByteArrayInputStream(bytes));
}
// ========== BASELINE PROTOCOL OPERATIONS (NO CACHE) ==========
/**
* Baseline: Read simple string without cache.
*/
@Benchmark
@OperationsPerInvocation(BATCH_SIZE)
public void readSimpleString(Blackhole blackhole) throws Exception {
for (int i = 0; i < BATCH_SIZE; i++) {
blackhole.consume(Protocol.read(stream(SIMPLE_STRING_BYTES)));
}
}
/**
* Baseline: Read bulk string without cache.
*/
@Benchmark
@OperationsPerInvocation(BATCH_SIZE)
public void readBulkString(Blackhole blackhole) throws Exception {
for (int i = 0; i < BATCH_SIZE; i++) {
blackhole.consume(Protocol.read(stream(BULK_STRING_BYTES)));
}
}
/**
* Baseline: Read array without cache.
*/
@Benchmark
@OperationsPerInvocation(BATCH_SIZE)
public void readArray(Blackhole blackhole) throws Exception {
for (int i = 0; i < BATCH_SIZE; i++) {
blackhole.consume(Protocol.read(stream(ARRAY_BYTES)));
}
}
/**
* Baseline: Read multi-bulk response without cache.
*/
@Benchmark
@OperationsPerInvocation(BATCH_SIZE)
public void readMultiBulkResponse(Blackhole blackhole) throws Exception {
for (int i = 0; i < BATCH_SIZE; i++) {
blackhole.consume(Protocol.read(stream(MULTI_BULK_BYTES)));
}
}
/**
* Encode SET command (baseline protocol performance).
*/
@Benchmark
@OperationsPerInvocation(BATCH_SIZE)
public void encodeSetCommand(Blackhole blackhole) throws Exception {
for (int i = 0; i < BATCH_SIZE; i++) {
RedisOutputStream out = new RedisOutputStream(new ByteArrayOutputStream(8192));
Protocol.sendCommand(out, new CommandArguments(Protocol.Command.SET).key(KEY).add(VAL));
blackhole.consume(out);
}
}
// ========== CACHE-AWARE READS: Overhead of checking for push messages ==========
/**
* Cache-aware: Read simple string with cache (checks for push messages but none present). Compare
* to readSimpleString to measure overhead.
*/
@Benchmark
@OperationsPerInvocation(BATCH_SIZE)
public void cacheAwareReadSimpleString(Blackhole blackhole) throws Exception {
for (int i = 0; i < BATCH_SIZE; i++) {
blackhole.consume(Protocol.read(stream(SIMPLE_STRING_BYTES), cache));
}
}
/**
* Cache-aware: Read bulk string with cache (checks for push messages but none present). Compare
* to readBulkString to measure overhead.
*/
@Benchmark
@OperationsPerInvocation(BATCH_SIZE)
public void cacheAwareReadBulkString(Blackhole blackhole) throws Exception {
for (int i = 0; i < BATCH_SIZE; i++) {
blackhole.consume(Protocol.read(stream(BULK_STRING_BYTES), cache));
}
}
/**
* Cache-aware: Read array with cache (checks for push messages but none present). Compare to
* readArray to measure overhead.
*/
@Benchmark
@OperationsPerInvocation(BATCH_SIZE)
public void cacheAwareReadArray(Blackhole blackhole) throws Exception {
for (int i = 0; i < BATCH_SIZE; i++) {
blackhole.consume(Protocol.read(stream(ARRAY_BYTES), cache));
}
}
/**
* Cache-aware: Read multi-bulk response with cache (checks for push messages but none present).
* Compare to readMultiBulkResponse to measure overhead.
*/
@Benchmark
@OperationsPerInvocation(BATCH_SIZE)
public void cacheAwareReadMultiBulkResponse(Blackhole blackhole) throws Exception {
for (int i = 0; i < BATCH_SIZE; i++) {
blackhole.consume(Protocol.read(stream(MULTI_BULK_BYTES), cache));
}
}
// ========== PUSH MESSAGE PROCESSING: Direct invalidation overhead ==========
/**
* Process single push invalidation message (1 key). Measures cost of processing one cache
* invalidation.
*/
@Benchmark
@OperationsPerInvocation(BATCH_SIZE)
public void processSinglePushInvalidation(Blackhole blackhole) throws Exception {
for (int i = 0; i < BATCH_SIZE; i++) {
blackhole.consume(Protocol.readPushes(stream(singlePushBytes), cache, true));
}
}
/**
* Process large push invalidation (100 keys in single message). Simulates mass invalidation event
* (e.g., FLUSHDB on tracked keys).
*/
@Benchmark
@OperationsPerInvocation(BATCH_SIZE)
public void processLargePushInvalidation(Blackhole blackhole) throws Exception {
for (int i = 0; i < BATCH_SIZE; i++) {
blackhole.consume(Protocol.readPushes(stream(largePushBytes), cache, true));
}
}
// ========== MIXED STREAMS: Realistic scenarios with push + response ==========
/**
* Read response preceded by 1 push invalidation. Most common realistic scenario.
*/
@Benchmark
@OperationsPerInvocation(BATCH_SIZE)
public void readWith1PushMessage(Blackhole blackhole) throws Exception {
for (int i = 0; i < BATCH_SIZE; i++) {
blackhole.consume(Protocol.read(stream(mixedSinglePushBytes), cache));
}
}
/**
* Read response preceded by 10 push invalidations. Moderate invalidation burst before response.
*/
@Benchmark
@OperationsPerInvocation(BATCH_SIZE)
public void readWith10PushMessages(Blackhole blackhole) throws Exception {
for (int i = 0; i < BATCH_SIZE; i++) {
blackhole.consume(Protocol.read(stream(mixedTenPushBytes), cache));
}
}
/**
* Read response preceded by 100 push invalidations. Heavy invalidation scenario (worst-case for
* read latency).
*/
@Benchmark
@OperationsPerInvocation(BATCH_SIZE)
public void readWith100PushMessages(Blackhole blackhole) throws Exception {
for (int i = 0; i < BATCH_SIZE; i++) {
blackhole.consume(Protocol.read(stream(mixedHundredPushBytes), cache));
}
}
}
Collaborator
Author
There was a problem hiding this comment.
Will try to avoid recreating the stream on each read, so let me try an alternative approach.
Work on batches, but pre-populate the stream with BATCH*RESPONSES,
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 5045cc4. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
JMH benchmark migration & expansion
Migrates legacy manual benchmarks to a unified JMH suite and adds new workload and pub/sub coverage.
Added benchmarks
CRC16Benchmark— hash slot CRC16 computation (string + byte[] paths).SafeEncoderBenchmark— UTF-8 encode/decode hot paths.ProtocolReadBenchmark— RESP2/RESP3 parser throughput, including cache-aware and push-frame variants.JedisGetSetBenchmark— single-threaded GET/SET + pipelined GET/SET baseline usingJedis.RedisClientGetSetBenchmark— pooled GET/SET + pipelined GET/SET usingRedisClientat T1 / T8 / T64.GetSetMixedR90W10Benchmark— fixed 90/10 read/write workload over a 1000-key working set; emits comparable troughput numbers forJedis(T1),JedisPool(T1/T8),RedisClient(T1/T8), andRedisClient+ client-side che (T1/T8). Replaces the legacyPoolBenchmarkandRedisClientCSCBenchmark.PubSubPushBenchmark— end-to-endpublish → onMessageround-trip throughput, complementing the protocolReadBenchmark` push-frame microbenchmark.Infrastructure
JmhMainIDE launcher with per-suite include patterns.jmhprofile wiressrc/test/jmhas an additional test-source root and runs JMH with JSON result output.Example CI run
https://github.com/redis/jedis/actions/runs/25842479419/job/75930489878
Note
Medium Risk
Adds a scheduled GitHub Action that runs benchmarks in CI, starts Redis in Docker, and auto-pushes results to the
benchmark-databranch; misconfiguration could impact CI stability or repo branches. Code changes are mostly test/benchmark-only but execute networked Redis workloads that flush DB during runs.Overview
Adds a new JMH benchmark suite under
src/test/jmh(protocol parsing/encoding, CRC16, UTF-8 encoding, GET/SET + pipelining forJedisandRedisClient, mixed 90/10 workload comparisons, and pub/sub round-trip throughput), plus an IDE launcher (JmhMain) and supporting utilities.Updates
pom.xmlto include JMH dependencies and ajmhMaven profile that compiles JMH sources, skips normal tests, and runsorg.openjdk.jmh.Mainto emitbenchmarks.json.Replaces the placeholder benchmark workflow with a nightly + manual GitHub Action that provisions Redis via Docker, runs
mvn -Pjmh, and auto-pushes results to thebenchmark-databranch;docs.ymlnow builds docs frommasterand embeds the benchmark dashboard frombenchmark-datainto the published site.Reviewed by Cursor Bugbot for commit 7b57c57. Bugbot is set up for automated code reviews on this repo. Configure here.