Skip to content

Commit 43237b3

Browse files
bm1549claude
andcommitted
Add memory-pressure benchmark for petclinic throughput testing
Adds a new benchmark category under benchmark/memory/petclinic/ that measures tracer throughput overhead across decreasing heap sizes using spring-petclinic 3.3.0 and JMeter. Based on Doug's methodology for evaluating memory overhead of the Java tracer agent. The script auto-downloads JMeter and builds petclinic, iterates over configurable heap sizes (default: 256/192/128/96/80/64 MiB), runs each with and without the agent, and outputs a comparison table with throughput delta. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 5ab378f commit 43237b3

3 files changed

Lines changed: 638 additions & 0 deletions

File tree

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
tools/
2+
results/
Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
# Memory Pressure Benchmark
2+
3+
Measures tracer throughput overhead across decreasing heap sizes using spring-petclinic and JMeter.
4+
5+
Based on Doug's methodology: run petclinic with `GET /owners/3` (DB-backed endpoint) at heap sizes from 256m down to 64m, comparing no-agent baseline vs candidate agent.
6+
7+
## Quick start
8+
9+
```bash
10+
# Full run: baseline + candidate at all heap sizes
11+
./run.sh
12+
13+
# With a specific agent jar
14+
./run.sh --agent /path/to/dd-java-agent.jar
15+
16+
# Quick test with fewer heap sizes and shorter durations
17+
./run.sh --heap-sizes 256,128,64 --warmup 15 --measure 30
18+
19+
# With JFR profiling
20+
./run.sh --jfr
21+
22+
# Only run candidate (skip baseline if you already have baseline numbers)
23+
./run.sh --skip-baseline
24+
25+
# Test a specific optimization with extra agent opts
26+
./run.sh --agent-opts "-Ddd.appsec.enabled=false -Ddd.profiling.enabled=false"
27+
```
28+
29+
## What it does
30+
31+
For each heap size (default: 256, 192, 128, 96, 80, 64 MiB):
32+
33+
1. Starts petclinic with `-Xms<N>m -Xmx<N>m` (no agent)
34+
2. Runs JMeter with 8 threads for warmup + measurement
35+
3. Records stabilized throughput (measurement window only)
36+
4. Repeats with `-javaagent:dd-java-agent.jar`
37+
5. If petclinic OOMs or fails to start, records "OOM"
38+
39+
## Output
40+
41+
```
42+
Heap Baseline Candidate Delta
43+
------ ----------- ----------- -----------
44+
256m 22400 req/s 19500 req/s -12.9%
45+
192m 21750 req/s 19000 req/s -12.6%
46+
128m 21500 req/s 14750 req/s -31.4%
47+
96m 16750 req/s 13000 req/s -22.4%
48+
80m 16350 req/s 11250 req/s -31.2%
49+
64m 15000 req/s 7500 req/s -50.0%
50+
```
51+
52+
Results are also saved as CSV in `results/<timestamp>/summary.csv` with per-run JTL files and logs.
53+
54+
## Dependencies
55+
56+
Auto-downloaded on first run:
57+
- **JMeter 5.6.3**`tools/apache-jmeter-5.6.3/`
58+
- **spring-petclinic 3.3.0**`tools/spring-petclinic/` (built with Maven)
59+
- **dd-java-agent** → auto-detected from `dd-java-agent/build/libs/` or built via Gradle
60+
61+
Requires: Java 17+, Python 3 (for JTL parsing), curl, git.
62+
63+
## Comparing versions
64+
65+
To compare two tracer versions, run separately and diff the CSVs:
66+
67+
```bash
68+
# Run with release version
69+
./run.sh --agent /path/to/dd-java-agent-1.61.0.jar --output ./results/v1.61.0
70+
71+
# Run with candidate
72+
./run.sh --agent /path/to/dd-java-agent-SNAPSHOT.jar --output ./results/candidate
73+
74+
# Compare
75+
paste -d, results/v1.61.0/summary.csv results/candidate/summary.csv
76+
```

0 commit comments

Comments
 (0)