|
| 1 | +# GraphCompose Benchmarks Module |
| 2 | + |
| 3 | +> **What this is.** A **manual performance harness** for GraphCompose — |
| 4 | +> a small set of Java programs that render representative documents |
| 5 | +> repeatedly and report rough numbers (latency, throughput, byte size, |
| 6 | +> peak memory) to a JSON / CSV / text report. |
| 7 | +> |
| 8 | +> **What this is _not_.** A JMH-grade benchmark. There is no warmup |
| 9 | +> control, no forked JVM, no per-measurement reset, no GC profiling |
| 10 | +> beyond what JFR / `-verbose:gc` can pick up out-of-band. Numbers |
| 11 | +> produced here are **rough local comparisons** suitable for "did this |
| 12 | +> change regress something obviously?" — not for public marketing |
| 13 | +> claims, not for cross-machine performance comparisons, and not for |
| 14 | +> answering "how does GraphCompose compare to iText / openHTMLToPDF / |
| 15 | +> JasperReports?" with rigour. |
| 16 | +> |
| 17 | +> A separate JMH layer (sibling chain Track C: B3 → B4 → B5 → B6 in the |
| 18 | +> 1.7.0 plan) will sit alongside this harness when it lands. Until |
| 19 | +> then, treat these numbers as **smoke-test fidelity, not benchmark |
| 20 | +> fidelity**. |
| 21 | +
|
| 22 | +## When to use the harness |
| 23 | + |
| 24 | +- **Smoke check before a release** — `CurrentSpeedBenchmark -Dgraphcompose.benchmark.profile=smoke` |
| 25 | + takes ~15 s, exercises the canonical render path through 5 fixture |
| 26 | + scenarios, and prints a single-page latency / throughput table. |
| 27 | + CI runs this on every PR (the `perf-smoke` job); the goal is "did |
| 28 | + this PR make a representative render visibly slower?" — *not* "is |
| 29 | + this number a publishable performance claim". |
| 30 | + |
| 31 | +- **Pre/post comparison on a single machine** — render a fixture |
| 32 | + before and after a layout change, run `BenchmarkDiffTool` against |
| 33 | + the two JSON reports, eyeball the delta. Variance per run is in |
| 34 | + single-digit percent; treat deltas inside ±5 % as noise on the |
| 35 | + default machine and tighten the threshold only when comparing on a |
| 36 | + quiescent system with a fixed CPU frequency. |
| 37 | + |
| 38 | +- **Stress / endurance check** — `GraphComposeStressTest` and |
| 39 | + `EnduranceTest` drive higher-cardinality fixtures over longer |
| 40 | + windows to catch GC pressure spikes or memory leaks that a single |
| 41 | + smoke run wouldn't surface. Run by hand; not on CI by default. |
| 42 | + |
| 43 | +## When **not** to use the harness |
| 44 | + |
| 45 | +- For a **published "X% faster than Y" claim** of any kind — the |
| 46 | + numbers are not statistically rigorous and the comparison setup is |
| 47 | + not reproducible across machines / JDKs. |
| 48 | +- For **deciding between two architecturally different approaches** — |
| 49 | + pick the right invariant (allocation count, big-O of the algorithm, |
| 50 | + layout-pass count) and reason about it; the harness is a sanity |
| 51 | + check after you've already chosen, not a decision tool before. |
| 52 | +- For **comparing GraphCompose to another PDF library** — |
| 53 | + `ComparativeBenchmark` does render the same fixture through iText / |
| 54 | + openHTMLToPDF / JasperReports for rough sizing, but the comparison |
| 55 | + is a manual smoke test: each library has different defaults |
| 56 | + (compression, font embedding, image resampling) and reading too much |
| 57 | + into a single number is the wrong call. |
| 58 | + |
| 59 | +## Files in this module |
| 60 | + |
| 61 | +| File | Role | |
| 62 | +|---|---| |
| 63 | +| `CurrentSpeedBenchmark` | Default scenario runner — what CI's `perf-smoke` job exercises. Takes a `-Dgraphcompose.benchmark.profile=smoke\|full\|stress` switch. | |
| 64 | +| `ComparativeBenchmark` | Renders the same fixtures through GraphCompose, iText, openHTMLToPDF, JasperReports. **Rough local comparison only** — see "When not to use" above. | |
| 65 | +| `FullCvBenchmark`, `ScalabilityBenchmark` | Fixture-specific runners for CV and table-heavy scenarios. | |
| 66 | +| `CanonicalBenchmarkSupport`, `BenchmarkSupport` | Shared fixture builders + measurement helpers. | |
| 67 | +| `BenchmarkReportWriter` | Writes JSON / CSV / text reports under `benchmarks/target/benchmarks/`. | |
| 68 | +| `BenchmarkDiffTool` | Compares two JSON reports and prints a delta table. Useful for pre/post comparisons. | |
| 69 | +| `BenchmarkMedianTool` | Median + dispersion across N runs of the same scenario. | |
| 70 | +| `GraphComposeStressTest`, `EnduranceTest` | Long-running stress / endurance harnesses. | |
| 71 | +| `GraphComposeBenchmark` | Legacy entry point preserved for one downstream caller. New work should target `CurrentSpeedBenchmark`. | |
| 72 | + |
| 73 | +## Running |
| 74 | + |
| 75 | +From the repo root: |
| 76 | + |
| 77 | +```bash |
| 78 | +# Smoke profile (~15s) — what CI runs on every PR |
| 79 | +./mvnw -B -ntp -f benchmarks/pom.xml -DskipTests \ |
| 80 | + exec:java \ |
| 81 | + -Dexec.mainClass=com.demcha.compose.CurrentSpeedBenchmark \ |
| 82 | + -Dgraphcompose.benchmark.profile=smoke |
| 83 | + |
| 84 | +# Diff two existing report runs under the same scenario |
| 85 | +./mvnw -B -ntp -f benchmarks/pom.xml -DskipTests \ |
| 86 | + exec:java \ |
| 87 | + -Dexec.mainClass=com.demcha.compose.BenchmarkDiffTool \ |
| 88 | + -Dexec.args="current-speed" |
| 89 | +``` |
| 90 | + |
| 91 | +Reports land in `benchmarks/target/benchmarks/<scenario>/`. The CI |
| 92 | +`perf-smoke` job uploads the smoke directory as an artifact for every |
| 93 | +PR run, so a regression can be diffed against the previous PR's run |
| 94 | +without reproducing locally. |
| 95 | + |
| 96 | +## How to read a report |
| 97 | + |
| 98 | +The JSON shape is intentionally simple — a top-level run record with |
| 99 | +per-scenario sub-records. Each sub-record carries: |
| 100 | + |
| 101 | +- `avgMs`, `p50Ms`, `p95Ms`, `maxMs` — latency distribution across |
| 102 | + iterations within the run. |
| 103 | +- `docsPerSec` — rough throughput; **not statistically rigorous**, |
| 104 | + intended only as a relative number against a sibling scenario or a |
| 105 | + previous run on the same machine. |
| 106 | +- `avgKB` — average output byte size. Stable across runs on the same |
| 107 | + fixture; useful for catching content corruption (size shifts by |
| 108 | + > a few hundred bytes are usually a bug, not a benchmark fluctuation). |
| 109 | +- `peakMB` — peak heap as observed by `MemoryMXBean`; coarse, do not |
| 110 | + use for memory-budget enforcement. |
| 111 | + |
| 112 | +## Roadmap |
| 113 | + |
| 114 | +The 1.7.0 plan (Track C, B3 → B4 → B5 → B6) introduces a sibling JMH |
| 115 | +layer: |
| 116 | + |
| 117 | +- **B3** — pull fixtures into a `fixtures/` package with deterministic |
| 118 | + seeds so the JMH layer can reuse them. |
| 119 | +- **B4** — JMH infrastructure (`jmh-core`, `jmh-generator-annprocess`, |
| 120 | + shade plugin) + first benchmark (`SimpleDocumentJmhBenchmark`). |
| 121 | +- **B5** — Invoice / CV / LargeTable / PdfRender JMH benchmarks. |
| 122 | +- **B6** — CI job that runs the JMH layer on a `workflow_dispatch` / |
| 123 | + weekly cadence and uploads `*.json` reports as artifacts. |
| 124 | + |
| 125 | +Once that chain is in place, any *public* performance claim should |
| 126 | +quote the JMH layer's numbers, with explicit warmup / measurement / |
| 127 | +fork configuration in the source. This manual harness will stay for |
| 128 | +the smoke / diff / endurance roles described above. |
| 129 | + |
| 130 | +--- |
| 131 | + |
| 132 | +*This page is the source of truth for what the manual benchmark layer |
| 133 | +is and is not. When in doubt — and especially before quoting a number |
| 134 | +in a public communication — re-read the "When not to use" section.* |
0 commit comments