Skip to content

README.md

Latest commit

 

History

History
140 lines (109 loc) · 5.49 KB

README.md

File metadata and controls

140 lines (109 loc) · 5.49 KB

benchmarks

JMH benchmarks comparing Modbus TCP client implementations: j2mod, modbus4j, and mymodbus FFM.

Prerequisites

  • JDK 25 with --enable-preview
  • Docker (for the pymodbus Testcontainer used by cross-library benchmarks)

Build

Important: Do NOT use -am (also-make) when building benchmarks. The -am flag recompiles mymodbus without JEXTRACT_BIN, stripping jextract-generated classes from the fat JAR.

# 1. Install mymodbus first (requires JEXTRACT_BIN)
JEXTRACT_BIN=/path/to/jextract mvn install -pl mymodbus -DskipTests

# 2. Build benchmarks fat JAR (without -am!)
mvn clean package -pl benchmarks -Pbenchmark -DskipTests

Run

# Full benchmark suite (requires Docker)
java --enable-preview --enable-native-access=ALL-UNNAMED \
     -jar benchmarks/target/benchmarks.jar \
     -rf json -rff results.json

# Quick smoke test
java --enable-preview --enable-native-access=ALL-UNNAMED \
     -jar benchmarks/target/benchmarks.jar \
     -wi 1 -i 1 -f 1

# Specific benchmark
java --enable-preview --enable-native-access=ALL-UNNAMED \
     -jar benchmarks/target/benchmarks.jar \
     "CrossLibraryRead.*" -p impl=J2MOD -p quantity=125

Benchmarks

Class What it measures Requires Docker
CrossLibraryReadBenchmark readHoldingRegisters latency per library and payload size (1, 32, 125 registers) Yes
CrossLibraryWriteBenchmark writeMultipleRegisters latency per library and payload size Yes
CrossLibraryReadConcurrentBenchmark Concurrent read scalability (4 threads, per-thread connections) Yes
InternalOverheadBenchmark mymodbus session/wrapper overhead using FakeModbusNative (no network) No

Architecture

es.omarall.bench/
  adapters/
    BenchModbusClient.java       Common interface for all implementations
    AdapterFactory.java          Factory creating clients by implementation name
    J2modAdapter.java            j2mod wrapper
    Modbus4jAdapter.java         modbus4j wrapper
    MymodbusAdapter.java         mymodbus FFM wrapper
  infra/
    PymodbusContainer.java       Testcontainers pymodbus server
  CrossLibraryReadBenchmark.java
  CrossLibraryWriteBenchmark.java
  CrossLibraryReadConcurrentBenchmark.java
  InternalOverheadBenchmark.java

Report Generation

After running benchmarks with -rf json -rff results.json, generate a rigorous statistical report:

# Build fat JAR (if not already built — see Build section above)

# Generate report
java --enable-preview -Djava.awt.headless=true \
     -cp benchmarks/target/benchmarks.jar \
     es.omarall.bench.report.BenchmarkReportGenerator \
     --input results.json --output report/

This produces:

  • report/BENCHMARK_REPORT.md — full Markdown report with statistical analysis
  • report/charts/ — PNG charts (box plots, bar charts with error bars, scaling charts)

CLI Options

Option Default Description
--input <file> (required) JMH JSON results file
--output <dir> (required) Output directory for report + charts
--alpha <value> 0.05 Significance level
--bootstrap-iterations <n> 10000 Bootstrap resampling iterations for percentile CIs

What the report includes

  • Descriptive statistics: mean, median, stdDev, CV, percentiles (p50/p95/p99/p99.9)
  • Bootstrap confidence intervals: 10,000-resample CIs for tail latencies (not just the mean)
  • Normality testing: Kolmogorov-Smirnov test against fitted normal
  • Pairwise comparisons: Welch's t-test + Mann-Whitney U (parametric + non-parametric)
  • Effect sizes: Cohen's d + Cliff's delta (magnitude, not just significance)
  • Multiple testing correction: Bonferroni-adjusted alpha
  • Cross-fork consistency: CV of per-fork means, flagged if > 10%
  • Charts: box-and-whisker plots, p99 bars with bootstrap CIs, payload scaling lines, VT vs PLATFORM grouped bars

Report architecture

es.omarall.bench.report/
  BenchmarkReportGenerator.java       main(), CLI, orchestrates pipeline
  model/
    JmhResult.java                    Parsed JMH JSON entry
    BenchmarkComparison.java          Pairwise comparison result
    StatsSummary.java                 Descriptive stats + bootstrap CIs
  parse/
    JmhJsonParser.java               Gson: JSON → List<JmhResult>
  stats/
    DistributionAnalyzer.java         Descriptive stats, bootstrap CIs, normality, fork consistency
    ComparisonAnalyzer.java           Welch t-test, Mann-Whitney U, Cohen's d, Cliff's delta, Bonferroni
  chart/
    ChartGenerator.java               Box plots, bar charts, scaling lines → PNG
    ChartTheme.java                   Color palette, fonts, dimensions
  markdown/
    MarkdownReportWriter.java         Markdown report assembly

Implementation Variants

The cross-library benchmarks compare four implementations via the impl @Param:

impl value Library Threading Notes
J2MOD j2mod Platform threads Pure-Java Modbus, reference implementation
MODBUS4J modbus4j Platform threads Pure-Java Modbus
MYMODBUS_FFM_VT mymodbus (FFM/libmodbus) Virtual threads Default mymodbus configuration
MYMODBUS_FFM_PT mymodbus (FFM/libmodbus) Platform threads For VT vs PT comparison

The InternalOverheadBenchmark uses a separate mode param (VT / PLATFORM) with a FakeModbusNative (no network) to isolate session/wrapper overhead.

Requirements for mymodbus variants: JEXTRACT_BIN env var set and libmodbus-dev installed.