rust-compile-bench

A comprehensive benchmark suite for measuring Rust compilation performance across multiple projects and optimization strategies.

Features

• 🦀 Three reference projects – small (embedded/generated code), medium (ripgrep), large (rust-analyzer)
• 🤖 Auto-detect mode – point at any Rust project; the tool analyses it and picks sensible parameters
• 💾 YAML config files – save/reload any benchmark configuration
• ⚡ 13 built-in optimization profiles – baseline, incremental, sccache, LTO, codegen-units, parallel frontend, Cranelift, mold linker, nightly, and more
• 🔁 N runs with optional warmup – discard warmup runs to reduce noise
• 🧙 Interactive wizard – guided configuration via terminal prompts
• 🚀 Direct CLI – all options available as command-line flags
• 📊 HTML reports with Chart.js diagrams – build time comparison, crate timings, per-project analysis
• 📦 JSON result storage – re-generate reports without re-running benchmarks

🤝 Contributing via AI assistants? See AGENTS.md for guidance for GitHub Copilot, Claude, and other coding agents.

---

Quick Start

Prerequisites

# Required
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh   # Rust
# Go 1.21+: https://go.dev/dl/

# Optional
cargo install sccache          # for sccache profile
rustup toolchain install nightly  # for nightly profiles
# mold linker: https://github.com/rui314/mold (for mold profile)

Build the CLI tool

cd bench
go build -o bench-tool .
# or: go run main.go [command]

Check dependencies

bash scripts/check-deps.sh

---

Usage

Interactive wizard

cd bench && go run main.go wizard

The wizard guides you through selecting projects, optimization profiles, run counts, and output options.

Direct CLI

# List available projects and profiles
go run main.go list

# Quick smoke test
go run main.go run --projects small --profiles baseline --runs 1 --warmup 0

# Compare optimization profiles
go run main.go run \
  --projects small \
  --profiles baseline,incremental,lto-thin,codegen-units-max,debug-info-off \
  --runs 3 --warmup 1 \
  --cargo-profile dev

# Run with nightly profiles (requires nightly toolchain)
go run main.go run --projects small --profiles baseline,nightly-baseline,parallel-frontend --runs 3

# Generate report from existing results
go run main.go report
go run main.go report --open   # open in browser

Auto-detect mode

Point the CLI at any Rust project on disk and it will analyse the codebase (size, workspace layout, build.rs, dependencies, available toolchains) and pick reasonable benchmark parameters automatically:

# Analyse + run benchmarks with auto-detected parameters
go run main.go auto /path/to/my-rust-project

# Print analysis + recommendation without running
go run main.go auto /path/to/my-rust-project --dry-run

# Auto-detect, save the recommendation, then exit
go run main.go auto /path/to/my-rust-project --dry-run --save-config my-bench.yaml

# Override the auto-picked runs/warmup
go run main.go auto /path/to/my-rust-project --runs 5 --warmup 2

The analyser uses a lightweight Cargo.toml parser plus a fast file walker and does not build the project during analysis.

Config files

Persist any configuration (from run, auto, or the wizard) to a YAML file and reload it later:

# Save the effective configuration from a run
go run main.go run --projects small --profiles baseline,incremental \
    --runs 3 --save-config ./my-bench.yaml

# Re-run exactly the same benchmark later
go run main.go run --config ./my-bench.yaml

# Load config, but override one value from the CLI
go run main.go run --config ./my-bench.yaml --runs 5

Config files are human-editable YAML; they reference built-in projects by ID (small, medium, large) or carry absolute paths for user-supplied projects under custom_projects:.

Taskfile (optional – requires task)

task list            # list available tasks
task bench           # launch wizard
task bench:quick     # fast smoke test
task bench:compare   # key profiles comparison
task bench:nightly   # nightly toolchain comparison
task bench:sccache   # sccache benchmark
task report          # generate report
task check           # verify dependencies

---

Projects

ID	Name	Size	Description
small	embedded-sensor-node	small	Custom embedded no_std project with build.rs code generation (CRC tables, sine lookup, ADC calibration, register maps)
medium	ripgrep	medium	BurntSushi/ripgrep v14.1.1 – fast recursive grep tool (~15k LoC Rust)
large	rust-analyzer	large	rust-lang/rust-analyzer 2025-01-13 – Rust language server (~350k LoC Rust)

Setting up reference projects

# Medium project (ripgrep)
bash projects/medium/setup.sh

# Large project (rust-analyzer)
bash projects/large/setup.sh

# Or both at once
cd bench && go run main.go setup all

---

Optimization Profiles

ID	Toolchain	Description
baseline	stable	Default cargo build
incremental	stable	CARGO_INCREMENTAL=1
no-incremental	stable	CARGO_INCREMENTAL=0
sccache	stable	RUSTC_WRAPPER=sccache – compiler cache
lto-thin	stable	-C lto=thin
lto-fat	stable	-C lto=fat – full LTO
codegen-units-1	stable	-C codegen-units=1 – max optimization
codegen-units-max	stable	-C codegen-units=256 – max parallelism
parallel-frontend	nightly	-Z threads=8 – parallel compiler frontend
cranelift	nightly	-Z codegen-backend=cranelift
mold	stable	-C link-arg=-fuse-ld=mold – mold linker
debug-info-off	stable	-C debuginfo=0
nightly-baseline	nightly	Nightly toolchain, no extra flags

Use all to run all profiles: --profiles all

---

CLI Reference

bench [command] [flags]

Commands:
  wizard    Interactive wizard to configure and run a benchmark
  run       Run the benchmark suite with the given options
  auto      Analyse a Rust project and run benchmarks with auto-detected parameters
  report    Generate an HTML report from the latest results
  setup     Clone and prepare reference projects
  list      List available projects and optimization profiles

Run flags:
  --projects strings      Projects to benchmark: small,medium,large,all (default [small])
  --profiles strings      Optimization profiles, comma-separated IDs or 'all' (default [baseline])
  --runs int              Number of measured runs per combination (default 3)
  --warmup int            Number of warmup runs, discarded (default 1)
  --cargo-profile string  Cargo build profile: dev or release (default "dev")
  --clean                 Clean build artifacts between runs (default true)
  --timings               Collect crate timings via --timings=json (default true)
  --output string         Output directory (default <repo>/results)
  --config string         Load benchmark configuration from a YAML file
  --save-config string    Save the effective configuration to a YAML file

Auto flags:
  --dry-run               Print analysis + recommendation without running
  --save-config string    Save the auto-detected configuration to YAML
  --runs int              Override auto-detected number of runs
  --warmup int            Override auto-detected number of warmup runs
  --cargo-profile string  Override auto-detected cargo profile

Report flags:
  --input string          Path to results JSON file (default: latest)
  --open                  Open the report in the default browser

---

Output

Results are written to the results/ directory:

• results-<timestamp>.json – raw results in JSON format
• report-<timestamp>.html – HTML report with Chart.js diagrams

HTML Report Sections

1. Summary – total runs, success rate, fastest/slowest build
2. Overall comparison chart – bar chart of all profiles across projects
3. Per-project analysis – tabbed view with:
   • Build time comparison chart
   • Profile statistics table (avg/min/max, speedup vs baseline)
   • Crate timing breakdown (slowest crates)
4. Raw results table – all individual runs

---

Project Structure

rust-compile-bench/
├── AGENTS.md                # Guidance for AI coding agents (Copilot / Claude / …)
├── bench/                   # Go CLI tool
│   ├── main.go
│   ├── cmd/                 # cobra commands (run, auto, wizard, report, setup, list)
│   └── internal/
│       ├── config/          # project/profile configuration types + YAML file load/save
│       ├── autodetect/      # Rust project analyser + parameter recommender
│       ├── runner/          # cargo build executor + result storage
│       ├── report/          # HTML report generator
│       │   └── templates/   # report.html (Chart.js template)
│       └── wizard/          # interactive terminal wizard
├── projects/
│   ├── small/               # Embedded sensor node (bundled, no setup needed)
│   │   ├── Cargo.toml
│   │   ├── build.rs         # generates CRC/sine/ADC lookup tables + register maps
│   │   └── src/             # adc, comm, dsp, scheduler, sensors modules
│   ├── medium/
│   │   └── setup.sh         # clones ripgrep v14.1.1
│   └── large/
│       └── setup.sh         # clones rust-analyzer 2025-01-13
├── results/                 # benchmark results (JSON + HTML, gitignored by default)
├── scripts/
│   └── check-deps.sh        # dependency checker
├── Taskfile.yml             # task orchestration (optional)
└── .gitignore

---

Development

Running tests

cd bench && go test ./...

Building for release

cd bench && go build -ldflags="-s -w" -o bench-tool .