Skip to content

benchmarking.md

Latest commit

 

History

History
73 lines (53 loc) · 2.27 KB

benchmarking.md

File metadata and controls

73 lines (53 loc) · 2.27 KB
Raw

Benchmarking

Sashiko includes a benchmark tool to evaluate AI review performance against known bugs. It ingests patches, waits for reviews to complete, then uses an AI judge to compare findings against ground-truth descriptions.

Prerequisites

  • A running sashiko daemon with a configured LLM provider
  • A clean database (move or remove any existing sashiko.db)
  • A benchmark JSON file (several are provided in benchmarks/)

Quick start

# Start with a clean database
mv sashiko.db sashiko.db.bak

# Run the benchmark
cargo run --bin benchmark -- --file benchmarks/benchmark_small.json

Benchmark files

File Description
benchmarks/benchmark_tiny.json Minimal set for quick smoke tests.
benchmarks/benchmark_small.json Small set for development iteration.
benchmarks/benchmark.json Full benchmark suite.
benchmarks/benchmark_preexisting.json Tests detection of pre-existing bugs.
benchmarks/benchmark_smoke.json CI smoke test set.

Each file contains entries with a commit hash, a Fixed-by reference, and a problem_description that the AI judge uses to evaluate whether sashiko detected the issue.

Command-line options

cargo run --bin benchmark -- [OPTIONS]
Flag Description
-f, --file <PATH> Path to the benchmark JSON file (required).
-p, --port <PORT> Override the daemon port (defaults to Settings.toml value).
-r, --repo <URL> Override the kernel repository URL.
--analyze-only Skip ingestion; only evaluate existing results in the database.

Output

The tool prints a summary to the console:

  • Detection rates: Detected, Missed, Partially Detected
  • Performance metrics: Average tokens in/out, average turns, average time per review
  • Counts: Total concerns and findings

Detailed results are written to benchmark_results.json in the current directory, including the AI judge's explanation for each finding.

Re-evaluating existing results

If you have already run ingestion and reviews but want to re-score with updated evaluation logic:

cargo run --bin benchmark -- --file benchmarks/benchmark_small.json --analyze-only

This skips patch submission and review, reading results directly from the database.