Skip to content

BENCHMARKS.md

Latest commit

 

History

History
575 lines (428 loc) · 22.3 KB

BENCHMARKS.md

File metadata and controls

575 lines (428 loc) · 22.3 KB
Raw

Performance Benchmarks

Purpose: Track and validate performance budgets for pi_agent_rust.

User-Perceived SLI Contract

Phase-0 canonical UX/SLI contract lives in docs/perf_sli_matrix.json (schema: pi.perf.sli_matrix.v1).

  • Primary release-deciding metrics are user-visible E2E/responsiveness SLIs.
  • Micro-benchmarks in this file are diagnostic/supporting metrics.
  • Scenario-to-SLI mapping is keyed by docs/e2e_scenario_matrix.json workflow IDs.
  • Downstream PERF-3X validation beads must consume SLI results directly from the contract artifact.

Quick Start

# Run all benchmarks
cargo bench

# Run specific benchmark
cargo bench "truncation"
cargo bench "sse_parsing"
cargo bench "ext_policy"
cargo bench "ext_js_runtime"

# Run with baseline comparison
cargo bench -- --save-baseline main
cargo bench -- --baseline main

Performance Budgets

These are the target performance metrics. Regressions beyond these thresholds should be investigated.

Core Metrics (Hard Budgets)

Benchmark Budget Current Status
startup/version <100ms (p95) ~11ms ✅
startup/help <150ms (p95) ~15ms ✅
startup/list_models <200ms (p95) ~25ms ✅
binary/size_mb <20MB ~7.6MB ✅
memory/version_peak <50MB RSS TBD ⬜

Micro-Benchmarks

Benchmark Budget Current Status
truncate_head (10K lines) <1ms ~250μs ✅
truncate_tail (10K lines) <1ms ~250μs ✅
sse_parse (100 events) <100μs ~50μs ✅
ext_policy/evaluate <1μs ~20ns ✅
ext_dispatch/decision <10μs ~100ns ✅
ext_protocol/parse <100μs ~5μs ✅
ext_js_runtime/cold_start <200ms ~308μs ✅
ext_js_runtime/warm_eval_noop <25ms ~3.50μs ✅
ext_js_runtime/warm_run_pending_jobs_empty <1μs ~84ns ✅
ext_js_runtime/tool_call_roundtrip <500μs ~43.9μs ✅

Extension Runtime (Baseline: 2026-02-07, debug build, 103 extensions)

Benchmark Budget Current (debug) Status
ext_cold_load_simple_p95 (100 extensions) p95 < 200ms 106ms ✅
ext_cold_load_per_ext_p99 (worst ext) p99 < 100ms 134ms (hjanuschka-plan-mode) ⬜*
ext_warm_load_p95 (100 extensions) p95 < 100ms 734μs ✅
ext_warm_load_per_ext_p99 (worst ext) p99 < 100ms 926μs (jyaunches-pi-canvas) ✅
event_dispatch_p99 (AgentStart, PR mode) p99 < 5ms 616μs ✅

*Cold load per-extension P99 exceeds debug-mode budget but is expected to pass in release (release cold loads are typically ~5-10ms). Budget assertions are release-only.

Baseline data: tests/perf/reports/ext_bench_baseline.json Outlier analysis: tests/perf/reports/BASELINE_REPORT.md

Extension Runtime Budget Definitions

These budgets target extension overhead, not end-to-end LLM latency.

  • Cold start: first time an extension runtime is created/initialized for a process (cold caches).
  • Warm start: extension runtime is already initialized (warm caches); measures steady-state overhead.
  • Hook overhead: incremental latency added by routing a tool call through a no-op extension hook.
  • Hostcall dispatch: cost to invoke a single hostcall across the connector boundary (no-op payload).

Measurement Methodology (bd-1ii)

  • Hardware class: GitHub Actions ubuntu-latest runner (x86_64). Treat numbers as CI budgets; local machines will vary.
  • Percentiles: budgets are specified as p95/p99 to avoid overfitting to median-only results on shared CI runners.
  • Benchmarks: extension benchmarks will live under benches/extensions.rs (planned) and should report:
    • cold vs warm timings separately
    • a baseline (no extension) vs no-op extension delta for hook overhead
    • enough samples to make percentile reporting meaningful on CI

Benchmark Results

Truncation Performance

Processing throughput for text truncation operations:

truncation/head/1000    time:   [32 µs]     thrpt:  [2.3 GiB/s]
truncation/head/10000   time:   [251 µs]    thrpt:  [3.0 GiB/s]
truncation/head/100000  time:   [2.3 ms]    thrpt:  [3.3 GiB/s]

truncation/tail/1000    time:   [~32 µs]    thrpt:  [~2.3 GiB/s]
truncation/tail/10000   time:   [~251 µs]   thrpt:  [~3.0 GiB/s]
truncation/tail/100000  time:   [~2.3 ms]   thrpt:  [~3.3 GiB/s]

Key observations:

  • Throughput is consistent at 2.3-3.3 GiB/s regardless of input size
  • Head and tail truncation have similar performance
  • Well within the 1ms budget for typical file sizes (10K lines)

SSE Parsing Performance

Server-Sent Events parsing throughput:

sse_parsing/parse/100   time:   [50.129 µs 50.315 µs 50.504 µs]
                         thrpt:  [1.9800 Melem/s 1.9875 Melem/s 1.9949 Melem/s]

sse_parsing/parse/1000  time:   [495.54 µs 495.96 µs 496.40 µs]
                         thrpt:  [2.0145 Melem/s 2.0163 Melem/s 2.0180 Melem/s]

Benchmark Structure

benches/
├── bench_env.rs      # Shared environment validation and fingerprinting
├── tools.rs          # Core operation benchmarks
│   ├── truncation    # Text truncation (head/tail)
│   ├── sse_parsing   # SSE event parsing
│   ├── sse_stream    # Streaming SSE parsing at various chunk sizes
│   └── streaming_clone  # Arc<AssistantMessage> vs deep clone
├── extensions.rs     # Connector dispatch + policy / protocol parsing
│   ├── ext_policy
│   ├── ext_required_capability
│   ├── ext_dispatch
│   ├── ext_protocol
│   ├── ext_js_runtime     # QuickJS cold/warm start + no-op eval
│   ├── hostcall_*         # Hostcall conversion, hashing, dispatch
│   └── js_serde_bridge    # JS↔Rust serialization roundtrip
├── system.rs         # System-level benchmarks (process spawn)
│   ├── startup       # Startup time (version, help, list_models)
│   ├── memory        # RSS memory measurement
│   └── binary        # Binary size tracking
├── tui_perf.rs       # TUI rendering benchmarks (PERF-8)
│   ├── build_conversation_content
│   ├── view          # Full TUI render
│   ├── viewport_operations
│   └── markdown_rendering
└── session_save.rs   # Session clone benchmarks
scripts/
└── bench_env_setup.sh  # OS-level benchmark environment standardization

Adding New Benchmarks

  1. Add benchmark function to benches/tools.rs:
fn bench_new_operation(c: &mut Criterion) {
    let mut group = c.benchmark_group("new_operation");

    // Test with different input sizes
    for size in [100, 1000, 10000] {
        let input = generate_input(size);
        group.throughput(Throughput::Elements(size as u64));
        group.bench_with_input(
            BenchmarkId::new("name", size),
            &input,
            |b, input| {
                b.iter(|| pi::module::function(black_box(input)));
            },
        );
    }

    group.finish();
}

// Add to criterion_group!
criterion_group!(benches, ..., bench_new_operation);
  1. Add performance budget to this document
  2. Run benchmark: cargo bench new_operation

CI Integration

Performance regression detection in GitHub Actions:

# .github/workflows/bench.yml
name: Benchmarks
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  CARGO_TERM_COLOR: always
  RUSTFLAGS: -D warnings

jobs:
  benchmark:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain@nightly

      - name: Build release binary
        run: cargo build --release

      - name: Check binary size budget
        run: |
          SIZE_MB=$(stat --printf="%s" target/release/pi | awk '{printf "%.2f", $1/1024/1024}')
          echo "Binary size: ${SIZE_MB}MB"
          if (( $(echo "$SIZE_MB > 20" | bc -l) )); then
            echo "::error::Binary size ${SIZE_MB}MB exceeds 20MB budget"
            exit 1
          fi

      - name: Run benchmarks
        run: |
          cargo bench --bench tools -- --noplot
          cargo bench --bench extensions -- --noplot
          cargo bench --bench system -- --noplot

      - name: Generate PiJS workload perf data (JSONL)
        run: |
          set -euxo pipefail
          mkdir -p target/perf/perf
          PI_BENCH_BUILD_PROFILE=perf cargo run --profile perf --example pijs_workload -- --iterations 2000 --tool-calls 1 > target/perf/perf/pijs_workload_perf.jsonl
          PI_BENCH_BUILD_PROFILE=perf cargo run --profile perf --example pijs_workload -- --iterations 2000 --tool-calls 10 >> target/perf/perf/pijs_workload_perf.jsonl

      - name: Perf budget gate
        run: cargo test --test perf_budgets -- --nocapture

      - name: Upload benchmark results
        uses: actions/upload-artifact@v4
        with:
          name: benchmark-results
          path: target/criterion/
          retention-days: 30

Regression Detection (Manual)

Compare against a known good baseline:

# Save baseline on main branch
cargo bench -- --save-baseline main

# After changes, compare
cargo bench -- --baseline main

# Look for regressions > 10%

Variance Handling

System benchmarks spawn real processes, so variance is higher than micro-benchmarks:

  • Micro-benchmarks (tools.rs, extensions.rs): Use criterion defaults (100+ samples)
  • System benchmarks (system.rs): Use 20 samples, 10s measurement time
  • CI runners: Expect 2-3x variance vs local machines; focus on relative changes
  • Percentiles: Report p95/p99 for budgets, not just mean

Environment Standardization (bd-3ar8v.5.4)

All benchmark suites use a shared environment module (benches/bench_env.rs) that:

  1. Validates the execution environment at startup (CPU governor, turbo boost, ASLR, THP)
  2. Fingerprints every run with OS, CPU, cores, memory, governor, turbo, ASLR, THP, and config hash
  3. Computes a noise score (0 = optimal) and warns when conditions are suboptimal

The scripts/bench_env_setup.sh script standardizes the OS for low-variance results:

# Check current environment suitability
./scripts/bench_env_setup.sh validate

# Apply optimal settings (requires root)
sudo ./scripts/bench_env_setup.sh apply

# Run benchmarks with CPU affinity and priority
./scripts/bench_env_setup.sh run cargo bench

# Emit JSON fingerprint for artifact tracking
./scripts/bench_env_setup.sh fingerprint

# Restore original settings
sudo ./scripts/bench_env_setup.sh restore

What it controls:

Setting Optimal Why
CPU governor performance Fixed frequency eliminates DVFS variance
Turbo boost disabled Prevents thermal-dependent frequency shifts
ASLR disabled Reproducible memory layouts
THP never Avoids latency spikes from page coalescing

Environment variables:

Variable Default Description
BENCH_CORES 0,1 CPU cores for taskset affinity
BENCH_GOVERNOR performance CPU frequency governor to set
BENCH_NICE -20 Nice priority for bench processes

Noise score interpretation:

Score Meaning
0 Optimal — all settings applied
1-2 Minor — THP or ASLR not ideal
3-5 Moderate — governor or turbo not controlled
6-7 High — multiple sources of variance

CI applies environment setup automatically before benchmarks. The [bench-env] banner in benchmark stderr output includes the noise score for every run.

Profiling Tips

bd-1pb: Profile-Driven Optimization Loop

This workstream uses a strict baseline → profile → prove → implement → verify loop.

1) Baseline

  • Use Criterion for stable micro-bench artifacts: cargo bench --bench extensions -- ext_js_runtime.
  • Use hyperfine for end-to-end CLI paths (if installed):
hyperfine --warmup 3 --runs 10 'target/release/pi --version'
  • Use the PiJS workload harness for deterministic extension roundtrips:
scripts/bench_extension_workloads.sh

Baseline Captures (2026-02-05)

Commands:

hyperfine --warmup 3 --runs 10 'target/perf/pijs_workload --iterations 200 --tool-calls 1'
hyperfine --warmup 3 --runs 10 'target/perf/pijs_workload --iterations 200 --tool-calls 10'

Summary (times in ms):

Scenario Mean ± σ Min / Max per_call_us calls/sec
pijs_workload_200x1 16.96 ± 0.98 15.78 / 19.00 44 22,716
pijs_workload_200x10 97.09 ± 4.27 93.08 / 105.57 43 22,883

JSONL logs (hyperfine + workload):

{"tool":"hyperfine","scenario":"pijs_workload_200x1","command":"target/perf/pijs_workload --iterations 200 --tool-calls 1","mean_ms":16.96,"stddev_ms":0.98,"min_ms":15.78,"max_ms":19.00}
{"tool":"hyperfine","scenario":"pijs_workload_200x10","command":"target/perf/pijs_workload --iterations 200 --tool-calls 10","mean_ms":97.09,"stddev_ms":4.27,"min_ms":93.08,"max_ms":105.57}
{"schema":"pi.perf.workload.v1","tool":"pijs_workload","scenario":"tool_call_roundtrip","iterations":200,"tool_calls_per_iteration":1,"total_calls":200,"elapsed_ms":8,"per_call_us":44,"calls_per_sec":22716,"build_profile":"perf"}
{"schema":"pi.perf.workload.v1","tool":"pijs_workload","scenario":"tool_call_roundtrip","iterations":200,"tool_calls_per_iteration":10,"total_calls":2000,"elapsed_ms":87,"per_call_us":43,"calls_per_sec":22883,"build_profile":"perf"}

Raw artifacts (local):

  • target/perf/perf/hyperfine_pijs_workload_200x1_perf.json
  • target/perf/perf/hyperfine_pijs_workload_200x10_perf.json
  • target/perf/perf/pijs_workload_perf.jsonl

2) Profile

  • CPU hotspots: cargo flamegraph --bench extensions (requires cargo install flamegraph).
  • Allocations: heaptrack cargo bench --bench extensions (Linux).
  • Flamegraph run (2026-02-05): cargo flamegraph --bench extensions -- ext_js_runtime --noplot compiled benches successfully, then failed during sampling because perf_event_paranoid=4 on this host (no perf access). Retry on a host with CAP_PERFMON (or lower perf_event_paranoid) and keep the resulting SVG as the flamegraph artifact.

Hotspot snapshot from Criterion new/estimates.json (mean point estimate):

Benchmark Mean (ns) Mean (μs) Relative cost vs warm_eval_noop
ext_js_runtime/cold_start 307,950.60 307.95 88.0×
ext_js_runtime/tool_call_roundtrip 43,915.12 43.92 12.6×
ext_js_runtime/warm_eval_noop 3,498.12 3.50 1.0×
ext_js_runtime/warm_run_pending_jobs_empty 84.45 0.08 0.02×

3) Prove (No “silent regressions”)

  • Keep outputs reproducible: record environment ([bench-env] ... config_hash=... emitted by benches/extensions.rs).
  • Store benchmark artifacts in target/criterion/ (Criterion JSON + reports).
  • Use --save-baseline / --baseline comparisons for regression detection.

4) Opportunity Matrix (Prioritized)

Opportunity Evidence Expected impact Confidence Effort Score Notes
Cache compiled extension setup program across repeated loads ext_js_runtime/cold_start = 307.95μs dominates runtime hotspot table -150μs to -220μs cold-start cost on repeated extension loads 4 3 5.33 Keep module hash keyed by source+runtime config; preserve deterministic teardown semantics
Reduce JSON bridge overhead in hostcall tool path ext_js_runtime/tool_call_roundtrip = 43.92μs and pijs_workload steady-state per-call = 43–46μs -8μs to -15μs per roundtrip 3 2 4.50 Target serialization/path allocation churn first; validate with criterion baseline diff
Keep run_pending_jobs empty fast path as invariant ext_js_runtime/warm_run_pending_jobs_empty = 84.45ns Avoid regressions in scheduler idle overhead 5 1 5.00 No optimization work needed; treat as guardrail metric in future PRs

CPU Profiling with perf

# Record profile
cargo bench -- --profile-time 10
perf record -g target/release/deps/tools-*

# Analyze
perf report

Memory Profiling with heaptrack

heaptrack cargo bench
heaptrack_gui heaptrack.tools.*.gz

Flame Graphs

cargo install flamegraph
cargo flamegraph --bench tools

Comparison with TypeScript

Target metrics for Rust vs TypeScript:

Operation TypeScript Rust Target Rust Actual
Startup ~200ms <100ms 11.2ms ✅
10K line truncate ~10ms <1ms 250μs ✅
100 SSE events ~5ms <100μs 50.3μs ✅
Binary size N/A (Node) <20MB 7.6MB ✅
Memory (idle) ~80MB <50MB TBD

Extension Load Time: Rust vs Legacy TS (bd-uah)

Per-extension load time comparison across all 60 official extensions. Both runtimes load the same unmodified .ts files. TS uses Bun/jiti (native V8-based eval). Rust uses QuickJS with SWC transpilation.

Metric Rust (QuickJS) TS (Bun/jiti)
Mean load time 103ms 2ms
Min load time 96ms 1ms
Max load time 131ms 51ms

Known regression: Extension loading in Rust is ~50-100x slower due to:

  1. SWC TypeScript-to-JavaScript transpilation per-load
  2. QuickJS bytecode compilation (no JIT)
  3. Virtual module system resolution overhead

Why this is acceptable: The loading cost is a one-time cold-start per session. Steady-state operations are orders of magnitude faster in Rust:

  • Tool call roundtrip: 44μs (Rust) vs ~5ms (TS)
  • Policy evaluation: 20ns (Rust)
  • Event hook dispatch: sub-50μs (Rust)

Planned mitigation: Compiled bytecode caching (see Opportunity Matrix above) to amortize cold-start across sessions.

Full per-extension data: tests/ext_conformance/reports/performance_comparison.json

Regenerate: cargo test --test performance_comparison generate_performance_comparison -- --nocapture

Extension Benchmark Harness (bd-20s9 / bd-2mb1)

The unified benchmark harness (tests/ext_bench_harness.rs) runs extension load and event dispatch scenarios with per-extension timeouts, budget checks, and full environment fingerprinting.

Running the Harness

# PR mode — diverse 10-extension subset, 10 iterations, ~3-4s in debug
PI_BENCH_MODE=pr cargo test --test ext_bench_harness --features ext-conformance -- --nocapture

# Nightly mode — full safe corpus, 50 iterations
PI_BENCH_MODE=nightly cargo test --test ext_bench_harness --features ext-conformance -- --nocapture

# Custom mode — tune all parameters
PI_BENCH_MODE=custom PI_BENCH_MAX=25 PI_BENCH_ITERATIONS=20 PI_BENCH_EVENT_COUNT=100 \
  cargo test --test ext_bench_harness --features ext-conformance -- --nocapture

Environment Variables

Variable Default Description
PI_BENCH_MODE pr Mode: pr, nightly, or custom
PI_BENCH_MAX 10 (pr) / 200 (nightly) / 20 (custom) Max extensions to benchmark
PI_BENCH_ITERATIONS 10 (pr) / 50 (nightly) / 20 (custom) Iterations per extension per scenario
PI_BENCH_EVENT_COUNT 50 (pr) / 200 (nightly) / 100 (custom) Event dispatch iterations
PI_BENCH_TIMEOUT_SECS 30 Per-extension timeout (skips slow extensions)

PR Subset Selection Policy

PR mode selects a diverse representative subset to maximize API surface coverage:

  • 2 official extensions (1 with tool registration, 1 with event subscriptions)
  • 2 community extensions (1 with commands+events, 1 with tools+commands+flags)
  • 2 npm-registry extensions (1 with commands, 1 with events)
  • Remaining slots filled from safe pool in manifest order

This ensures each run exercises tools, commands, flags, and event hooks.

Scenarios

Scenario What it measures Method
cold_load Fresh runtime + context creation per iteration New ExtensionManager + JsExtensionRuntimeHandle::start() + load_js_extensions()
warm_load Repeated load on shared runtime (cache-hit path) Single runtime, repeated load_js_extensions() after warmup
event_dispatch Event hook dispatch latency across loaded extensions dispatch_event(AgentStart, payload) on loaded corpus

Budget Checks

Budget Threshold Enforced
ext_cold_load_simple_p95 200ms Release builds only
event_dispatch_p99 5ms Release builds only
ext_warm_load_p95 100ms Release builds only

Budget assertions are skipped in debug builds (debug mode is naturally 5-10x slower).

Output Artifacts

All outputs go to target/perf/:

File Format Content
ext_bench_harness.jsonl JSONL One pi.ext.rust_bench.v1 record per extension per scenario
ext_bench_harness_report.json JSON Full report with env, config, summaries, budget checks
BENCH_HARNESS_REPORT.md Markdown Human-readable summary with tables

Interpreting Results

  • P50/P95/P99 are computed per-extension from raw microsecond samples
  • Cold load times include QuickJS runtime creation (~70ms in debug, ~5ms in release)
  • Warm load times measure only the load_js_extensions() call (~300-800us)
  • Event dispatch measures dispatch_event() latency (~40-700us depending on loaded extensions)
  • Aggregate budget checks use the P95 across all per-extension P95 values

Updating Baselines

To intentionally update baseline thresholds:

  1. Run the harness in release mode to get accurate numbers: 
    cargo test --release --test ext_bench_harness --features ext-conformance -- --nocapture
  2. Review target/perf/ext_bench_harness_report.json for actual P95/P99 values
  3. Update the threshold constants in check_budgets() in tests/ext_bench_harness.rs
  4. Document the justification in the commit message

Detecting Noise vs Real Regressions

  • Run the harness 3 times and compare P95 values
  • Variance > 20% between runs indicates environmental noise
  • Consistent P95 increase > 50% across runs indicates a real regression
  • Check the env fingerprint in JSONL to ensure same hardware/build profile

Notes

  • Benchmarks run in release mode with LTO enabled
  • Times measured on standard CI hardware (GitHub Actions)
  • Throughput measured in GiB/s or elements/sec
  • Use --save-baseline and --baseline for regression detection