diff --git a/.github/workflows/bench-205.yml b/.github/workflows/bench-205.yml index 1abc1cc6..de7bf3f1 100644 --- a/.github/workflows/bench-205.yml +++ b/.github/workflows/bench-205.yml @@ -28,6 +28,9 @@ jobs: - crate: fbuild-library-select bench: resolve_cold target: P-02 (<= 200 ms cold for typical project) + - crate: fbuild-library-select + bench: resolve_warm + target: P-01-mini (warm cache against MiniFramework, baseline TBD) steps: - uses: actions/checkout@v6 - uses: astral-sh/setup-uv@v3 diff --git a/bench/README.md b/bench/README.md index c6a75f9f..91102efe 100644 --- a/bench/README.md +++ b/bench/README.md @@ -7,20 +7,22 @@ Per-crate criterion benches live alongside their crate, e.g.: - `crates/fbuild-header-scan/benches/scan_throughput.rs` - `crates/fbuild-library-select/benches/resolve_cold.rs` +- `crates/fbuild-library-select/benches/resolve_warm.rs` Run those with: ```bash uv run soldr cargo bench -p fbuild-library-select --bench resolve_cold +uv run soldr cargo bench -p fbuild-library-select --bench resolve_warm uv run soldr cargo bench -p fbuild-header-scan --bench scan_throughput ``` ## Subdirectories -- [`fastled-examples/`](fastled-examples/README.md) — placeholder for the - warm-cache library-selection harness across the FastLED examples matrix - (`FastLED/fbuild#205` AC#5, P-01). Awaits Phase 4 zccache K/V memoization - before there's a warm path to measure. +- [`fastled-examples/`](fastled-examples/README.md) — reserved for the + real-FastLED warm-cache library-selection matrix (`FastLED/fbuild#205` + AC#5, P-01) once `~/dev/fastled` is wired in. The synthetic warm-path + baseline already lives in `crates/fbuild-library-select/benches/resolve_warm.rs`. Other end-to-end matrices (whole-build wall-clock, deploy+flash latency, emulator boot) may join this directory in the future. Each subdirectory diff --git a/bench/fastled-examples/README.md b/bench/fastled-examples/README.md index a7216cea..ba958fab 100644 --- a/bench/fastled-examples/README.md +++ b/bench/fastled-examples/README.md @@ -9,17 +9,19 @@ criterion **AC#5 / P-01**: ## Status: empty placeholder -There is no harness here yet, and on purpose. P-01 measures the **warm** -path through the resolver, which depends on the zccache K/V memoization -delivered by `#205` Phase 4 (gated on `zackees/zccache#130`). Until that -lands, every `resolve()` call is cold and there is no warm number to -gate against. Adding a "warm-ish" harness today would just measure cold -work twice and produce a misleading baseline. +There is no harness in this directory yet — the real per-board, per-example +matrix needs a checked-out FastLED tree (`~/dev/fastled`) and orchestrator +wiring that routes through `resolve_cached`. That work is tracked +separately. The synthetic warm baseline (`MiniFramework`-backed cache hit, +no real FastLED) already exists at +[`../../crates/fbuild-library-select/benches/resolve_warm.rs`](../../crates/fbuild-library-select/benches/resolve_warm.rs) +and is the first-pass regression guard for the cache-hit path. -When Phase 4 ships and a `zccache` release is cut, a follow-up PR drops -the actual harness into this directory and wires it into CI. +Phase 4 K/V memoization itself shipped in PR #212, so the warm path is +real and measurable today; what is missing here is the multi-board, +real-sketch matrix that AC#5 requires. -## The plan once Phase 4 lands +## The plan once the FastLED tree is wired in 1. Iterate the FastLED examples tree (`~/dev/fastled/examples/**`) under each supported board: at minimum `teensyLC`, `teensy30`, `teensy41`, @@ -38,29 +40,30 @@ the actual harness into this directory and wires it into CI. 4. Emit a structured JSON report (`bench/fastled-examples/report.json`) that future PR comments can diff. Format TBD with the harness. -## Running a partial version today +## Running the synthetic mini benches today -The closest signal available right now is the per-crate cold-resolve -criterion bench: +The closest signal available right now without a FastLED checkout is the +per-crate cold and warm criterion benches against `MiniFramework`: ```bash uv run soldr cargo bench -p fbuild-library-select --bench resolve_cold +uv run soldr cargo bench -p fbuild-library-select --bench resolve_warm ``` -That bench drives a synthetic ~30-library Teensyduino-class tree built +Those benches drive a synthetic ~30-library Teensyduino-class tree built from `fbuild-test-support`'s `MiniFramework` rather than real FastLED -sketches, and it measures the cold path only. It is a useful regression -guard for the resolver itself, but it does **not** satisfy AC#5. +sketches. They are useful regression guards for the resolver and its +cache layer respectively, but they do **not** satisfy AC#5 on their own. ## Cross-links - Issue: `FastLED/fbuild#205` -- Phase 4 design note (the prerequisite for this directory): +- Phase 4 K/V memoization (shipped in #212): [`../../tasks/zccache-kv-design.md`](../../tasks/zccache-kv-design.md) - Subsystem architecture: [`../../docs/architecture/library-selection.md`](../../docs/architecture/library-selection.md) - Foundation baseline that the warm threshold compares against: [`../../tasks/baseline-205.md`](../../tasks/baseline-205.md) -- Per-crate cold benches (different scope, same subsystem): +- Per-crate cold + warm benches (different scope, same subsystem): [`../../crates/fbuild-library-select/benches/README.md`](../../crates/fbuild-library-select/benches/README.md), [`../../crates/fbuild-header-scan/benches/README.md`](../../crates/fbuild-header-scan/benches/README.md) diff --git a/crates/fbuild-library-select/Cargo.toml b/crates/fbuild-library-select/Cargo.toml index 184478c0..141a7c51 100644 --- a/crates/fbuild-library-select/Cargo.toml +++ b/crates/fbuild-library-select/Cargo.toml @@ -24,3 +24,7 @@ fbuild-test-support = { path = "../fbuild-test-support" } [[bench]] name = "resolve_cold" harness = false + +[[bench]] +name = "resolve_warm" +harness = false diff --git a/crates/fbuild-library-select/benches/README.md b/crates/fbuild-library-select/benches/README.md index 3d711942..ddcef97a 100644 --- a/crates/fbuild-library-select/benches/README.md +++ b/crates/fbuild-library-select/benches/README.md @@ -9,8 +9,8 @@ End-to-end cold-path measurement of `resolve()` against a synthetic 5-deep transitive include chain forces the two-pass LDF reconciliation; the remaining libraries are unreferenced and must be rejected — that doubles as a guard against the #204 over-selection regression. Walks the tempdir on every -iteration since no cache sits in front of `resolve()` today (Phase 4 -memoization waits on zccache#130). +iteration; the bench calls `resolve()` directly, so no cache layer ever sits +in front of it. The Phase 7 P-02 threshold from FastLED/fbuild#205 is **≤ 200 ms cold for a typical teensy41 project**. This bench captures the baseline; future PRs gate @@ -21,3 +21,26 @@ Run: ```bash uv run soldr cargo bench -p fbuild-library-select --bench resolve_cold ``` + +## resolve_warm + +Warm cache-hit path through `resolve_cached()` against the same synthetic +~30-library `MiniFramework` tree. The bench builds the fixture once, opens a +`KvStore` in a tempdir, primes the cache with one untimed `resolve_cached` +call (which misses), then times only the second invocation. Each iteration +asserts `from_cache == true` and panics otherwise — that way we can never +silently regress to measuring miss work. + +This is the Phase 7 / #215 P-01-mini bench. The real per-board, per-example +warm matrix lives at `bench/fastled-examples/` and depends on a checked-out +FastLED tree; the synthetic mini bench here is the cache-hit regression +guard that runs on every CI of this crate. + +Run: + +```bash +uv run soldr cargo bench -p fbuild-library-select --bench resolve_warm +``` + +Compare against `resolve_cold`: warm should be orders of magnitude faster +(K/V lookup + bincode decode vs. full filesystem walk + LDF reconciliation). diff --git a/crates/fbuild-library-select/benches/resolve_warm.rs b/crates/fbuild-library-select/benches/resolve_warm.rs new file mode 100644 index 00000000..ecb93cee --- /dev/null +++ b/crates/fbuild-library-select/benches/resolve_warm.rs @@ -0,0 +1,121 @@ +//! P-01 (mini) warm-resolver benchmark. +//! +//! Phase 4 of shipped +//! [`fbuild_library_select::cache::resolve_cached`], a `KvStore`-backed memo +//! in front of [`fbuild_library_select::resolve`]. AC#5 / P-01 of #205 sets +//! a "warm library-selection ≤ current fbuild + 50 ms" goal; the real +//! per-board matrix lives under `bench/fastled-examples/` and waits on a +//! checked-out FastLED tree. This bench is the per-crate counterpart: it +//! measures the synthetic warm path against the same `MiniFramework` +//! fixture as `resolve_cold`, so we have a regression guard for the cache- +//! hit code path independent of the larger matrix. +//! +//! Structure mirrors `resolve_cold.rs`: build the ~30-library tree once, +//! open a `KvStore` once, prime the cache with one untimed `resolve_cached` +//! call, then time only the second invocation (the hit path). +//! +//! Run with: +//! +//! ```text +//! uv run soldr cargo bench -p fbuild-library-select --bench resolve_warm +//! ``` +//! +//! Follows up #215 (mini bench) and #205 Phase 7 (perf budgets). + +use std::path::PathBuf; + +use criterion::{black_box, criterion_group, criterion_main, Criterion, Throughput}; +use fbuild_library_select::cache::{resolve_cached, CacheKeyInputs}; +use fbuild_packages::library::framework_library::discover_framework_libraries; +use fbuild_packages::library::FrameworkLibrary; +use fbuild_test_support::MiniFramework; +use zccache_artifact::KvStore; + +const LIB_COUNT: usize = 30; +const CHAIN_LEN: usize = 5; + +fn build_fixture() -> ( + MiniFramework, + Vec, + Vec, + Vec, +) { + let mut mf = MiniFramework::new(); + for i in 0..LIB_COUNT { + let name = format!("Lib{i:02}"); + let next = if i + 1 < CHAIN_LEN { + Some(format!("Lib{:02}", i + 1)) + } else { + None + }; + let header = if let Some(n) = &next { + format!("#pragma once\n#include <{n}.h>\n") + } else { + "#pragma once\n".to_string() + }; + let cpp = format!("#include <{name}.h>\nvoid {name}_func() {{}}\n"); + mf.add_library(&name).header(&header).cpp(&cpp).done(); + } + mf.sketch("#include \nvoid setup() {}\nvoid loop() {}\n"); + + let libs = discover_framework_libraries(&mf.libraries_dir()); + let seeds = mf.project_seeds(); + let search_paths = mf.project_search_paths(); + (mf, seeds, search_paths, libs) +} + +fn bench_resolve_warm(c: &mut Criterion) { + let (mf, seeds, search_paths, libs) = build_fixture(); + let kv_dir = tempfile::tempdir().expect("resolve_warm: failed to create kv tempdir"); + let kv = KvStore::open(kv_dir.path().join("kv")).expect("resolve_warm: KvStore::open failed"); + + let framework_root = mf.framework_root().to_path_buf(); + let inputs = CacheKeyInputs { + toolchain_triple: "avr-unknown-none", + framework_install_path: &framework_root, + framework_version: "1.59.0", + }; + + // Prime the cache so the timed loop measures the hit path only. + let primed = resolve_cached(&seeds, &search_paths, &libs, &inputs, &kv) + .expect("resolve_warm: prime resolve_cached failed"); + assert!( + !primed.from_cache, + "resolve_warm: priming call must miss (got hit)" + ); + + // WHY: confirm the very next call hits before entering the bench loop; + // otherwise we'd silently measure cold work and report a misleading + // baseline. + let probe = resolve_cached(&seeds, &search_paths, &libs, &inputs, &kv) + .expect("resolve_warm: probe resolve_cached failed"); + assert!( + probe.from_cache, + "resolve_warm: second call did not hit cache; bench would measure misses" + ); + + let mut group = c.benchmark_group("resolve"); + group.throughput(Throughput::Elements(libs.len() as u64)); + group.bench_function("warm_30_libs_chain_5", |b| { + b.iter(|| { + let res = resolve_cached( + black_box(&seeds), + black_box(&search_paths), + black_box(&libs), + black_box(&inputs), + black_box(&kv), + ) + .expect("resolve_warm: resolve_cached failed inside bench loop"); + if !res.from_cache { + panic!("resolve_warm: bench iteration missed the cache"); + } + black_box(res); + }); + }); + group.finish(); + drop(mf); + drop(kv_dir); +} + +criterion_group!(benches, bench_resolve_warm); +criterion_main!(benches);