feat(library-selection): #215 add resolve_warm criterion bench (P-01 mini)

.github/workflows/bench-205.yml

@@ -28,6 +28,9 @@ jobs:
           - crate: fbuild-library-select
             bench: resolve_cold
             target: P-02 (<= 200 ms cold for typical project)
+          - crate: fbuild-library-select
+            bench: resolve_warm
+            target: P-01-mini (warm cache against MiniFramework, baseline TBD)
     steps:
       - uses: actions/checkout@v6
       - uses: astral-sh/setup-uv@v3

bench/README.md

@@ -7,20 +7,22 @@ Per-crate criterion benches live alongside their crate, e.g.:
 
 - `crates/fbuild-header-scan/benches/scan_throughput.rs`
 - `crates/fbuild-library-select/benches/resolve_cold.rs`
+- `crates/fbuild-library-select/benches/resolve_warm.rs`
 
 Run those with:
 
 ```bash
 uv run soldr cargo bench -p fbuild-library-select --bench resolve_cold
+uv run soldr cargo bench -p fbuild-library-select --bench resolve_warm
 uv run soldr cargo bench -p fbuild-header-scan  --bench scan_throughput
 ```
 
 ## Subdirectories
 
-- [`fastled-examples/`](fastled-examples/README.md) — placeholder for the
-  warm-cache library-selection harness across the FastLED examples matrix
-  (`FastLED/fbuild#205` AC#5, P-01). Awaits Phase 4 zccache K/V memoization
-  before there's a warm path to measure.
+- [`fastled-examples/`](fastled-examples/README.md) — reserved for the
+  real-FastLED warm-cache library-selection matrix (`FastLED/fbuild#205`
+  AC#5, P-01) once `~/dev/fastled` is wired in. The synthetic warm-path
+  baseline already lives in `crates/fbuild-library-select/benches/resolve_warm.rs`.
 
 Other end-to-end matrices (whole-build wall-clock, deploy+flash latency,
 emulator boot) may join this directory in the future. Each subdirectory

bench/fastled-examples/README.md

@@ -9,17 +9,19 @@ criterion **AC#5 / P-01**:
 
 ## Status: empty placeholder
 
-There is no harness here yet, and on purpose. P-01 measures the **warm**
-path through the resolver, which depends on the zccache K/V memoization
-delivered by `#205` Phase 4 (gated on `zackees/zccache#130`). Until that
-lands, every `resolve()` call is cold and there is no warm number to
-gate against. Adding a "warm-ish" harness today would just measure cold
-work twice and produce a misleading baseline.
+There is no harness in this directory yet — the real per-board, per-example
+matrix needs a checked-out FastLED tree (`~/dev/fastled`) and orchestrator
+wiring that routes through `resolve_cached`. That work is tracked
+separately. The synthetic warm baseline (`MiniFramework`-backed cache hit,
+no real FastLED) already exists at
+[`../../crates/fbuild-library-select/benches/resolve_warm.rs`](../../crates/fbuild-library-select/benches/resolve_warm.rs)
+and is the first-pass regression guard for the cache-hit path.
 
-When Phase 4 ships and a `zccache` release is cut, a follow-up PR drops
-the actual harness into this directory and wires it into CI.
+Phase 4 K/V memoization itself shipped in PR #212, so the warm path is
+real and measurable today; what is missing here is the multi-board,
+real-sketch matrix that AC#5 requires.
 
-## The plan once Phase 4 lands
+## The plan once the FastLED tree is wired in
 
 1. Iterate the FastLED examples tree (`~/dev/fastled/examples/**`) under
    each supported board: at minimum `teensyLC`, `teensy30`, `teensy41`,
@@ -38,29 +40,30 @@ the actual harness into this directory and wires it into CI.
 4. Emit a structured JSON report (`bench/fastled-examples/report.json`)
    that future PR comments can diff. Format TBD with the harness.
 
-## Running a partial version today
+## Running the synthetic mini benches today
 
-The closest signal available right now is the per-crate cold-resolve
-criterion bench:
+The closest signal available right now without a FastLED checkout is the
+per-crate cold and warm criterion benches against `MiniFramework`:
 
 ```bash
 uv run soldr cargo bench -p fbuild-library-select --bench resolve_cold
+uv run soldr cargo bench -p fbuild-library-select --bench resolve_warm
 ```
 
-That bench drives a synthetic ~30-library Teensyduino-class tree built
+Those benches drive a synthetic ~30-library Teensyduino-class tree built
 from `fbuild-test-support`'s `MiniFramework` rather than real FastLED
-sketches, and it measures the cold path only. It is a useful regression
-guard for the resolver itself, but it does **not** satisfy AC#5.
+sketches. They are useful regression guards for the resolver and its
+cache layer respectively, but they do **not** satisfy AC#5 on their own.
 
 ## Cross-links
 
 - Issue: `FastLED/fbuild#205`
-- Phase 4 design note (the prerequisite for this directory):
+- Phase 4 K/V memoization (shipped in #212):
   [`../../tasks/zccache-kv-design.md`](../../tasks/zccache-kv-design.md)
 - Subsystem architecture:
   [`../../docs/architecture/library-selection.md`](../../docs/architecture/library-selection.md)
 - Foundation baseline that the warm threshold compares against:
   [`../../tasks/baseline-205.md`](../../tasks/baseline-205.md)
-- Per-crate cold benches (different scope, same subsystem):
+- Per-crate cold + warm benches (different scope, same subsystem):
   [`../../crates/fbuild-library-select/benches/README.md`](../../crates/fbuild-library-select/benches/README.md),
   [`../../crates/fbuild-header-scan/benches/README.md`](../../crates/fbuild-header-scan/benches/README.md)

crates/fbuild-library-select/Cargo.toml

@@ -24,3 +24,7 @@ fbuild-test-support = { path = "../fbuild-test-support" }
 [[bench]]
 name = "resolve_cold"
 harness = false
+
+[[bench]]
+name = "resolve_warm"
+harness = false

crates/fbuild-library-select/benches/README.md

@@ -9,8 +9,8 @@ End-to-end cold-path measurement of `resolve()` against a synthetic
 5-deep transitive include chain forces the two-pass LDF reconciliation; the
 remaining libraries are unreferenced and must be rejected — that doubles as a
 guard against the #204 over-selection regression. Walks the tempdir on every
-iteration since no cache sits in front of `resolve()` today (Phase 4
-memoization waits on zccache#130).
+iteration; the bench calls `resolve()` directly, so no cache layer ever sits
+in front of it.
 
 The Phase 7 P-02 threshold from FastLED/fbuild#205 is **≤ 200 ms cold for a
 typical teensy41 project**. This bench captures the baseline; future PRs gate
@@ -21,3 +21,26 @@ Run:
 ```bash
 uv run soldr cargo bench -p fbuild-library-select --bench resolve_cold
 ```
+
+## resolve_warm
+
+Warm cache-hit path through `resolve_cached()` against the same synthetic
+~30-library `MiniFramework` tree. The bench builds the fixture once, opens a
+`KvStore` in a tempdir, primes the cache with one untimed `resolve_cached`
+call (which misses), then times only the second invocation. Each iteration
+asserts `from_cache == true` and panics otherwise — that way we can never
+silently regress to measuring miss work.
+
+This is the Phase 7 / #215 P-01-mini bench. The real per-board, per-example
+warm matrix lives at `bench/fastled-examples/` and depends on a checked-out
+FastLED tree; the synthetic mini bench here is the cache-hit regression
+guard that runs on every CI of this crate.
+
+Run:
+
+```bash
+uv run soldr cargo bench -p fbuild-library-select --bench resolve_warm
+```
+
+Compare against `resolve_cold`: warm should be orders of magnitude faster
+(K/V lookup + bincode decode vs. full filesystem walk + LDF reconciliation).

crates/fbuild-library-select/benches/resolve_warm.rs

@@ -0,0 +1,121 @@
+//! P-01 (mini) warm-resolver benchmark.
+//!
+//! Phase 4 of <https://github.com/FastLED/fbuild/issues/205> shipped
+//! [`fbuild_library_select::cache::resolve_cached`], a `KvStore`-backed memo
+//! in front of [`fbuild_library_select::resolve`]. AC#5 / P-01 of #205 sets
+//! a "warm library-selection ≤ current fbuild + 50 ms" goal; the real
+//! per-board matrix lives under `bench/fastled-examples/` and waits on a
+//! checked-out FastLED tree. This bench is the per-crate counterpart: it
+//! measures the synthetic warm path against the same `MiniFramework`
+//! fixture as `resolve_cold`, so we have a regression guard for the cache-
+//! hit code path independent of the larger matrix.
+//!
+//! Structure mirrors `resolve_cold.rs`: build the ~30-library tree once,
+//! open a `KvStore` once, prime the cache with one untimed `resolve_cached`
+//! call, then time only the second invocation (the hit path).
+//!
+//! Run with:
+//!
+//! ```text
+//! uv run soldr cargo bench -p fbuild-library-select --bench resolve_warm
+//! ```
+//!
+//! Follows up #215 (mini bench) and #205 Phase 7 (perf budgets).
+
+use std::path::PathBuf;
+
+use criterion::{black_box, criterion_group, criterion_main, Criterion, Throughput};
+use fbuild_library_select::cache::{resolve_cached, CacheKeyInputs};
+use fbuild_packages::library::framework_library::discover_framework_libraries;
+use fbuild_packages::library::FrameworkLibrary;
+use fbuild_test_support::MiniFramework;
+use zccache_artifact::KvStore;
+
+const LIB_COUNT: usize = 30;
+const CHAIN_LEN: usize = 5;
+
+fn build_fixture() -> (
+    MiniFramework,
+    Vec<PathBuf>,
+    Vec<PathBuf>,
+    Vec<FrameworkLibrary>,
+) {
+    let mut mf = MiniFramework::new();
+    for i in 0..LIB_COUNT {
+        let name = format!("Lib{i:02}");
+        let next = if i + 1 < CHAIN_LEN {
+            Some(format!("Lib{:02}", i + 1))
+        } else {
+            None
+        };
+        let header = if let Some(n) = &next {
+            format!("#pragma once\n#include <{n}.h>\n")
+        } else {
+            "#pragma once\n".to_string()
+        };
+        let cpp = format!("#include <{name}.h>\nvoid {name}_func() {{}}\n");
+        mf.add_library(&name).header(&header).cpp(&cpp).done();
+    }
+    mf.sketch("#include <Lib00.h>\nvoid setup() {}\nvoid loop() {}\n");
+
+    let libs = discover_framework_libraries(&mf.libraries_dir());
+    let seeds = mf.project_seeds();
+    let search_paths = mf.project_search_paths();
+    (mf, seeds, search_paths, libs)
+}
+
+fn bench_resolve_warm(c: &mut Criterion) {
+    let (mf, seeds, search_paths, libs) = build_fixture();
+    let kv_dir = tempfile::tempdir().expect("resolve_warm: failed to create kv tempdir");
+    let kv = KvStore::open(kv_dir.path().join("kv")).expect("resolve_warm: KvStore::open failed");
+
+    let framework_root = mf.framework_root().to_path_buf();
+    let inputs = CacheKeyInputs {
+        toolchain_triple: "avr-unknown-none",
+        framework_install_path: &framework_root,
+        framework_version: "1.59.0",
+    };
+
+    // Prime the cache so the timed loop measures the hit path only.
+    let primed = resolve_cached(&seeds, &search_paths, &libs, &inputs, &kv)
+        .expect("resolve_warm: prime resolve_cached failed");
+    assert!(
+        !primed.from_cache,
+        "resolve_warm: priming call must miss (got hit)"
+    );
+
+    // WHY: confirm the very next call hits before entering the bench loop;
+    // otherwise we'd silently measure cold work and report a misleading
+    // baseline.
+    let probe = resolve_cached(&seeds, &search_paths, &libs, &inputs, &kv)
+        .expect("resolve_warm: probe resolve_cached failed");
+    assert!(
+        probe.from_cache,
+        "resolve_warm: second call did not hit cache; bench would measure misses"
+    );
+
+    let mut group = c.benchmark_group("resolve");
+    group.throughput(Throughput::Elements(libs.len() as u64));
+    group.bench_function("warm_30_libs_chain_5", |b| {
+        b.iter(|| {
+            let res = resolve_cached(
+                black_box(&seeds),
+                black_box(&search_paths),
+                black_box(&libs),
+                black_box(&inputs),
+                black_box(&kv),
+            )
+            .expect("resolve_warm: resolve_cached failed inside bench loop");
+            if !res.from_cache {
+                panic!("resolve_warm: bench iteration missed the cache");
+            }
+            black_box(res);
+        });
+    });
+    group.finish();
+    drop(mf);
+    drop(kv_dir);
+}
+
+criterion_group!(benches, bench_resolve_warm);
+criterion_main!(benches);