Skip to content

Commit fffb1b4

Browse files
zackeesclaude
zackees
and
claude
authored
feat(library-selection): #218 add bench-fastled-examples Phase 7 warm-cache harness (#219)
Adds a real-FastLED-matrix bench under `bench/fastled-examples/`, complementing the synthetic per-crate `resolve_warm` from #216. For each of a curated subset of FastLED examples (Blink, Pacifica, Animartrix, Audio, BlurBenchmark, Chromancer), runs `resolve_cached` cold + warm against a fresh `KvStore` and reports timings. Asserts `from_cache=true` on the warm pass so silent re-misses surface immediately. Local numbers (Windows release, FastLED master, 6 examples × synthetic 6-lib Teensyduino-class framework): | example | cold (ms) | warm (ms) | speedup | |---------------|----------:|----------:|--------:| | Blink | 923.58 | 11.36 | 81.3x | | Pacifica | 915.98 | 12.64 | 72.4x | | Animartrix | 970.14 | 11.76 | 82.5x | | Audio | 830.51 | 11.74 | 70.7x | | BlurBenchmark | 827.46 | 10.48 | 79.0x | | Chromancer | 844.13 | 10.89 | 77.5x | Warm path comfortably clears AC#5 (≤ +50 ms over current fbuild) at ~11 ms per example. The asymmetry reflects the cost of walking FastLED src/ (~1000 files) cold vs. a `KvStore::get` + bincode decode warm. CI lane added as `workflow_dispatch`-only (needs FastLED checkout step). No threshold gate yet — capture a stable cross-runner baseline first. Closes #218. Refs #205 Phase 7. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 86bb2d9 commit fffb1b4

8 files changed

Lines changed: 442 additions & 63 deletions

File tree

‎.github/workflows/bench-205.yml‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,3 +55,47 @@ jobs:
5555
bench-${{ matrix.bench }}.txt
5656
if-no-files-found: warn
5757
retention-days: 14
58+
59+
fastled-examples:
60+
name: Bench (fastled-examples — workflow_dispatch only)
61+
if: github.event_name == 'workflow_dispatch'
62+
runs-on: ubuntu-latest
63+
timeout-minutes: 30
64+
steps:
65+
- uses: actions/checkout@v6
66+
- uses: astral-sh/setup-uv@v3
67+
- name: Setup soldr
68+
uses: zackees/setup-soldr@v0
69+
with:
70+
cache: true
71+
build-cache: true
72+
target-cache: true
73+
74+
- name: Checkout FastLED
75+
uses: actions/checkout@v6
76+
with:
77+
repository: FastLED/FastLED
78+
path: external/fastled
79+
# Pinned to a FastLED release tag for reproducibility. Bump in
80+
# lockstep with the sample-numbers table in
81+
# bench/fastled-examples/README.md when retaking baseline.
82+
ref: "3.10.3"
83+
84+
- name: Run bench-fastled-examples
85+
env:
86+
FASTLED_DIR: ${{ github.workspace }}/external/fastled
87+
run: |
88+
mkdir -p bench/fastled-examples
89+
soldr cargo run --release -p fbuild-bench-fastled-examples -- \
90+
--json bench/fastled-examples/report.json | tee bench/fastled-examples/report.md
91+
92+
- name: Upload report
93+
if: always()
94+
uses: actions/upload-artifact@v4
95+
with:
96+
name: bench-fastled-examples
97+
path: |
98+
bench/fastled-examples/report.md
99+
bench/fastled-examples/report.json
100+
if-no-files-found: warn
101+
retention-days: 30

‎Cargo.lock‎

Lines changed: 12 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ members = [
1414
"crates/fbuild-test-support",
1515
"crates/fbuild-header-scan",
1616
"crates/fbuild-library-select",
17+
"bench/fastled-examples",
1718
]
1819

1920
[workspace.package]

‎bench/README.md‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -19,10 +19,13 @@ uv run soldr cargo bench -p fbuild-header-scan --bench scan_throughput
1919

2020
## Subdirectories
2121

22-
- [`fastled-examples/`](fastled-examples/README.md) — reserved for the
23-
real-FastLED warm-cache library-selection matrix (`FastLED/fbuild#205`
24-
AC#5, P-01) once `~/dev/fastled` is wired in. The synthetic warm-path
25-
baseline already lives in `crates/fbuild-library-select/benches/resolve_warm.rs`.
22+
- [`fastled-examples/`](fastled-examples/README.md) — real-FastLED
23+
warm-cache library-selection matrix (`FastLED/fbuild#205` AC#5, P-01).
24+
Discovers examples under `$FASTLED_DIR` (default `~/dev/fastled`),
25+
runs the resolver cold + warm per example, and reports timings.
26+
Run with `uv run soldr cargo run --release -p fbuild-bench-fastled-examples`.
27+
The synthetic warm-path baseline lives in
28+
`crates/fbuild-library-select/benches/resolve_warm.rs`.
2629

2730
Other end-to-end matrices (whole-build wall-clock, deploy+flash latency,
2831
emulator boot) may join this directory in the future. Each subdirectory

‎bench/fastled-examples/Cargo.toml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
[package]
2+
name = "fbuild-bench-fastled-examples"
3+
description = "Warm-cache library-selection bench across the FastLED examples matrix (#205 Phase 7)"
4+
version.workspace = true
5+
edition.workspace = true
6+
rust-version.workspace = true
7+
license.workspace = true
8+
publish = false
9+
10+
[[bin]]
11+
name = "bench-fastled-examples"
12+
path = "src/main.rs"
13+
14+
[dependencies]
15+
fbuild-library-select = { path = "../../crates/fbuild-library-select" }
16+
fbuild-packages = { path = "../../crates/fbuild-packages" }
17+
fbuild-test-support = { path = "../../crates/fbuild-test-support" }
18+
zccache-artifact = { workspace = true }
19+
tempfile = { workspace = true }
20+
serde_json = { workspace = true }

‎bench/fastled-examples/README.md‎

Lines changed: 88 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -1,69 +1,98 @@
11
# bench/fastled-examples
22

3-
Warm-cache library-selection benchmarks across the FastLED examples matrix.
4-
This is the harness referenced by `FastLED/fbuild#205` for acceptance
5-
criterion **AC#5 / P-01**:
6-
7-
> Warm library-selection on FastLED examples matrix `<= current fbuild
8-
> + 50 ms`.
9-
10-
## Status: empty placeholder
11-
12-
There is no harness in this directory yet — the real per-board, per-example
13-
matrix needs a checked-out FastLED tree (`~/dev/fastled`) and orchestrator
14-
wiring that routes through `resolve_cached`. That work is tracked
15-
separately. The synthetic warm baseline (`MiniFramework`-backed cache hit,
16-
no real FastLED) already exists at
17-
[`../../crates/fbuild-library-select/benches/resolve_warm.rs`](../../crates/fbuild-library-select/benches/resolve_warm.rs)
18-
and is the first-pass regression guard for the cache-hit path.
19-
20-
Phase 4 K/V memoization itself shipped in PR #212, so the warm path is
21-
real and measurable today; what is missing here is the multi-board,
22-
real-sketch matrix that AC#5 requires.
23-
24-
## The plan once the FastLED tree is wired in
25-
26-
1. Iterate the FastLED examples tree (`~/dev/fastled/examples/**`) under
27-
each supported board: at minimum `teensyLC`, `teensy30`, `teensy41`,
28-
`stm32f103c8`, `esp32-s3`, `uno`, `ws2812`. The matrix expands with
29-
board coverage.
30-
2. For each `(example, board)` pair, run the resolver twice:
31-
- **Cold pass.** Empty `~/.zccache/`. Captures the K/V miss path and
32-
the underlying scan + walk + LDF cost. This is the P-02 lane.
33-
- **Warm pass.** Populated `~/.zccache/`. Captures the K/V hit path,
34-
where the only work should be cache lookup + result deserialization.
35-
This is the P-01 lane.
36-
3. Diff the warm scan time against a captured baseline checked into
37-
`tasks/baseline-205.md`. CI fails the job if any `(example, board)`
38-
regresses the warm path by more than 50 ms vs. that baseline (the
39-
`#205` AC#5 threshold).
40-
4. Emit a structured JSON report (`bench/fastled-examples/report.json`)
41-
that future PR comments can diff. Format TBD with the harness.
42-
43-
## Running the synthetic mini benches today
44-
45-
The closest signal available right now without a FastLED checkout is the
46-
per-crate cold and warm criterion benches against `MiniFramework`:
3+
Warm-cache library-selection benchmark across a curated FastLED examples
4+
matrix. This is the AC#5 / P-01 measurement for
5+
[`FastLED/fbuild#205`](https://github.com/FastLED/fbuild/issues/205).
6+
7+
## What it measures
8+
9+
For each example sketch under `$FASTLED_DIR/examples/`, runs the
10+
`fbuild_library_select::cache::resolve_cached` resolver twice against a
11+
fresh `KvStore`:
12+
13+
- **Cold** — empty cache. Wall-clock includes the scanner walk over the
14+
FastLED `src/` tree (~1000 files), the 2-pass LDF reconciliation, and
15+
the cache write. This dominates total time.
16+
- **Warm** — cache pre-populated. Wall-clock includes the cache-key
17+
compute (sorted seed/header content hashing, bounded by `cache_key`
18+
itself) and the bincode decode of the cached `Selection`.
19+
`from_cache = true` is asserted so silent re-misses surface
20+
immediately.
21+
22+
The framework library set is a synthetic Teensyduino-class stub built
23+
via `MiniFramework`. The bench measures resolver throughput, not the
24+
correctness of which libraries get selected — that is the acceptance-test
25+
layer (`crates/fbuild-build/tests/teensylc_acceptance.rs`).
26+
27+
## Running
28+
29+
`FASTLED_DIR` is required — there is no implicit default, since the
30+
correct path is host-dependent (CI uses `external/fastled` from the
31+
workflow checkout, developers use whatever convention they like) and a
32+
silent fallback would mask configuration mistakes.
4733

4834
```bash
49-
uv run soldr cargo bench -p fbuild-library-select --bench resolve_cold
50-
uv run soldr cargo bench -p fbuild-library-select --bench resolve_warm
35+
FASTLED_DIR=/path/to/fastled \
36+
uv run soldr cargo run --release -p fbuild-bench-fastled-examples
37+
38+
# Emit a JSON report alongside stdout
39+
FASTLED_DIR=/path/to/fastled \
40+
uv run soldr cargo run --release -p fbuild-bench-fastled-examples \
41+
-- --json bench/fastled-examples/report.json
5142
```
5243

53-
Those benches drive a synthetic ~30-library Teensyduino-class tree built
54-
from `fbuild-test-support`'s `MiniFramework` rather than real FastLED
55-
sketches. They are useful regression guards for the resolver and its
56-
cache layer respectively, but they do **not** satisfy AC#5 on their own.
44+
If any example fails to measure (missing sketch, KvStore error, warm
45+
miss) the binary exits non-zero rather than skipping the row. CI must
46+
treat a partial matrix as a failure, not a pass.
47+
48+
## Sample numbers
49+
50+
Captured 2026-05-10 on Windows / Ryzen workstation, FastLED `main`,
51+
release build:
52+
53+
| example | cold (ms) | warm (ms) | speedup |
54+
|---------------|----------:|----------:|--------:|
55+
| Blink | 923.58 | 11.36 | 81.3x |
56+
| Pacifica | 915.98 | 12.64 | 72.4x |
57+
| Animartrix | 970.14 | 11.76 | 82.5x |
58+
| Audio | 830.51 | 11.74 | 70.7x |
59+
| BlurBenchmark | 827.46 | 10.48 | 79.0x |
60+
| Chromancer | 844.13 | 10.89 | 77.5x |
61+
62+
The warm path comfortably clears AC#5 (≤ +50 ms over current fbuild) at
63+
~11 ms per example. The ~75x speedup reflects the cost asymmetry between
64+
walking the FastLED `src/` tree (~1000 files) and a `KvStore`
65+
get + bincode decode of a serialized `Selection`.
66+
67+
## Curated example set
68+
69+
The harness intentionally runs a small representative subset rather than
70+
all 80+ examples. Adding more is cheap — see `EXAMPLES` in `src/main.rs`.
71+
The current set spans:
72+
73+
- Trivial single-strip sketches (`Blink`)
74+
- Animation-heavy sketches (`Pacifica`, `Animartrix`)
75+
- I/O-heavy sketches (`Audio`)
76+
- Throughput stress sketches (`BlurBenchmark`, `Chromancer`)
77+
78+
## CI
79+
80+
The `fastled-examples` job in `.github/workflows/bench-205.yml` is
81+
`workflow_dispatch`-only because it requires a FastLED checkout. CI
82+
checks out FastLED at a pinned release tag (currently `3.10.3`) so
83+
measurements are reproducible, then runs the bench and uploads the JSON
84+
report as an artifact. Bumping the pin is a deliberate baseline event —
85+
update both the workflow `ref:` and the sample-numbers table above in
86+
lockstep.
87+
88+
There is no automatic CI gate on the warm timings yet — first capture a
89+
stable cross-runner baseline, then a follow-up adds the threshold gate.
5790

5891
## Cross-links
5992

60-
- Issue: `FastLED/fbuild#205`
61-
- Phase 4 K/V memoization (shipped in #212):
62-
[`../../tasks/zccache-kv-design.md`](../../tasks/zccache-kv-design.md)
93+
- Issue: [`FastLED/fbuild#205`](https://github.com/FastLED/fbuild/issues/205)
94+
- This harness: [`FastLED/fbuild#218`](https://github.com/FastLED/fbuild/issues/218)
95+
- Per-crate synthetic warm bench:
96+
[`crates/fbuild-library-select/benches/resolve_warm.rs`](../../crates/fbuild-library-select/benches/resolve_warm.rs)
6397
- Subsystem architecture:
64-
[`../../docs/architecture/library-selection.md`](../../docs/architecture/library-selection.md)
65-
- Foundation baseline that the warm threshold compares against:
66-
[`../../tasks/baseline-205.md`](../../tasks/baseline-205.md)
67-
- Per-crate cold + warm benches (different scope, same subsystem):
68-
[`../../crates/fbuild-library-select/benches/README.md`](../../crates/fbuild-library-select/benches/README.md),
69-
[`../../crates/fbuild-header-scan/benches/README.md`](../../crates/fbuild-header-scan/benches/README.md)
98+
[`docs/architecture/library-selection.md`](../../docs/architecture/library-selection.md)

‎bench/fastled-examples/src/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# bench/fastled-examples/src
2+
3+
Source for the `bench-fastled-examples` binary. See the parent
4+
[`README.md`](../README.md) for what the harness does and how to run it.

0 commit comments

Comments
 (0)