Skip to content

Commit 23772e8

Browse files
zackeesclaude
zackees
and
claude
committed
ci(library-selection): #205 acceptance + bench workflows + bench/ stub
Wires the gates we already shipped into CI and stubs the warm-cache benchmark home that #205 calls for. No runtime changes. ## What ships ### CI workflows - `.github/workflows/acceptance-205.yml` — runs the `#[ignore]`'d Phase 6 gates from PR #208 (`teensylc_acceptance` + `stm32_acceptance`) on a manual trigger, nightly schedule, and PRs that touch resolver-affecting paths. 45-minute timeout per job; `fail-fast: false` so a teensy regression doesn't mask an stm32 one. - `.github/workflows/bench-205.yml` — runs the criterion benches from PR #210 (`scan_throughput` + `resolve_cold`) and uploads the HTML reports as 14-day artifacts. Manual + scoped-PR triggers; no CI threshold gates yet (P-02 / P-03 numbers captured for review until runner variance is characterized). ### `bench/fastled-examples/` stub - `bench/README.md` — top-level orientation. - `bench/fastled-examples/README.md` — placeholder for the warm-cache matrix benchmark called for by #205 P-01 / AC#5. Marked as awaiting Phase 4 (zccache K/V cache, gated on zackees/zccache#130) since there's no warm path to measure today. - `docs/INDEX.md` row added. ### Issue housekeeping - Closed #204 with a comment referencing PR #207 (foundation) + PR #208 (acceptance gate at `tests/teensylc_acceptance.rs`). - Commented on already-closed #202 noting the same resolution path for STM32 SPI auto-discovery. ## Verification - YAML files parse via `pyyaml.safe_load`. - Workflow paths-filters point at real files (`crates/fbuild-build/tests/{teensylc,stm32}_acceptance.rs`, the two `benches/*.rs` files). - `[[bench]]` names match `Cargo.toml` (`scan_throughput`, `resolve_cold`). ## Out of scope (still tracked) - Phase 4 — zccache K/V memoization (zackees/zccache#130). - P-01 (warm matrix) and P-04 (cache-hit) perf gates — depend on Phase 4. - CI threshold gates on P-02 / P-03 — wait for runner-variance data. - Phase 8.b — final `framework_libs.rs` cleanup once Phase 4 lands. - Baseline numeric capture (`tasks/baseline-205.md` placeholder; needs the new acceptance workflow to finish a successful nightly run). Refs: #202 (commented), #204 (closed), #205, zackees/zccache#130 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 0bf4b2f commit 23772e8

5 files changed

Lines changed: 204 additions & 0 deletions

File tree

‎.github/workflows/acceptance-205.yml‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
name: Acceptance gates (#205)
2+
3+
on:
4+
# Don't run on every PR -- these tests take 10+ minutes each because they
5+
# download Teensyduino + arm-gcc + STM32duino. Trigger explicitly.
6+
workflow_dispatch:
7+
schedule:
8+
# Nightly at 04:00 UTC. Catches drift in Teensyduino / STM32duino
9+
# framework downloads and fbuild's resolver behaviour.
10+
- cron: '0 4 * * *'
11+
pull_request:
12+
paths:
13+
# Run when something that could affect resolver behaviour changes.
14+
- 'crates/fbuild-header-scan/**'
15+
- 'crates/fbuild-library-select/**'
16+
- 'crates/fbuild-build/src/framework_libs.rs'
17+
- 'crates/fbuild-build/src/teensy/**'
18+
- 'crates/fbuild-build/src/stm32/**'
19+
- 'crates/fbuild-build/tests/teensylc_acceptance.rs'
20+
- 'crates/fbuild-build/tests/stm32_acceptance.rs'
21+
- '.github/workflows/acceptance-205.yml'
22+
23+
env:
24+
CARGO_TERM_COLOR: always
25+
RUSTFLAGS: "-D warnings"
26+
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
27+
28+
jobs:
29+
acceptance:
30+
name: Acceptance (${{ matrix.gate }})
31+
runs-on: ubuntu-latest
32+
timeout-minutes: 45
33+
strategy:
34+
fail-fast: false
35+
matrix:
36+
include:
37+
- gate: teensyLC
38+
test_bin: teensylc_acceptance
39+
test_fn: teensylc_blink_meets_205_acceptance_criteria
40+
- gate: stm32f103c8 SPI
41+
test_bin: stm32_acceptance
42+
test_fn: stm32f103c8_blink_with_spi_auto_discovers_library_205_ac4
43+
steps:
44+
- uses: actions/checkout@v6
45+
- uses: astral-sh/setup-uv@v3
46+
- name: Setup soldr
47+
uses: zackees/setup-soldr@v0
48+
with:
49+
cache: true
50+
build-cache: true
51+
target-cache: true
52+
53+
- name: Run ${{ matrix.gate }} acceptance gate
54+
run: |
55+
soldr cargo test -p fbuild-build --test ${{ matrix.test_bin }} -- --ignored --exact ${{ matrix.test_fn }} --nocapture --test-threads=1

‎.github/workflows/bench-205.yml‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
name: Perf benches (#205)
2+
3+
on:
4+
workflow_dispatch:
5+
pull_request:
6+
paths:
7+
- 'crates/fbuild-header-scan/**'
8+
- 'crates/fbuild-library-select/**'
9+
- 'crates/fbuild-test-support/src/mini_framework.rs'
10+
- '.github/workflows/bench-205.yml'
11+
12+
env:
13+
CARGO_TERM_COLOR: always
14+
RUSTFLAGS: "-D warnings"
15+
16+
jobs:
17+
bench:
18+
name: Bench (${{ matrix.bench }})
19+
runs-on: ubuntu-latest
20+
timeout-minutes: 30
21+
strategy:
22+
fail-fast: false
23+
matrix:
24+
include:
25+
- crate: fbuild-header-scan
26+
bench: scan_throughput
27+
target: P-03 (>= 50 MB/s single-thread)
28+
- crate: fbuild-library-select
29+
bench: resolve_cold
30+
target: P-02 (<= 200 ms cold for typical project)
31+
steps:
32+
- uses: actions/checkout@v6
33+
- uses: astral-sh/setup-uv@v3
34+
- name: Setup soldr
35+
uses: zackees/setup-soldr@v0
36+
with:
37+
cache: true
38+
build-cache: true
39+
target-cache: true
40+
41+
- name: "Run ${{ matrix.bench }} (target: ${{ matrix.target }})"
42+
run: |
43+
soldr cargo bench -p ${{ matrix.crate }} --bench ${{ matrix.bench }} -- --output-format bencher | tee bench-${{ matrix.bench }}.txt
44+
45+
- name: Upload criterion HTML report
46+
if: always()
47+
uses: actions/upload-artifact@v4
48+
with:
49+
name: criterion-${{ matrix.bench }}
50+
path: |
51+
target/criterion/**
52+
bench-${{ matrix.bench }}.txt
53+
if-no-files-found: warn
54+
retention-days: 14

‎bench/README.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# bench
2+
3+
End-to-end performance benchmarks that drive the `fbuild` CLI against real
4+
sketches and frameworks, as opposed to per-crate micro-benchmarks.
5+
6+
Per-crate criterion benches live alongside their crate, e.g.:
7+
8+
- `crates/fbuild-header-scan/benches/scan_throughput.rs`
9+
- `crates/fbuild-library-select/benches/resolve_cold.rs`
10+
11+
Run those with:
12+
13+
```bash
14+
uv run soldr cargo bench -p fbuild-library-select --bench resolve_cold
15+
uv run soldr cargo bench -p fbuild-header-scan --bench scan_throughput
16+
```
17+
18+
## Subdirectories
19+
20+
- [`fastled-examples/`](fastled-examples/README.md) — placeholder for the
21+
warm-cache library-selection harness across the FastLED examples matrix
22+
(`FastLED/fbuild#205` AC#5, P-01). Awaits Phase 4 zccache K/V memoization
23+
before there's a warm path to measure.
24+
25+
Other end-to-end matrices (whole-build wall-clock, deploy+flash latency,
26+
emulator boot) may join this directory in the future. Each subdirectory
27+
must carry its own `README.md` explaining what it measures, how to run it,
28+
and which CI gate (if any) it feeds.

‎bench/fastled-examples/README.md‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# bench/fastled-examples
2+
3+
Warm-cache library-selection benchmarks across the FastLED examples matrix.
4+
This is the harness referenced by `FastLED/fbuild#205` for acceptance
5+
criterion **AC#5 / P-01**:
6+
7+
> Warm library-selection on FastLED examples matrix `<= current fbuild
8+
> + 50 ms`.
9+
10+
## Status: empty placeholder
11+
12+
There is no harness here yet, and on purpose. P-01 measures the **warm**
13+
path through the resolver, which depends on the zccache K/V memoization
14+
delivered by `#205` Phase 4 (gated on `zackees/zccache#130`). Until that
15+
lands, every `resolve()` call is cold and there is no warm number to
16+
gate against. Adding a "warm-ish" harness today would just measure cold
17+
work twice and produce a misleading baseline.
18+
19+
When Phase 4 ships and a `zccache` release is cut, a follow-up PR drops
20+
the actual harness into this directory and wires it into CI.
21+
22+
## The plan once Phase 4 lands
23+
24+
1. Iterate the FastLED examples tree (`~/dev/fastled/examples/**`) under
25+
each supported board: at minimum `teensyLC`, `teensy30`, `teensy41`,
26+
`stm32f103c8`, `esp32-s3`, `uno`, `ws2812`. The matrix expands with
27+
board coverage.
28+
2. For each `(example, board)` pair, run the resolver twice:
29+
- **Cold pass.** Empty `~/.zccache/`. Captures the K/V miss path and
30+
the underlying scan + walk + LDF cost. This is the P-02 lane.
31+
- **Warm pass.** Populated `~/.zccache/`. Captures the K/V hit path,
32+
where the only work should be cache lookup + result deserialization.
33+
This is the P-01 lane.
34+
3. Diff the warm scan time against a captured baseline checked into
35+
`tasks/baseline-205.md`. CI fails the job if any `(example, board)`
36+
regresses the warm path by more than 50 ms vs. that baseline (the
37+
`#205` AC#5 threshold).
38+
4. Emit a structured JSON report (`bench/fastled-examples/report.json`)
39+
that future PR comments can diff. Format TBD with the harness.
40+
41+
## Running a partial version today
42+
43+
The closest signal available right now is the per-crate cold-resolve
44+
criterion bench:
45+
46+
```bash
47+
uv run soldr cargo bench -p fbuild-library-select --bench resolve_cold
48+
```
49+
50+
That bench drives a synthetic ~30-library Teensyduino-class tree built
51+
from `fbuild-test-support`'s `MiniFramework` rather than real FastLED
52+
sketches, and it measures the cold path only. It is a useful regression
53+
guard for the resolver itself, but it does **not** satisfy AC#5.
54+
55+
## Cross-links
56+
57+
- Issue: `FastLED/fbuild#205`
58+
- Phase 4 design note (the prerequisite for this directory):
59+
[`../../tasks/zccache-kv-design.md`](../../tasks/zccache-kv-design.md)
60+
- Subsystem architecture:
61+
[`../../docs/architecture/library-selection.md`](../../docs/architecture/library-selection.md)
62+
- Foundation baseline that the warm threshold compares against:
63+
[`../../tasks/baseline-205.md`](../../tasks/baseline-205.md)
64+
- Per-crate cold benches (different scope, same subsystem):
65+
[`../../crates/fbuild-library-select/benches/README.md`](../../crates/fbuild-library-select/benches/README.md),
66+
[`../../crates/fbuild-header-scan/benches/README.md`](../../crates/fbuild-header-scan/benches/README.md)

‎docs/INDEX.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ A grep-friendly FAQ that maps common questions to the file that answers them. Bo
3232
| Why does warm-pass build take ~30 s per sketch? (#91) | [PERF_WARM_BUILD.md](PERF_WARM_BUILD.md) |
3333
| What does `FBUILD_PERF_LOG=1` do? | [PERF_WARM_BUILD.md](PERF_WARM_BUILD.md#instrumentation) |
3434
| How fast is `soldr` when building fbuild itself? | [SOLDR_BUILD_PERF.md](SOLDR_BUILD_PERF.md) |
35+
| Where do end-to-end perf benchmarks live (FastLED matrix, P-01)? | [../bench/fastled-examples/README.md](../bench/fastled-examples/README.md) |
3536
| What architecture docs should I read for a given crate? | [CLAUDE.md](CLAUDE.md) |
3637

3738
## Conventions

0 commit comments

Comments
 (0)