perf: reduce test-suite execution time (tracking)

[Chemaclass]

Tracking: reduce test-suite execution time

Status 2026-07-12 (round 1 complete, commit 1ae3e82)

Round 1 landed (#767-#771). Re-measured on the same machine (macOS, default bash 3.2.57, Apple Silicon):

• 100 trivial tests: ~0.68s (was ~5.0s) — per-test framework overhead down from ~48ms to ~6.8ms (~7x).
• Full suite --parallel: ~52s for 1250 tests (baseline was ~48s for ~1160 tests, so roughly flat wall-clock while the suite grew ~8%). The dominant remaining cost is the 258 nested ./bashunit cold starts in the acceptance suite, not per-test overhead.

Child issues — round 1 (done)

• perf(state): avoid base64 forks for empty title and hook message in export_subshell_context #762 perf(state): avoid base64 forks for empty title and hook message (PR perf(state): skip base64 forks for empty subshell-context fields #768)
• perf(runner): cache data-provider lookup per file instead of grep+sed per test #763 perf(runner): cache data-provider lookup per file (PR perf(runner): cache data-provider lookup per file #767)
• perf(runner): remove avoidable subshell forks from the run_test hot path #764 perf(runner): remove avoidable subshell forks from run_test (PR perf(runner): remove avoidable subshell forks from the run_test hot path #770)
• perf(clock): skip per-test clock reads when the clock impl forks an interpreter #765 perf(clock): skip per-test clock reads when the clock forks an interpreter (PR perf(runner): auto-skip per-test timing when the clock forks an interpreter #769)
• perf(parallel): detect CPU count and benchmark a default --jobs cap #766 perf(parallel): --jobs auto CPU-count cap (PR feat(parallel): add --jobs auto to cap concurrency at CPU cores #771)

Child issues — round 2

Measured and filed at commit 1ae3e82, ordered by expected impact:

• perf(assert): remove per-assertion subshell forks (strip_ansi, label, alias probe) #772 perf(assert): per-assertion subshell forks — strip_ansi capture (~0.77ms per assert_equals), failure-path label forks, alias probe
• perf(tags): build per-file function-to-tags map in one pass instead of per-function grep/sed walk #773 perf(tags): per-function grep/sed walk when --tag is used (~14ms extra per test, runs twice per function)
• perf(runner): remaining run_test hot-path forks (labels, clock capture, parallel::is_enabled) #774 perf(runner): remaining run_test forks — 2x normalize label captures, clock $(...) capture for pure-bash impls, parallel::is_enabled recomputed 3x/test
• perf(snapshot): remove success-path forks from assert_match_snapshot #775 perf(snapshot): ~6-8 forks per assert_match_snapshot even on the success path

Smaller leftovers, not worth their own issue yet (fold into a round-2 PR if touching the area): per-arg base64 -d in data-provider parsing (src/runner.sh:816), jobs -r | wc -l fork per wait_for_job_slot iteration on bash >= 4.3 (src/runner.sh:305), check_duplicate_functions grep+awk+sort per file (src/helpers.sh:174).

Next frontier (needs measurement before filing)

The acceptance suite's 258 nested ./bashunit subprocess runs now dominate wall-clock. Candidate directions, each needs profiling first: trimming cold-start cost (sourcing + env resolution), sharing one built artifact across nested runs, or converting acceptance tests that don't exercise the CLI surface into functional tests.

Constraints for all work

• Bash 3.0+ compatibility is non-negotiable (no declare -A, no [[ ]], no ${var,,}, no negative array indexing, no &>>). The perf target machine IS bash 3.2 on macOS.
• TDD, quality gate (./bashunit tests/, ./bashunit --parallel tests/, make sa, make lint, shfmt -w .) on every PR.
• One issue per PR; no batching unrelated changes.

Shared benchmark protocol

Run before and after each change, report numbers in the PR:

mkdir -p /tmp/bashunit-bench
for i in $(seq 1 100); do echo "function test_noop_$i() { assert_true true; }"; done > /tmp/bashunit-bench/noop100_test.sh
time ./bashunit --no-parallel /tmp/bashunit-bench/noop100_test.sh   # current: ~0.68s

# fork census (external binaries per run)
PS4='+X ${BASH_SOURCE##*/}:$LINENO ' bash -x ./bashunit --no-parallel /tmp/bashunit-bench/noop100_test.sh \
  2>/tmp/bashunit-bench/trace.txt >/dev/null
grep -oE ' (perl|base64|grep|sed|mktemp|rm|date) ' /tmp/bashunit-bench/trace.txt | sort | uniq -c

For assertion-heavy paths (#772) use 100 tests x 10 assert_equals (current: ~1.45s); for tags (#773) use 100 tagged tests with --tag foo (current: ~2.10s).

Also considered and NOT filed: lazy-sourcing large optional modules (coverage.sh, learn.sh) at startup. Sourcing all of src/ measures only ~28ms, so the win across 258 nested cold starts is ~5-8s and it complicates the single-file release build. Revisit as part of the "next frontier" item above.

Original baseline (commit 4d80e7c, for history)

• 100 trivial tests took ~5.0s (~48ms/test): 2x perl clock forks, 3x base64 pipelines, 1x grep+sed provider probe, 1x base64 -d decode, ~12-14 $(...) subshells per test.
• Sequential full suite >4 min; --parallel ~48s with no default job cap.

Line references valid at commit 1ae3e82 and will drift.