Skip to content

Support multiple benchmark packages per macOS workflow invocation#3602

Open
rnro wants to merge 3 commits into
apple:mainfrom
rnro:macos_benchmarks_multiple_packages
Open

Support multiple benchmark packages per macOS workflow invocation#3602
rnro wants to merge 3 commits into
apple:mainfrom
rnro:macos_benchmarks_multiple_packages

Conversation

@rnro
Copy link
Copy Markdown
Contributor

@rnro rnro commented May 20, 2026

Motivation

  • External callers of macos_benchmarks.yml (e.g. swift-log) currently invoke the reusable workflow once per benchmark package, paying self-hosted macOS runner-recycling cost for each invocation.

Modifications

  • Add a new benchmark_package_paths input to .github/workflows/macos_benchmarks.yml and mark the singular benchmark_package_path as deprecated. The run step forwards both inputs straight to the script via BENCHMARK_PACKAGE_PATH and BENCHMARK_PACKAGE_PATHS environment variables; the workflow itself does no path parsing.
  • Teach scripts/check_benchmark_thresholds.sh to accept BENCHMARK_PACKAGE_PATHS as either a JSON array of strings (auto-detected by a leading [, requires jq) or a newline-separated list, falling back to the legacy BENCHMARK_PACKAGE_PATH when unset or when the JSON form is an empty array.
  • The script now loops over the resolved package list, wraps each iteration in a GitHub Actions ::group::, and emits an ::error:: summary listing failed packages at the end, exiting non-zero if any failed.

Result

  • A single workflow invocation runs all listed packages serially on each Swift/Xcode runner, paying runner-recycling cost once per Swift version instead of once per (version x package).
  • Existing single-package callers keep working unchanged.
  • Adopters of the script in other contexts (e.g. the Linux benchmarks.yml) can pick up multi-package support by passing BENCHMARK_PACKAGE_PATHS without reimplementing parsing or failure-collection logic.

An example of this working apple/swift-log#466

Motivation

* External callers of `macos_benchmarks.yml` (e.g. swift-log) currently invoke the reusable workflow once per benchmark package, paying self-hosted macOS runner-recycling cost for each invocation.
* There was no way to consolidate multiple packages into a single workflow call.

Modifications

* Add a new `benchmark_package_paths` input to `.github/workflows/macos_benchmarks.yml` and mark the singular `benchmark_package_path` as deprecated. The run step forwards both inputs straight to the script via `BENCHMARK_PACKAGE_PATH` and `BENCHMARK_PACKAGE_PATHS` environment variables; the workflow itself does no path parsing.
* Teach `scripts/check_benchmark_thresholds.sh` to accept `BENCHMARK_PACKAGE_PATHS` as either a JSON array of strings (auto-detected by a leading `[`, requires `jq`) or a newline-separated list, falling back to the legacy `BENCHMARK_PACKAGE_PATH` when unset or when the JSON form is an empty array.
* The script now loops over the resolved package list, wraps each iteration in a GitHub Actions `::group::`, and emits an `::error::` summary listing failed packages at the end, exiting non-zero if any failed.

Result

* A single workflow invocation runs all listed packages serially on each Swift/Xcode runner, paying runner-recycling cost once per Swift version instead of once per (version x package).
* Existing single-package callers keep working unchanged.
* Adopters of the script in other contexts (e.g. the Linux `benchmarks.yml`) can pick up multi-package support by passing `BENCHMARK_PACKAGE_PATHS` without reimplementing parsing or failure-collection logic.
@rnro rnro added the semver/none No version bump required. label May 20, 2026
@rnro rnro marked this pull request as ready for review May 20, 2026 21:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

semver/none No version bump required.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants