Skip to content

dev-cmd/tests: add except flag to skip tests#23089

Closed
cho-m wants to merge 3 commits into
mainfrom
tests-except
Closed

dev-cmd/tests: add except flag to skip tests#23089
cho-m wants to merge 3 commits into
mainfrom
tests-except

Conversation

@cho-m

@cho-m cho-m commented Jul 13, 2026

Copy link
Copy Markdown
Member

  • Have you followed our Contributing guidelines?
  • Have you checked for other open Pull Requests for the same change?
  • Have you explained what your changes do? Performance claims (e.g. "this is faster") must include Hyperfine benchmarks.
  • Have you explained why you'd like these changes included, not just what they do?
  • For bug fixes, have you given step-by-step brew commands to reproduce the bug?
  • Have you written new tests (excluding integration tests)? Here's an example.
  • Have you successfully run brew lgtm (style, typechecking and tests) locally?

  • AI was used to generate or assist with generating this PR.

Will likely use this to bypass any remaining test failures in portable Ruby PR to fix up our workflows. Many failures seem to be environment differences that are difficult to reconcile (e.g. Tier 3 vs Tier 1, container setup, etc).

Flag name was chosen based on brew audit which also has --only and --except.

The handling is based as --only flag (other than not providing line number support):

only.split(",").flat_map do |test|
test_name, line = test.split(":", 2)
tests = if line.present?
parallel = false
["test/#{test_name}_spec.rb:#{line}"]
else
Dir.glob("test/{#{test_name},#{test_name}/**/*}_spec.rb")


Needed to manually update RBI as brew tc --update kept failing with stacktrace.

Copilot AI review requested due to automatic review settings July 13, 2026 21:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an --except flag to brew tests to allow skipping specific test specs or directories, mirroring the existing --only selection behavior and updating Sorbet RBI accordingly.

Changes:

  • Add --except= CLI flag to brew tests and exclude matching spec files/directories from the run list.
  • Update option conflict rules to prevent combining --except with --changed/--only.
  • Extend the Sorbet DSL RBI for Homebrew::DevCmd::Tests::Args with the new except accessor.

Reviewed changes

Copilot reviewed 1 out of 2 changed files in this pull request and generated 3 comments.

File Description
Library/Homebrew/dev-cmd/tests.rb Adds --except flag parsing and applies exclusions to the computed RSpec file list.
Library/Homebrew/sorbet/rbi/dsl/homebrew/dev_cmd/tests.rbi Updates the generated RBI to include args.except typing.
Files not reviewed (1)
  • Library/Homebrew/sorbet/rbi/dsl/homebrew/dev_cmd/tests.rbi: File type not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread Library/Homebrew/dev-cmd/tests.rb
Comment thread Library/Homebrew/dev-cmd/tests.rb
Comment thread Library/Homebrew/dev-cmd/tests.rb
cho-m and others added 2 commits July 13, 2026 17:56
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 7 changed files in this pull request and generated 2 comments.

Files not reviewed (1)
  • Library/Homebrew/sorbet/rbi/dsl/homebrew/dev_cmd/tests.rbi: File type not supported

Comment on lines +96 to +104
if (except = args.except)
except_tests = except.split(",").flat_map do |test_name|
tests = Dir.glob("test/{#{test_name},#{test_name}/**/*}_spec.rb")
raise UsageError, "Invalid `--except` argument: #{test_name}" if tests.blank?

tests
end
files -= except_tests
end
Comment on lines +96 to +104
if (except = args.except)
except_tests = except.split(",").flat_map do |test_name|
tests = Dir.glob("test/{#{test_name},#{test_name}/**/*}_spec.rb")
raise UsageError, "Invalid `--except` argument: #{test_name}" if tests.blank?

tests
end
files -= except_tests
end
@cho-m
cho-m marked this pull request as draft July 13, 2026 22:44

@MikeMcQuaid MikeMcQuaid left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I'm not sure this really makes sense as an approach, though. Instead can we skip those environments/tests where we're getting failures.

It also might be that it doesn't make sense to run brew tests for Portable Ruby when building in non-Tier 1 environments or we use --only to run a small subset instead.

@cho-m

cho-m commented Jul 14, 2026

Copy link
Copy Markdown
Member Author

@cho-m cho-m closed this Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants