[rb] Run unit tests as a single Bazel target instead of per-file by titusfortner · Pull Request #17616 · SeleniumHQ/selenium

titusfortner · 2026-06-02T22:46:31Z

Ruby unit tests take a long time under Bazel, especially on Windows.
Measured impact on the unit-test jobs on GitHub runner:

Job	Trunk (73 targets)	PR (1 target)	Δ
3.3.11, windows	13m 34s	7m 10s	−47%
truffleruby, ubuntu	7m 47s	3m 01s	−61%
jruby, ubuntu	7m 31s	3m 42s	−51%
3.3.11, ubuntu	4m 56s	3m 17s	−33%
3.3.11, macos	3m 47s	3m 08s	−17%
4.0.4, ubuntu	3m 03s	3m 08s	+3%

On RBE:
Aggregate compute goes from 24 minutes to 1 minute. Wall clock doesn't change much because everything is in parallel on RBE, but reducing total compute is good.

💥 What does this PR do?

Runs the Ruby unit specs as a single Bazel target (//rb/spec/unit:unit) instead of generating
one rb_test target per spec file (~73 targets).

🔧 Implementation Notes

No CI changes were needed: ci-ruby.yml selects unit tests via --test_size_filters small //rb/spec/...,
which still matches the single size = "small" target.

The motivation was Windows CI time. The cost on Windows/JRuby/TruffleRuby isn't test execution — it's
the per-target fixed overhead (no symlinks on Windows means the runfiles tree is copied per target,
plus interpreter/JVM startup) paid ~73×. Consolidating to one target pays that tax once.

Trade-offs, since this is a deliberate change:

One failure fails the whole target, and editing any spec invalidates the cache for all of them
(full re-run)
Cross-target parallelism is lost, but on Windows that parallelism was net-negative anyway (each
parallel target re-paying the copy/startup tax).

🤖 AI assistance

No substantial AI assistance used
AI assisted (complete below)
- Tool(s): Claude Code
- What was generated:
- I reviewed all AI output and can explain the change

🔄 Types of changes

Cleanup (formatting, renaming)

qodo-code-review · 2026-06-02T22:46:49Z

Review Summary by Qodo

Consolidate Ruby unit tests into single Bazel target for faster CI

✨ Enhancement

Walkthroughs

Description

• Consolidate 73 Ruby unit test targets into single //rb/spec/unit:unit target
• Remove per-file rb_unit_test macro and individual BUILD.bazel files
• Significantly reduce CI execution time (17-61% improvement across platforms)
• Update documentation and test references to reflect new structure

Diagram

flowchart LR
  A["73 individual unit test targets"] -->|consolidate| B["Single //rb/spec/unit:unit target"]
  B -->|reduces| C["CI execution time"]
  C -->|Windows| D["-47%"]
  C -->|TruffleRuby| E["-61%"]
  C -->|JRuby| F["-51%"]

File Changes

1. rb/spec/tests.bzl Refactoring +0/-14

Remove rb_unit_test macro definition

rb/spec/tests.bzl

2. rb/spec/unit/BUILD.bazel ✨ Enhancement +48/-0

Create consolidated unit test target

rb/spec/unit/BUILD.bazel

3. rb/spec/BUILD.bazel Refactoring +2/-61

Replace individual unit targets with consolidated target

rb/spec/BUILD.bazel

View more (19)

4. rb/spec/unit/selenium/BUILD.bazel Refactoring +0/-26

Remove individual devtools and server test targets

rb/spec/unit/selenium/BUILD.bazel

5. rb/spec/unit/selenium/devtools/BUILD.bazel Refactoring +0/-13

Remove cdp_client_generator test target

rb/spec/unit/selenium/devtools/BUILD.bazel

6. rb/spec/unit/selenium/webdriver/BUILD.bazel Refactoring +0/-47

Remove all individual webdriver unit test targets

rb/spec/unit/selenium/webdriver/BUILD.bazel

7. rb/spec/unit/selenium/webdriver/bidi/BUILD.bazel Refactoring +0/-10

Remove individual bidi test targets

rb/spec/unit/selenium/webdriver/bidi/BUILD.bazel

8. rb/spec/unit/selenium/webdriver/chrome/BUILD.bazel Refactoring +0/-14

Remove individual chrome test targets

rb/spec/unit/selenium/webdriver/chrome/BUILD.bazel

9. rb/spec/unit/selenium/webdriver/common/BUILD.bazel Refactoring +0/-58

Remove individual common module test targets

rb/spec/unit/selenium/webdriver/common/BUILD.bazel

10. rb/spec/unit/selenium/webdriver/common/fedcm/BUILD.bazel Refactoring +0/-16

Remove individual fedcm test targets

rb/spec/unit/selenium/webdriver/common/fedcm/BUILD.bazel

11. rb/spec/unit/selenium/webdriver/common/interactions/BUILD.bazel Refactoring +0/-13

Remove individual interactions test targets

rb/spec/unit/selenium/webdriver/common/interactions/BUILD.bazel

12. rb/spec/unit/selenium/webdriver/devtools/BUILD.bazel Refactoring +0/-10

Remove individual devtools test targets

rb/spec/unit/selenium/webdriver/devtools/BUILD.bazel

13. rb/spec/unit/selenium/webdriver/edge/BUILD.bazel Refactoring +0/-14

Remove individual edge test targets

rb/spec/unit/selenium/webdriver/edge/BUILD.bazel

14. rb/spec/unit/selenium/webdriver/firefox/BUILD.bazel Refactoring +0/-19

Remove individual firefox test targets

rb/spec/unit/selenium/webdriver/firefox/BUILD.bazel

15. rb/spec/unit/selenium/webdriver/ie/BUILD.bazel Refactoring +0/-14

Remove individual ie test targets

rb/spec/unit/selenium/webdriver/ie/BUILD.bazel

16. rb/spec/unit/selenium/webdriver/remote/BUILD.bazel Refactoring +0/-14

Remove individual remote test targets

rb/spec/unit/selenium/webdriver/remote/BUILD.bazel

17. rb/spec/unit/selenium/webdriver/remote/http/BUILD.bazel Refactoring +0/-10

Remove individual http test targets

rb/spec/unit/selenium/webdriver/remote/http/BUILD.bazel

18. rb/spec/unit/selenium/webdriver/safari/BUILD.bazel Refactoring +0/-14

Remove individual safari test targets

rb/spec/unit/selenium/webdriver/safari/BUILD.bazel

19. rb/spec/unit/selenium/webdriver/support/BUILD.bazel Refactoring +0/-28

Remove individual support module test targets

rb/spec/unit/selenium/webdriver/support/BUILD.bazel

20. rb/spec/unit/selenium/webdriver/remote/features_spec.rb 🐞 Bug fix +2/-2

Use described_class instead of hardcoded Features

rb/spec/unit/selenium/webdriver/remote/features_spec.rb

21. README.md 📝 Documentation +5/-4

Update test documentation for consolidated unit tests

README.md

22. third_party/firebug/BUILD.bazel ⚙️ Configuration changes +1/-1

Update visibility to consolidated unit test target

third_party/firebug/BUILD.bazel

qodo-code-review · 2026-06-02T22:46:50Z

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (1) 🔗 Cross-repo conflicts (0)

1. Edits under third_party/firebug 📘 Rule violation ⚙ Maintainability

Description

This PR modifies third_party/firebug/BUILD.bazel by changing exports_files visibility, which
violates the requirement that third_party/ be treated as read-only. Such edits can cause untracked
divergence from vendored dependencies and complicate dependency updates.

Code

third_party/firebug/BUILD.bazel[12]

Evidence
PR Compliance ID 3 forbids modifications under third_party/. The diff shows
third_party/firebug/BUILD.bazel was changed to update exports_files visibility to include
//rb/spec/unit:__pkg__.
AGENTS.md: Treat third_party/ as Read-only and Avoid Editing Generated Bazel Output
third_party/firebug/BUILD.bazel[9-13]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The PR modifies a file under `third_party/` (`third_party/firebug/BUILD.bazel`) to change `exports_files` visibility, but `third_party/` must be treated as read-only.

## Issue Context
The visibility change appears to be needed so the new consolidated Ruby unit test target (`//rb/spec/unit:unit`) can depend on the XPI files as `data`. Instead of changing `third_party`, add an in-repo wrapper target in a package that already has visibility to these files (e.g., keep/restore a small package at the previously-allowed path) and have `//rb/spec/unit:unit` depend on that wrapper.

## Fix Focus Areas
- third_party/firebug/BUILD.bazel[9-13]
- rb/spec/unit/BUILD.bazel[17-47]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

2. Flaky retries dropped 🐞 Bug ☼ Reliability

Description

Some unit specs were previously run in Bazel targets marked flaky = True, but consolidation into a
single //rb/spec/unit:unit target (not marked flaky) removes Bazel’s ability to retry only those
flaky specs and will stop --flaky_test_attempts from applying to them. This increases CI
brittleness (or forces rerunning the entire unit suite) if those specs continue to be intermittent.

Code

rb/spec/unit/selenium/webdriver/BUILD.bazel[L27-31]

Evidence
The PR removes per-spec unit test targets where flaky = True could be applied, but the new
consolidated unit target is not marked flaky; CI uses --flaky_test_attempts in other Bazel runs,
which only affects targets marked flaky.
rb/spec/unit/BUILD.bazel[17-30]
scripts/github-actions/ci-build.sh[17-21]
.github/workflows/ci-ruby.yml[80-88]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
Previously, some Ruby unit specs were executed in Bazel targets explicitly marked `flaky = True`. With this PR, unit specs run under a single `rb_test(name = "unit")` target that is not marked flaky, so Bazel flaky retries can no longer be applied only to the historically flaky subset.

### Issue Context
Bazel’s `flaky` attribute is per-target, and CI commonly uses `--flaky_test_attempts` for retry behavior. With a single monolithic target, the only ways to regain retries are (a) mark the entire unit suite flaky (expensive/noisy) or (b) split the known-flaky specs into a small separate target that remains `flaky = True`.

### Fix Focus Areas
- rb/spec/unit/BUILD.bazel[17-48]

Suggested implementation:
- Keep `//rb/spec/unit:unit` as the main non-flaky suite.
- Add a second target (e.g. `//rb/spec/unit:unit_flaky`) with `flaky = True` and `srcs = [ ...specific flaky *_spec.rb paths... ]`.
- Exclude those files from the main `unit` target’s `glob()` (use `exclude = [...]`).
- Ensure `//rb/spec/...` still selects both targets under `--test_size_filters small` (they will, as long as both are `size = "small"`).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

selenium-ci · 2026-06-02T22:46:55Z

Thank you, @titusfortner for this code suggestion.

The support packages contain example code that many users find helpful, but they do not necessarily represent
the best practices for using Selenium, and the Selenium team is not currently merging changes to them.

After reviewing the change, unless it is a critical fix or a feature that is needed for Selenium
to work, we will likely close the PR.

We actively encourage people to add the wrapper and helper code that makes sense for them to their own frameworks.
If you have any questions, please contact us

[rb] Run unit tests as a single Bazel target instead of per-file

1248995

selenium-ci added C-rb Ruby Bindings B-build Includes scripting, bazel and CI integrations B-devtools Includes everything BiDi or Chrome DevTools related B-support Issue or PR related to support classes labels Jun 2, 2026

qodo-code-review Bot reviewed Jun 2, 2026

View reviewed changes

Comment thread third_party/firebug/BUILD.bazel

titusfortner merged commit f98e870 into trunk Jun 3, 2026
43 checks passed

titusfortner deleted the rb_unit_bazel branch June 3, 2026 00:09

titusfortner mentioned this pull request Jun 3, 2026

[rb] streamline tests on github actions runners #17550

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[rb] Run unit tests as a single Bazel target instead of per-file#17616

[rb] Run unit tests as a single Bazel target instead of per-file#17616
titusfortner merged 1 commit into
trunkfrom
rb_unit_bazel

titusfortner commented Jun 2, 2026

Uh oh!

qodo-code-review Bot commented Jun 2, 2026

Uh oh!

qodo-code-review Bot commented Jun 2, 2026 •

edited

Loading

Uh oh!

selenium-ci commented Jun 2, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

titusfortner commented Jun 2, 2026

💥 What does this PR do?

🔧 Implementation Notes

🤖 AI assistance

🔄 Types of changes

Uh oh!

qodo-code-review Bot commented Jun 2, 2026

Review Summary by Qodo

Walkthroughs

File Changes

Uh oh!

qodo-code-review Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review by Qodo

Uh oh!

selenium-ci commented Jun 2, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

qodo-code-review Bot commented Jun 2, 2026 •

edited

Loading