test: Fix flaky full-stack tests by mocking service ports#7244
Draft
okurz wants to merge 1 commit into
Draft
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #7244 +/- ##
==========================================
- Coverage 99.97% 99.96% -0.01%
==========================================
Files 419 419
Lines 44141 44155 +14
==========================================
+ Hits 44128 44138 +10
- Misses 13 17 +4 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
perlpunk
reviewed
Apr 2, 2026
| # reuse it in listen to prevent race condition | ||
| # | ||
| # Potentially this approach can also be used in production code. | ||
| sub setup_random_base_port { |
Contributor
There was a problem hiding this comment.
Now I'm wondering why mock_service_ports was removed: 8d7b9a5
I guess the according comment block was overlooked.
Member
Author
There was a problem hiding this comment.
huh. That's why this comment block was a bit dangling :D
Member
Author
There was a problem hiding this comment.
replaced all the randomization by using re-introduced mock_service_ports
Martchus
approved these changes
Apr 10, 2026
3fd7398 to
f8dae91
Compare
f8dae91 to
a92fc66
Compare
8818363 to
be83974
Compare
Motivation: Multiple full-stack tests were failing intermittently when run in parallel due to "Address already in use" errors. They all defaulted to the same base port (9526), causing conflicts in concurrent test environments. Design Choices: - Introduced assign_free_service_ports in OpenQA::Test::Utils. It sets per-service environment variables (OPENQA_PORT_*) using OS-allocated free ports via Mojo::IOLoop::Server->generate_port instead of blindly guessing or using a fixed base port. - Applied assign_free_service_ports to t/05-scheduler-full.t, t/25-cache-service.t, t/33-developer_mode.t, t/43-scheduling-and-worker-scalability.t, and t/full-stack.t, replacing the prior setup_random_base_port approach. - Each service gets an independently chosen free port, reducing the risk that one occupied port breaks all services. - Removed duplicate randomization logic and redundant comments across multiple test files. - Cleaned up BEGIN blocks and imports in full-stack tests. - Re-ordered assertions and imports in affected test files for better maintainability and reliability. - Added code comments to clarify complex logic, such as \U usage in env var lookups. Benefits: - Greatly reduces port conflicts via OS-level allocation, addressing race conditions that setup_random_base_port could not prevent. - Improves test stability and reliability for parallel execution in CI/CD. - Centralizes logic for consistent port selection strategy across tests. - Cleaner and more maintainable test code.
be83974 to
3eac708
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation:
Multiple full-stack tests were failing intermittently when run in parallel
due to "Address already in use" errors. They all defaulted to the same
base port (9526), causing conflicts in concurrent test environments.
Design Choices:
removed as dead code in 8d7b9a5. It mocks OpenQA::Utils::service_port
via Test::MockModule to allocate independent free ports per service using
Mojo::IOLoop::Server->generate_port, which asks the OS for a truly free
port instead of blindly guessing.
t/33-developer_mode.t, t/43-scheduling-and-worker-scalability.t, and
t/full-stack.t, replacing the prior setup_random_base_port approach.
block), eliminating the risk that one occupied port breaks all services.
multiple test files.
Benefits:
that setup_random_base_port could not prevent.
port selection strategy.