Skip to content

test: Fix flaky full-stack tests by mocking service ports#7244

Draft
okurz wants to merge 1 commit into
os-autoinst:masterfrom
okurz:feature/randomize_ports_tests
Draft

test: Fix flaky full-stack tests by mocking service ports#7244
okurz wants to merge 1 commit into
os-autoinst:masterfrom
okurz:feature/randomize_ports_tests

Conversation

@okurz
Copy link
Copy Markdown
Member

@okurz okurz commented Apr 2, 2026

Motivation:
Multiple full-stack tests were failing intermittently when run in parallel
due to "Address already in use" errors. They all defaulted to the same
base port (9526), causing conflicts in concurrent test environments.

Design Choices:

  • Restored mock_service_ports in OpenQA::Test::Utils which was previously
    removed as dead code in 8d7b9a5. It mocks OpenQA::Utils::service_port
    via Test::MockModule to allocate independent free ports per service using
    Mojo::IOLoop::Server->generate_port, which asks the OS for a truly free
    port instead of blindly guessing.
  • Applied mock_service_ports to t/05-scheduler-full.t, t/25-cache-service.t,
    t/33-developer_mode.t, t/43-scheduling-and-worker-scalability.t, and
    t/full-stack.t, replacing the prior setup_random_base_port approach.
  • Each service gets an independently chosen free port (not a fixed-offset
    block), eliminating the risk that one occupied port breaks all services.
  • Removed duplicate randomization logic and redundant comments across
    multiple test files.
  • Cleaned up BEGIN blocks and imports in full-stack tests.

Benefits:

  • Guarantees free ports via OS-level allocation, eliminating race conditions
    that setup_random_base_port could not prevent.
  • Eliminates port conflicts during parallel test execution.
  • Improves test stability and reliability for CI/CD.
  • Centralizes logic for easier future maintenance and consistent
    port selection strategy.
  • Cleaner and more maintainable test code.

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 2, 2026

Codecov Report

❌ Patch coverage is 81.81818% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 99.96%. Comparing base (91af0a8) to head (a92fc66).
⚠️ Report is 87 commits behind head on master.

Files with missing lines Patch % Lines
t/lib/OpenQA/Test/Utils.pm 42.85% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7244      +/-   ##
==========================================
- Coverage   99.97%   99.96%   -0.01%     
==========================================
  Files         419      419              
  Lines       44141    44155      +14     
==========================================
+ Hits        44128    44138      +10     
- Misses         13       17       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread t/lib/OpenQA/Test/Utils.pm Outdated
# reuse it in listen to prevent race condition
#
# Potentially this approach can also be used in production code.
sub setup_random_base_port {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now I'm wondering why mock_service_ports was removed: 8d7b9a5
I guess the according comment block was overlooked.

Copy link
Copy Markdown
Member Author

@okurz okurz Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

huh. That's why this comment block was a bit dangling :D

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

replaced all the randomization by using re-introduced mock_service_ports

@okurz okurz force-pushed the feature/randomize_ports_tests branch from 3fd7398 to f8dae91 Compare April 13, 2026 11:22
@okurz okurz changed the title test: Fix flaky full-stack tests by randomizing base ports test: Fix flaky full-stack tests by mocking service ports Apr 13, 2026
@okurz okurz force-pushed the feature/randomize_ports_tests branch from f8dae91 to a92fc66 Compare April 13, 2026 20:24
@okurz okurz force-pushed the feature/randomize_ports_tests branch 2 times, most recently from 8818363 to be83974 Compare April 24, 2026 21:42
Motivation:
Multiple full-stack tests were failing intermittently when run in parallel
due to "Address already in use" errors. They all defaulted to the same
base port (9526), causing conflicts in concurrent test environments.

Design Choices:
- Introduced assign_free_service_ports in OpenQA::Test::Utils. It sets
  per-service environment variables (OPENQA_PORT_*) using OS-allocated
  free ports via Mojo::IOLoop::Server->generate_port instead of blindly
  guessing or using a fixed base port.
- Applied assign_free_service_ports to t/05-scheduler-full.t,
  t/25-cache-service.t, t/33-developer_mode.t,
  t/43-scheduling-and-worker-scalability.t, and t/full-stack.t,
  replacing the prior setup_random_base_port approach.
- Each service gets an independently chosen free port, reducing the risk
  that one occupied port breaks all services.
- Removed duplicate randomization logic and redundant comments across
  multiple test files.
- Cleaned up BEGIN blocks and imports in full-stack tests.
- Re-ordered assertions and imports in affected test files for better
  maintainability and reliability.
- Added code comments to clarify complex logic, such as \U usage in env
  var lookups.

Benefits:
- Greatly reduces port conflicts via OS-level allocation, addressing
  race conditions that setup_random_base_port could not prevent.
- Improves test stability and reliability for parallel execution in CI/CD.
- Centralizes logic for consistent port selection strategy across tests.
- Cleaner and more maintainable test code.
@okurz okurz force-pushed the feature/randomize_ports_tests branch from be83974 to 3eac708 Compare May 21, 2026 18:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants