Add option to run the tests in parallel.#18
Open
ParveezBaig wants to merge 1 commit into
Open
Conversation
ParveezBaig
force-pushed
the
run-tests-parallel
branch
4 times, most recently
from
June 16, 2026 07:28
2a8b70d to
d3ba2ab
Compare
ParveezBaig
changed the title
There was a problem hiding this comment.
Pull request overview
This PR enhances the QA framework to support running suite tests in parallel workers, improving isolation (per-worker workdirs/ports/logs) and adding more robust timeouts/log handling to better detect and report hangs/failures.
Changes:
- Added parallel execution via
qa_framework.py --number-of-workers/-w, plus CSV support for--tests/-tand--suites/-s. - Introduced worker isolation primitives (per-worker workdirs, bounded port allocation bands, and per-test log files).
- Added/updated timeouts and error handling in several utility/startup paths; improved docs and logging.
Reviewed changes
Copilot reviewed 33 out of 33 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| util/utility.py | Adds port allocation helpers, updates sysbench PID detection, and introduces more timeout-based waits/logging. |
| util/sysbench_run.py | Adds sysbench run timeout handling via subprocess and moves log output under the selected workdir. |
| util/rqg_datagen.py | Runs RQG data generation with a timeout to fail gracefully on hangs. |
| util/pxc_startup.py | Refactors PXC startup to be worker-aware (workdirs/ports/keyrings) and adds timeouts around startup. |
| util/ps_startup.py | Refactors PS startup to be worker-aware and switches to deterministic port allocation for parallel runs. |
| util/prepared_statements.sql | Replaces MD5+RAND string generation with SHA2-based generation. |
| util/db_connection.py | Adds connector timeouts and wraps query execution to raise clearer query/timeout errors. |
| util/data_generator.py | Ensures stdout restoration happens reliably via finally. |
| util/createsql.py | Tracks/restores redirected stdout and closes the output file handle safely. |
| test/test_startup.py | Updates startup test flow for new cluster start behavior. |
| suite/upgrade/pxc_upgrade.py | Updates SysbenchRun constructor usage and sysbench PID handling. |
| suite/upgrade/pxc_replication_upgrade.py | Updates SysbenchRun constructor usage and sysbench PID handling. |
| suite/sysbench_run/sysbench_read_only_test.py | Passes workdir into SysbenchRun for correct per-worker logging. |
| suite/sysbench_run/sysbench_oltp_test.py | Passes workdir into SysbenchRun for correct per-worker logging. |
| suite/sysbench_run/sysbench_customized_dataload_test.py | Passes workdir into SysbenchRun for correct per-worker logging. |
| suite/ssl/ssl_qa.py | Passes workdir into SysbenchRun for correct per-worker logging. |
| suite/ssl/encryption_qa.py | Passes workdir/worker_id into startup/sysbench paths and adjusts logging behavior. |
| suite/replication/replication.py | Adjusts replication worker configuration logic and adds stricter mode handling. |
| suite/replication/backup_replication.py | Passes workdir into SysbenchRun for correct per-worker logging. |
| suite/random_qa/random_mysqld_option_test.py | Switches random QA artifacts to use the active workdir and updates SysbenchRun usage. |
| suite/random_qa/pstress_crash_recovery_qa.py | Updates cluster kill helper to accept node list (worker-safe). |
| suite/loadtest/sysbench_wsrep_provider_option_random_test.py | Updates SysbenchRun usage and sysbench PID handling for worker runs. |
| suite/loadtest/sysbench_random_load_test.py | Passes workdir into SysbenchRun for correct per-worker logging. |
| suite/loadtest/sysbench_load_test.py | Passes workdir into SysbenchRun for correct per-worker logging. |
| suite/galera_sr/thread_pool_qa.py | Uses per-node admin port and worker-aware cluster startup. |
| suite/galera_sr/galera_basic_sr_qa.py | Passes workdir into SysbenchRun and adds lock-wait timeout to procedure calls. |
| suite/correctness/crash_recovery.py | Updates SysbenchRun usage and sysbench PID handling for worker runs. |
| suite/correctness/consistency_check.py | Passes workdir into SysbenchRun for correct per-worker logging. |
| suite/correctness/cluster_interaction.py | Updates SysbenchRun usage and sysbench PID handling for worker runs. |
| suite/correctness/chaosmonkey-test.py | Updates SysbenchRun usage and sysbench PID handling for worker runs. |
| README.md | Documents new CLI options, parallel worker layout, logging, and disabled test support. |
| qa_framework.py | Implements test discovery, disabled filtering, per-test logging, and parallel worker execution. |
| base_test.py | Adds worker-id parsing and redirects per-test stdout/stderr to worker-specific log files. |
Comments suppressed due to low confidence (1)
util/ps_startup.py:35
component_keyring_file_path()builds the keyring path under the globalWORKDIR, which is not worker-isolated. With parallel workers (workdir/wN), this can cause different workers to share/overwrite the same keyring files during encryption runs.
def component_keyring_file_path(node_number: int):
keyring_dir = os.path.join(workdir, 'keyring')
os.makedirs(keyring_dir, exist_ok=True)
return os.path.join(keyring_dir, 'psnode' + str(node_number) + '_' + comp_name)
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
ParveezBaig
force-pushed
the
run-tests-parallel
branch
2 times, most recently
from
June 18, 2026 07:17
f30a38a to
0956ed0
Compare
ParveezBaig
force-pushed
the
run-tests-parallel
branch
from
June 19, 2026 09:52
0956ed0 to
6be00bc
Compare
ParveezBaig
force-pushed
the
run-tests-parallel
branch
2 times, most recently
from
June 29, 2026 07:47
d372ec8 to
b56ae2c
Compare
tplavcic
requested changes
Jun 29, 2026
ParveezBaig
force-pushed
the
run-tests-parallel
branch
from
July 7, 2026 11:26
b56ae2c to
4201407
Compare
ParveezBaig
force-pushed
the
run-tests-parallel
branch
from
July 7, 2026 12:15
4201407 to
3b422bd
Compare
Comment on lines
+56
to
+61
| if comment == "sta": | ||
| if int(version) >= int("080027"): | ||
| my_extra = my_extra + ' --replica-parallel-workers=0' | ||
| else: | ||
| if int(version) < int("080027"): | ||
| my_extra = my_extra + ' --slave-parallel-workers=4' |
| print(query) | ||
| if int(os.system(EXPORT_LUA_PATH + ";" + query)) != 0: | ||
| print(full_cmd) | ||
| process = subprocess.Popen(full_cmd, shell=True, start_new_session=True, stderr=subprocess.DEVNULL) |
Comment on lines
+68
to
+71
| if _is_query_timeout(query_error): | ||
| raise QueryTimeoutError( | ||
| "Query timed out after " + str(QUERY_TIMEOUT) + " seconds: " + str(query) | ||
| ) from query_error |
Comment on lines
19
to
22
| def test_initialize_cluster(self): | ||
| self.assertIsNot(cluster.initialize_cluster(), 1, | ||
| 'Could not initialize database directory. ' | ||
| 'Please check error log') |
Comment on lines
32
to
36
| def test_connection_check(self): | ||
| self.assertIsNot(connection_check.connection_check(), 1, | ||
| global pxc_nodes | ||
| for node in pxc_nodes: | ||
| self.assertEqual(node.connection_check() , 0, | ||
| 'Could not establish DB connection') |
| if debug == 'YES': | ||
| print(backup_cmd) | ||
| os.system(backup_cmd) | ||
| run_with_timeout(backup_cmd, "xtrabackup backup") |
| if debug == 'YES': | ||
| print(prepare_backup) | ||
| os.system(prepare_backup) | ||
| run_with_timeout(prepare_backup, "xtrabackup prepare") |
| if debug == 'YES': | ||
| print(copy_backup) | ||
| os.system(copy_backup) | ||
| run_with_timeout(copy_backup, "xtrabackup copy-back") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Modified qa_framework.py to make it possible to run the tests in parallel.
Added new option -w to qa_framework.py to run the tests in parallel.
Updated -t option to accept multiple tests.
Updated -s option to accept multiple suites.
The output of specific tests is now printed to test log files. Whereas the final result pass/fail is printed in the main output log file and console.
Added a way to skip a test from running if it is listed under disabled.list file in script directory which would make it easier to temporarily disable any failing test.
Updated many commands to run with timeout and fail gracefully in case of timeout exceeds which would help in detection of failures in case of server hang or environment issues.
Running the tests will automatically cleanup the directory set in workdir under config.ini file if it already exists before running any tests.
Fixed failing backup_replication.py test in encryption mode.
Test results:
Cmd: python3 qa_framework.py --suite=correctness -e -w 2
################################################################################
Adding tests from correctness suite
Work directory /home/parveez.baig/tests_data already exists, removing it
Creating work directory /home/parveez.baig/tests_data
################################################################################
Test correctness.cluster_interaction.py w2 [Pass]
Test correctness.chaosmonkey-test.py w2 [Pass]
Test correctness.crash_recovery.py w1 [Pass]
Test correctness.consistency_check.py w2 [Pass]
Cmd: python3 qa_framework.py --suite=galera_sr,sysbench_run -e -w 2
################################################################################
Adding tests from galera_sr suite
Adding tests from sysbench_run suite
Work directory /home/parveez.baig/tests_data already exists, removing it
Creating work directory /home/parveez.baig/tests_data
################################################################################
Test galera_sr.thread_pool_qa.py w1 [Pass]
Test galera_sr.galera_basic_sr_qa.py w2 [Pass]
Test sysbench_run.sysbench_customized_dataload_test.py w1 [Pass]
Test sysbench_run.sysbench_read_only_test.py w2 [Pass]
Test sysbench_run.sysbench_oltp_test.py w1 [Pass]`