Add CI/Pipeline Automation for MySQL (issue #872)

[crprashant]

Add CI/Pipeline Automation for MySQL (Issue #872)

Port of the MariaDB CI pipeline to MySQL. Follows the same shape as the
PostgreSQL port (PR #875).

What this adds

• src/mysql/mysqlci.tcl — MySQL provider (clone / build / package /
  install / init / start / ping / run_sql / shutdown)
• config/ci.xml — new <mysql> provider block:
  • source: https://github.com/mysql/mysql-server.git (tag mysql-8.0.39)
  • build_cmd uses -DFORCE_INSOURCE_BUILD=1 (MySQL 8.0 refuses in-source
    builds without this; the CI build tree is the source tree)
  • <pipeline> = clone build package commit_msg install init run_sql:shutdown start ping run_sql:change_password start_tests:oltp run_sql:shutdown
• scripts/tcl/mysql/tprocc/mysql_tprocc_run_profile.tcl — TPROC-C profile
  script used by start_tests:oltp

Checklist from issue #872

• ci.xml entries for MySQL (clone URL, tag, build/install commands)
• Triggered by GitHub webhook, Run Benchmark button, and cipush
  CLI (uses the existing process_request dispatcher; no changes needed
  to the trigger layer)
• Clone, build and install MySQL
• Run HammerDB TPROC-C workloads (profile script present; executed via
  the existing start_tests:oltp step)
• Capture results via the jobs framework (NOPM/TPM through the standard
  jobs module; identical path to PG/MariaDB)
• Comparison charts / CI output reuse the existing web-service rendering

E2E validation (WSL Ubuntu 24.04, 16-core, 32 GB)

Ran cipush end-to-end against a MySQL 8.0.39 build compiled from the
upstream source. Steps validated via a targeted harness that replays each
pipeline step against the preserved build tree:

Step	Result
clone	✅ upstream tag mysql-8.0.39
build (-DFORCE_INSOURCE_BUILD=1)	✅ ~35 min, Release config
package / install	✅ mysql-8.0.39-linux-x86_64
init (--initialize-insecure)	✅ data + redo dirs populated
start (--daemonize)	✅ pid + /tmp/mysql.sock
ping (SELECT @@version)	✅ returns 8.0.39
change_password	✅ ALTER USER … IDENTIFIED BY 'mysql'
shutdown	✅ clean exit

Bugs found and fixed during E2E validation

All of these surfaced only when running against an actual MySQL build; none
are reachable from static review.

1. In-source build refusal — MySQL 8.0 aborts CMake unless
   -DFORCE_INSOURCE_BUILD=1 is set. Added to ci.xml build_cmd.
2. Missing apt deps — documented libaio-dev, libncurses-dev,
   pkg-config as prerequisites (same set as upstream BUILD/INSTALL).
3. Glob case — upstream tarball names use lowercase mysql-*, not
   MySQL-*. Fixed in 6 glob sites.
4. Binary name — client is bin/mysql, not bin/MySQL.
5. Unix-socket vs. native auth — MariaDB's default is the
   unix_socket plugin (OS user ⇒ DB root). MySQL does not auto-map; the
   client has to pass -u root explicitly.
6. Post-change_password shutdown — after ALTER USER … IDENTIFIED BY 'mysql', subsequent admin calls need the password. Added a
   || <cmd with -pmysql> bash fallback so the same run_sql wrapper works
   before and after change_password.
7. --daemonize required for start — without it, the Tcl pipe close
   tears down the mysqld parent. MariaDB uses a bash -c '... &' fork; MySQL
   8.0 supports the native --daemonize flag.
8. --defaults-file must be the first CLI argument — if anything else
   (even --daemonize) precedes it, mysqld parses it as an unknown
   variable. mysql_start now hoists it to the front of the argv.
9. --thread-pool-size is Enterprise-only — MySQL Community does not
   ship the thread_pool plugin; passing the flag causes
   Plugin 'thread_pool' is not loaded and an abort. Removed from the
   MySQL arglist (MariaDB still has it — MariaDB bundles thread_pool).
10. Socket path casing — scripts used /tmp/MySQL.sock; MySQL defaults
    to /tmp/mysql.sock. Unified throughout.

Build prerequisites (Linux)

sudo apt-get install -y build-essential cmake bison pkg-config \
    libssl-dev libaio-dev libncurses-dev libreadline-dev zlib1g-dev

Notes for reviewers

• No changes to src/generic/ — the port is entirely additive, as
  requested in the issue.
• Pipeline order matches MariaDB's: the run_sql:shutdown right after
  init exists so change_password can run against a freshly started
  daemon with a known root password.
• start_tests:oltp reuses the normal TPROC-C profile code path; no
  MySQL-specific scheduling knobs are required.
• Pass the same webhook / CLI triggers as other providers — no new entry
  points are introduced.

[sm-shaw]

Issues are similar to #875
In genci cifix wont run as there is no rule for my.cnf or myio.cnf. This is update for PostgreSQL but not for MySQL.

PostgreSQL
 set name [file tail $f]
        if {$name ni {"maria.cnf" "mariaio.cnf"}} {
        if {$name ni {"maria.cnf" "mariaio.cnf" "postgresql.conf" "postgresqlio.conf"}} {
            putsci "CI FIX ERROR: no download rule for missing file $f"
            return 1
        }

---
MySQL
        set name [file tail $f] 
        if {$name ni {"maria.cnf" "mariaio.cnf"}} {
            putsci "CI FIX ERROR: no download rule for missing file $f"
            return 1
        }

When you run through the web interface there is the same error as PostgreSQL

in the pipelines module there is a similar error to PostgreSQL

613c613
< if {$dbprefix ne "maria"} {
---
> if {$dbprefix ne "maria" && $dbprefix ne "my"} {
874c874
<             if {$dbprefix ne "maria"} {
---
>             if {$dbprefix ne "maria" && $dbprefix ne "my" } {

Note that when enabled. We need to support maria my and pg as prefixes.

For consistency it will be best if we add all of the supported databases in the initial dropdown in the same order as the GUI)

and report the same message for Oracle, SQL Server and Db2 for the time being. There will be plans to support these but only for already running instances as we will not be able to build them.

This could be added to either PR but we want to make sure that for both it is not blocked as it currently is.

[crprashant]

Thanks @sm-shaw — addressed in 245c107. Summary of changes and verification below.

Changes

src/generic/genci.tcl — cifix allowlist (line 767)
Extended the missing-file allowlist so MySQL (and PostgreSQL) configs are pulled from hammerdb.com/ci-config instead of failing:

-        if {$name ni {"maria.cnf" "mariaio.cnf"}} {
+        if {$name ni {"maria.cnf" "mariaio.cnf" "my.cnf" "myio.cnf" "postgresql.conf" "postgresqlio.conf"}} {

Included the PostgreSQL entries here too so this benefits #875 even before that PR merges, as you suggested.

modules/pipelines-1.0.tm — dbprefix gate at lines 613 and 874
Inverted the gate from "must be maria" to "block ora/mssqls/db2", so MariaDB, MySQL, and PostgreSQL all flow through the run form:

-if {$dbprefix ne "maria"} {
+if {$dbprefix in {ora mssqls db2}} {

The friendly banner text was updated to:

<DB Name> pipelines are not yet enabled.
Support for already-running Oracle, SQL Server, and Db2 instances is planned for a future release.

The Windows-only banner was also generalised from "MariaDB CI pipelines are not available on Windows" to "CI pipelines are not available on Windows in this release".

modules/pipelines-1.0.tm — database dropdown (line 858)
Now lists all six supported databases in the same order as the GUI tree (Oracle → SQL Server → Db2 → MySQL → PostgreSQL → MariaDB):

-        foreach {pfx label} [ list maria MariaDB mysql MySQL pg PostgreSQL ] {
+        foreach {pfx label} [ list ora Oracle mssqls "SQL Server" db2 Db2 mysql MySQL pg PostgreSQL maria MariaDB ] {

modules/pipelines-1.0.tm — __db_label helper
Added Oracle, SQL Server, and Db2 cases so banner titles read "Oracle pipelines are not yet enabled." instead of "ora pipelines are not yet enabled." Switched the run-action / banner code paths from [string totitle $dbprefix] to [__db_label $dbprefix] for consistent labelling.

Verification

• info complete — both modified files pass Tcl syntax check.
• Label resolution (smoke-tested with tclsh):

  ora    -> Oracle
  mssqls -> SQL Server
  db2    -> Db2
  mysql  -> MySQL
  pg     -> PostgreSQL
  maria  -> MariaDB
  

• Gate behaviour:

  ora    blocked=1   mssqls blocked=1   db2   blocked=1
  mysql  blocked=0   pg     blocked=0   maria blocked=0
  

• git diff --stat: modules/pipelines-1.0.tm +15/-12, src/generic/genci.tcl +1/-1.

Net effect: MariaDB / MySQL / PostgreSQL pipelines all run through the Web Service page; Oracle / SQL Server / Db2 show the friendly "support planned for already-running instances" banner. The same fixes naturally unblock the PG flow once #875 lands as well.

[sm-shaw]

Many thanks for the update will retest.

[sm-shaw]

One additional issue with the MySQL implementation. The parameter innodb-log-file-size is no longer supported so in 2 places in mysqlci this parameter needs to be replaced with the following which is supported from 8.0.30.

lappend arglist "--innodb-redo-log-capacity=[calc_redo_mb]M"

This will prevent running against MySQL 5.7 but that is EOL.
Although I have not tested the update yet, still working with the previous. One update thought on the previous comment is that if we have the commercial databases in the list then by default the page will show not supported. So we should comment this out and only have the following in the available databases
mysql -> MySQL
pg -> PostgreSQL
maria -> MariaDB
And add the others when enabled.

[crprashant]

Thanks @sm-shaw — both items addressed in 701ccf9.

1. --innodb-log-file-size → --innodb-redo-log-capacity

Replaced in both call sites in src/mysql/mysqlci.tcl (mysql_init line 725, mysql_start line 942):

-        lappend arglist "--innodb-log-file-size=[calc_redo_mb]M"
+        lappend arglist "--innodb-redo-log-capacity=[calc_redo_mb]M"

Agree on dropping MySQL 5.7 — it's been EOL since Oct 2023, and 8.0.30 is over three years old at this point. MariaDB still uses --innodb-log-file-size so src/mariadb/mariaci.tcl is intentionally left untouched.

2. Trim dropdown to only enabled databases

Reverted the dropdown to just the three databases that actually run pipelines today. Oracle / SQL Server / Db2 are commented out with a TODO so they can be re-added later when build/install (or already-running-instance) support lands:

-        foreach {pfx label} [ list ora Oracle mssqls "SQL Server" db2 Db2 mysql MySQL pg PostgreSQL maria MariaDB ] {
+        # Future entries to add to the dropdown when supported:
+        #   ora    Oracle
+        #   mssqls "SQL Server"
+        #   db2    Db2
+        foreach {pfx label} [ list mysql MySQL pg PostgreSQL maria MariaDB ] {

The defensive if {$dbprefix in {ora mssqls db2}} gate at lines 613 / 874 is intentionally left in place — if anyone navigates to pipelines?db=ora (or mssqls/db2) directly, they still get the friendly "… pipelines are not yet enabled. Support for already-running … instances is planned for a future release." banner instead of falling through to a broken run form.

Verification

• info complete — modules/pipelines-1.0.tm and src/mysql/mysqlci.tcl both pass Tcl syntax check.
• git diff --stat: modules/pipelines-1.0.tm +8/-2, src/mysql/mysqlci.tcl +2/-2.
• Dropdown now renders only MySQL · PostgreSQL · MariaDB, matching the set of DBs that have working CI pipelines.

The matching dropdown / __db_label updates are also live on PR #875, so both PRs stay in sync.

Pushed as 701ccf9. Ready for re-test.

[sm-shaw]

This is getting close. I am still testing the previous commit to get MySQL to successfully run and identifying issues. Once it has run successfully I will add a note and then after a further commit to fix there all issues should be identified and it should run successfully.

So now there are 2 current issues:

1. In the following script, the database is set incorrectly so it fails trying to load the MariaDB library and fails to recognise any of the diset settings which is correct as it is setting mysql settings when the database is set to the maria prefix.

scripts/tcl/mysql/tprocc/mysql_tprocc_run_profile.tcl

set tmpdir $::env(TMP)
puts "SETTING CONFIGURATION"
dbset db maria
dbset bm TPC-C

This should be changed to:

dbset db mysql

Also MySQL does not support dynamic purge, so this line can be removed:
diset tpcc mysql_purge true

Dictionary "tpcc" for MySQL exists but key "mysql_purge" doesn't, key "mysql_purge" cannot be found in any MySQL dictionary
Type "print dict" for valid dictionaries and keys for MySQL

2. There are equivalent Python scripts to the Tcl ones. These are not run directly by automation but need to be provided for consistency as the functionality will also run under the Python CLI.

Note that in these scripts it should not copy directly the unfixed MySQL scripts as above i.e. it should set the database to MySQL and not MariaDB.

The highlighted MariaDB files are in bold. These need to be added for PostgreSQL and MySQL.

Maria
tprocc$
maria_tprocc_buildschema.py maria_tprocc_py.sh
maria_tprocc_checkschema.py maria_tprocc_result.py
maria_tprocc_deleteschema.py maria_tprocc_run_profile.py
maria_tprocc_profile_py.sh maria_tprocc_run.py
maria_tprocc_py.ps1 maria_tprocc_single_py.sh

MySQL
tprocc$ ls
mysql_tprocc_buildschema.py mysql_tprocc_py.ps1 mysql_tprocc_run.py
mysql_tprocc_checkschema.py mysql_tprocc_py.sh
mysql_tprocc_deleteschema.py mysql_tprocc_result.py

PostgreSQL
/tprocc$ ls
pg_tprocc_buildschema.py pg_tprocc_py.ps1 pg_tprocc_run.py
pg_tprocc_checkschema.py pg_tprocc_py.sh
pg_tprocc_deleteschema.py pg_tprocc_result.py

Maria
ls tproch
maria_tproch_buildschema.py maria_tproch_py.ps1 maria_tproch_run.py
maria_tproch_checkschema.py maria_tproch_py.sh maria_tproch_single_py.sh
maria_tproch_deleteschema.py maria_tproch_result.py

MySQL
tprocc$ ls
mysql_tprocc_buildschema.py mysql_tprocc_py.ps1 mysql_tprocc_run.py
mysql_tprocc_checkschema.py mysql_tprocc_py.sh
mysql_tprocc_deleteschema.py mysql_tprocc_result.py

PostgreSQL
tproch$ ls
pg_tproch_buildschema.py pg_tproch_py.ps1 pg_tproch_run.py
pg_tproch_checkschema.py pg_tproch_py.sh
pg_tproch_deleteschema.py pg_tproch_result.py

[sm-shaw]

OK, confirm that I have now successfully run a single TPROC-C against MySQL with the changes above.

Still need to test TPROC-H, profile, compare and manual cipush trigger but it looks like most of the practical issues in building and installing MySQL are resolved so once the updates are done I will switch to testing the newer version.

[crprashant]

@sm-shaw thanks for the catch and for confirming the TPROC-C single run worked locally with the patch. Pushed 1e1c0ee addressing both items:

1. mysql_tprocc_run_profile.tcl fixes

• dbset db maria → dbset db mysql (was incorrectly inherited from the MariaDB template)
• Removed diset tpcc mysql_purge true — mysql_purge doesn't exist in the MySQL TPC-C dict (it's a MariaDB-only key)

2. Added Python siblings (mirroring MariaDB layout)

MySQL (this PR):

• scripts/python/mysql/tprocc/mysql_tprocc_run_profile.py
• scripts/python/mysql/tprocc/mysql_tprocc_profile_py.sh
• scripts/python/mysql/tprocc/mysql_tprocc_single_py.sh
• scripts/python/mysql/tproch/mysql_tproch_single_py.sh

The new mysql_tprocc_run_profile.py correctly uses dbset("db", "mysql") and does not include the bogus mysql_purge key.

PostgreSQL: I'll push the equivalent pg_tprocc_run_profile.py, pg_tprocc_profile_py.sh, pg_tprocc_single_py.sh, pg_tproch_single_py.sh to PR #875 next.

Verification

• info complete on the modified Tcl file → 1
• python3 -m py_compile on the new .py → OK
• bash -n on the three new .sh files → OK

Ready for re-review.

[crprashant]

Rebased feature/872-mysql-ci-pipeline onto latest master to clear the conflict that appeared after #875 was merged.

What changed

• The conflict was in modules/pipelines-1.0.tm (database dropdown). Master keeps the full GUI-ordered list (Oracle / SQL Server / Db2 / MySQL / PostgreSQL / MariaDB); my earlier "dropdown trim" hunk has been dropped so this branch matches master exactly there. The dbprefix gate already shows the friendly "not yet enabled" banner for ora / mssqls / db2, so behaviour is preserved.
• git rebase master also auto-dropped one earlier commit (the cifix allowlist enable for my/pg) because it was already upstream via Add PostgreSQL CI/Pipeline automation (#871) #875.
• Force-pushed (--force-with-lease) at 61e79fc.

Verification

• tclsh info complete on the touched Tcl files (modules/pipelines-1.0.tm, src/mysql/mysqlci.tcl, src/generic/genci.tcl, scripts/tcl/mysql/tprocc/mysql_tprocc_run_profile.tcl) → OK.
• gh pr view 876 reports mergeable: MERGEABLE (was conflicted before).

Final layered diff vs master is just the MySQL CI port (mysqlci.tcl, config/ci.xml MySQL block, scripts under scripts/tcl/mysql/ + scripts/python/mysql/).

[sm-shaw]

Approved after changes.