Fix DB upgrade machinery

[sjpb]

• Version of pymysql depends on python version so need this to be configurable.
• ansible.builtin.expect requires system python to have pexpect package installed

[gemini-code-assist]

Code Review

This pull request updates tasks/upgrade.yml to specify a non-default Python interpreter (/usr/bin/python3.9) for a database query task to ensure a recent pymysql version is used. The feedback suggests avoiding hardcoding this path directly in the task to maintain flexibility, recommending instead the use of a variable with a fallback default.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[sjpb]

Worked ok in slurm appliance CI here: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/29017171482/job/86137980900?pr=994

TASK [stackhpc.openhpc : Check if slurm database requires an upgrade] **********
Slurm Database current version is '15' needs to be at '16'. Conversion needed.
...
TASK [stackhpc.openhpc : Backup Slurm database] ********************************
STDOUT:

+-------------+----------------------------------------+
| Field       | Value                                  |
+-------------+----------------------------------------+
| created_at  | 2026-07-09T13:33:33.709914             |
| description | None                                   |
| id          | 7cdb52ec-6215-43f6-a752-b5f64edd51da   |
| name        | slurmci-RL8-1366-state-20260709T133115 |
| properties  |                                        |
| size        | 10                                     |
| status      | creating                               |
| updated_at  | None                                   |
| volume_id   | e2e1cafe-37dc-4dc8-b702-0a7f28ef5dd1   |
+-------------+----------------------------------------+
...
TASK [stackhpc.openhpc : Ensure Slurm database service started] ****************
changed: [slurmci-RL8-1366-control] => {
...
TASK [stackhpc.openhpc : Run slurmdbd in foreground for upgrade] ***************
STDOUT:

[2026-07-09T13:33:35.934] debug:  Log file re-opened

[2026-07-09T13:33:35.934] error: Unable to open pidfile `/var/run/slurmdbd.pid': Permission denied

[2026-07-09T13:33:35.937] debug:  auth/munge: init: loaded

[2026-07-09T13:33:35.941] debug:  hash/k12: init: init: KangarooTwelve hash plugin loaded

[2026-07-09T13:33:35.942] debug:  tls/none: init: tls/none loaded

[2026-07-09T13:33:35.942] debug2: accounting_storage/as_mysql: init: mysql_connect() called for db slurm_acct_db

[2026-07-09T13:33:35.943] debug2: Attempting to connect to slurmci-RL8-1366-control:3306

[2026-07-09T13:33:35.945] accounting_storage/as_mysql: _check_mysql_concat_is_sane: MySQL server version is: 8.0.30

[2026-07-09T13:33:35.946] debug2: accounting_storage/as_mysql: _check_database_variables: innodb_buffer_pool_size: 1073741824

[2026-07-09T13:33:35.946] debug2: accounting_storage/as_mysql: _check_database_variables: innodb_redo_log_capacity: 104857600

[2026-07-09T13:33:35.946] debug2: accounting_storage/as_mysql: _check_database_variables: innodb_lock_wait_timeout: 900

[2026-07-09T13:33:35.946] debug2: accounting_storage/as_mysql: _check_database_variables: max_allowed_packet: 67108864

[2026-07-09T13:33:35.946] error: Database settings not recommended values: innodb_buffer_pool_size

[2026-07-09T13:33:35.963] adding column id after name in table cluster_table

[2026-07-09T13:33:35.963] dropping column plugin_id_select from table cluster_table

[2026-07-09T13:33:35.963] debug:  Table cluster_table has changed.  Updating...

[2026-07-09T13:33:35.963] debug2: query

alter table cluster_table modify `creation_time` bigint unsigned not null, modify `mod_time` bigint unsigned default 0 not null, modify `deleted` tinyint default 0, modify `name` tinytext not null, add `id` smallint after name, modify `control_host` tinytext not null default '', modify `control_port` int unsigned not null default 0, modify `last_port` int unsigned not null default 0, modify `rpc_version` smallint unsigned not null default 0, modify `classification` smallint unsigned default 0, modify `dimensions` smallint unsigned default 1, modify `flags` int unsigned default 0, modify `federation` tinytext not null, modify `features` text not null default '', modify `fed_id` int unsigned default 0 not null, modify `fed_state` smallint unsigned not null, drop plugin_id_select, drop primary key, add primary key (name(42));

[2026-07-09T13:33:35.990] adding column deleted at the beginning of table txn_table

[2026-07-09T13:33:35.990] adding key archive_delete (deleted) to table txn_table

[2026-07-09T13:33:35.990] debug:  Table txn_table has changed.  Updating...

[2026-07-09T13:33:37.447] debug2: TrackSlurmctldDown     = no

[2026-07-09T13:33:37.447] debug2: accounting_storage/as_mysql: acct_storage_p_get_connection: request new connection 1

[2026-07-09T13:33:37.447] debug2: Attempting to connect to slurmci-RL8-1366-control:3306

[2026-07-09T13:33:37.451] debug2: assoc 2(root, root) has direct parent of 1(root, (null)) /0-root/

[2026-07-09T13:33:37.452] debug2: user root default acct is root

[2026-07-09T13:33:37.453] slurmdbd version 25.05.8 started

[2026-07-09T13:33:37.453] debug2: running rollup

[2026-07-09T13:33:37.454] debug2: _slurm_connect: failed to connect to 192.168.10.90:6817: Connection refused

[2026-07-09T13:33:37.454] debug2: Error connecting slurm stream socket at 192.168.10.90:6817: Connection refused

[2026-07-09T13:33:37.459] debug2: Attempting to connect to slurmci-RL8-1366-control:3306

[2026-07-09T13:33:37.462] debug2: accounting_storage/as_mysql: _cluster_rollup_usage: No need to roll cluster slurmci-rl8-1366 this hour 1783602000 <= 1783602000

[2026-07-09T13:33:37.462] debug2: accounting_storage/as_mysql: _cluster_rollup_usage: No need to roll cluster slurmci-rl8-1366 this day 1783555200 <= 1783555200

[2026-07-09T13:33:37.462] debug2: accounting_storage/as_mysql: _cluster_rollup_usage: No need to roll cluster slurmci-rl8-1366 this month 1782864000 <= 1782864000

[2026-07-09T13:33:37.465] debug2: accounting_storage/as_mysql: as_mysql_roll_usage: Got 1 of 1 rolled up

[2026-07-09T13:33:37.465] debug2: accounting_storage/as_mysql: as_mysql_roll_usage: Everything rolled up

TASK [stackhpc.openhpc : Notify handler for slurmd restart] ********************
...

[wtripp180901]

LGTM