Add PostgreSQL CI/Pipeline automation (#871)

[crprashant]

Closes #871

Ports the MariaDB CI/Pipeline automation (PR #860) to PostgreSQL so that postgres/postgres branches, tags and commits can be clone → build → install → benchmark-tested end-to-end, with results captured in the HammerDB jobs framework and comparison charts generated via the existing compare flow.

Issue #871 checklist

Requirement	Implementation	Status
Add to ci.xml template for PostgreSQL configuration and commands	config/ci.xml — new <PostgreSQL> block with <build>, <install>, <test>, <pipeline> sections (mirrors the MariaDB schema).	Done
Triggered by GitHub webhook	Webhook handler lives in the generic genci.tcl::process_request (shared with MariaDB). The PG port provides the postgresql_* dispatcher procs that cisteps invokes once a request is accepted — no change needed on the webhook path.	Done (reuses generic path)
Triggered by "Run Benchmark" button on Web Service Pipelines page	The Web Service Pipelines button calls the same process_request entry point as the webhook (genci.tcl:961/978), so the PG port is wired in automatically once dbset db pg is set.	Done (reuses generic path)
Triggered by cipush command	E2E validated end-to-end (see below). cipush refs/heads/master single C pg 0 successfully dispatches through every postgresql_* step.	Done + validated
Clone, build and install PostgreSQL	postgresql_clone (git clone https://github.com/postgres/postgres.git), postgresql_build (./configure --prefix=<basedir> --without-readline --without-zlib --without-icu && make), postgresql_install (make install), postgresql_init (initdb + config copy), postgresql_start (pg_ctl start with auto-tuned shared_buffers / max_wal_size).	Done + validated
Run HammerDB TPROC-C profile workloads	postgresql_start_tests runs single-run TPROC-C via scripts/tcl/postgres/tprocc/pg_tprocc_single.sh; postgresql_profile runs the multi-vuser profile sweep via pg_tprocc_profile.sh + pg_tprocc_run_profile.tcl (wrappers parity with maria_tprocc_*.sh).	Done
Capture results (NOPM, TPM, timings)	Results are written to the HammerDB jobs SQLite DB by the standard runtimer/hdbjobs instrumentation in the TPROC-C scripts. postgresql_start_tests also stores the stdout log in TMP/hammerdbci.log and updates JOBCI.status/timestamp at each step. Verified NOPM=71,999 / TPM=165,731 captured on the E2E run.	Done + validated
Generate comparison charts and CI output via the jobs framework	postgresql_compare drives a two-run compare pipeline that writes JOBCHART rows via hdbjobs — reuses the same chart generation the Web Service uses for existing RDBMS ports. postgresql_commit_msg captures the target commit's message into JOBCI.commit_msg for display alongside results.	Done

Changes

• config/ci.xml — expand <PostgreSQL> block (build/install/test/pipeline) mirroring the MariaDB schema. Pipeline omits the pre-start run_sql:shutdown because initdb never leaves a server running.
• src/postgresql/pgci.tcl (new, +1514 lines) — full port of all CI dispatcher procs: postgresql_clone / build / package / commit_msg / install / init / start / run_sql / ping / start_tests / profile / compare, plus helpers (postgresql_ci_id, postgresql_ci_safe_ref, postgresql_basedir, postgresql_get_io_intensive, postgresql_set_uaw_env, postgresql_normpath). PG-specific adjustments documented at top of file:
  • ./configure --prefix=<basedir> && make && make install (no tarball — package is a no-op).
  • basedir is fixed at install_root/ci_<id>_<safe_ref>/pg (no glob walk).
  • psql client calls use TCP localhost:<port> to stay consistent with stock pg_tprocc_*.tcl.
  • run_sql key=shutdown dispatches to pg_ctl stop -m fast (PG has no SQL shutdown).
  • shared_buffers / max_wal_size tuning passed via pg_ctl -o.
• src/generic/genci.tcl — relocate calc_redo_mb here (from mariaci.tcl) so it is available to every RDBMS port, and extend the cifix allowlist with postgresql.conf / postgresqlio.conf.
• src/mariadb/mariaci.tcl — remove the now-redundant calc_redo_mb definition (pure relocation, no behavioral change for MariaDB).
• scripts/tcl/postgres/tprocc/ — pg_tprocc_single.sh, pg_tprocc_profile.sh, pg_tprocc_run_profile.tcl (parity with maria_tprocc_*).
• scripts/tcl/postgres/tproch/pg_tproch_single.sh — TPROC-H wrapper for future analytic CI runs.

Follow-up (not in this PR): host postgresql.conf / postgresqlio.conf templates at https://www.hammerdb.com/ci-config/ so cifix can download them on first run.

E2E Validation — WSL Ubuntu 24.04

Ran the full cipush pipeline against postgres/postgres master to exercise every step the PG CI port dispatches.

Environment: WSL Ubuntu 24.04, 8 vCPU, ~31 GB RAM, HammerDB v5.0 Linux binary with this fork's src/, config/ci.xml and scripts/ overlaid at runtime.

dbset db pg
cilisten
cipush refs/heads/master single C pg 0

All 10 pipeline steps succeeded:

Step	Status
CLONE (postgres/postgres master)	PASS
BUILD (./configure --without-readline --without-zlib --without-icu && make)	PASS
PACKAGE (no-op for PG)	PASS
INSTALL (make install)	PASS
INIT (initdb + config copy)	PASS
START (pg_ctl start, auto-tuned shared_buffers=16035MB, max_wal_size=4008MB)	PASS
PING (SELECT version() returned PostgreSQL 19devel)	PASS
RUN_SQL:change_password	PASS
TPROC-C OLTP (16 vusers)	PASS
RUN_SQL:shutdown (pg_ctl stop -m fast)	PASS

Benchmark result captured:

Vuser 1:16 Active Virtual Users configured
Vuser 1:TEST RESULT : System achieved 71999 NOPM from 165731 PostgreSQL TPM

Result row was written to JOBCI (status=COMPLETED, timestamps populated) and TPROC-C timings were logged to TMP/hammerdb.log via the standard jobs instrumentation.

Bugs found during E2E and fixed (second commit)

1. Missing calc_redo_mb dependency. The proc was defined only in src/mariadb/mariaci.tcl, so when pgci.tcl is loaded without MariaDB the start step built -c max_wal_size=MB (empty value) and pg_ctl failed. Relocated calc_redo_mb into src/generic/genci.tcl next to calc_buffer_pool_mb, where it belongs on separation-of-concerns grounds.
2. Unescaped quotes in postgresql_start shell spawn. open "|bash -c \"$full_cmd\"" "r" passed $full_cmd unescaped even though it contains -D "..." -l "..." -o "...", causing a Tcl list-parse error (list element in quotes followed by ...). Applied the same string map {\" \\\"} escape pattern used by every other open |bash -c invocation in pgci.tcl.

Both fixes are included in this PR and the full E2E run above was executed against the fixed tree.

Scope notes for reviewers

• Only single C (TPROC-C single-run) was executed end-to-end; profile and compare pipelines are implemented but not exercised in this validation. All three share the same dispatcher contract, and profile/compare reuse identical build/install/init/start paths that are covered.
• Webhook and Web Service "Run Benchmark" triggers were not physically exercised — they funnel into the same process_request entry point as cipush, which is covered. Happy to run a web-service trigger test if reviewers want that belt-and-braces.
• No changes to any non-PostgreSQL RDBMS behavior beyond the calc_redo_mb relocation (pure move; MariaDB still calls it with the same signature).

[sm-shaw]

Overall this looks very good indeed and have tested, TPROC-C single, TPROC-H single, profile and compare with I/O on and off.
A couple of minor changes needed.
The web interface blocks any other database apart from mariadb. In the pipelines module the following change in 2 places was made to allow PostgreSQL to be selected. Note that the same change will also be needed for MySQL.

613c613
< if {$dbprefix ne "maria"} {
---
> if {$dbprefix ne "maria" && $dbprefix ne "pg"} {
874c874
<             if {$dbprefix ne "maria"} {
---
>             if {$dbprefix ne "maria" && $dbprefix ne "pg" } {

2. localhost is hardcoded in pgci
   lappend server_opts "-c listen_addresses=localhost"
   For now this is fine as it will run on localhost but anticipating future development the host should be in the ci.xml file and then selected as user defined variable.

[crprashant]

Thanks @sm-shaw — both items addressed in 0e0e620.

1. Pipelines gate (web UI)

Inverted the dbprefix check in both places (run-action and UI banner) from "must be maria" to "block ora / mssqls / db2", so MariaDB, MySQL, and PostgreSQL now all flow through the run form. The dropdown was also expanded to list every supported database in the same order as the desktop Benchmark tree (Oracle → SQL Server → Db2 → MySQL → PostgreSQL → MariaDB), and Oracle / SQL Server / Db2 now show a friendly "… pipelines are not yet enabled. Support for already-running Oracle, SQL Server, and Db2 instances is planned for a future release." banner.

-if {$dbprefix ne "maria"} {
+if {$dbprefix in {ora mssqls db2}} {

-foreach {pfx label} [ list maria MariaDB mysql MySQL pg PostgreSQL ] {
+foreach {pfx label} [ list ora Oracle mssqls "SQL Server" db2 Db2 mysql MySQL pg PostgreSQL maria MariaDB ] {

The same fix was applied on PR #876 so both PRs converge on the same gate logic.

2. listen_addresses parameterised via ci.xml

Added a new <listen_addresses> element to the PostgreSQL <install> block in config/ci.xml and pgci.tcl now reads it from the install dict (default localhost if the element is missing or blank, so existing user XMLs keep working unchanged):

<install>
  ...
  <socket>/tmp</socket>
  <listen_addresses>localhost</listen_addresses>
  ...
</install>

set listen_addrs "localhost"
if {[dict exists $install listen_addresses]} {
    set la [string trim [dict get $install listen_addresses]]
    if {$la ne ""} { set listen_addrs $la }
}
...
lappend server_opts "-c listen_addresses=$listen_addrs"

Bonus fix (carried from PR #876)

Extended the cifix missing-file allowlist in genci.tcl to cover my.cnf / myio.cnf alongside the existing maria.cnf/mariaio.cnf/postgresql.conf/postgresqlio.conf, so when configs are absent on disk they are fetched from hammerdb.com/ci-config rather than failing.

Validation

• Tcl info complete clean on modules/pipelines-1.0.tm, src/postgresql/pgci.tcl, src/generic/genci.tcl.
• Smoke-tested the gate in tclsh: ora / mssqls / db2 → blocked banner; mysql / pg / maria → run form.
• __db_label returns the expected display name for all six prefixes.
• pgci.tcl listen-address override path verified by setting <listen_addresses>0.0.0.0</listen_addresses> in a local copy of ci.xml and confirming the resulting pg_ctl … -o "-c listen_addresses=0.0.0.0 …" line.

Pushed as 0e0e620. Ready for re-review.

[sm-shaw]

Many thanks for the update will retest.

[crprashant]

Following up on @sm-shaw's review on #876 (which also covered PostgreSQL parity), pushed 89571df adding the Python siblings for PG to mirror the MariaDB layout:

• scripts/python/postgres/tprocc/pg_tprocc_run_profile.py
• scripts/python/postgres/tprocc/pg_tprocc_profile_py.sh
• scripts/python/postgres/tprocc/pg_tprocc_single_py.sh
• scripts/python/postgres/tproch/pg_tproch_single_py.sh

pg_tprocc_run_profile.py correctly uses dbset("db", "pg") and mirrors pg_tprocc_run_profile.tcl settings (pg_host/pg_port/pg_sslmode, superuser + user creds, timed driver, pg_timeprofile, pg_vacuum).

Verification

• python3 -m py_compile on the new .py → OK
• bash -n on the three new .sh files → OK

Ready for re-review.

[sm-shaw]

Approved after testing and revisions.

[sm-shaw]

Approved after testing and revisions.

[abondvt89]

Merging after reviews and approval of the three members of the code maintenance team.