Add T1 (envelope-sized) and T2 (workload-aware) tuning tiers

README.md

@@ -50,6 +50,28 @@ We currently run five benchmarks:
 
 The three BenchBase benchmarks cover **5** engines — there is no Db2 BenchBase profile, so Db2 has TPC-C/TPC-H only.
 
+## Tuning tiers
+
+To study energy vs. tuning effort, each scenario comes in tiers. Tier files sit next to the default and share the
+default's flow (only the *engine configuration* changes between tiers), so `run_on_cluster.py` discovers them
+automatically:
+
+- **T0 — default**: `benchmarks/<benchmark>/<db>.yml` (stock container).
+- **T1 — envelope-sized**: `benchmarks/<benchmark>/<db>.t1.yml` — each vendor's own rules-of-thumb sized to the fixed
+  4-CPU / 8-GB container; durability left at default so the T0→T1 delta is pure resource sizing.
+
+```sh
+# default vs. envelope-sized, Postgres TPC-C
+./run_on_cluster.py --machine-id N --filter 'tpcc/pg.yml'
+./run_on_cluster.py --machine-id N --filter 'tpcc/pg.t1.yml'
+./run_on_cluster.py --machine-id N -t 0                    # every T0 scenario
+./run_on_cluster.py --machine-id N -t 1                    # every T1 scenario
+./run_on_cluster.py --machine-id N -t 0 -n                 # preview without submitting
+```
+
+See [TUNING.md](TUNING.md) for per-engine settings, provenance, threats to validity (Oracle Free / Db2 Community
+edition caps, OOM headroom, Oracle/Db2 verify-on-first-run), and the planned T2 (advisor) and T3 (auto-tuner) tiers.
+
 ## Db2 setup
 
 Db2 needs a one-time prep the other engines don't. The `tpcorg/hammerdb` image has no Db2 client, and

benchmarks/chbenchmark/maria.t1.yml

@@ -0,0 +1,46 @@
+---
+name: BenchBase MariaDB CH-benCHmark benchmark (T1 envelope-sized)
+author: Didi Hoffmann <didi@green-coding.berlin)
+description: >
+  CH-benCHmark HTAP (TPC-C + TPC-H, BenchBase) on MariaDB, tuning tier T1
+  (envelope-sized). Same hardware-sized InnoDB config as tpcc/maria.t1.yml; durability at
+  default. See TUNING.md.
+
+compose-file: !include compose.yml
+
+custom_metrics:
+  chbenchmark_requests:
+    unit: CH-benCHmark request
+    sci: true
+
+services:
+  # T1 tuning: flags appended to mariadbd by the image entrypoint. Merged over
+  # compose.yml. (buffer_pool_instances is omitted: a no-op on modern MariaDB.)
+  #
+  # Every literal below is recomputable from the 4-CPU / 8-GB envelope alone --
+  # see the derivation table in TUNING.md ("How each T1 number is derived").
+  # Same InnoDB rule as mysql.t1.yml: O_DIRECT makes the buffer pool the only
+  # cache, so it takes ~62% of the cgroup directly (vendor band: 50-75% of RAM).
+  mariadb:
+    command:
+      - "--innodb-buffer-pool-size=5G"           # 62.5% of the 8-GB cgroup
+      - "--innodb-log-file-size=2G"              # redo log sizing
+      - "--innodb-flush-method=O_DIRECT"         # avoid double-buffering w/ page cache
+      - "--innodb-flush-log-at-trx-commit=1"     # full durability (engine default, explicit)
+      - "--innodb-io-capacity=2000"              # SSD-class
+      - "--innodb-io-capacity-max=4000"          # SSD-class
+      - "--max-connections=200"
+      - "--table-open-cache=4000"
+  postgres: # unused — empty key removes the service
+  mysql: # unused — empty key removes the service
+  oracle: # unused — empty key removes the service
+  mssql: # unused — empty key removes the service
+  db2: # unused — empty key removes the service
+  hammerdb: # unused — empty key removes the service
+  benchbase: # load driver; wait for the database to be healthy first
+    depends_on:
+      mariadb:
+        condition: service_healthy
+
+# Same benchmark flow as the default (T0) scenario.
+flow: !include ['maria.yml', 'flow']

benchmarks/chbenchmark/maria.t2.yml

@@ -0,0 +1,43 @@
+---
+name: BenchBase MariaDB CH-benCHmark benchmark (T2 workload-aware)
+author: Didi Hoffmann <didi@green-coding.berlin)
+description: >
+  CH-benCHmark (BenchBase, HTAP) on MariaDB, tuning tier T2. T1 InnoDB sizing + OLAP
+  session buffers + OLTP write-path tuning. Config-only; shares the T0 flow.
+  See TUNING.md.
+
+compose-file: !include compose.yml
+
+custom_metrics:
+  chbenchmark_requests:
+    unit: CH-benCHmark request
+    sci: true
+
+services:
+  mariadb:
+    command:
+      - "--innodb-buffer-pool-size=5G"
+      - "--innodb-log-file-size=2G"
+      - "--innodb-flush-method=O_DIRECT"
+      - "--innodb-flush-log-at-trx-commit=1"
+      - "--innodb-io-capacity=2000"
+      - "--innodb-io-capacity-max=4000"
+      - "--max-connections=200"
+      - "--table-open-cache=4000"
+      - "--join-buffer-size=128M"
+      - "--sort-buffer-size=32M"
+      - "--tmp-table-size=512M"
+      - "--max-heap-table-size=512M"
+      - "--innodb-purge-threads=4"
+  postgres: # unused — empty key removes the service
+  mysql: # unused — empty key removes the service
+  oracle: # unused — empty key removes the service
+  mssql: # unused — empty key removes the service
+  db2: # unused — empty key removes the service
+  hammerdb: # unused — empty key removes the service
+  benchbase:
+    depends_on:
+      mariadb:
+        condition: service_healthy
+
+flow: !include ['maria.yml', 'flow']

benchmarks/chbenchmark/mssql.t1.yml

@@ -0,0 +1,40 @@
+---
+name: BenchBase Microsoft SQL Server CH-benCHmark benchmark (T1 envelope-sized)
+author: Didi Hoffmann <didi@green-coding.berlin)
+description: >
+  CH-benCHmark HTAP (TPC-C + TPC-H, BenchBase) on Microsoft SQL Server, tuning tier T1
+  (envelope-sized). max server memory via MSSQL_MEMORY_LIMIT_MB, as in tpcc/mssql.t1.yml.
+  See TUNING.md.
+
+compose-file: !include compose.yml
+
+custom_metrics:
+  chbenchmark_requests:
+    unit: CH-benCHmark request
+    sci: true
+
+services:
+  # T1 tuning: merged onto compose.yml's mssql environment (ACCEPT_EULA etc. kept).
+  #
+  # SQL Server manages its own buffer cache, so "max server memory" is the only
+  # envelope lever here; MAXDOP / cost-threshold are workload-specific -> T2.
+  # See the derivation table in TUNING.md ("How each T1 number is derived").
+  # Deliberate deviation from the vendor rule (leave 25-35% of RAM to the OS and
+  # SQLOS): we leave 37.5%, because under a hard cgroup limit an over-commit is a
+  # silent OOM-kill, not a swap slowdown.
+  mssql:
+    environment:
+      MSSQL_MEMORY_LIMIT_MB: "5120"   # max server memory; 62.5% of the 8-GB cgroup
+  postgres: # unused — empty key removes the service
+  mariadb: # unused — empty key removes the service
+  mysql: # unused — empty key removes the service
+  oracle: # unused — empty key removes the service
+  db2: # unused — empty key removes the service
+  hammerdb: # unused — empty key removes the service
+  benchbase: # load driver; wait for the database to be healthy first
+    depends_on:
+      mssql:
+        condition: service_healthy
+
+# Same benchmark flow as the default (T0) scenario.
+flow: !include ['mssql.yml', 'flow']

benchmarks/chbenchmark/mssql.t2.yml

@@ -0,0 +1,33 @@
+---
+name: BenchBase MSSQL CH-benCHmark benchmark (T2 workload-aware)
+author: Didi Hoffmann <didi@green-coding.berlin)
+description: >
+  CH-benCHmark (BenchBase, HTAP) on SQL Server, tuning tier T2. The natural T2+ for
+  SQL Server HTAP would be a nonclustered columnstore index on the analytical
+  tables, but creating it post-load needs sqlcmd (absent in the Linux image) and
+  BenchBase won't emit it — so that columnar variant is deferred. Injectable lever
+  is max server memory (set in T1): CONFIG-PARITY (thin) T2 = T1. See TUNING.md.
+
+compose-file: !include compose.yml
+
+custom_metrics:
+  chbenchmark_requests:
+    unit: CH-benCHmark request
+    sci: true
+
+services:
+  mssql:
+    environment:
+      MSSQL_MEMORY_LIMIT_MB: "5120"
+  postgres: # unused — empty key removes the service
+  mariadb: # unused — empty key removes the service
+  mysql: # unused — empty key removes the service
+  oracle: # unused — empty key removes the service
+  db2: # unused — empty key removes the service
+  hammerdb: # unused — empty key removes the service
+  benchbase:
+    depends_on:
+      mssql:
+        condition: service_healthy
+
+flow: !include ['mssql.yml', 'flow']

benchmarks/chbenchmark/mysql.t1.yml

@@ -0,0 +1,47 @@
+---
+name: BenchBase MySQL CH-benCHmark benchmark (T1 envelope-sized)
+author: Didi Hoffmann <didi@green-coding.berlin)
+description: >
+  CH-benCHmark HTAP (TPC-C + TPC-H, BenchBase) on MySQL, tuning tier T1
+  (envelope-sized). Same hardware-sized InnoDB config as tpcc/mysql.t1.yml; durability at
+  default. See TUNING.md.
+
+compose-file: !include compose.yml
+
+custom_metrics:
+  chbenchmark_requests:
+    unit: CH-benCHmark request
+    sci: true
+
+services:
+  # T1 tuning: flags appended to mysqld by the official image entrypoint (it
+  # prepends `mysqld` when the command starts with `-`). Merged over compose.yml.
+  #
+  # Every literal below is recomputable from the 4-CPU / 8-GB envelope alone --
+  # see the derivation table in TUNING.md ("How each T1 number is derived").
+  # InnoDB with O_DIRECT owns the only page cache, so the buffer pool takes ~62%
+  # of the cgroup directly (vendor band: 50-75% of RAM).
+  mysql:
+    command:
+      - "--innodb-buffer-pool-size=5G"           # 62.5% of the 8-GB cgroup
+      - "--innodb-buffer-pool-instances=4"       # 5G / 4 = 1.25 GB each (>= 1 GB rule)
+      - "--innodb-redo-log-capacity=2G"          # MySQL 8.0.30+/9.x redo sizing
+      - "--innodb-flush-method=O_DIRECT"         # avoid double-buffering w/ page cache
+      - "--innodb-flush-log-at-trx-commit=1"     # full durability (engine default, explicit)
+      - "--innodb-io-capacity=2000"              # SSD-class
+      - "--innodb-io-capacity-max=4000"          # SSD-class
+      - "--max-connections=200"
+      - "--table-open-cache=4000"
+  postgres: # unused — empty key removes the service
+  mariadb: # unused — empty key removes the service
+  oracle: # unused — empty key removes the service
+  mssql: # unused — empty key removes the service
+  db2: # unused — empty key removes the service
+  hammerdb: # unused — empty key removes the service
+  benchbase: # load driver; wait for the database to be healthy first
+    depends_on:
+      mysql:
+        condition: service_healthy
+
+# Same benchmark flow as the default (T0) scenario.
+flow: !include ['mysql.yml', 'flow']

benchmarks/chbenchmark/mysql.t2.yml

@@ -0,0 +1,44 @@
+---
+name: BenchBase MySQL CH-benCHmark benchmark (T2 workload-aware)
+author: Didi Hoffmann <didi@green-coding.berlin)
+description: >
+  CH-benCHmark (BenchBase, HTAP) on MySQL, tuning tier T2. T1 InnoDB sizing + OLAP
+  session buffers (for the analytical queries) + OLTP write-path tuning.
+  Config-only; shares the T0 flow. See TUNING.md.
+
+compose-file: !include compose.yml
+
+custom_metrics:
+  chbenchmark_requests:
+    unit: CH-benCHmark request
+    sci: true
+
+services:
+  mysql:
+    command:
+      - "--innodb-buffer-pool-size=5G"
+      - "--innodb-buffer-pool-instances=4"
+      - "--innodb-redo-log-capacity=2G"
+      - "--innodb-flush-method=O_DIRECT"
+      - "--innodb-flush-log-at-trx-commit=1"
+      - "--innodb-io-capacity=2000"
+      - "--innodb-io-capacity-max=4000"
+      - "--max-connections=200"
+      - "--table-open-cache=4000"
+      - "--join-buffer-size=128M"
+      - "--sort-buffer-size=32M"
+      - "--tmp-table-size=512M"
+      - "--max-heap-table-size=512M"
+      - "--innodb-purge-threads=4"
+  postgres: # unused — empty key removes the service
+  mariadb: # unused — empty key removes the service
+  oracle: # unused — empty key removes the service
+  mssql: # unused — empty key removes the service
+  db2: # unused — empty key removes the service
+  hammerdb: # unused — empty key removes the service
+  benchbase:
+    depends_on:
+      mysql:
+        condition: service_healthy
+
+flow: !include ['mysql.yml', 'flow']

benchmarks/chbenchmark/oracle.t1.yml

@@ -0,0 +1,61 @@
+---
+name: BenchBase Oracle Database Free CH-benCHmark benchmark (T1 envelope-sized)
+author: Didi Hoffmann <didi@green-coding.berlin)
+description: >
+  CH-benCHmark HTAP (TPC-C + TPC-H, BenchBase) on Oracle Database Free, tuning tier T1
+  (envelope-sized). SGA/PGA sized by a prepended ALTER SYSTEM step, as in tpcc/oracle.t1.yml.
+  CAVEAT: Oracle Free is licence-capped at ~2 GB RAM / 2 CPU threads, so the
+  T0->T1 delta is expected to be small; ALTERs are skipped under Automatic
+  Memory Management. VERIFY on first run; see TUNING.md.
+
+compose-file: !include compose.yml
+
+custom_metrics:
+  chbenchmark_requests:
+    unit: CH-benCHmark request
+    sci: true
+
+services:
+  # oracle service intentionally NOT listed -> kept as-is from compose.yml
+  # (Oracle exposes no command-flag config; T1 is applied at runtime by the
+  # prepended flow step below).
+  postgres: # unused — empty key removes the service
+  mariadb: # unused — empty key removes the service
+  mysql: # unused — empty key removes the service
+  mssql: # unused — empty key removes the service
+  db2: # unused — empty key removes the service
+  hammerdb: # unused — empty key removes the service
+  benchbase: # load driver; wait for the database to be healthy first
+    depends_on:
+      oracle:
+        condition: service_healthy
+
+# T1 tuning runs as a prepended flow step (Oracle has no command-flag config).
+# Retries until the listener answers, continues past ORA errors, never fails the
+# run. Connects to the CDB root (service FREE) as SYSDBA.
+#
+# WHY THESE NUMBERS LOOK NOTHING LIKE THE OTHER ENGINES': Oracle Database Free is
+# licence-capped at ~2 GB of database RAM (SGA+PGA) and 2 CPU threads, so T1 is
+# sized against THAT cap, not the 8-GB cgroup every other engine gets:
+#     1300M (SGA) + 500M (PGA) = 1800M of the 2048M cap, ~250M headroom.
+# The procedure is identical (vendor rule applied to the effective memory limit);
+# only the effective limit differs. Oracle Free therefore cannot fill the shared
+# envelope, and its absolute J/op is not comparable on equal-hardware terms.
+# See TUNING.md: "How each T1 number is derived" + "Threats to validity".
+flow-prepend:
+  - name: Tune Oracle (T1 envelope-sized)
+    container: oracle_container
+    commands:
+      - type: console
+        note: APPLY T1 SGA/PGA SIZING (capped by Oracle Free 2 GB limit)
+        command: |
+          i=0; while [ $i -lt 30 ]; do
+            printf 'WHENEVER SQLERROR CONTINUE\nALTER SYSTEM SET sga_target=1300M SCOPE=BOTH;\nALTER SYSTEM SET pga_aggregate_target=500M SCOPE=BOTH;\nSHOW PARAMETER sga_target\nSHOW PARAMETER pga_aggregate_target\nEXIT\n' \
+              | sqlplus -s -L sys/oracle@//localhost:1521/FREE as sysdba && break
+            i=$((i+1)); sleep 3
+          done; true
+        shell: bash
+        log-stdout: true
+
+# Same benchmark flow as the default (T0) scenario.
+flow: !include ['oracle.yml', 'flow']

benchmarks/chbenchmark/oracle.t2.yml

@@ -0,0 +1,44 @@
+---
+name: BenchBase Oracle CH-benCHmark benchmark (T2 workload-aware)
+author: Didi Hoffmann <didi@green-coding.berlin)
+description: >
+  CH-benCHmark (BenchBase, HTAP) on Oracle Database Free, tuning tier T2. T1
+  SGA/PGA sizing (prepended). Free's 2 GB / 2-thread cap bounds the analytical side
+  and there is no usable native advisor, so this is a CONFIG-PARITY (thin) T2 = T1.
+  See TUNING.md.
+
+compose-file: !include compose.yml
+
+custom_metrics:
+  chbenchmark_requests:
+    unit: CH-benCHmark request
+    sci: true
+
+services:
+  postgres: # unused — empty key removes the service
+  mariadb: # unused — empty key removes the service
+  mysql: # unused — empty key removes the service
+  mssql: # unused — empty key removes the service
+  db2: # unused — empty key removes the service
+  hammerdb: # unused — empty key removes the service
+  benchbase:
+    depends_on:
+      oracle:
+        condition: service_healthy
+
+flow-prepend:
+  - name: Tune Oracle (T1 envelope-sized)
+    container: oracle_container
+    commands:
+      - type: console
+        note: APPLY T1 SGA/PGA SIZING (capped by Oracle Free 2 GB limit)
+        command: |
+          i=0; while [ $i -lt 30 ]; do
+            printf 'WHENEVER SQLERROR CONTINUE\nALTER SYSTEM SET sga_target=1300M SCOPE=BOTH;\nALTER SYSTEM SET pga_aggregate_target=500M SCOPE=BOTH;\nSHOW PARAMETER sga_target\nSHOW PARAMETER pga_aggregate_target\nEXIT\n' \
+              | sqlplus -s -L sys/oracle@//localhost:1521/FREE as sysdba && break
+            i=$((i+1)); sleep 3
+          done; true
+        shell: bash
+        log-stdout: true
+
+flow: !include ['oracle.yml', 'flow']