Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,9 @@ updates:
directory: "/" # Location of package manifests
schedule:
interval: "weekly"
ignore:
# MySQL must stay on the 8.4 LTS line: 9.x removed mysql_native_password and
# HammerDB 5.0 cannot authenticate to caching_sha2_password over plaintext TCP.
# See the comment on the mysql service in compose.yml.
- dependency-name: "mysql"
update-types: ["version-update:semver-major"]
101 changes: 96 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,114 @@
A tool that benchmarks various DBMS systems agains each other and looks at energy usage

The `compose.yml` defines all the containers we want to use to benchmark. We use the std containers provided by the
various DBMS and the [hammerdb](https://www.hammerdb.com/) container. For benchmarking we use the
hammerDB system.
various DBMS plus a load-driver container per benchmark family:

- [HammerDB](https://www.hammerdb.com/) drives the TPC-C and TPC-H benchmarks.
- [BenchBase](https://github.com/cmu-db/benchbase) (CMU; the maintained OLTP-Bench successor) drives the Wikipedia,
YCSB and CH-benCHmark benchmarks. One image bakes in a distribution per engine, so the same driver runs every
database in a benchmark — see [BenchBase setup](#benchbase-setup).

Currently we support:

- IBM Db2 (needs a one-time image build — see [Db2 setup](#db2-setup))
- MariaDB
- Microsoft SQL Server
- MySql
- Oracle Database Free
- PostgreSQL

For each DB vendor we have a benchmarking `usage_scenario_xxx.yml` that you can execute with the Green Metrics Tool to
get energy readings.
The repo is split into two trees: `benchmarks/` holds the GMT usage scenarios (one folder per benchmark), and `db/`
holds the per-engine driver scripts (one folder per database engine).

For each DB vendor and benchmark we have a `benchmarks/<benchmark>/<db>.yml` (e.g. `benchmarks/tpcc/pg.yml`,
`benchmarks/tpch/oracle.yml`) that you can execute with the Green Metrics Tool to get energy readings. The driver's
per-DB scripts live under `db/<db>/<benchmark>/`: HammerDB TCL scripts for TPC-C/TPC-H (e.g. `db/pg/tpcc/`), BenchBase
config XMLs for Wikipedia/YCSB/CH-benCHmark (e.g. `db/pg/wikipedia/pg_wikipedia_config.xml`).

Each benchmark directory keeps its **own copy of `compose.yml`** (`benchmarks/tpcc/compose.yml`,
`benchmarks/tpch/compose.yml`, …): GMT's `!include` only resolves files inside the scenario's own directory, so
`!include ../compose.yml` is rejected. Treat the root `compose.yml` as the source of truth and re-sync the copies
whenever it changes:

```sh
for d in tpcc tpch wikipedia ycsb chbenchmark; do cp compose.yml "benchmarks/$d/compose.yml"; done
```

We currently run five benchmarks:

- [TPC-C](https://en.wikipedia.org/wiki/TPC-C) (HammerDB TPROC-C) — write-heavy OLTP; recorded metric is NOPM
- [TPC-H](https://en.wikipedia.org/wiki/TPC-H) (HammerDB TPROC-H) — analytical query workload; the SCI functional unit
is the number of queries completed (i.e. carbon per query)
- [Wikipedia](https://github.com/cmu-db/benchbase) (BenchBase) — read-mostly web-serving workload over real Wikipedia
article traces; the SCI functional unit is the number of requests completed
- [YCSB](https://github.com/cmu-db/benchbase) (BenchBase) — cloud key-value serving (point reads/writes/scans); the
SCI functional unit is the number of requests completed
- [CH-benCHmark](https://github.com/cmu-db/benchbase) (BenchBase, composite `tpcc,chbenchmark`) — HTAP:
TPC-C transactions and 22 TPC-H-style analytical queries run *concurrently* on one schema; the SCI functional unit is
the number of requests completed (OLTP txns + OLAP queries — see the per-transaction CSVs to split them)

The three BenchBase benchmarks cover **5** engines — there is no Db2 BenchBase profile, so Db2 has TPC-C/TPC-H only.

## Db2 setup

Db2 needs a one-time prep the other engines don't. The `tpcorg/hammerdb` image has no Db2 client, and
HammerDB's `db2tcl` binding needs the *full* Db2 client (the free CLI driver lacks the `sqlefrce_api`
symbol). HammerDB also talks to Db2 through a *catalogued* database, so the client needs a real Db2
instance. `db/db2/build-image.sh` handles this: it lifts the Db2 install + an initialised client instance out
of the `icr.io/db2_community/db2` image and bakes them onto the hammerdb (Ubuntu) base, then pushes the
result to Docker Hub as `ribalba/hammerdb-db2`, which the Db2 usage scenarios reference.

Before running the Db2 scenarios (one-time):

```sh
docker login # as the account that owns the image (ribalba)
./db/db2/build-image.sh # builds and pushes ribalba/hammerdb-db2:latest
```

Notes:

- The Db2 server container requires the `CAP_IPC_OWNER` capability (granted via `--cap-add IPC_OWNER` in
`docker-run-args`), and the host must allow it. Full `privileged: true` is not needed.
- Db2 is slow to start (minutes) and loads via INSERTs, so it builds slower than the bulk-loaded engines.
Comment on lines +69 to +73

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
fd -e yml . benchmarks/tpcc benchmarks/tpch --exec rg -n -A2 -B2 'privileged|cap_add' {} \;
rg -n -A2 -B2 'privileged|cap_add' compose.yml

Repository: green-coding-solutions/DBMS-bench

Length of output: 1401


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n== README excerpt ==\n'
sed -n '60,78p' README.md

printf '\n== Db2 compose files ==\n'
for f in benchmarks/tpcc/db2.yml benchmarks/tpch/db2.yml compose.yml; do
  echo "--- $f ---"
  sed -n '1,180p' "$f"
done

Repository: green-coding-solutions/DBMS-bench

Length of output: 13513


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n== locate Db2 service definitions ==\n'
rg -n -A4 -B4 'db2|privileged|cap_add|CAP_IPC_OWNER|sysctl' benchmarks/tpcc/db2.yml benchmarks/tpch/db2.yml compose.yml

Repository: green-coding-solutions/DBMS-bench

Length of output: 13362


🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
from pathlib import Path
for p in [Path('benchmarks/tpcc/db2.yml'), Path('benchmarks/tpch/db2.yml'), Path('compose.yml')]:
    print(f"\n### {p}")
    text = p.read_text()
    for key in ['privileged', 'cap_add', 'CAP_IPC_OWNER', '--privileged']:
        if key in text:
            print(f"contains {key}")
PY

Repository: green-coding-solutions/DBMS-bench

Length of output: 311


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# show exact lines around any Db2 capability notes in YAMLs
for f in benchmarks/tpcc/db2.yml benchmarks/tpch/db2.yml compose.yml; do
  echo "--- $f ---"
  nl -ba "$f" | sed -n '85,120p'
done

Repository: green-coding-solutions/DBMS-bench

Length of output: 243


Replace privileged: true with the actual Db2 capability requirement README.md:69-72 — the runtime compose setup only adds CAP_IPC_OWNER via cap_add; it doesn’t require full privileged: true.


## BenchBase setup

The Wikipedia, YCSB and CH-benCHmark scenarios are driven by a `benchbase` container. BenchBase builds one
self-contained distribution per database (each bundles that engine's JDBC driver), so `benchmarks/benchbase/build-image.sh` builds
every engine's profile into a single image and pushes it to Docker Hub as `ribalba/benchbase:latest`, which the
scenarios pull. Each profile lands in `/benchbase/benchbase-<profile>/` and the scenario `cd`s into the right one.

Before running the Wikipedia/YCSB/CH-benCHmark scenarios (one-time):

```sh
docker login # as the account that owns the image (ribalba)
./benchmarks/benchbase/build-image.sh # builds + pushes ribalba/benchbase:latest
BENCHBASE_REF=<commit-sha> ./benchmarks/benchbase/build-image.sh # pin a commit for a reproducible paper build
```

Notes:

We are currently only using the [TPC-C benchmark](https://en.wikipedia.org/wiki/TPC-C)
- Each scenario runs three flow steps: create the `benchbase` database/user on the DB container, then BenchBase
`--create --load` (schema + data), then `--execute` (the measured run). The functional-unit metric
(`wikipedia_requests` / `ycsb_requests` / `chbenchmark_requests`) is parsed from BenchBase's `measuredRequests` on
the execute step.
- CH-benCHmark is the composite `-b tpcc,chbenchmark` with a single config: the `tpcc` plugin supplies the OLTP schema
and transactions, the `chbenchmark` plugin adds the TPC-H tables and the 22 analytical queries. OLTP and OLAP run in
*separate terminal pools* (`<terminals bench="tpcc">` vs `<terminals bench="chbenchmark">`) — that split is the HTAP
knob. The execute step reports one combined `measuredRequests`; for the OLTP-vs-OLAP breakdown read BenchBase's
per-transaction `results/*.results.<Txn>.csv` files (TPC-C's five vs `Q1`…`Q22`).
- The workload (scalefactor, terminals, isolation, weights) is fixed in `db/<db>/<benchmark>/<db>_<benchmark>_config.xml`
so only the *database* configuration is the variable when comparing default vs. tuned. Defaults: Wikipedia
`scalefactor=10`, YCSB `scalefactor=1000` (≈1 GB), CH-benCHmark `scalefactor=10` (TPC-C warehouses, ≈1 GB) — all
deliberately larger than default buffer pools so tuning shows.
- BenchBase requires Java 23 to build; the image handles this. It has no Db2 profile.

## Some background reading

- https://www.hammerdb.com/about.html
- https://www.hammerdb.com/blog/uncategorized/how-to-deploy-hammerdb-cli-fast-with-docker/
- https://www.tpc.org/tpcc/default5.asp
- https://www.hammerdb.com/blog/uncategorized/how-to-run-a-fixed-throughput-workloads/
- https://github.com/cmu-db/benchbase (BenchBase)
- OLTP-Bench paper (BenchBase lineage): https://www.vldb.org/pvldb/vol7/p277-difallah.pdf
- CH-benCHmark paper (Cole et al., DBTest 2011): https://doi.org/10.1145/1988842.1988850
85 changes: 85 additions & 0 deletions benchmarks/benchbase/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# BenchBase load-driver image for the Wikipedia and YCSB scenarios.
#
# BenchBase builds one self-contained distribution per database profile (each
# bundles that engine's JDBC driver), so we build every profile we need and copy
# them all into one runtime image. A scenario then runs:
# cd /benchbase/benchbase-<profile> && java -jar benchbase.jar -b <bench> -c <cfg> ...
#
# Build via ./benchmarks/benchbase/build-image.sh (sets the build args below).
# syntax=docker/dockerfile:1

# --- build stage: compile each requested profile from source ---------------
# BenchBase's pom.xml pins Java 23 (maven.compiler.release=23).
FROM eclipse-temurin:23-jdk AS build

# Moving default; pin a commit SHA via build-image.sh for a reproducible build.
ARG BENCHBASE_REF=main
# Space-separated BenchBase Maven profiles to bake in.
ARG BENCHBASE_PROFILES="postgres mysql mariadb sqlserver oracle"

RUN apt-get update \
&& apt-get install -y --no-install-recommends git ca-certificates \
&& rm -rf /var/lib/apt/lists/*

WORKDIR /src
RUN git clone https://github.com/cmu-db/benchbase.git . \
&& git checkout "${BENCHBASE_REF}"

# Patch an upstream Wikipedia data-generator bug: UpdatePage truncates the
# generated revComment to 254 chars before writing revision.rev_comment, but
# writes the *untruncated* value into recentchanges.rc_comment (also
# varchar(255)). On any strict engine (PostgreSQL, and MySQL/MariaDB/Oracle/
# SQL Server in their default strict modes) the occasional >255-char comment
# makes the rc_comment insert fail with "value too long" (SQLSTATE 22001),
# silently dropping that UpdatePage txn. We apply the same substring truncation
# already used for rev_comment. Anchored on the `// rc_comment` trailing comment
# so it survives line-number drift; the grep guard fails the build loudly if
# upstream changes shape and the patch silently no-ops.
# Upstream: https://github.com/cmu-db/benchbase (procedures/UpdatePage.java)
RUN set -eux; \
f=src/main/java/com/oltpbenchmark/benchmarks/wikipedia/procedures/UpdatePage.java; \
sed -i 's|revComment); // rc_comment|revComment.substring(0, Math.min(revComment.length(), 255 - 1))); // rc_comment|' "$f"; \
grep -q 'revComment.substring(0, Math.min(revComment.length(), 255 - 1))); // rc_comment' "$f"; \
! grep -q 'param++, revComment); // rc_comment' "$f"

# Patch the upstream Oracle Wikipedia DDL: five varchar2 columns (ipb_timestamp,
# page_touched, rev_timestamp, user_touched, user_token) carry a MySQL-style
# binary default of N literal NUL bytes written as '\0\0...'. Oracle does not
# process backslash escapes in string literals, so it reads each '\0' as the two
# characters backslash+zero -- e.g. varchar2(14) DEFAULT '\0\0\0\0\0\0\0\0\0\0\0\0\0\0'
# is a 28-char default into a 14-char column, which fails CREATE TABLE with
# ORA-01401 (inserted value too large for column). The sibling timestamp columns
# in the same file use DEFAULT '', so we normalise every varchar2 NUL-byte
# default to ''. The grep guard fails the build loudly if upstream changes shape
# and the patch silently no-ops.
# Upstream: https://github.com/cmu-db/benchbase (benchmarks/wikipedia/ddl-oracle.sql)
RUN set -eux; \
f=src/main/resources/benchmarks/wikipedia/ddl-oracle.sql; \
sed -i "s/\(varchar2([0-9]*)\) DEFAULT '[\\0]*'/\1 DEFAULT ''/g" "$f"; \
! grep -qF "\0" "$f"

# Build each profile and unpack its tarball into /dist/benchbase-<profile>/.
# The Maven dependency cache (~/.m2) persists across profiles within this layer,
# so only the first profile pays the full download cost. Tests are skipped.
RUN set -eux; \
mkdir -p /dist; \
for p in ${BENCHBASE_PROFILES}; do \
echo ">> building BenchBase profile: $p"; \
./mvnw -q -B clean package -P "$p" -DskipTests; \
tar -xzf "target/benchbase-$p.tgz" -C /dist; \
done; \
ls -d /dist/benchbase-*

# --- runtime stage: JRE + the unpacked distributions -----------------------
FROM eclipse-temurin:23-jre
LABEL org.opencontainers.image.source="https://github.com/cmu-db/benchbase" \
org.opencontainers.image.description="CMU BenchBase, multi-profile, load driver for DBMS-bench"

WORKDIR /benchbase
COPY --from=build /dist/ /benchbase/
# Fail the build early if no profile made it in.
RUN ls -d /benchbase/benchbase-* >/dev/null

# Stay alive so GMT can exec each flow step (create+load, then execute) into the
# running container, the same way it drives the hammerdb container.
CMD ["sleep", "infinity"]
45 changes: 45 additions & 0 deletions benchmarks/benchbase/build-image.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
#!/usr/bin/env bash
# Builds the `benchbase` image used to drive the Wikipedia and YCSB scenarios.
#
# BenchBase (https://github.com/cmu-db/benchbase) builds one self-contained
# distribution per database profile (each bundles that engine's JDBC driver).
# This image bakes in a distribution for every engine we benchmark so a SINGLE
# image/container drives all databases of a benchmark — keeping the load
# generator identical across engines. Each profile lands in
# /benchbase/benchbase-<profile>/ inside the image; the usage scenarios cd into
# the right one and run `java -jar benchbase.jar ...`.
#
# Run this ONCE before running the Wikipedia/YCSB scenarios (after `docker login`):
# ./benchmarks/benchbase/build-image.sh
# It builds and pushes the image to Docker Hub as ribalba/benchbase:latest, which
# the usage scenarios pull. Set NO_PUSH=1 to build only.
#
# For a reproducible paper build, pin BENCHBASE_REF to a commit SHA:
# BENCHBASE_REF=<sha> ./benchmarks/benchbase/build-image.sh
set -euo pipefail

OUT_IMAGE="${OUT_IMAGE:-ribalba/benchbase:latest}"
# BenchBase has no GitHub releases; `main` is the moving default. Pin a commit
# SHA via BENCHBASE_REF for a reproducible build.
BENCHBASE_REF="${BENCHBASE_REF:-main}"
# Engines we benchmark. BenchBase profile names (note: postgres->pg, mariadb->maria,
# sqlserver->mssql in this repo's directory naming). No Db2 profile exists.
BENCHBASE_PROFILES="${BENCHBASE_PROFILES:-postgres mysql mariadb sqlserver oracle}"
HERE="$(cd "$(dirname "$0")" && pwd)"

echo ">> building $OUT_IMAGE"
echo " ref: $BENCHBASE_REF"
echo " profiles: $BENCHBASE_PROFILES"
docker build \
--build-arg BENCHBASE_REF="$BENCHBASE_REF" \
--build-arg BENCHBASE_PROFILES="$BENCHBASE_PROFILES" \
-t "$OUT_IMAGE" \
"$HERE"

if [ "${NO_PUSH:-0}" = 1 ]; then
echo ">> built $OUT_IMAGE (push skipped via NO_PUSH=1)"
else
echo ">> pushing $OUT_IMAGE to Docker Hub (requires prior 'docker login')"
docker push "$OUT_IMAGE"
echo ">> done: pushed $OUT_IMAGE"
fi
Loading