Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
test/
usage_scenario_*_test.yml
46 changes: 35 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,45 @@
# DBMS-bench
A tool that benchmarks various DBMS systems agains each other and looks at energy usage

The `compose.yml` defines all the containers we want to use to benchmark. We use the std containers provided by the
various DBMS and the [hammerdb](https://www.hammerdb.com/) container. For benchmarking we use the
hammerDB system.
A tool that benchmarks various DBMS systems against each other and looks at energy usage.

Currently we support:
The `compose.yml` defines all the containers use to benchmark. it uses the std containers provided by the various DBMS and the __[hammerdb](https://www.hammerdb.com/)__ container. For benchmarking we use the hammerDB system.

- MariaDB
- MySql
- PostgreSQL
## Supported Databases

For each DB vendor we have a benchmarking `usage_scenario_xxx.yml` that you can execute with the Green Metrics Tool to
get energy readings.
* MariaDB
* MySQL
* PostgreSQL
* Microsoft SQL Server
* Oracle Free Edition
* CockroachDB (Distributed)

We are currently only using the [TPC-C benchmark](https://en.wikipedia.org/wiki/TPC-C)
## Structure

- `production/` — Full benchmark scripts (rampup 2, duration 5, all VUs)
- `test/` — Minimal test scripts (rampup 0, duration 1, 1 VU)
- `usage_scenario_<db>.yml` — Production GMT scenarios
- `usage_scenario_<db>_test.yml` — Minimal test GMT scenarios

## Usage

Run a benchmark with the Green Metrics Tool:

python3 runner.py --uri /path/to/DBMS-bench --filename usage_scenario_maria_test.yml --allow-unsafe

## Benchmarks

We are currently only using the TPC-C benchmark.

## Notes

### Credentials
All database credentials are dummy credentials for local Docker containers only. Never use these in production.

### Oracle
Oracle requires the freepdb1 Pluggable Database instead of the default CDB. The connection string is //oracle_container:1521/freepdb1. Oracle needs ~120 seconds to start up, handled via a sleep in the usage scenario. To pull the Oracle image you need a free Oracle account and must accept the license at container-registry.oracle.com.

### SQL Server
SQL Server needs ~30 seconds to start up, handled via a sleep in the usage scenario.
## Some background reading

- https://www.hammerdb.com/about.html
Expand Down
22 changes: 22 additions & 0 deletions cockroach/cockroach_tprocc_buildschema.tcl
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#!/bin/tclsh
puts "SETTING CONFIGURATION"
dbset db pg
dbset bm TPC-C
diset connection pg_host cockroachdb_container
diset connection pg_port 26257
diset connection pg_sslmode disable
diset tpcc pg_superuser root
diset tpcc pg_superuserpass ""
diset tpcc pg_defaultdbase defaultdb
diset tpcc pg_user root
diset tpcc pg_pass ""
diset tpcc pg_dbase defaultdb
set vu [ numberOfCPUs ]
set warehouse [ expr {$vu * 20} ]
diset tpcc pg_count_ware $warehouse
diset tpcc pg_num_vu $vu
diset tpcc pg_partition false
diset tpcc pg_raiseerror true
puts "SCHEMA BUILD STARTED"
buildschema
puts "SCHEMA BUILD COMPLETED"
17 changes: 17 additions & 0 deletions cockroach/cockroach_tprocc_deleteschema.tcl
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
#!/bin/tclsh
puts "SETTING CONFIGURATION"
dbset db pg
dbset bm TPC-C
diset connection pg_host cockroachdb_container
diset connection pg_port 26257
diset connection pg_sslmode disable
diset tpcc pg_superuser root
diset tpcc pg_superuserpass ""
diset tpcc pg_defaultdbase defaultdb
diset tpcc pg_user root
diset tpcc pg_pass ""
diset tpcc pg_dbase defaultdb
diset tpcc pg_raiseerror true
puts "DROP SCHEMA STARTED"
deleteschema
puts "DROP SCHEMA COMPLETED"
33 changes: 33 additions & 0 deletions cockroach/cockroach_tprocc_run.tcl
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
#!/bin/tclsh
puts "SETTING CONFIGURATION"
dbset db pg
dbset bm TPC-C
diset connection pg_host cockroachdb_container
diset connection pg_port 26257
diset connection pg_sslmode disable
diset tpcc pg_superuser root
diset tpcc pg_superuserpass ""
diset tpcc pg_defaultdbase defaultdb
diset tpcc pg_user root
diset tpcc pg_pass ""
diset tpcc pg_dbase defaultdb
diset tpcc pg_driver timed
diset tpcc pg_rampup 2
diset tpcc pg_duration 5
diset tpcc pg_allwarehouse true
diset tpcc pg_timeprofile true
diset tpcc pg_keyandthink true
diset tpcc pg_raiseerror true
loadscript
puts "TEST STARTED"
vuset vu vcpu
vucreate
tcstart
tcstatus
set jobid [ vurun ]
vudestroy
tcstop
puts "TEST COMPLETE"
set of [ open /tmp/cockroach_tprocc w ]
puts $of $jobid
close $of
25 changes: 24 additions & 1 deletion compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ services:
mysql:
image: mysql:9.5.0-oraclelinux9
container_name: mysql_container
command: --default-authentication-plugin=mysql_native_password
environment:
MYSQL_ROOT_PASSWORD: mysql
MYSQL_DATABASE: mysql
Expand All @@ -34,3 +33,27 @@ services:
hammerdb:
image: tpcorg/hammerdb:latest
container_name: hammerdb_container
environment:
LD_LIBRARY_PATH: /home/instantclient_21_18
ORACLE_HOME: /home/instantclient_21_18
mssql:
image: mcr.microsoft.com/mssql/server:2022-latest
container_name: mssql_container
environment:
ACCEPT_EULA: Y
MSSQL_SA_PASSWORD: Mssql1234!
ports:
- "1433:1433"
cockroachdb:
image: cockroachdb/cockroach:latest
container_name: cockroachdb_container
command: start-single-node --insecure
ports:
- "26257:26257"
oracle:
image: container-registry.oracle.com/database/free:latest
container_name: oracle_container
environment:
ORACLE_PWD: manager
ports:
- "1521:1521"
23 changes: 23 additions & 0 deletions mssql/mssql_tprocc_buildschema.tcl
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
#!/bin/tclsh
puts "SETTING CONFIGURATION"
dbset db mssqls
dbset bm TPC-C
diset connection mssqls_tcp true
diset connection mssqls_port 1433
diset connection mssqls_azure false
diset connection mssqls_encrypt_connection true
diset connection mssqls_trust_server_cert true
diset connection mssqls_linux_server mssql_container
diset connection mssqls_uid sa
diset connection mssqls_pass Mssql1234!
diset connection mssqls_linux_authent sql
diset connection mssqls_linux_odbc {ODBC Driver 18 for SQL Server}
set vu [ numberOfCPUs ]
set warehouse [ expr {$vu * 5} ]
diset tpcc mssqls_count_ware $warehouse
diset tpcc mssqls_num_vu $vu
diset tpcc mssqls_dbase tpcc
diset tpcc mssqls_raiseerror true
puts "SCHEMA BUILD STARTED"
buildschema
puts "SCHEMA BUILD COMPLETED"
19 changes: 19 additions & 0 deletions mssql/mssql_tprocc_deleteschema.tcl
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
#!/bin/tclsh
puts "SETTING CONFIGURATION"
dbset db mssqls
dbset bm TPC-C
diset connection mssqls_tcp true
diset connection mssqls_port 1433
diset connection mssqls_azure false
diset connection mssqls_encrypt_connection true
diset connection mssqls_trust_server_cert true
diset connection mssqls_linux_server mssql_container
diset connection mssqls_uid sa
diset connection mssqls_pass Mssql1234!
diset connection mssqls_linux_authent sql
diset connection mssqls_linux_odbc {ODBC Driver 18 for SQL Server}
diset tpcc mssqls_dbase tpcc
diset tpcc mssqls_raiseerror true
puts "DROP SCHEMA STARTED"
deleteschema
puts "DROP SCHEMA COMPLETED"
34 changes: 34 additions & 0 deletions mssql/mssql_tprocc_run.tcl
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#!/bin/tclsh
puts "SETTING CONFIGURATION"
dbset db mssqls
dbset bm TPC-C
diset connection mssqls_tcp true
diset connection mssqls_port 1433
diset connection mssqls_azure false
diset connection mssqls_encrypt_connection true
diset connection mssqls_trust_server_cert true
diset connection mssqls_linux_server mssql_container
diset connection mssqls_uid sa
diset connection mssqls_pass Mssql1234!
diset connection mssqls_linux_authent sql
diset connection mssqls_linux_odbc {ODBC Driver 18 for SQL Server}
diset tpcc mssqls_dbase tpcc
diset tpcc mssqls_driver timed
diset tpcc mssqls_rampup 2
diset tpcc mssqls_duration 5
diset tpcc mssqls_allwarehouse true
diset tpcc mssqls_timeprofile true
diset tpcc mssqls_raiseerror true
loadscript
puts "TEST STARTED"
vuset vu vcpu
vucreate
tcstart
tcstatus
set jobid [ vurun ]
vudestroy
tcstop
puts "TEST COMPLETE"
set of [ open /tmp/mssql_tprocc w ]
puts $of $jobid
close $of
26 changes: 26 additions & 0 deletions oracle/oracle_tprocc_buildschema.tcl
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
#!/bin/tclsh
puts "SETTING CONFIGURATION"
dbset db ora
dbset bm TPC-C
diset connection system_user system
diset connection system_password manager
diset connection instance //oracle_container:1521/freepdb1
set vu [ numberOfCPUs ]
set warehouse [ expr {$vu * 5} ]
diset tpcc count_ware $warehouse
diset tpcc num_vu $vu
diset tpcc tpcc_user tpcc
diset tpcc tpcc_pass tpcc
diset tpcc tpcc_def_tab users
diset tpcc tpcc_def_temp temp
if { $warehouse >= 200 } {
diset tpcc partition true
diset tpcc hash_clusters true
diset tpcc tpcc_ol_tab users
} else {
diset tpcc partition false
diset tpcc hash_clusters false
}
puts "SCHEMA BUILD STARTED"
buildschema
puts "SCHEMA BUILD COMPLETED"
12 changes: 12 additions & 0 deletions oracle/oracle_tprocc_deleteschema.tcl
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
#!/bin/tclsh
puts "SETTING CONFIGURATION"
dbset db ora
dbset bm TPC-C
diset connection system_user system
diset connection system_password manager
diset connection instance //oracle_container:1521/freepdb1
diset tpcc tpcc_user tpcc
diset tpcc tpcc_pass tpcc
puts "DROP SCHEMA STARTED"
deleteschema
puts "DROP SCHEMA COMPLETED"
29 changes: 29 additions & 0 deletions oracle/oracle_tprocc_run.tcl
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
#!/bin/tclsh
puts "SETTING CONFIGURATION"
dbset db ora
dbset bm TPC-C
diset connection system_user system
diset connection system_password manager
diset connection instance //oracle_container:1521/freepdb1
diset tpcc tpcc_user tpcc
diset tpcc tpcc_pass tpcc
diset tpcc ora_driver timed
diset tpcc total_iterations 10000000
diset tpcc rampup 2
diset tpcc duration 5
diset tpcc ora_timeprofile true
diset tpcc allwarehouse true
diset tpcc checkpoint false
loadscript
puts "TEST STARTED"
vuset vu vcpu
vucreate
tcstart
tcstatus
set jobid [ vurun ]
vudestroy
tcstop
puts "TEST COMPLETE"
set of [ open /tmp/oracle_tprocc w ]
puts $of $jobid
close $of
Loading