More tools

Author: AdamGS

datafusion-vortex-partitioned/README.md

@@ -15,6 +15,12 @@ bash benchmark.sh
 
 The benchmark script builds `vortex-datafusion-cli`, downloads the partitioned Parquet files, converts each `partitioned/hits_N.parquet` file into exactly one `vortex/hits_N.vortex` file, and runs the query set.
 
+You can update/preview the results by running:
+
+```bash
+./make-json.sh <machine-name> # Example. ./make-json.sh c6a.xlarge
+```
+
 ## Parquet to Vortex conversion
 
 Each input file is converted independently through `vortex-datafusion-cli`:

datafusion-vortex-partitioned/make-json.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+
+# This script converts the raw `result.csv` data from `benchmark.sh` into the
+# final json format used by the benchmark dashboard.
+#
+# usage : ./make-json.sh <machine>
+#
+# example (save results/c6a.4xlarge.json)
+#         ./make-json.sh c6a.4xlarge
+
+MACHINE=$1
+OUTPUT_FILE="results/${MACHINE}.json"
+SYSTEM_NAME="DataFusion (Vortex, partitioned)"
+DATE=$(date +%Y-%m-%d)
+LOAD_TIME=${LOAD_TIME:-$(cat load-time.txt 2>/dev/null || echo null)}
+DATA_SIZE=${DATA_SIZE:-$(du -bcs vortex/*.vortex 2>/dev/null | awk '/total/ { print $1 }')}
+DATA_SIZE=${DATA_SIZE:-null}
+
+mkdir -p results
+
+# Read the CSV and build the result array using sed
+RESULT_ARRAY=$(awk -F, '{arr[$1]=arr[$1]","$3} END {for (i=1;i<=length(arr);i++) {gsub(/^,/, "", arr[i]); printf "        ["arr[i]"]"; if (i<length(arr)) printf ",\n"}}' result.csv)
+
+# form the final JSON structure from the template
+cat <<EOF > $OUTPUT_FILE
+{
+    "system": "$SYSTEM_NAME",
+    "date": "$DATE",
+    "machine": "$MACHINE",
+    "cluster_size": 1,
+    "proprietary": "no",
+    "tuned": "no",
+    "hardware": "cpu",
+    "tags": ["Rust","column-oriented","embedded","stateless"],
+    "load_time": $LOAD_TIME,
+    "data_size": $DATA_SIZE,
+    "result": [
+        $RESULT_ARRAY
+    ]
+}
+EOF

datafusion-vortex/README.md

@@ -15,6 +15,12 @@ bash benchmark.sh
 
 The benchmark script builds `vortex-datafusion-cli`, downloads `hits.parquet`, converts it to `vortex/hits.vortex`, and runs the query set.
 
+You can update/preview the results by running:
+
+```bash
+./make-json.sh <machine-name> # Example. ./make-json.sh c6a.xlarge
+```
+
 ## Parquet to Vortex conversion
 
 The conversion intentionally goes through the DataFusion CLI path:

datafusion-vortex/make-json.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+
+# This script converts the raw `result.csv` data from `benchmark.sh` into the
+# final json format used by the benchmark dashboard.
+#
+# usage : ./make-json.sh <machine>
+#
+# example ./make-json.sh c6a.4xlarge # saves results/c6a.4xlarge.json
+#
+
+MACHINE=$1
+OUTPUT_FILE="results/${MACHINE}.json"
+SYSTEM_NAME="DataFusion (Vortex, single)"
+DATE=$(date +%Y-%m-%d)
+LOAD_TIME=${LOAD_TIME:-$(cat load-time.txt 2>/dev/null || echo null)}
+DATA_SIZE=${DATA_SIZE:-$(du -bcs vortex/*.vortex 2>/dev/null | awk '/total/ { print $1 }')}
+DATA_SIZE=${DATA_SIZE:-null}
+
+mkdir -p results
+
+# Read the CSV and build the result array using sed
+RESULT_ARRAY=$(awk -F, '{arr[$1]=arr[$1]","$3} END {for (i=1;i<=length(arr);i++) {gsub(/^,/, "", arr[i]); printf "        ["arr[i]"]"; if (i<length(arr)) printf ",\n"}}' result.csv)
+
+# form the final JSON structure from the template
+cat <<EOF > $OUTPUT_FILE
+{
+    "system": "$SYSTEM_NAME",
+    "date": "$DATE",
+    "machine": "$MACHINE",
+    "cluster_size": 1,
+    "proprietary": "no",
+    "tuned": "no",
+    "hardware": "cpu",
+    "tags": ["Rust","column-oriented","embedded","stateless"],
+    "load_time": $LOAD_TIME,
+    "data_size": $DATA_SIZE,
+    "result": [
+        $RESULT_ARRAY
+    ]
+}
+EOF