diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/_index.md b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/_index.md new file mode 100644 index 0000000000..a7bc631f48 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/_index.md @@ -0,0 +1,36 @@ +--- +title: Go Benchmarks with Sweet and Benchstat + +minutes_to_complete: 60 + +who_is_this_for: This learning path is for developers who are interested in measuring the performance of Go-based applications on their servers. + +learning_objectives: + - Bring up two instances of GCP VMs + - Install Go, benchmarks, benchstat, and sweet on the two VMs + - Using sweet and benchstat, compare performance of Go applications on the two VMs + +prerequisites: + - A [Google Cloud account](https://console.cloud.google.com/). This learning path can be run on-prem or on any cloud provider, but specifically documents the process on GCP's Google Axion platform. + - A local machine with [Google Cloud CLI](/install-guides/gcloud/) installed. + +author: Geremy Cohen + +### Tags +skilllevels: Introductory +subjects: Performance and Architecture +armips: + - Neoverse +tools_software_languages: + +operatingsystems: + - Linux + + + +### FIXED, DO NOT MODIFY +# ================================================================================ +weight: 1 # _index.md always has weight of 1 to order correctly +layout: "learningpathall" # All files under learning paths have this same wrapper +learning_path_main_page: "yes" # This should be surfaced when looking for related content. Only set for _index.md of learning path content. +--- diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/_next-steps.md b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/_next-steps.md new file mode 100644 index 0000000000..3b237f0d1c --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/_next-steps.md @@ -0,0 +1,8 @@ +--- +# ================================================================================ +# FIXED, DO NOT MODIFY THIS FILE +# ================================================================================ +weight: 999 # Set to always be larger than the content in this path to be at the end of the navigation. +title: "Next Steps" # Always the same, html page title. +layout: "learningpathall" # All files under learning paths have this same wrapper for Hugo processing. +--- diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/add_c4_vm.md b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/add_c4_vm.md new file mode 100644 index 0000000000..43aa54edaa --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/add_c4_vm.md @@ -0,0 +1,30 @@ +--- +title: Launching an Intel C4 Instance +weight: 30 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Section Overview +In this section, you will set up the second benchmarking system, an Intel-based c4-standard-8 instance. + +## Creating the Instance + +To create the second system, follow the previous lesson's c4a install instructions, but make the following changes: + +1. **Name your instance:** For the `Name` field, enter "c4". + +2. **Select machine series:** Scroll down to the Machine series section, and select the C4 radio button. + +![](images/launch_c4/3.png) + +3. **View machine types:** Scroll down to the Machine type dropdown, and click it to show all available options. + +![](images/launch_c4/4.png) + +4. **Choose machine size:** Select "c4-standard-8" under the Standard tab. + +![](images/launch_c4/5.png) + +After the c4 instance starts up, you are ready to continue to the next section, where you'll install the benchmarking software. \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/add_c4a_vm.md b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/add_c4a_vm.md new file mode 100644 index 0000000000..fed6b3de2a --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/add_c4a_vm.md @@ -0,0 +1,67 @@ +--- +title: Launching an Arm Axion C4a Instance +weight: 20 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Section Overview +In this section, you'll learn how to spin up the first of two different systems used in our benchmarking tests, an Arm-based Google Axion c4a-standard-4 (c4a for short). + +## Creating the c4a-standard-4 Instance + +1. **Access Google Cloud Console:** Navigate to [https://console.cloud.google.com/welcome](https://console.cloud.google.com/welcome) + +2. **Search for VM instances:** Click into the Search field. + +3. **Find VM Instances:** Start typing `vm` until the UI auto-completes `VM Instances`, then click it. + +![](images/launch_c4a/3.png) + +The VM Instances page appears. + +4. **Create a new instance:** Click `Create instance` + +![](images/launch_c4a/4.png) + +The Machine configuration page appears. + +5. **Name your instance:** Click the `Name` field, and enter "c4a" for the `Name`. + +![](images/launch_c4a/5.png) + +6. **Select machine series:** Scroll down to the Machine series section, and select the C4A radio button. + +![](images/launch_c4a/7.png) + +7. **View machine types:** Scroll down to the Machine type dropdown, and click it to show all available options. + +![](images/launch_c4a/8.png) + +8. **Choose machine size:** Select "c4a-standard-4" under the Standard tab. + +![](images/launch_c4a/9.png) + +9. **Configure storage:** Click the "OS and Storage" tab. + +![](images/launch_c4a/10.png) + +10. **Modify storage settings:** Click "Change" + +![](images/launch_c4a/11.png) + +11. **Set disk size:** Double-click the "Size (GB)" field, then enter "1000" for the value. + +![](images/launch_c4a/16.png) + +12. **Confirm storage settings:** Click "Select" to continue. + +![](images/launch_c4a/18.png) + +13. **Launch the instance:** Click "Create" to bring up the instance. + +![](images/launch_c4a/19.png) + +After a few seconds, your c4a instance starts up, and you are ready to continue to the next section. In the next step, you will launch the second system, an Intel-based Emerald Rapids c4-standard-8 (c4 for short), which will serve as the comparison system for our benchmarking tests. + diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/11.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/11.png new file mode 100644 index 0000000000..daf435ca54 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/11.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/12.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/12.png new file mode 100644 index 0000000000..2272290d64 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/12.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/13.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/13.png new file mode 100644 index 0000000000..9011f185f3 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/13.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/15.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/15.png new file mode 100644 index 0000000000..c4acc35c47 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/15.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/16.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/16.png new file mode 100644 index 0000000000..a8b317beae Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/16.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/3.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/3.png new file mode 100644 index 0000000000..108c89e021 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/3.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/4.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/4.png new file mode 100644 index 0000000000..502d70c81e Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/4.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/5.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/5.png new file mode 100644 index 0000000000..0cc7da7c9c Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4/5.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/0.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/0.png new file mode 100644 index 0000000000..854c12ef57 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/0.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/1.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/1.png new file mode 100644 index 0000000000..9a854d354f Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/1.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/10.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/10.png new file mode 100644 index 0000000000..d3db445885 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/10.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/11.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/11.png new file mode 100644 index 0000000000..fd9649cdaa Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/11.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/16.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/16.png new file mode 100644 index 0000000000..121dc7b71a Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/16.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/18.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/18.png new file mode 100644 index 0000000000..2dc9951856 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/18.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/19.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/19.png new file mode 100644 index 0000000000..d46999f111 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/19.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/3.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/3.png new file mode 100644 index 0000000000..91df042524 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/3.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/4.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/4.png new file mode 100644 index 0000000000..cadc235255 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/4.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/5.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/5.png new file mode 100644 index 0000000000..709c0bd493 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/5.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/7.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/7.png new file mode 100644 index 0000000000..8879a456ec Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/7.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/8.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/8.png new file mode 100644 index 0000000000..58c5ae19e9 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/8.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/9.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/9.png new file mode 100644 index 0000000000..fdd0e1fa25 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/launch_c4a/9.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_auto/1.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_auto/1.png new file mode 100644 index 0000000000..272bd23f2b Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_auto/1.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_auto/2.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_auto/2.png new file mode 100644 index 0000000000..a67077541d Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_auto/2.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/11.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/11.png new file mode 100644 index 0000000000..03b8028518 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/11.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/16.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/16.png new file mode 100644 index 0000000000..0e8ec05f54 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/16.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/17.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/17.png new file mode 100644 index 0000000000..4863740bd2 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/17.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/2.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/2.png new file mode 100644 index 0000000000..56d9f59e13 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/2.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/3.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/3.png new file mode 100644 index 0000000000..503f9b341c Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/3.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/4.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/4.png new file mode 100644 index 0000000000..f1888b0a57 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/4.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/5.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/5.png new file mode 100644 index 0000000000..3794c19753 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/5.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/6.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/6.png new file mode 100644 index 0000000000..084c5444e7 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/6.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/7.png b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/7.png new file mode 100644 index 0000000000..57cbd90d53 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/images/run_manually/7.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/installing_go_and_sweet.md b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/installing_go_and_sweet.md new file mode 100644 index 0000000000..736b1ebf24 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/installing_go_and_sweet.md @@ -0,0 +1,120 @@ +--- +title: Installing Go and Sweet +weight: 40 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Section Overview + +In this section, you'll install Go, Sweet, and the Benchstat comparison tool on both VMs. + +## Installation Script + +Sweet is a Go benchmarking tool that provides a standardized way to run performance tests across different systems. Benchstat is a companion tool that analyzes and compares benchmark results, helping you understand performance differences between systems. Together, these tools will allow us to accurately measure and compare Go performance on Arm and x86 architectures. + + +{{% notice Note %}} +Subsequent steps in the learning path assume you are running this script (installing) from your home directory (`~`), resulting in the creation of a `~/benchmarks/sweet` final install path. If you decide to install elsewhere, adjust the path accordingly when prompted to run the benchmark logic later in the learning path. +{{% /notice %}} + + +Copy and paste this script to **both** of your GCP VMs. + +**You don't need to run it after pasting**, just paste it into your home directory and press enter to install all needed dependencies: + +```bash +#!/usr/bin/env bash + +# Write the script to filesystem using a HEREDOC +cat <<'EOF' > install_go_and_sweet.sh + +sudo apt-get -y update +sudo apt-get -y install git build-essential + +# Detect architecture - this allows the same script to work on both +# our Arm (c4a) and x86 (c4) VMs without modification +ARCH=$(uname -m) +case "$ARCH" in + arm64|aarch64) + GO_PKG="go1.24.2.linux-arm64.tar.gz" + ;; + x86_64|amd64) + GO_PKG="go1.24.2.linux-amd64.tar.gz" + ;; + *) + echo "Unsupported architecture: $ARCH" + exit 1 + ;; +esac + +# Download and install architecture-specific Go environments + +URL="https://go.dev/dl/${GO_PKG}" +echo "Downloading $URL..." +wget -q --show-progress "$URL" + +echo "Extracting $GO_PKG to /usr/local..." +sudo tar -C /usr/local -xzf "$GO_PKG" + + +echo "Go 1.24.2 installed successfully for $ARCH." + +export GOPATH=$HOME/go +export GOBIN=$GOPATH/bin +export PATH=$PATH:$GOBIN:/usr/local/go/bin + +# Install Sweet, benchmarks, and benchstat tools +go install golang.org/x/benchmarks/sweet/cmd/sweet@latest +go install golang.org/x/perf/cmd/benchstat@latest + +git clone https://github.com/golang/benchmarks +cd benchmarks/sweet +sweet get -force # to get assets + +# Create a configuration file + +cat < config.toml +[[config]] + name = "arm-benchmarks" + goroot = "/usr/local/go" + +CONFFILE + +EOF + +# Make the script executable +chmod 755 install_go_and_sweet.sh + +# Run the script +./install_go_and_sweet.sh + +``` + + + +## Verify Installation + +To test that everything is installed correctly, run the following command on each VM after the script completes: + +```bash +# Run the benchmarks +export GOPATH=$HOME/go +export GOBIN=$GOPATH/bin +export PATH=$PATH:$GOBIN:/usr/local/go/bin +cd benchmarks/sweet +sweet run -count 10 -run="markdown" config.toml # run one, 1X +``` + +You should see output similar to the following: + +```bash +# Example output: +[sweet] Work directory: /tmp/gosweet3444550660 +[sweet] Benchmarks: markdown (10 runs) +[sweet] Setting up benchmark: markdown +[sweet] Running benchmark markdown for arm-benchmarks: run 1 +... +[sweet] Running benchmark markdown for arm-benchmarks: run 10 +``` \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/manual_run_benchmark.md b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/manual_run_benchmark.md new file mode 100644 index 0000000000..4c9c250140 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/manual_run_benchmark.md @@ -0,0 +1,64 @@ +--- +title: Manually running benchmarks +weight: 51 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Section Overview + +In this section, you'll run benchmarks manually to understand how `sweet` and `benchstat` work together. + +## Run Benchmarks on Each Machine +The following explains how to run the benchmarks by hand on each machine. + +1. **Access VM instances:** Navigate to the GCP [VM Instances](https://console.cloud.google.com/compute/instances) console. + +2. **Connect to c4a instance:** Click on the `SSH` button next to your `c4a` instance. + An SSH terminal will open in a new tab. + + ![](images/run_manually/2.png) + +3. **Set up environment:** Copy and paste the following into the SSH terminal to setup the environment and change to the sweet directory: + + ```bash + cd benchmarks/sweet + export GOPATH=$HOME/go + export GOBIN=$GOPATH/bin + export PATH=$PATH:$GOBIN:/usr/local/go/bin + ``` + + ![](images/run_manually/3.png) + +{{% notice Note %}} +The above instructions assume you installed the benchmarks in the `~/benchmarks/sweet` directory. If you installed them elsewhere, adjust the path accordingly. +{{% /notice %}} + + +4. **Run the benchmark:** Copy and paste the following command to run the `markdown` benchmark with `sweet`: + + ```bash + sweet run -count 10 -run="markdown" config.toml + ``` + +5. **Locate results:** After the benchmark completes, cd to the `results/markdown` directory and list the files to see the `arm-benchmarks.result` file: + + ```bash + cd results/markdown + ls -d $PWD/* + ``` + +6. **Copy result path:** Copy the absolute pathname of `arm-benchmarks.result`. + +7. **Download results:** Click `DOWNLOAD FILE`, and paste the **ABSOLUTE PATHNAME** you just copied for the filename, and then click `Download`. This will download the benchmark results to your local machine. + + ![](images/run_manually/6.png) + +8. **Rename the file:** Once downloaded, on your local machine, rename this file to `c4a.result` so you can distinguish it from the x86 results you'll download later. This naming convention will help you clearly identify which results came from which architecture. You'll know the file downloaded successfully if you see the file in your Downloads directory with the name `c4a.result`, as well as the confirmation dialog in your browser: + + ![](images/run_manually/7.png) + +9. **Repeat for c4 instance:** Repeat steps 2-8 with your `c4` (x86) instance. Do everything the same, except after downloading the c4's `arm-benchmarks.result` file, rename it to `c4.result`. + +Now that you have the results from both VMs, in the next section, you'll learn how to use benchstat to analyze these results and understand the performance differences between the two architectures. \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/manual_run_benchstat.md b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/manual_run_benchstat.md new file mode 100755 index 0000000000..b0986a4216 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/manual_run_benchstat.md @@ -0,0 +1,128 @@ +--- +title: Manually running benchstat +weight: 52 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Section Overview + +You've successfully run and downloaded the benchmark results from both your Arm-based and x86-based VMs. In this section, you'll compare them to each other using the benchstat tool. + + +## Inspecting the Results Files + +With the results files downloaded to your local machine, if you're curious to what they look like, you can inspect them to understand better what `benchstat` is analyzing. + +1. **View raw results:** Open the `c4a.result` file in a text editor, and you'll see something like this: + + ![](images/run_manually/11.png) + +The file contains the results of the `markdown` benchmark run on the Arm-based c4a VM, showing time and memory stats taken for each iteration. If you open the `c4.result` file, you'll see similar results for the x86-based c4 VM. + +2. **Close the editor:** Close the text editor when done. + +## Running Benchstat to Compare Results + +To compare the results, you'll use `benchstat` to analyze the two result files you downloaded. Since all the prerequisites are already installed on the `c4` and `c4a` instances, benchstat will be run from one of those instances. + +1. **Connect to VM:** As demonstrated in the previous chapter, SSH into the `c4a` instance. + +2. **Create working directory:** Make a temporary benchstat directory to hold the results files, and cd into it: + + ```bash + mkdir benchstat_results + cd benchstat_results + ``` + +3. **Upload result files:** Click the `UPLOAD FILE` button in the GCP console, and upload the `c4a.results` AND `c4.results` files you downloaded earlier. (This uploads them to your home directory, not to the current directory.) + + ![](images/run_manually/16.png) + +4. **Verify upload:** You'll know it worked correctly via the confirmation dialog in your terminal: + + ![](images/run_manually/17.png) + +5. **Move files to working directory:** Move the results files to the `benchstat_results` directory, and confirm their presence: + + ```bash + mv ~/c4a.results ~/c4.results . + ls -al + ``` + + You should see both files in the benchstat_results directory: + + ```bash + # Example output: + c4a-48:~/benchstat_results$ ls + c4.results c4a.results + ``` + +6. **Run benchstat:** Now you can run `benchstat` to compare the two results files: + + ```bash + export GOPATH=$HOME/go + export GOBIN=$GOPATH/bin + export PATH=$PATH:$GOBIN:/usr/local/go/bin + benchstat c4a.results c4.results > c4a_vs_c4.txt + ``` + +7. **View comparison results:** Run the `cat` command to view the results: + + ```bash + cat c4a_vs_c4.txt + ``` + + You should see output similar to the following: + + ```output + # Example output: + geremy_cohen_arm_com@c4a-48:~/benchstat_results$ cat c4a_vs_c4.txt + │ c4a.results │ c4.results │ + │ sec/op │ sec/op vs base │ + MarkdownRenderXHTML-48 143.9m ± 1% + MarkdownRenderXHTML-96 158.3m ± 0% + geomean 143.9m 158.3m ? ¹ ² + ¹ benchmark set differs from baseline; geomeans may not be comparable + ² ratios must be >0 to compute geomean + + │ c4a.results │ c4.results │ + │ average-RSS-bytes │ average-RSS-bytes vs base │ + MarkdownRenderXHTML-48 22.49Mi ± 6% + MarkdownRenderXHTML-96 24.78Mi ± 2% + geomean 22.49Mi 24.78Mi ? ¹ ² + ¹ benchmark set differs from baseline; geomeans may not be comparable + ² ratios must be >0 to compute geomean + + │ c4a.results │ c4.results │ + │ peak-RSS-bytes │ peak-RSS-bytes vs base │ + MarkdownRenderXHTML-48 23.67Mi ± 4% + MarkdownRenderXHTML-96 25.11Mi ± 7% + geomean 23.67Mi 25.11Mi ? ¹ ² + ¹ benchmark set differs from baseline; geomeans may not be comparable + ² ratios must be >0 to compute geomean + + │ c4a.results │ c4.results │ + │ peak-VM-bytes │ peak-VM-bytes vs base │ + MarkdownRenderXHTML-48 1.176Gi ± 0% + MarkdownRenderXHTML-96 1.176Gi ± 0% + geomean 1.176Gi 1.176Gi ? ¹ ² + ¹ benchmark set differs from baseline; geomeans may not be comparable + ² ratios must be >0 to compute geomean + ``` + + This output shows the performance differences between the two VMs for the `markdown` benchmark in text format. The key metrics to observe are: + + - **sec/op**: Shows the execution time per operation (lower is better) + - **average-RSS-bytes**: Shows the average resident set size memory usage (lower is better) + - **peak-RSS-bytes**: Shows the maximum resident set size memory usage (lower is better) + - **peak-VM-bytes**: Shows the maximum virtual memory usage (lower is better) + + In this example, you can see that the c4a (Arm) instance completed the markdown benchmark in 143.9m seconds, while the c4 (x86) instance took 158.3m seconds, indicating better performance on the Arm system for this particular workload. + + If you wanted the results in CSV format, you could run the `benchstat` command with the `-format csv` option instead. + +At this point, you can download the `c4a_vs_c4.txt` for further analysis or reporting. You can also run the same or different benchmarks with the same, or different combinations of VMs, and continue comparing results using `benchstat`. + +Continuing on to the next section, you will learn how to automate and gain enhanced visuals with sweet and benchstat. \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/overview.md b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/overview.md new file mode 100644 index 0000000000..9dd1d48001 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/overview.md @@ -0,0 +1,48 @@ +--- +title: Overview +weight: 10 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +# Go Benchmarking Learning Path Overview + +In this module, you'll learn how to measure, collect, and compare Go performance data across different CPU architectures. This knowledge is essential for developers and system architects who need to make informed decisions about infrastructure choices for their Go applications. + +You'll gain hands-on experience with: + +- **Go Benchmarks**, a collection of pre-written benchmark definitions that standardizes performance tests for popular Go applications, leveraging Go's built-in benchmark support. + +- **Sweet**, a benchmark runner that automates running Go benchmarks across multiple environments, collecting and formatting results for comparison. + +- **Benchstat**, a statistical comparison tool that analyzes benchmark results to identify meaningful performance differences between systems. + +Benchmarking is critical for modern software development because it allows you to: +- Quantify the performance impact of code changes +- Compare different hardware platforms objectively +- Make data-driven decisions about infrastructure investments +- Identify optimization opportunities in your applications + +You'll use Intel c4-standard-8 and Arm-based c4a-standard-4 (both four-core) instances running on GCP to run and compare benchmarks using these tools. + +{{% notice Note %}} +Arm-based c4a-standard-4 instances and Intel-based c4-standard-8 instances both utilize four cores. Both instances are categorized by GCP as members of the **consistently high performing** series; the main difference between the two is that the c4a has 16 GB of RAM, while the c4 has 30 GB of RAM. We've chosen to keep CPU cores equivalent across the two instances of the same series to keep the comparison as close as possible. +{{% /notice %}} + + +## Prerequisites + +Before starting this learning path, you should have: +- A Google Cloud Platform account with permissions to create VMs +- Basic familiarity with Linux command line +- Basic understanding of what benchmarking is and why it's useful + +## What You'll Accomplish + +When you are finished running through this learning path, you'll have: + +1. Brought up Arm and x86 instances of GCP-based VMs +2. Installed Go, benchmarks, benchstat, and sweet on the two VMs +3. Used sweet and benchstat to compare performance of Go applications on the two VMs +4. Extrapolated the knowledge to create Go Benchmarks for your own workloads and systems \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/pytest.ini b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/pytest.ini new file mode 100644 index 0000000000..38be796943 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/pytest.ini @@ -0,0 +1,5 @@ +[pytest] +testpaths = tests +python_files = test_*.py +python_classes = Test* +python_functions = test_* \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/requirements.txt b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/requirements.txt new file mode 100644 index 0000000000..447c5731b6 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/requirements.txt @@ -0,0 +1,8 @@ +numpy==2.0.2 +pandas==2.3.0 +plotly==5.13.1 +python-dateutil==2.9.0.post0 +pytz==2025.2 +six==1.17.0 +tenacity==9.1.2 +tzdata==2025.2 diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet.py b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet.py new file mode 100644 index 0000000000..5fd734e3ba --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet.py @@ -0,0 +1,27 @@ +#!/usr/bin/env python3 +""" +Entry point for the rexec_sweet tool. +""" +import sys +import os + +def main(): + try: + from rexec_sweet.cli import main as cli_main + sys.exit(cli_main()) + except ImportError as e: + print(f"Error importing modules: {e}") + print("\nTry running with PYTHONPATH set:") + script_dir = os.path.dirname(os.path.abspath(__file__)) + print(f"PYTHONPATH={script_dir} python {os.path.join(script_dir, 'rexec_sweet.py')}") + import traceback + traceback.print_exc() + sys.exit(1) + except Exception as e: + print(f"Unexpected error: {e}") + import traceback + traceback.print_exc() + sys.exit(1) + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet/__init__.py b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet/__init__.py new file mode 100644 index 0000000000..06fe9336e3 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet/__init__.py @@ -0,0 +1,21 @@ +""" +Remote execution tool for Sweet Go benchmarking. +""" +from .benchstat_report import BenchstatReport +from .gcp_utils import get_running_instances, choose_instance, get_instance_zone, scp_results +from .benchmark_runner import BenchmarkRunner +from .visualization import generate_html +from .config import Config + +__version__ = "0.1.0" + +__all__ = [ + 'BenchstatReport', + 'get_running_instances', + 'choose_instance', + 'get_instance_zone', + 'scp_results', + 'BenchmarkRunner', + 'generate_html', + 'Config', +] \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet/benchmark_runner.py b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet/benchmark_runner.py new file mode 100644 index 0000000000..6f81c7790d --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet/benchmark_runner.py @@ -0,0 +1,262 @@ +""" +Benchmark runner module for executing Sweet benchmarks on remote instances. +""" +import os +import subprocess +import sys +import threading +import webbrowser +from typing import Dict, List, Optional, Tuple + +from .config import COLORS, Config + +class BenchmarkRunner: + """ + Handles running benchmarks and processing results. + """ + def __init__(self): + self.config = Config() + self.reset = COLORS['reset'] + + def run_benchstat_report(self, benchstat_file: str, output_dir: str) -> None: + """ + Run benchstat report generation for the given benchstat_file into output_dir. + """ + # Strip any '/tmp/tmp...' prefix from filenames when printing + short_input = os.path.basename(benchstat_file) + short_outdir = os.path.relpath(output_dir, start=os.getcwd()) + print(f"Generating report for {short_input}…") + + from .benchstat_report import BenchstatReport + reporter = BenchstatReport() + reporter.generate_report(benchstat_file, output_dir) + + print(f"Report generated in {short_outdir}") + # Automatically open the generated report in the default web browser + html_path = os.path.join(output_dir, 'report.html') + abs_path = os.path.abspath(html_path) + webbrowser.open(f"file://{abs_path}") + + def run_remote(self, name: str, zone: str, remote_dir: str, + benchmark_name: str, color_idx: int = 0) -> int: + """ + SSH to the instance and run the sweet benchmark, streaming output. + + Args: + name: Instance name + zone: GCP zone + remote_dir: Remote directory path + benchmark_name: Name of the benchmark to run + color_idx: Index for color selection (0 or 1) + + Returns: + Return code (0 for success, non-zero for failure) + """ + benchmark_cmd = self.config.get_benchmark_command(benchmark_name) + if not benchmark_cmd: + print(f"Unknown benchmark: {benchmark_name}") + return 1 + + color = list(COLORS.values())[color_idx % 2] # blue or green + + ssh_cmd = [ + "gcloud", "compute", "ssh", name, + "--zone", zone, + "--command", + f"{self.config.env_setup} cd {remote_dir} && {benchmark_cmd}" + ] + + p = subprocess.Popen(ssh_cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) + + def reader(): + for line in iter(p.stdout.readline, b""): + sys.stdout.write(f"{color}[{name}]{self.reset} {line.decode()}") + + t = threading.Thread(target=reader, daemon=True) + t.start() + p.wait() + + if p.returncode != 0: + print(f"[{name}] ⚠️ exited with code {p.returncode}") + else: + print(f"[{name}] ✅ benchmark completed") + + return p.returncode + + def run_benchmark(self, benchmark_name: str, instance_name: str, + zone: str, remote_dir: str) -> bool: + """ + Run a specific benchmark on a remote instance. + + Args: + benchmark_name: Name of the benchmark to run + instance_name: GCP instance name + zone: GCP zone + remote_dir: Remote directory path + + Returns: + True if successful, False otherwise + """ + # Create remote directory if needed + ssh_cmd = [ + "gcloud", "compute", "ssh", + instance_name, + "--zone", zone, + "--command", f"mkdir -p {remote_dir}" + ] + try: + subprocess.check_call(ssh_cmd) + except subprocess.CalledProcessError: + print(f"Failed to create remote directory {remote_dir}") + return False + + # Run the benchmark with Go environment setup + color_idx = 0 if instance_name.endswith("1") else 1 + return self.run_remote(instance_name, zone, remote_dir, benchmark_name, color_idx) == 0 + + def collect_results(self, primary: Dict, secondary: Dict, benchmark_name: str, + local_dir: str) -> Optional[str]: + """ + Collect benchmark results from both instances and run benchstat. + + Args: + primary: Primary instance configuration + secondary: Secondary instance configuration + benchmark_name: Name of the benchmark + local_dir: Local directory to store results + + Returns: + Path to benchstat results file or None if failed + """ + # Create a remote temp directory on the primary VM + mktemp_cmd = [ + "gcloud", "compute", "ssh", primary["name"], + "--zone", primary["zone"], + "--command", "mktemp -d" + ] + try: + remote_tmp = subprocess.check_output(mktemp_cmd, universal_newlines=True).strip() + print(f"Created remote temp dir on {primary['name']}: {remote_tmp}") + except subprocess.CalledProcessError: + print(f"Failed to create temporary directory on {primary['name']}") + return None + + # Copy result files locally and push back to primary + try: + self._transfer_result_files(primary, secondary, remote_tmp, benchmark_name) + except subprocess.CalledProcessError as e: + print(f"Failed to transfer result files: {e}") + return None + + # Run benchstat on the primary VM + try: + self._run_benchstat(primary, secondary, remote_tmp) + except subprocess.CalledProcessError as e: + print(f"Failed to run benchstat: {e}") + return None + + # Pull all results back to local directory + print(f"Please wait, copying over result files to local directory: {local_dir}") + + try: + for fname in [ + f"{secondary['name']}.results", + f"{primary['name']}.results", + "benchstat.results" + ]: + subprocess.check_call([ + "gcloud", "compute", "scp", + f"{primary['name']}:{remote_tmp}/{fname}", + local_dir, "--zone", primary["zone"] + ]) + except subprocess.CalledProcessError as e: + print(f"Failed to copy results to local directory: {e}") + return None + + # Return path to benchstat results file + benchstat_file = os.path.join(local_dir, "benchstat.results") + if os.path.exists(benchstat_file): + return benchstat_file + else: + print(f"Warning: Benchstat results file not found at {benchstat_file}") + return None + + def _transfer_result_files(self, primary: Dict, secondary: Dict, + remote_tmp: str, benchmark_name: str) -> None: + """ + Transfer result files between instances. + + Args: + primary: Primary instance configuration + secondary: Secondary instance configuration + remote_tmp: Remote temporary directory + benchmark_name: Name of the benchmark + """ + import tempfile + with tempfile.TemporaryDirectory() as tmp: + # Copy secondary results to local + local_sec = os.path.join(tmp, f"{secondary['name']}.results") + print(f"Copying {secondary['name']} results to local: {local_sec}") + subprocess.check_call([ + "gcloud", "compute", "scp", + f"{secondary['name']}:{secondary['remote_dir']}/results/{benchmark_name}/*.results", + local_sec, + "--zone", secondary["zone"] + ]) + + # Copy primary results to local + local_prim = os.path.join(tmp, f"{primary['name']}.results") + print(f"Copying {primary['name']} results to local: {local_prim}") + subprocess.check_call([ + "gcloud", "compute", "scp", + f"{primary['name']}:{primary['remote_dir']}/results/{benchmark_name}/*.results", + local_prim, + "--zone", primary["zone"] + ]) + + # Push secondary results to primary + print(f"Pushing {secondary['name']} results to {primary['name']}:{remote_tmp}") + subprocess.check_call([ + "gcloud", "compute", "scp", + local_sec, + f"{primary['name']}:{remote_tmp}/{secondary['name']}.results", + "--zone", primary["zone"] + ]) + + # Push primary results to primary + print(f"Pushing {primary['name']} results to {primary['name']}:{remote_tmp}") + subprocess.check_call([ + "gcloud", "compute", "scp", + local_prim, + f"{primary['name']}:{remote_tmp}/{primary['name']}.results", + "--zone", primary["zone"] + ]) + + def _run_benchstat(self, primary: Dict, secondary: Dict, remote_tmp: str) -> None: + """ + Run benchstat on the primary VM. + + Args: + primary: Primary instance configuration + secondary: Secondary instance configuration + remote_tmp: Remote temporary directory + """ + benchstat_cmd = ( + f"benchstat -format csv " + f"{remote_tmp}/{secondary['name']}.results " + f"{remote_tmp}/{primary['name']}.results > {remote_tmp}/benchstat.results" + ) + ls_cmd = ( + f"ls -al {remote_tmp}/{secondary['name']}.results " + f"{remote_tmp}/{primary['name']}.results {remote_tmp}/benchstat.results" + ) + ssh_cmd = [ + "gcloud", "compute", "ssh", primary["name"], + "--zone", primary["zone"], + "--command", + ( + f"{self.config.env_setup} " + f"{benchstat_cmd} && echo \"Results files:\" && {ls_cmd}" + ) + ] + subprocess.check_call(ssh_cmd) \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet/benchstat_report.py b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet/benchstat_report.py new file mode 100644 index 0000000000..f983d79e52 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet/benchstat_report.py @@ -0,0 +1,37 @@ +#!/usr/bin/env python3 +""" +Module for generating reports from benchstat results. +""" +import os +from typing import List, Tuple + +from .visualization import parse_benchstat, plot_metric_group, generate_html + +class BenchstatReport: + """ + Encapsulate benchstat report generation logic. + """ + def generate_report(self, benchstat_file: str, out_dir: str) -> List[str]: + """ + Generate HTML report from benchstat results file. + + Args: + benchstat_file: Path to benchstat results file + out_dir: Output directory for report files + + Returns: + List of generated image paths + """ + if not os.path.exists(benchstat_file): + print(f"Error: Benchstat file {benchstat_file} not found") + return [] + + groups = parse_benchstat(benchstat_file) + images = [] + + for idx, (inst_a, inst_b, metric_name, df) in enumerate(groups): + img_path = plot_metric_group(inst_a, inst_b, metric_name, df, out_dir, idx) + images.append((img_path, metric_name, inst_a, inst_b)) + + generate_html(images, out_dir) + return [img[0] for img in images] \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet/cli.py b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet/cli.py new file mode 100644 index 0000000000..1a7e7c5e33 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet/cli.py @@ -0,0 +1,253 @@ +""" +Command-line interface for rexec_sweet. +""" +import os +import sys +import argparse +import tempfile +import threading +from datetime import datetime +from typing import Dict, List, Optional, Tuple + +from .gcp_utils import get_running_instances, choose_instance, get_instance_zone +from .benchmark_runner import BenchmarkRunner +from .config import Config, COLORS + +def parse_args(): + """Parse command line arguments.""" + config = Config() + parser = argparse.ArgumentParser(description="Run Sweet benchmarks on GCP instances") + parser.add_argument("--benchmark", choices=config.get_benchmark_names(), + help="Benchmark to run") + parser.add_argument("--instance1", help="First GCP instance name") + parser.add_argument("--instance2", help="Second GCP instance name") + parser.add_argument("--report", help="Generate report from benchstat file") + parser.add_argument("--output-dir", help="Output directory for results") + parser.add_argument("--debug", action="store_true", help="Show detailed error messages") + return parser.parse_args() + +def get_remote_path(instance_name: str) -> str: + """Ask user for remote path, defaulting to ~/benchmarks/sweet""" + default_path = "~/benchmarks/sweet" + path = input(f"Enter remote path for {instance_name} [default: {default_path}]: ").strip() + if not path: + return default_path + return path + +def select_benchmark(config: Config) -> str: + """Prompt user to select a benchmark.""" + print("Available benchmarks:") + benchmark_names = config.get_benchmark_names() + for idx, name in enumerate(benchmark_names, 1): + default_marker = " (default)" if name == config.default_benchmark else "" + print(f"{idx}. {name}{default_marker}") + + while True: + choice = input(f"Enter number (1-{len(benchmark_names)}) [default: {config.default_benchmark}]: ").strip() + if not choice: # Default + return config.default_benchmark + if choice.isdigit(): + idx = int(choice) + if 1 <= idx <= len(benchmark_names): + return benchmark_names[idx-1] + print("Invalid selection, try again.") + +def select_instance(prompt_text: str, instances: List[str], instance_name: Optional[str] = None) -> Tuple[str, str, str]: + """ + Select an instance and get its zone and remote path. + + Args: + prompt_text: Text to display when prompting for instance selection + instances: List of available instances + instance_name: Optional pre-selected instance name + + Returns: + Tuple of (instance_name, zone, remote_path) + """ + if not instance_name: + print(f"\n{prompt_text}:") + if not instances: + print("Error: No running instances found.") + sys.exit(1) + instance_name = choose_instance(instances) + + # Get remote path and zone + remote_path = get_remote_path(instance_name) + zone = get_instance_zone(instance_name) + + return instance_name, zone, remote_path + +def display_instances_and_prompt(instances: List[str]) -> bool: + """Display instance list with first two highlighted and ask if user wants to use them.""" + if len(instances) < 2: + print("Error: Need at least two running instances.") + sys.exit(1) + + print("\nAvailable instances:") + blue = COLORS['blue'] + green = COLORS['green'] + reset = COLORS['reset'] + + for i, name in enumerate(instances): + if i == 0: + print(f"{blue}1. {name} (will be used as first instance){reset}") + elif i == 1: + print(f"{green}2. {name} (will be used as second instance){reset}") + else: + print(f"{i+1}. {name}") + + choice = input(f"\nDo you want to run the first two instances found with default install directories? [Y/n]: ").strip().lower() + return choice == "" or choice == "y" + +def get_default_instances(instances: List[str]) -> List[Tuple[str, str, str]]: + """Get the first two instances with default directories.""" + if len(instances) < 2: + print("Error: Need at least two running instances.") + sys.exit(1) + + default_path = "~/benchmarks/sweet" + result = [] + + # Cache zones to avoid multiple API calls + zones = {} + + for i in range(2): + instance_name = instances[i] + if instance_name not in zones: + zones[instance_name] = get_instance_zone(instance_name) + result.append((instance_name, zones[instance_name], default_path)) + + print(f"\nUsing instances: {instances[0]} and {instances[1]} with default path: {default_path}") + return result + +def run_benchmarks_parallel(runner: BenchmarkRunner, systems: List[Dict]) -> List[str]: + """ + Run benchmarks in parallel on multiple systems. + + Args: + runner: BenchmarkRunner instance + systems: List of system configurations + + Returns: + List of systems that failed + """ + print("\nRunning benchmarks on the selected instances...") + errors = [] + codes = {} + threads = [] + + for syscfg in systems: + def run_and_store(syscfg=syscfg): + code = runner.run_benchmark( + syscfg["bench"], + syscfg["name"], + syscfg["zone"], + syscfg["remote_dir"] + ) + codes[syscfg["name"]] = 0 if code else 1 + + t = threading.Thread(target=run_and_store, daemon=True) + t.start() + threads.append(t) + + # Wait for benchmarks to finish + for t in threads: + t.join() + + # Collect failures + for name, code in codes.items(): + if code != 0: + errors.append(name) + + return errors + +def main(): + """Main entry point for the CLI.""" + args = parse_args() + runner = BenchmarkRunner() + config = Config() + + try: + # Just generate a report from an existing benchstat file + if args.report: + out_dir = args.output_dir or tempfile.mkdtemp() + runner.run_benchstat_report(args.report, out_dir) + return 0 + + # Choose benchmark to run + benchmark_name = args.benchmark or select_benchmark(config) + + # Setup systems for benchmarking + systems = [] + + # Get instances only once + instances = get_running_instances() if not args.instance1 and not args.instance2 else [] + + # Check if user wants to use default instances + if instances and display_instances_and_prompt(instances): + # Get the first two instances with default directories + instance_configs = get_default_instances(instances) + + for instance_name, zone, remote_path in instance_configs: + systems.append({ + "name": instance_name, + "zone": zone, + "bench": benchmark_name, + "remote_dir": remote_path + }) + else: + # Get first instance + instance1_name, zone1, remote_path1 = select_instance( + "Select FIRST instance", instances, args.instance1) + + systems.append({ + "name": instance1_name, + "zone": zone1, + "bench": benchmark_name, + "remote_dir": remote_path1 + }) + + # Get second instance + instance2_name, zone2, remote_path2 = select_instance( + "Select SECOND instance", instances, args.instance2) + + systems.append({ + "name": instance2_name, + "zone": zone2, + "bench": benchmark_name, + "remote_dir": remote_path2 + }) + + # Create output directory + timestamp = datetime.now().strftime('%Y%m%dT%H%M%S') + local_dir = config.get_results_dir(systems[0]["name"], systems[1]["name"], benchmark_name, timestamp) + print(f"Output directory: {local_dir}") + + # Run benchmarks in parallel + errors = run_benchmarks_parallel(runner, systems) + + if errors: + print("\n⚠️ Some benchmarks failed. Aborting benchstat step.") + return 1 + + # Identify primary and secondary systems + primary = systems[1] + secondary = systems[0] + + # Collect results and run benchstat + benchstat_file = runner.collect_results(primary, secondary, benchmark_name, local_dir) + + # Generate report + if benchstat_file: + runner.run_benchstat_report(benchstat_file, local_dir) + + return 0 + except Exception as e: + print(f"Error: {str(e)}") + if args.debug: + import traceback + traceback.print_exc() + return 1 + +if __name__ == "__main__": + sys.exit(main()) \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet/config.py b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet/config.py new file mode 100644 index 0000000000..5e701639f1 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet/config.py @@ -0,0 +1,63 @@ +""" +Configuration management for rexec_sweet. +""" +import os +from typing import Dict, List, Optional + +# ANSI color codes for visual differentiation +COLORS = { + 'blue': '\033[94m', + 'green': '\033[92m', + 'yellow': '\033[93m', + 'red': '\033[91m', + 'reset': '\033[0m' +} + +# Default benchmark configurations +DEFAULT_BENCHMARKS: Dict[str, str] = { + "biogo-igor": 'sweet run -count 10 -run="biogo-igor" config.toml', + "biogo-krishna": 'sweet run -count 10 -run="biogo-krishna" config.toml', + "bleve-index": 'sweet run -count 10 -run="bleve-index" config.toml', + "cockroachdb": 'sweet run -count 10 -run="cockroachdb" config.toml', + "esbuild": 'sweet run -count 10 -run="esbuild" config.toml', + "etcd": 'sweet run -count 10 -run="etcd" config.toml', + "go-build": 'sweet run -count 10 -run="go-build" config.toml', + "gopher-lua": 'sweet run -count 10 -run="gopher-lua" config.toml', + "markdown": 'sweet run -count 10 -run="markdown" config.toml', + "tile38": 'sweet run -count 10 -run="tile38" config.toml', +} + +# Default remote directory for benchmarks +DEFAULT_REMOTE_DIR = "~/benchmarks/sweet" + +# Default environment setup for remote execution +DEFAULT_ENV_SETUP = ( + "export GOPATH=$HOME/go; " + "export GOBIN=$GOPATH/bin; " + "export PATH=$PATH:$GOBIN:/usr/local/go/bin; " +) + +class Config: + """Configuration manager for rexec_sweet.""" + + def __init__(self): + self.benchmarks = DEFAULT_BENCHMARKS.copy() + self.default_benchmark = "markdown" + self.results_dir = os.path.join(os.getcwd(), "results") + self.env_setup = DEFAULT_ENV_SETUP + + def get_benchmark_command(self, benchmark_name: str) -> Optional[str]: + """Get the command for a specific benchmark.""" + return self.benchmarks.get(benchmark_name) + + def get_benchmark_names(self) -> List[str]: + """Get a sorted list of available benchmark names.""" + return sorted(self.benchmarks.keys()) + + def get_results_dir(self, instance1: str, instance2: str, benchmark: str, timestamp: str) -> str: + """Generate the results directory path.""" + os.makedirs(self.results_dir, exist_ok=True) + subdir = f"{instance1}-{instance2}-{benchmark}-{timestamp}" + result_dir = os.path.join(self.results_dir, subdir) + os.makedirs(result_dir, exist_ok=True) + return result_dir \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet/gcp_utils.py b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet/gcp_utils.py new file mode 100644 index 0000000000..1393b13118 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet/gcp_utils.py @@ -0,0 +1,142 @@ +""" +Utilities for interacting with Google Cloud Platform. +""" +import subprocess +import sys +import time +import threading +from typing import List, Optional, Dict + +# Cache for instance zones to avoid redundant API calls +_zone_cache: Dict[str, str] = {} + +def show_spinner(stop_event, message="Fetching instances"): + """Display a simple spinner while a task is running.""" + spinner = ['|', '/', '-', '\\'] + i = 0 + while not stop_event.is_set(): + sys.stdout.write(f"\r{message}... {spinner[i % len(spinner)]}") + sys.stdout.flush() + i += 1 + time.sleep(0.1) + sys.stdout.write("\r \r") # Clear the spinner line + +def get_running_instances() -> List[str]: + """ + Return a list of names of GCP VMs that are currently RUNNING. + + Returns: + List of instance names + """ + stop_event = threading.Event() + spinner_thread = threading.Thread(target=show_spinner, args=(stop_event, "Fetching instances")) + spinner_thread.daemon = True + spinner_thread.start() + + try: + output = subprocess.check_output( + ["gcloud", "compute", "instances", "list", + "--filter=status=RUNNING", + "--format=value(name)"], + universal_newlines=True + ) + instances = [line.strip() for line in output.splitlines() if line.strip()] + return instances + except subprocess.CalledProcessError as e: + print(f"Error fetching running instances: {e}", file=sys.stderr) + return [] + finally: + stop_event.set() + spinner_thread.join(timeout=0.5) + +def choose_instance(instances: List[str]) -> str: + """ + Prompt user to choose an instance from the list. + + Args: + instances: List of instance names + + Returns: + Selected instance name + """ + if not instances: + print("No running instances found.", file=sys.stderr) + sys.exit(1) + + for idx, name in enumerate(instances, 1): + print(f"{idx}. {name}") + + while True: + choice = input(f"Enter number (1-{len(instances)}): ").strip() + if choice.isdigit(): + idx = int(choice) + if 1 <= idx <= len(instances): + return instances[idx - 1] + print("Invalid selection, try again.") + +def get_instance_zone(name: str) -> str: + """ + Return the zone of the given GCP VM instance. + + Args: + name: Instance name + + Returns: + Zone name + """ + # Check cache first + if name in _zone_cache: + return _zone_cache[name] + + stop_event = threading.Event() + spinner_thread = threading.Thread(target=show_spinner, args=(stop_event, f"Fetching zone for {name}")) + spinner_thread.daemon = True + spinner_thread.start() + + try: + output = subprocess.check_output( + ["gcloud", "compute", "instances", "list", + "--filter", f"name={name}", + "--format=value(zone)"], + universal_newlines=True + ) + zone = output.strip() + if not zone: + raise ValueError(f"No zone found for instance {name}") + + # Cache the result + _zone_cache[name] = zone + return zone + except (subprocess.CalledProcessError, ValueError) as e: + print(f"Error fetching zone for instance {name}: {e}", file=sys.stderr) + sys.exit(1) + finally: + stop_event.set() + spinner_thread.join(timeout=0.5) + +def scp_results(name: str, zone: str, remote_pattern: str, local_out: str) -> bool: + """ + Copy back all files in remote_dir that match the pattern. + + Args: + name: Instance name + zone: GCP zone + remote_pattern: Remote file pattern + local_out: Local output directory + + Returns: + True if successful, False otherwise + """ + print(f"Copying {remote_pattern} from {name} to {local_out}") + scp_cmd = [ + "gcloud", "compute", "scp", + f"{name}:{remote_pattern}", + local_out, + "--zone", zone + ] + try: + subprocess.check_call(scp_cmd) + return True + except subprocess.CalledProcessError as e: + print(f"Error copying files: {e}", file=sys.stderr) + return False \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet/visualization.py b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet/visualization.py new file mode 100644 index 0000000000..59c760ed6f --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/rexec_sweet/visualization.py @@ -0,0 +1,134 @@ +import os +import textwrap +import plotly.graph_objs as go +from io import StringIO +import pandas as pd + +def wrap_label(lbl, width=20): + parts = lbl.split('/') + wrapped = [] + for part in parts: + wrapped.extend(textwrap.wrap(part, width=width) or ['']) + return '
'.join(wrapped) + +def make_bar_chart(x_labels, y1, y2, name1, name2, title, xaxis_title, yaxis_title, out_path): + fig = go.Figure() + fig.add_trace(go.Bar( + x=x_labels, + y=y1, + name=name1, + marker_color='blue', + hovertemplate="%{x}
%{name}: %{y:.4g}" + )) + fig.add_trace(go.Bar( + x=x_labels, + y=y2, + name=name2, + marker_color='orange', + hovertemplate="%{x}
%{name}: %{y:.4g}" + )) + fig.update_layout( + title=title, + xaxis_title=xaxis_title, + yaxis_title=yaxis_title, + barmode='group', + xaxis_tickfont=dict(size=12), + xaxis_tickangle=-45, + xaxis_tickmode='array', + xaxis_tickvals=list(range(len(x_labels))), + xaxis_ticktext=x_labels, + xaxis_automargin=True, + template='plotly_white', + margin=dict( + t=50, + b=250 if xaxis_title == 'Benchmark' else 200, + l=100, + r=100 + ) + ) + fig.write_html(out_path) + return out_path + +def parse_benchstat(file_path): + with open(file_path, 'r') as f: + content = f.read() + sections = [sec.strip() for sec in content.split('\n\n') if sec.strip()] + groups = [] + for sec in sections: + lines = sec.splitlines() + header_line = lines[0] + parts = header_line.split(',') + if len(parts) < 4: + raise ValueError(f"Unexpected header format: {header_line}") + raw_a = parts[1] + raw_b = parts[3] if len(parts) > 3 else 'Unknown' + # Strip any directory prefix so we only keep the basename + base_a = os.path.basename(raw_a) + base_b = os.path.basename(raw_b) + # Now split on '-' to get just the instance name (e.g. 'c4' from 'c4-96.results') + inst_a = base_a.split('-')[0] if '-' in base_a else base_a.split('.')[0] + inst_b = base_b.split('-')[0] if '-' in base_b else base_b.split('.')[0] + metric_name = lines[1].split(',')[1] + csv_text = '\n'.join(lines[1:]) + df = pd.read_csv(StringIO(csv_text), engine='python') + df.rename(columns={df.columns[0]: 'Benchmark'}, inplace=True) + df['Benchmark'] = df['Benchmark'].str.replace(r'-\d+$', '', regex=True) + groups.append((inst_a, inst_b, metric_name, df)) + return groups + +def plot_metric_group(inst_a, inst_b, metric_name, df, out_dir, idx): + benchmarks = df['Benchmark'].unique() + cols = df.columns.tolist() + baseline_col = cols[1] + compare_col = cols[3] + grouped_data = df.groupby('Benchmark')[[baseline_col, compare_col]].mean() + x = grouped_data.index.tolist() + values_a = grouped_data[baseline_col].values + values_b = grouped_data[compare_col].values + wrapped_names = [wrap_label(lbl) for lbl in x] + custom_labels = [] + for i, name in enumerate(wrapped_names): + a = values_a[i] + b = values_b[i] + if a == 0: + comparison = "undefined" + else: + pct_diff = ((b - a) / a) * 100 + dir_sym = ">" if pct_diff > 0 else "<" + comparison = f"{abs(pct_diff):.1f}% {dir_sym}" + custom_labels.append( + f"{name}
" + f"{inst_b} is {comparison} {inst_a}" + ) + html_filename = f"{metric_name}_{idx}.html".replace('/', '_').replace(' ', '_') + html_path = os.path.join(out_dir, html_filename) + return make_bar_chart( + custom_labels, + values_a, + values_b, + inst_a, + inst_b, + f"{metric_name} comparison", + "Benchmark", + metric_name, + html_path + ) + +def generate_html(images, out_dir): + html_path = os.path.join(out_dir, 'report.html') + with open(html_path, 'w') as html: + html.write('Benchmark Report\n') + html.write('

Benchmark Comparison Report

\n') + for img_path, metric_name, inst_a, inst_b in images: + html.write(f'

{metric_name} comparison between hosts {inst_a} and {inst_b}

\n') + html.write(f'\n') + html.write( + f'

This visualization compares {metric_name} for benchmark cases ' + f'on instances {inst_a} and {inst_b}. ' + 'Each bar represents the metric for a specific benchmark test.

\n' + ) + html.write('') + # Only print the path relative to CWD (no /tmp/tmp... prefix) + rel_html = os.path.relpath(html_path, start=os.getcwd()) + print(f'Generated report at {rel_html}') + return html_path \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/run.py b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/run.py new file mode 100644 index 0000000000..5a63ba8571 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/run.py @@ -0,0 +1,16 @@ +#!/usr/bin/env python3 + +import sys +import os + +# Add the current directory to Python path +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +try: + from rexec_sweet.cli import main + sys.exit(main()) +except Exception as e: + print(f"Error: {e}") + import traceback + traceback.print_exc() + sys.exit(1) \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/setup.py b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/setup.py new file mode 100644 index 0000000000..425dbde7fd --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/setup.py @@ -0,0 +1,24 @@ +from setuptools import setup, find_packages + +setup( + name="rexec_sweet", + version="0.1.0", + packages=find_packages(), + install_requires=[ + "numpy>=2.0.0", + "pandas>=2.3.0", + "plotly>=5.13.0", + ], + entry_points={ + "console_scripts": [ + "rexec-sweet=rexec_sweet.cli:main", + ], + }, + python_requires=">=3.8", + extras_require={ + "dev": [ + "pytest>=7.0.0", + "pytest-cov>=4.0.0", + ], + }, +) \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/__init__.py b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/__init__.py new file mode 100644 index 0000000000..dc6420ffb0 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/__init__.py @@ -0,0 +1,3 @@ +""" +Test package for rexec_sweet. +""" \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/conftest.py b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/conftest.py new file mode 100644 index 0000000000..db06d19a45 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/conftest.py @@ -0,0 +1,31 @@ +""" +Pytest configuration file. +""" +import os +import pytest +import tempfile + +@pytest.fixture +def sample_benchstat_file(): + """Create a sample benchstat file for testing.""" + with tempfile.NamedTemporaryFile(mode='w', delete=False) as f: + f.write("name,c4-96.results,±,c4-64.results,±\n") + f.write("BenchmarkTest1,100,10%,120,5%\n") + f.write("BenchmarkTest2,200,5%,180,10%\n") + f.write("\n") + f.write("name,c4-96.results,±,c4-64.results,±\n") + f.write("BenchmarkTest1,5,10%,4,5%\n") + f.write("BenchmarkTest2,10,5%,8,10%\n") + path = f.name + + yield path + + # Cleanup + if os.path.exists(path): + os.unlink(path) + +@pytest.fixture +def temp_output_dir(): + """Create a temporary directory for test outputs.""" + with tempfile.TemporaryDirectory() as tmpdir: + yield tmpdir \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/mock_data/benchstat.csv b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/mock_data/benchstat.csv new file mode 100644 index 0000000000..d2effce17e --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/mock_data/benchstat.csv @@ -0,0 +1,7 @@ +name,c4-96.results,±,c4-64.results,± +BenchmarkTest1,100,10%,120,5% +BenchmarkTest2,200,5%,180,10% + +name,c4-96.results,±,c4-64.results,± +BenchmarkTest1,5,10%,4,5% +BenchmarkTest2,10,5%,8,10% \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/test_benchmark_runner.py b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/test_benchmark_runner.py new file mode 100644 index 0000000000..0c707c0dda --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/test_benchmark_runner.py @@ -0,0 +1,103 @@ +""" +Tests for the benchmark_runner module. +""" +import os +import unittest +from unittest import mock +import subprocess +import tempfile + +from rexec_sweet.benchmark_runner import BenchmarkRunner + +class TestBenchmarkRunner(unittest.TestCase): + """Test cases for the BenchmarkRunner class.""" + + def setUp(self): + """Set up test fixtures.""" + self.runner = BenchmarkRunner() + + def test_init(self): + """Test initialization.""" + self.assertIsNotNone(self.runner.config) + + @mock.patch('subprocess.check_call') + def test_run_benchmark_success(self, mock_check_call): + """Test successful benchmark run.""" + # Mock the run_remote method to return success + with mock.patch.object(self.runner, 'run_remote', return_value=0) as mock_run: + result = self.runner.run_benchmark("markdown", "instance1", "us-central1-a", "~/benchmarks") + self.assertTrue(result) + mock_run.assert_called_once() + + @mock.patch('subprocess.check_call') + def test_run_benchmark_failure_mkdir(self, mock_check_call): + """Test benchmark run with mkdir failure.""" + mock_check_call.side_effect = subprocess.CalledProcessError(1, "cmd") + result = self.runner.run_benchmark("markdown", "instance1", "us-central1-a", "~/benchmarks") + self.assertFalse(result) + + @mock.patch('subprocess.check_call') + def test_run_benchmark_failure_run(self, mock_check_call): + """Test benchmark run with execution failure.""" + # Mock the run_remote method to return failure + with mock.patch.object(self.runner, 'run_remote', return_value=1) as mock_run: + result = self.runner.run_benchmark("markdown", "instance1", "us-central1-a", "~/benchmarks") + self.assertFalse(result) + + @mock.patch('subprocess.Popen') + def test_run_remote(self, mock_popen): + """Test remote command execution.""" + # Setup mock process + mock_process = mock.MagicMock() + mock_process.returncode = 0 + mock_process.stdout.readline.side_effect = [b"output line 1", b"output line 2", b""] + mock_popen.return_value = mock_process + + # Run the method + result = self.runner.run_remote("instance1", "us-central1-a", "~/benchmarks", "markdown", 0) + + # Verify results + self.assertEqual(result, 0) + mock_popen.assert_called_once() + + @mock.patch('webbrowser.open') + @mock.patch('rexec_sweet.benchstat_report.BenchstatReport.generate_report') + def test_run_benchstat_report(self, mock_generate, mock_browser): + """Test benchstat report generation.""" + with tempfile.TemporaryDirectory() as tmpdir: + # Create a dummy benchstat file + benchstat_file = os.path.join(tmpdir, "benchstat.results") + with open(benchstat_file, "w") as f: + f.write("dummy data") + + # Run the method + self.runner.run_benchstat_report(benchstat_file, tmpdir) + + # Verify results + mock_generate.assert_called_once_with(benchstat_file, tmpdir) + mock_browser.assert_called_once() + + @mock.patch('subprocess.check_output') + @mock.patch('subprocess.check_call') + def test_collect_results_success(self, mock_check_call, mock_check_output): + """Test successful results collection.""" + # Setup mocks + mock_check_output.return_value = "/tmp/remote_dir\n" + + # Setup test data + primary = {"name": "instance1", "zone": "us-central1-a", "remote_dir": "~/benchmarks"} + secondary = {"name": "instance2", "zone": "us-central1-b", "remote_dir": "~/benchmarks"} + + # Mock the transfer and benchstat methods + with mock.patch.object(self.runner, '_transfer_result_files') as mock_transfer: + with mock.patch.object(self.runner, '_run_benchstat') as mock_benchstat: + with mock.patch('os.path.exists', return_value=True): + result = self.runner.collect_results(primary, secondary, "markdown", "/tmp/local_dir") + + # Verify results + self.assertIsNotNone(result) + mock_transfer.assert_called_once() + mock_benchstat.assert_called_once() + +if __name__ == '__main__': + unittest.main() \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/test_benchstat_report.py b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/test_benchstat_report.py new file mode 100644 index 0000000000..5003c70fa1 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/test_benchstat_report.py @@ -0,0 +1,62 @@ +""" +Tests for the benchstat_report module. +""" +import os +import unittest +from unittest import mock +import tempfile + +from rexec_sweet.benchstat_report import BenchstatReport + +class TestBenchstatReport(unittest.TestCase): + """Test cases for the BenchstatReport class.""" + + def setUp(self): + """Set up test fixtures.""" + self.reporter = BenchstatReport() + + def test_generate_report_success(self): + """Test successful report generation.""" + with tempfile.TemporaryDirectory() as output_dir: + # Create a real test file + test_file = os.path.join(output_dir, "benchstat.csv") + with open(test_file, "w") as f: + f.write("name,c4-96.results,±,c4-64.results,±\n") + f.write("BenchmarkTest1,100,10%,120,5%\n") + f.write("BenchmarkTest2,200,5%,180,10%\n") + + # Mock the functions that would process the file + with mock.patch('rexec_sweet.benchstat_report.parse_benchstat') as mock_parse, \ + mock.patch('rexec_sweet.benchstat_report.plot_metric_group') as mock_plot, \ + mock.patch('rexec_sweet.benchstat_report.generate_html') as mock_generate_html: + + # Setup mock return values + mock_parse.return_value = [ + ("c4-96", "c4-64", "time/op", mock.MagicMock()), + ("c4-96", "c4-64", "alloc/op", mock.MagicMock()) + ] + mock_plot.side_effect = [ + os.path.join(output_dir, "chart1.html"), + os.path.join(output_dir, "chart2.html") + ] + + # Run the method with the real file + result = self.reporter.generate_report(test_file, output_dir) + + # Verify the length of the result matches the number of charts + self.assertEqual(len(result), 2) + + def test_generate_report_file_not_found(self): + """Test report generation with missing file.""" + with tempfile.TemporaryDirectory() as tmpdir: + # Use a non-existent file + benchstat_file = os.path.join(tmpdir, "nonexistent.results") + + # Run the method + result = self.reporter.generate_report(benchstat_file, tmpdir) + + # Verify results + self.assertEqual(result, []) # Empty result + +if __name__ == '__main__': + unittest.main() \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/test_cli.py b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/test_cli.py new file mode 100644 index 0000000000..7a1875cc7e --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/test_cli.py @@ -0,0 +1,190 @@ +""" +Tests for the CLI module. +""" +import unittest +from unittest import mock +import argparse +import sys +import tempfile + +from rexec_sweet.cli import parse_args, get_remote_path, select_benchmark, select_instance, run_benchmarks_parallel, main + +class TestCLI(unittest.TestCase): + """Test cases for CLI functions.""" + + @mock.patch('argparse.ArgumentParser.parse_args') + def test_parse_args(self, mock_parse_args): + """Test argument parsing.""" + # Setup mock return value + args = argparse.Namespace( + benchmark="markdown", + instance1="instance1", + instance2="instance2", + report=None, + output_dir=None, + debug=False + ) + mock_parse_args.return_value = args + + # Call the function + result = parse_args() + + # Verify results + self.assertEqual(result.benchmark, "markdown") + self.assertEqual(result.instance1, "instance1") + self.assertEqual(result.instance2, "instance2") + + @mock.patch('builtins.input') + def test_get_remote_path_default(self, mock_input): + """Test getting remote path with default value.""" + mock_input.return_value = "" + path = get_remote_path("instance1") + self.assertEqual(path, "~/benchmarks/sweet") + + @mock.patch('builtins.input') + def test_get_remote_path_custom(self, mock_input): + """Test getting remote path with custom value.""" + mock_input.return_value = "/custom/path" + path = get_remote_path("instance1") + self.assertEqual(path, "/custom/path") + + @mock.patch('builtins.input') + def test_select_benchmark_default(self, mock_input): + """Test benchmark selection with default.""" + from rexec_sweet.config import Config + config = Config() + mock_input.return_value = "" + + benchmark = select_benchmark(config) + self.assertEqual(benchmark, "markdown") + + @mock.patch('builtins.input') + def test_select_benchmark_by_number(self, mock_input): + """Test benchmark selection by number.""" + from rexec_sweet.config import Config + config = Config() + mock_input.return_value = "1" # First benchmark in sorted list + + benchmark = select_benchmark(config) + self.assertEqual(benchmark, config.get_benchmark_names()[0]) + + @mock.patch('rexec_sweet.cli.get_running_instances') + @mock.patch('rexec_sweet.cli.choose_instance') + @mock.patch('rexec_sweet.cli.get_instance_zone') + @mock.patch('builtins.input') + def test_select_instance(self, mock_input, mock_zone, mock_choose, mock_instances): + """Test instance selection.""" + # Setup mocks + mock_instances.return_value = ["instance1", "instance2"] + mock_choose.return_value = "instance1" + mock_zone.return_value = "us-central1-a" + mock_input.return_value = "" # Default path + + # Call with no pre-selected instance + name, zone, path = select_instance("Select instance") + + # Verify results + self.assertEqual(name, "instance1") + self.assertEqual(zone, "us-central1-a") + self.assertEqual(path, "~/benchmarks/sweet") + + # Call with pre-selected instance + name, zone, path = select_instance("Select instance", "instance2") + + # Verify results + self.assertEqual(name, "instance2") + self.assertEqual(zone, "us-central1-a") + mock_choose.assert_called_once() # Should only be called in the first test + + @mock.patch('threading.Thread') + def test_run_benchmarks_parallel(self, mock_thread): + """Test parallel benchmark execution.""" + # Setup mock + mock_thread_instance = mock.MagicMock() + mock_thread.return_value = mock_thread_instance + + # Setup test data + runner = mock.MagicMock() + runner.run_benchmark.return_value = True # Success + + systems = [ + {"name": "instance1", "zone": "us-central1-a", "bench": "markdown", "remote_dir": "~/benchmarks"}, + {"name": "instance2", "zone": "us-central1-b", "bench": "markdown", "remote_dir": "~/benchmarks"} + ] + + # Run the function + errors = run_benchmarks_parallel(runner, systems) + + # Verify results + self.assertEqual(len(errors), 0) # No errors + self.assertEqual(mock_thread.call_count, 2) # Two threads created + self.assertEqual(mock_thread_instance.start.call_count, 2) # Both threads started + self.assertEqual(mock_thread_instance.join.call_count, 2) # Both threads joined + + @mock.patch('rexec_sweet.cli.parse_args') + @mock.patch('rexec_sweet.cli.select_benchmark') + @mock.patch('rexec_sweet.cli.select_instance') + @mock.patch('rexec_sweet.cli.run_benchmarks_parallel') + def test_main_success(self, mock_run, mock_select_instance, mock_select_benchmark, mock_parse_args): + """Test successful main execution.""" + # Setup mocks + args = argparse.Namespace( + benchmark=None, + instance1=None, + instance2=None, + report=None, + output_dir=None, + debug=False + ) + mock_parse_args.return_value = args + mock_select_benchmark.return_value = "markdown" + mock_select_instance.side_effect = [ + ("instance1", "us-central1-a", "~/benchmarks"), + ("instance2", "us-central1-b", "~/benchmarks") + ] + mock_run.return_value = [] # No errors + + # Mock the BenchmarkRunner + with mock.patch('rexec_sweet.cli.BenchmarkRunner') as mock_runner_class: + mock_runner = mock.MagicMock() + mock_runner_class.return_value = mock_runner + mock_runner.collect_results.return_value = "/tmp/benchstat.results" + + # Run main + result = main() + + # Verify results + self.assertEqual(result, 0) # Success + mock_runner.run_benchstat_report.assert_called_once() + + @mock.patch('rexec_sweet.cli.parse_args') + def test_main_report_only(self, mock_parse_args): + """Test main with report-only mode.""" + # Setup mocks + args = argparse.Namespace( + benchmark=None, + instance1=None, + instance2=None, + report="/tmp/benchstat.results", + output_dir=None, + debug=False + ) + mock_parse_args.return_value = args + + # Mock the BenchmarkRunner + with mock.patch('rexec_sweet.cli.BenchmarkRunner') as mock_runner_class: + mock_runner = mock.MagicMock() + mock_runner_class.return_value = mock_runner + + # Run main + result = main() + + # Verify results + self.assertEqual(result, 0) # Success + mock_runner.run_benchstat_report.assert_called_once() + # Should not call other methods + mock_runner.run_benchmark.assert_not_called() + mock_runner.collect_results.assert_not_called() + +if __name__ == '__main__': + unittest.main() \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/test_config.py b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/test_config.py new file mode 100644 index 0000000000..99c84fa158 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/test_config.py @@ -0,0 +1,51 @@ +""" +Tests for the config module. +""" +import os +import unittest +from unittest import mock + +from rexec_sweet.config import Config + +class TestConfig(unittest.TestCase): + """Test cases for the Config class.""" + + def test_init_default_values(self): + """Test that Config initializes with default values.""" + config = Config() + self.assertEqual(config.default_benchmark, "markdown") + self.assertTrue("markdown" in config.benchmarks) + self.assertTrue("go-build" in config.benchmarks) + self.assertEqual(len(config.benchmarks), 11) # Check all benchmarks are loaded + + def test_get_benchmark_command(self): + """Test retrieving benchmark commands.""" + config = Config() + # Test valid benchmark + cmd = config.get_benchmark_command("markdown") + self.assertEqual(cmd, 'sweet run -count 10 -run="markdown" config.toml') + + # Test invalid benchmark + cmd = config.get_benchmark_command("nonexistent") + self.assertIsNone(cmd) + + def test_get_benchmark_names(self): + """Test getting sorted benchmark names.""" + config = Config() + names = config.get_benchmark_names() + self.assertEqual(len(names), 11) + self.assertEqual(names[0], "biogo-igor") # Should be alphabetically sorted + self.assertEqual(names[-1], "tile38") + + def test_get_results_dir(self): + """Test results directory creation.""" + config = Config() + with mock.patch('os.makedirs') as mock_makedirs: + result_dir = config.get_results_dir("inst1", "inst2", "bench", "20250101") + # Check directory structure + self.assertTrue(result_dir.endswith("inst1-inst2-bench-20250101")) + # Check that makedirs was called twice (once for results dir, once for subdir) + self.assertEqual(mock_makedirs.call_count, 2) + +if __name__ == '__main__': + unittest.main() \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/test_gcp_utils.py b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/test_gcp_utils.py new file mode 100644 index 0000000000..bb629d28ee --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/test_gcp_utils.py @@ -0,0 +1,77 @@ +""" +Tests for the gcp_utils module. +""" +import unittest +from unittest import mock +import sys +import subprocess +from io import StringIO + +from rexec_sweet.gcp_utils import get_running_instances, choose_instance, get_instance_zone, scp_results + +class TestGcpUtils(unittest.TestCase): + """Test cases for GCP utility functions.""" + + @mock.patch('subprocess.check_output') + def test_get_running_instances_success(self, mock_check_output): + """Test getting running instances when successful.""" + mock_check_output.return_value = "instance1\ninstance2\n" + instances = get_running_instances() + self.assertEqual(instances, ["instance1", "instance2"]) + mock_check_output.assert_called_once() + + @mock.patch('subprocess.check_output') + def test_get_running_instances_error(self, mock_check_output): + """Test getting running instances when command fails.""" + mock_check_output.side_effect = subprocess.CalledProcessError(1, "cmd") + instances = get_running_instances() + self.assertEqual(instances, []) + + @mock.patch('builtins.input') + def test_choose_instance(self, mock_input): + """Test instance selection.""" + instances = ["instance1", "instance2", "instance3"] + mock_input.return_value = "2" + selected = choose_instance(instances) + self.assertEqual(selected, "instance2") + + @mock.patch('builtins.input') + def test_choose_instance_invalid_then_valid(self, mock_input): + """Test instance selection with initial invalid input.""" + instances = ["instance1", "instance2"] + mock_input.side_effect = ["3", "abc", "1"] + selected = choose_instance(instances) + self.assertEqual(selected, "instance1") + self.assertEqual(mock_input.call_count, 3) + + @mock.patch('subprocess.check_output') + def test_get_instance_zone(self, mock_check_output): + """Test getting instance zone.""" + mock_check_output.return_value = "us-central1-a\n" + zone = get_instance_zone("instance1") + self.assertEqual(zone, "us-central1-a") + + @mock.patch('subprocess.check_output') + @mock.patch('sys.exit') + def test_get_instance_zone_error(self, mock_exit, mock_check_output): + """Test error handling when getting instance zone.""" + mock_check_output.side_effect = subprocess.CalledProcessError(1, "cmd") + get_instance_zone("instance1") + mock_exit.assert_called_once_with(1) + + @mock.patch('subprocess.check_call') + def test_scp_results_success(self, mock_check_call): + """Test successful SCP operation.""" + result = scp_results("instance1", "us-central1-a", "remote/path/*.results", "local/path") + self.assertTrue(result) + mock_check_call.assert_called_once() + + @mock.patch('subprocess.check_call') + def test_scp_results_failure(self, mock_check_call): + """Test failed SCP operation.""" + mock_check_call.side_effect = subprocess.CalledProcessError(1, "cmd") + result = scp_results("instance1", "us-central1-a", "remote/path/*.results", "local/path") + self.assertFalse(result) + +if __name__ == '__main__': + unittest.main() \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/test_visualization.py b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/test_visualization.py new file mode 100644 index 0000000000..bc170549a9 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/refactor1/tests/test_visualization.py @@ -0,0 +1,123 @@ +""" +Tests for the visualization module. +""" +import os +import unittest +import tempfile +import pandas as pd +from unittest import mock +from io import StringIO + +from rexec_sweet.visualization import ( + wrap_label, make_bar_chart, parse_benchstat, + plot_metric_group, generate_html +) + +class TestVisualization(unittest.TestCase): + """Test cases for visualization functions.""" + + def test_wrap_label(self): + """Test label wrapping function.""" + # Test simple label + result = wrap_label("simple") + self.assertEqual(result, "simple") + + # Test label with slashes + result = wrap_label("category/subcategory/name") + self.assertEqual(result, "category
subcategory
name") + + # Test long label that needs wrapping + result = wrap_label("this_is_a_very_long_label_that_needs_wrapping", width=10) + self.assertTrue("
" in result) + self.assertTrue(len(result.split("
")[0]) <= 10) + + @mock.patch('plotly.graph_objs.Figure.write_html') + def test_make_bar_chart(self, mock_write_html): + """Test bar chart generation.""" + output_path = "/tmp/chart.html" + + # Create a simple chart + result = make_bar_chart( + x_labels=["Test1", "Test2"], + y1=[1.0, 2.0], + y2=[1.5, 2.5], + name1="System A", + name2="System B", + title="Test Chart", + xaxis_title="Tests", + yaxis_title="Values", + out_path=output_path + ) + + # Verify mock was called + mock_write_html.assert_called_once() + self.assertEqual(result, output_path) + + def test_parse_benchstat(self): + """Test parsing benchstat CSV output.""" + # Use the mock data file + mock_file = os.path.join(os.path.dirname(__file__), "mock_data", "benchstat.csv") + + # Parse the file + groups = parse_benchstat(mock_file) + + # Verify results + self.assertEqual(len(groups), 2) # Two groups + inst_a, inst_b, metric_name, df = groups[0] + self.assertEqual(inst_a, "c4") + self.assertEqual(inst_b, "c4") + + @mock.patch('rexec_sweet.visualization.make_bar_chart') + def test_plot_metric_group(self, mock_make_bar_chart): + """Test plotting a metric group.""" + # Create test data with the correct column name + df = pd.DataFrame({ + 'name': ['Test1', 'Test2'], # This should be 'name' not 'Benchmark' + 'c4-96.results': [100, 200], + '±': ['10%', '5%'], + 'c4-64.results': [120, 180], + '±.1': ['5%', '10%'] + }) + + # Rename 'name' to 'Benchmark' as the function expects + df = df.rename(columns={'name': 'Benchmark'}) + + mock_make_bar_chart.return_value = "/tmp/chart.html" + + # Plot the data + result = plot_metric_group("c4-96", "c4-64", "time/op", df, "/tmp", 0) + + # Verify results + self.assertEqual(result, "/tmp/chart.html") + mock_make_bar_chart.assert_called_once() + + def test_generate_html(self): + """Test HTML report generation.""" + with tempfile.TemporaryDirectory() as tmpdir: + # Create test data + images = [ + (os.path.join(tmpdir, "chart1.html"), "time/op", "c4-96", "c4-64"), + (os.path.join(tmpdir, "chart2.html"), "alloc/op", "c4-96", "c4-64") + ] + + # Create dummy image files + for img_path, _, _, _ in images: + with open(img_path, "w") as f: + f.write("") + + # Generate the report + result = generate_html(images, tmpdir) + + # Verify results + self.assertTrue(os.path.exists(result)) + self.assertEqual(result, os.path.join(tmpdir, "report.html")) + + # Check content + with open(result, "r") as f: + content = f.read() + self.assertTrue("

Benchmark Comparison Report

" in content) + self.assertTrue("time/op" in content) + self.assertTrue("alloc/op" in content) + +if __name__ == '__main__': + unittest.main() \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/requirements.txt b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/requirements.txt new file mode 100644 index 0000000000..62f9d150da --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/requirements.txt @@ -0,0 +1,3 @@ +pandas==2.1.0 +plotly==5.13.1 +typer==0.9.0 \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/rexec_sweet_install.md b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/rexec_sweet_install.md new file mode 100644 index 0000000000..99f04da546 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/rexec_sweet_install.md @@ -0,0 +1,64 @@ +--- +title: Installing the Automated Benchmark and Benchstat Runner +weight: 53 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Section Overview + +In the last section, you learned how to run benchmarks and benchstat manually. Now you'll learn how to run them automatically, with enhanced visualization of the results. + +## Introducing rexec_sweet.py + +The `rexec_sweet.py` script is a powerful automation tool that simplifies the benchmarking workflow. This tool connects to your GCP instances, runs the benchmarks, collects the results, and generates comprehensive reports—all in one seamless operation. It provides several key benefits: + +- **Automation**: Runs benchmarks on multiple VMs without manual SSH connections +- **Consistency**: Ensures benchmarks are executed with identical parameters +- **Visualization**: Generates HTML reports with interactive charts for easier analysis + +The only dependency you are responsible for satisfying before the script runs is completion of the "Installing Go and Sweet" sections of this learning path. Additional dependencies are dynamically loaded at install time by the install script. + +## Setting up rexec_sweet + +1. **Create a working directory:** On your local machine, open a terminal, then create and cd into a directory to store the `rexec_sweet.py` script and related files: + + ```bash + mkdir rexec_sweet + cd rexec_sweet + ``` + +2. **Clone the repository inside the directory:** Get the `rexec_sweet.py` script from the GitHub repository: + + ```bash + git clone https://github.com/geremyCohen/go_benchmarks.git + cd go_benchmarks + ``` + +3. **Run the installer:** Copy and paste this command into your terminal to run the installer: + + ```bash + ./install.sh + ``` + + If the install.sh script detects that you already have dependencies installed, it may ask you if you wish to reinstall them with the following prompt: + + ```bash + # Example output: + $ ./install.sh + ... + pyenv: /Users/gercoh01/.pyenv/versions/3.9.22 already exists + continue with installation? (y/N) + ``` + + If you see this prompt, enter `N` (not `Y`!) to continue with the installation without modifying the existing installed dependencies. + +4. **Verify VM status:** Make sure the GCP VM instances you created in the previous section are running. If not, start them now, and give them a few minutes to come up. + +{{% notice Note %}} +The install script will prompt you to authenticate with Google Cloud Platform (GCP) using the gcloud command-line tool at the end of install. If after installing you have issues running the script and/or get GCP authentication errors, you can manually authenticate with GCP by running the following command: `gcloud auth login` +{{% /notice %}} + + +Continue on to the next section to run the script and see how it simplifies the benchmarking process. \ No newline at end of file diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/rexec_sweet_run.md b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/rexec_sweet_run.md new file mode 100644 index 0000000000..2d71628d43 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/rexec_sweet_run.md @@ -0,0 +1,108 @@ +--- +title: Running the Automated Benchmark and Benchstat Runner +weight: 54 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Section Overview + +With `rexec_sweet` installed, your benchmarking instances running, and your localhost authenticated with GCP, you'll now see how to run benchmarks in an automated fashion. + +## Run an Automated Benchmark and Analysis + +1. **Run the script:** Execute the `rexec_sweet` script from your local terminal: + +```bash +rexec_sweet +``` + +2. **Select a benchmark:** The script will prompt you for the name of the benchmark you want to run. Press enter to run the default benchmark, which is `markdown` (this is the recommended benchmark to run the first time.) + +```bash +# Example output: +Available benchmarks: +1. biogo-igor +2. biogo-krishna +3. bleve-index +4. cockroachdb +5. esbuild +6. etcd +7. go-build +8. gopher-lua +9. markdown (default) +10. tile38 +Enter number (1-10) [default: markdown]: +``` + +3. **Select instances:** The script will call into GCP to detect all running VMs. + +```bash +# Example output: +Available instances: +1. c4 (will be used as first instance) +2. c4a (will be used as second instance) + +Do you want to run the first two instances found with default install directories? [Y/n]: +``` + +4. **Choose your configuration:** You have two options: + + - **Use default settings:** If you want to run benchmarks on the instances labeled with "will be used as nth instance", and you installed Go and Sweet into the default directories as noted in the tutorial, you can press Enter to accept the defaults. + + - **Custom configuration:** If you are running more than two instances, and the script doesn't suggest the correct two to autorun, or you installed Go and Sweet to non-default folders, select "n" and press Enter. The script will then prompt you to select the instances and runtime paths. + +In this example, we'll manually select the instances and paths: + +```bash +# Example output: +Available instances: +1. c4 (will be used as first instance) +2. c4a (will be used as second instance) + +Do you want to run the first two instances found with default install directories? [Y/n]: n + +Select 1st instance: +1. c4 +2. c4a +Enter number (1-2): 1 +Enter remote path for c4 [default: ~/benchmarks/sweet]: + +Select 2nd instance: +1. c4 +2. c4a +Enter number (1-2): 2 +Enter remote path for c4a [default: ~/benchmarks/sweet]: +Output directory: /private/tmp/a/go_benchmarks/results/c4-c4a-markdown-20250610T190407 +... +``` + +Upon entering instance names and paths for the VMs, the script will automatically: + - Run the benchmark on both VMs + - Run `benchstat` to compare the results + - Push the results to your local machine + +```bash +# Example output: +Running benchmarks on the selected instances... +[c4a] [sweet] Work directory: /tmp/gosweet3216239593 +[c4] [sweet] Work directory: /tmp/gosweet2073316306... +[c4a] ✅ benchmark completed +[c4] ✅ benchmark completed +... +Report generated in results/c4-c4a-markdown-20250610T190407 +``` + +5. **View the report:** Once on your local machine, `rexec_sweet` will generate an HTML report that will open automatically in your web browser. + + If you close the tab or browser, you can always reopen the report by navigating to the `results` subdirectory of the current working directory of the `rexec_sweet.py` script, and opening `report.html`. + +![](images/run_auto/2.png) + + +{{% notice Note %}} +If you see output messages from `rexec_sweet.py` similar to "geomeans may not be comparable" or "Dn: ratios must be >0 to compute geomean", this is expected and can be ignored. These messages indicate that the benchmark sets differ between the two VMs, which is common when running benchmarks on different hardware or configurations. +{{% /notice %}} + +6. **Analyze results:** Upon completion, the script will generate a report in the `results` subdirectory of the current working directory of the `rexec_sweet.py` script, which opens automatically in your web browser to view the benchmark results and comparisons. diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/running_benchmarks.md b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/running_benchmarks.md new file mode 100644 index 0000000000..66815cc77d --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/running_benchmarks.md @@ -0,0 +1,113 @@ +--- +title: Benchmark Types and Metrics +weight: 50 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Section Overview + +With setup complete, you can now run and analyze the benchmarks. Before you do, it's good to understand all the moving pieces in more detail. + +## Choosing a Benchmark to Run + +Whether running manually or automatic, the benchmarking process consists of two main steps: + +1. **Running benchmarks with Sweet**: `sweet` executes the benchmarks on each VM, generating raw performance data + +2. **Analyzing results with Benchstat**: `benchstat` compares the results from different VMs to identify performance differences. Benchstat can output results in text format (default) or CSV format. The text format provides a human-readable tabular view, while CSV allows for further processing with other tools. + +Sweet comes ready to run with the following benchmarks: + +| Benchmark | Description | Command | +|-----------------|-------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------| +| **biogo-igor** | Processes pairwise alignment data using the biogo library, grouping repeat feature families and outputting results in JSON format. | `sweet run -count 10 -run="biogo-igor" config.toml` | +| **biogo-krishna** | Pure-Go implementation of the PALS algorithm for pairwise sequence alignment, measuring alignment runtime performance. | `sweet run -count 10 -run="biogo-krishna" config.toml` | +| **bleve-index** | Indexes a subset of Wikipedia articles into a Bleve full-text search index to assess indexing throughput and resource usage. | `sweet run -count 10 -run="bleve-index" config.toml` | +| **cockroachdb** | Executes CockroachDB KV workloads with varying read percentages (0%, 50%, 95%) and node counts (1 & 3) to evaluate database performance. | `sweet run -count 10 -run="cockroachdb" config.toml` | +| **esbuild** | Bundles and minifies JavaScript/TypeScript code using esbuild on a representative codebase to measure build speed and efficiency. | `sweet run -count 10 -run="esbuild" config.toml` | +| **etcd** | Uses the official etcd benchmarking tool to stress-test an etcd cluster, measuring request latency and throughput for key-value operations. | `sweet run -count 10 -run="etcd" config.toml` | +| **go-build** | Compiles a representative Go module (or the Go toolchain) to measure compilation time and memory (RSS) usage on supported platforms. | `sweet run -count 10 -run="go-build" config.toml` | +| **gopher-lua** | Executes Lua scripts using the GopherLua VM to benchmark the performance of a pure-Go Lua interpreter. | `sweet run -count 10 -run="gopher-lua" config.toml` | +| **markdown** | Parses and renders Markdown documents to HTML using a Go-based markdown library to evaluate parsing and rendering throughput. | `sweet run -count 10 -run="markdown" config.toml` | +| **tile38** | Stress-tests a Tile38 geospatial database with WITHIN, INTERSECTS, and NEARBY queries to measure spatial query performance. | `sweet run -count 10 -run="tile38" config.toml` | + +## Metrics Summary + +When running benchmarks, several key metrics are collected to evaluate performance. The following summarizes the most common metrics and their significance: + +### Seconds per Operation - Lower is better + +This metric measures the time taken to complete a single operation, indicating the raw speed of execution. It directly reflects the performance efficiency of a system for a specific task, making it one of the most fundamental benchmarking metrics. + +A system with lower seconds per operation completes tasks faster. + This metric primarily reflects CPU performance but can also be influenced by memory access speeds and I/O operations. If seconds per operation is the only metric showing significant difference while memory metrics are similar, the performance difference is likely CPU-bound. + +### Operations per Second - Higher is better +This metric provides a clear measure of system performance capacity, making it essential for understanding raw processing power and scalability potential. A system performing more operations per second has greater processing capacity. + +This metric reflects overall system performance including CPU speed, memory access efficiency, and I/O capabilities. + +If operations per second is substantially higher while memory usage remains proportional, the system likely has superior CPU performance. High operations per second with disproportionately high memory usage may indicate performance gains through memory-intensive optimizations. A system showing higher operations per second but also higher resource consumption may be trading efficiency for raw speed. + +This metric is essentially the inverse of "seconds per operation" and provides a more intuitive way to understand throughput capacity. + +### Average RSS Bytes - Lower is better + +Resident Set Size (RSS) represents the portion of memory occupied by a process that is held in RAM (not swapped out). It shows the typical memory footprint during operation, indicating memory efficiency and potential for scalability. + +Lower average RSS indicates more efficient memory usage. A system with lower average RSS can typically handle more concurrent processes before memory becomes a bottleneck. This metric reflects both algorithm efficiency and memory management capabilities. + +If one VM has significantly higher seconds per operation but lower RSS, it may be trading speed for memory efficiency. Systems with similar CPU performance but different RSS values indicate different memory optimization approaches; lower RSS with similar CPU performance suggests better memory management, which is a critical indicator of performance in memory-constrained environments. + +### Peak RSS Bytes - Lower is better + +Peak RSS bytes is the maximum Resident Set Size reached during execution, representing the worst-case memory usage scenario. The peak RSS metric helps to understand memory requirements and potential for memory-related bottlenecks during intensive operations. + +Lower peak RSS indicates better handling of memory-intensive operations. High peak RSS values can indicate memory spikes that might cause performance degradation through swapping or out-of-memory conditions. + +Large differences between average and peak RSS suggest memory usage volatility. A system with lower peak RSS but similar performance is better suited for memory-constrained environments. + +### Peak VM Bytes - Lower is better + +Peak VM Bytes is the maximum Virtual Memory size used, including both RAM and swap space allocated to the process. + +Lower peak VM indicates more efficient use of the total memory address space. High VM usage with low RSS suggests the system is effectively using virtual memory management. Extremely high VM with proportionally low RSS might indicate memory fragmentation issues. + +If peak VM is much higher than peak RSS, the system is relying heavily on virtual memory management. Systems with similar performance but different VM usage patterns may have different memory allocation strategies. High VM with performance degradation suggests potential memory-bound operations due to excessive paging. + +## Summary of Efficiency Indicators + +When comparing metrics across two systems, keep the following in mind: + +### CPU-bound vs Memory-bound +A system is likely CPU-bound if seconds per operation differs significantly while memory metrics remain similar. + +A system is likely memory-bound if performance degrades as memory metrics increase, especially when peak RSS approaches available physical memory. + +### Efficiency Indicators +The ideal system shows lower values across all metrics - faster execution with smaller memory footprint. Systems with similar seconds per operation but significantly different memory metrics indicate different optimization priorities. + +### Scalability Potential +Lower memory metrics (especially peak values) suggest better scalability for concurrent workloads. Systems with lower seconds per operation but higher memory usage may perform well for single tasks but scale poorly. + +### Optimization Targets +Large gaps between average and peak memory usage suggest opportunities for memory optimization. High seconds per operation with low memory usage suggests CPU optimization potential. + +## Best Practices when benchmarking across different instance types + +Here are some general tips to keep in mind as you explore benchmarking across different apps and instance types: + +- Unlike Intel and AMD processors that use hyper-threading, Arm processors provide single-threaded cores without hyper-threading. A four-core Arm processor has four independent cores running four threads, while an four-core Intel processor provides eight logical cores through hyper-threading. This means each Arm vCPU represents a full physical core, while each Intel/AMD vCPU represents half a physical core. For fair comparison, this learning path uses a 4-vCPU Arm instance against an 8-vCPU Intel instance. When scaling up instance sizes during benchmarking, make sure to keep a 2:1 Intel/AMD:Arm VCPU ratio if you wish to keep parity on CPU resources. + +- It's suggested to run each benchmark at least 10 times (specified via the `count` parameter) to handle outlier/errant runs and ensure statistical significance. + +- Results may be bound by CPU, memory, or I/O performance. If you see significant differences in one metric but not others, it may indicate a bottleneck in that area; running the same benchmark with different configurations (e.g., more CPU cores, more memory) can help identify the bottleneck. + + + + + + +