Getting Started with Automated Performance Regression Testing

Introduction to CyBench Automated Performance Regression Testing

CyBench allows users to set up automated performance regression testing. Assuming you have a private workspace configured on the CyBench UI, all you have to do is start benchmarking a project, sending reports to that workspace, and eventually CyBench will start running regression tests without you having to do anything else! You also have the option to configure your own tests, that will run every single time you run a benchmark and send it to CyBench.

In this case, you are integrating CyBench into your CI/CD pipeline. You can specify the degree to what you will allow a performance drop in your benchmarks until CyBench runner halts, stopping your pipeline. You will be able to prevent code deployments with potentially critical impacts to your applications performance.

Prerequisites

• GitHub account/CyBench account
  • The CyBench website uses GitHub for authentication, and allows you to save private reports to the CyBench database. You're also granted a private workspace where your reports will be uploaded to, with the option of making reports public. Essentially, your GitHub account also becomes your CyBench account.
  • In order to setup automated performance regression testing, you must have a private workspace setup where your project's reports and comparison info will all be stored.

Setting Up Email Notifications

In order to set up email notification, sign into CyBench and then click on your username at the top right. This will bring you to the workspace admin page. At the top left, you will see a green highlighted box titled "Email for Notifications." Enter your email here, and click verify. You will get an email and will be asked to verify that email address.

Once verified, you will receive email notifications for failures in performance regression testing that you configure! Emails will point you to the details of the test that was ran and will provide bits of information regarding the failures.

Comparisons Explained

Automated Comparisons That Run Without User Defined Configurations

Configuration from within UI

Configuration with CyBench Runner (Using Property File)

Configuration with CyBench Maven Plugin

Configuration with CyBench Gradle Plugin

Configuration with CyBench Eclipse Plugin

Configuration with CyBench IntelliJ Plugin