Skip to content

README.md

Latest commit

 

History

History
89 lines (60 loc) · 2.61 KB

README.md

File metadata and controls

89 lines (60 loc) · 2.61 KB
Raw

Substrate Benchmarking

This is the nascent suite for benchmarking Substrate's performance at scale.

Deploy benchmarks

Important

Source the environment configuration file (e.g., source .ate-dev-env.sh) first so PROJECT_ID, BUCKET_NAME, etc. are set.

Note that deploying the benchmarks does not run them. You must visit Locust's web UI to start a test.

A single wrapper deploys the scale workloads, builds and pushes the Locust image, then deploys the Locust workers:

./benchmarking/deploy_locust.sh --deploy

Useful flags:

  • --worker-count N — number of WorkerPool replicas (default 1).
  • --skip-build — reuse the existing :latest locust image (skip the docker build && docker push step).

To tear everything down (locust then workloads, in reverse order):

./benchmarking/deploy_locust.sh --delete

The same operations are also reachable from the top-level installer for convenience:

./hack/install-ate.sh --deploy-benchmarks
./hack/install-ate.sh --delete-benchmarks

The installer accepts --benchmark-worker-count N (default 1). --skip-build is only available when invoking benchmarking/deploy_locust.sh directly.

Running Tests

Locust Web UI

  • Run kubectl port-forward svc/locust -n benchmarking 8089:8089
  • Visit http://localhost:8089 in your browser to configure and start the load test.

The different user classes you can select are different types of load behaviors you can throw at the system. Note that the "CounterUser" load type requires that the counter demo be installed.

You can also configure things like the number of users, how quickly those users are spawned, the frequency with which requests are made and whether or not tracing is enabled.

Viewing Traces

You must have enabled otel tracing for your cluster to view traces.

You can find trace IDs by viewing the logs tab in the Locust UI

Optional: Prometheus + Grafana

Locust provides graphs, statistics, etc. via the UI. However, you can install Prometheus/Grafana if you want richer details or the ability to perform deeper analysis. Skip this section if you're only using the Locust web UI.

kubectl apply -f benchmarking/monitoring.yaml

Once installed:

  • Run kubectl port-forward svc/grafana -n benchmarking 3000:3000
  • Visit http://localhost:3000 in your browser.

Development

Rebuilding gRPC Python clients

Make sure you have a virtual environment created (python3 -m venv venv) and activated (source venv/bin/activate).

Install project requirements: pip install -r requirements.txt

Then run generate_protos.sh to generate the Python proto clients.