docs(benchmarks): improve benchmarking quickstart for users (kroxylicious#3811)

* docs(benchmarks): improve quickstart for upstream users

- Clarify that jbang and mvn are only needed for compare-results.sh,
  not for running benchmarks
- Add --cluster-overrides section with example for non-default storage
  classes and cluster-specific settings
- Add encryption to the available scenarios list
- Add 1topic-10kb and 1topic-100kb to available workloads list
- Add rate sweep section showing how to find the throughput ceiling
- Add troubleshooting entry for compare-results.sh mvn prerequisite

Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>

* docs(benchmarks): fix jbang/mvn prerequisites scope

jbang and mvn process-sources are required for all benchmark scripts,
not just compare-results.sh. run-benchmark.sh uses CollectResults via
jbang, check-backpressure.sh, check-significance.sh, and
collect-results.sh all require the generated sources too.

Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>

* docs(benchmarks): restore gh CLI prerequisite

gh is used by setup-cluster.sh to download the Kroxylicious operator
release and was accidentally dropped in the previous commit.

Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>

* docs(benchmarks): remove 10kb/100kb workloads from quickstart

These workloads are not yet on main — they live on feat/omb-hit-it-harder
and will be added to the quickstart as part of that PR.

Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>

* docs(benchmarks): expand mvn process-sources troubleshooting entry

Add context explaining why the step is needed (JBang tools generated
from annotated Maven sources) and show the actual error message so
users can recognise it.

Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>

---------

Signed-off-by: Sam Barker <sam@quadrocket.co.uk>

Author: SamBarker

kroxylicious-openmessaging-benchmarks/QUICKSTART.md

@@ -10,9 +10,12 @@ then run the benchmark suite.
   - Full benchmark run: 8 CPU cores, 16 GB RAM across the cluster (3 brokers + 3 OMB workers)
 - `kubectl` configured to access the cluster
 - `helm` 3.0+
-- `gh` (GitHub CLI) — for downloading the Kroxylicious operator release
-- `jbang` — for result comparison scripts ([install](https://www.jbang.dev/download/))
-- `mvn` — for generating JBang source filters
+- `gh` (GitHub CLI) — used by `setup-cluster.sh` to download the Kroxylicious operator release
+- `jbang` — used by the benchmark and result analysis scripts ([install](https://www.jbang.dev/download/))
+- `mvn` — run once before using any benchmark scripts to generate JBang source filters:
+  ```bash
+  mvn process-sources -pl kroxylicious-openmessaging-benchmarks
+  ```
 
 ### Start a local cluster (if needed)
 
@@ -68,14 +71,37 @@ To run one scenario and workload manually:
 ```bash
 ./scripts/run-benchmark.sh baseline 1topic-1kb ./results/baseline/
 ./scripts/run-benchmark.sh proxy-no-filters 1topic-1kb ./results/proxy/
+./scripts/run-benchmark.sh encryption 1topic-1kb ./results/encryption/
 
-# Compare afterwards
-./scripts/compare-results.sh ./results/baseline/*.json ./results/proxy/*.json
+# Compare afterwards (requires mvn process-sources to have been run once)
+./scripts/compare-results.sh ./results/baseline/result.json ./results/proxy/result.json
 ```
 
-Available scenarios: `baseline`, `proxy-no-filters`
+Available scenarios: `baseline`, `proxy-no-filters`, `encryption`
+
 Available workloads: `1topic-1kb`, `10topics-1kb`, `100topics-1kb`
 
+### Cluster-specific settings
+
+If your cluster requires non-default storage classes, image pull secrets, or resource limits,
+create a cluster overrides file and pass it with `--cluster-overrides`:
+
+```bash
+# cluster-overrides.yaml example
+kafka:
+  storage:
+    storageClass: my-storage-class
+```
+
+```bash
+./scripts/run-benchmark.sh \
+  --cluster-overrides ./cluster-overrides.yaml \
+  baseline 1topic-1kb ./results/baseline/
+```
+
+The `--cluster-overrides` file is applied last, so it always wins over scenario and profile
+defaults. Pass it to both `run-benchmark.sh` and `rate-sweep.sh`.
+
 ### Quick validation (smoke profile)
 
 The smoke profile uses 1 broker, 2 workers, and a 1-minute test — not suitable for
@@ -88,6 +114,33 @@ It fits comfortably within a 16 GB single-node cluster (minikube/kind).
   baseline 1topic-1kb ./results/smoke/
 ```
 
+### Finding the saturation point (rate sweep)
+
+To find the throughput ceiling for a scenario, use `rate-sweep.sh`. It deploys infrastructure
+once, then probes at increasing rates until the achieved throughput drops below the target,
+indicating saturation.
+
+```bash
+./scripts/rate-sweep.sh \
+  --scenarios baseline,encryption \
+  --min-rate 10000 \
+  --max-rate 60000 \
+  --step-percent 10 \
+  --output-dir ./results/sweep-$(date +%Y%m%d-%H%M%S)/
+```
+
+This prints a summary table at the end showing achieved rate and p99 latency at each probe,
+with saturation flagged where applicable. Use `--dry-run` to preview the rate sequence
+without deploying anything:
+
+```bash
+./scripts/rate-sweep.sh \
+  --scenarios encryption \
+  --min-rate 30000 --max-rate 50000 --step-percent 5 \
+  --output-dir /tmp/preview/ \
+  --dry-run
+```
+
 ## Interpreting Results
 
 `compare-results.sh` outputs a table with three sections:
@@ -145,3 +198,18 @@ kubectl get configmap omb-driver-baseline -n kafka \
 # baseline:       bootstrap.servers=kafka-kafka-bootstrap:9092
 # proxy scenario: bootstrap.servers=kafka-cluster-ip-bootstrap:9292
 ```
+
+**Scripts fail with "run mvn process-sources first"**
+
+The benchmark scripts use JBang to run Java-based result analysis tools. These tools are
+generated from annotated source files during the Maven build. If you see an error like:
+
+```
+Error: run 'mvn process-sources -pl kroxylicious-openmessaging-benchmarks' first to generate filtered sources
+```
+
+Run the following once from the repository root before using the scripts:
+
+```bash
+mvn process-sources -pl kroxylicious-openmessaging-benchmarks
+```