Skip to content

HOWRUN.md

Latest commit

 

History

History
88 lines (66 loc) · 3.63 KB

HOWRUN.md

File metadata and controls

88 lines (66 loc) · 3.63 KB

Use the current Python interpreter with oxia>=0.2.0 and boto3 installed.

For the EC2/public-S3 path, prefer the wrapper instead of manual export flows:

python3 -m loadtest.ec2 bootstrap
python3 -m loadtest.ec2 preflight --scenario loadtest/run-configs/public-s3-demo-5m.json --bucket "$S3_BUCKET"
python3 -m loadtest.ec2 smoke --bucket "$S3_BUCKET"
python3 -m loadtest.ec2 run --scenario loadtest/run-configs/public-s3-demo-5m-scaleout.json --bucket "$S3_BUCKET"
python3 -m loadtest.ec2 report latest
python3 -m loadtest.ec2 bundle latest

To bypass managed MinIO and use an existing S3 bucket instead:

export LLOG_LOADTEST_EXTERNAL_S3=1
export LLOG_S3_BUCKET="replace-with-your-bucket"
export AWS_REGION="us-east-1"

Set LLOG_S3_ENDPOINT_URL as well when the target is S3-compatible storage rather than AWS S3 itself. The public external-S3 demo flows and exact commands are in PUBLIC_S3_DEMO.md.

client-summary.json and consume-client-summary.json now report only the steady-state measurement window by default and keep a full_window section for warmup-inclusive debugging.

Sizing http_clients_per_broker:

  • request_payload_bytes = record_size_bytes * topic_partitions_per_request * records_per_topic_partition_per_request
  • target_requests_per_sec_per_broker = (target_mib_per_sec_per_broker * 1048576) / request_payload_bytes
  • required_inflight_per_broker ~= target_requests_per_sec_per_broker * p50_latency_seconds
  • if client-summary.json shows inflight_requests.per_broker.<broker>.peak stuck at the configured cap and pacing_lag_ms is large or rising, increase http_clients_per_broker before blaming the broker

Quick Grafana flow:

cd "$HOME/code/diskless-python-kafka" && ./quick_run.sh
cd "$HOME/code/diskless-python-kafka" && ./quick_run.sh down
cd "$HOME/code/diskless-python-kafka" && python3 -m loadtest.orchestrate up --scenario loadtest/run-configs/sanity-clustered.json

If LLOG_LOADTEST_PROMETHEUS_SYNC_SCRIPT points at a sync script, up will run it after the brokers are healthy so Prometheus sees the new loopback targets immediately instead of waiting for cron.

cd "$HOME/code/diskless-python-kafka" && python3 -m loadtest.orchestrate run --scenario loadtest/run-configs/sanity-clustered.json

run invokes the same sync after broker startup and again after teardown. That keeps short-lived load tests visible in Prometheus and cleans stale target files and bridge proxies when the stack exits.

Set "compactor_enabled": true in the scenario JSON when the run should start managed compactors alongside MinIO, Oxia, and the brokers. Use "compactor_count" and "compactor_workers" to choose how many processes to start and how many workers each one runs with. The compactors will be tracked in stack.json, ports.json, and processes.json under the run directory.

cd "$HOME/code/diskless-python-kafka" && python3 -m loadtest.orchestrate status --run-dir loadtest/results/<run-id>
cd "$HOME/code/diskless-python-kafka" && python3 -m loadtest.orchestrate precreate --run-dir loadtest/results/<run-id>
cd "$HOME/code/diskless-python-kafka" && python3 -m loadtest.metrics_collector --run-dir loadtest/results/<run-id> --once
cd "$HOME/code/diskless-python-kafka" && python3 -m loadtest.metrics_collector --run-dir loadtest/results/<run-id> --samples 10
cd "$HOME/code/diskless-python-kafka" && python3 -m loadtest.orchestrate down --run-dir loadtest/results/<run-id>

down also re-runs the sync so Prometheus target files and loopback bridge proxies are cleaned up right after shutdown.