Skip to content

how_run_test.md

Latest commit

 

History

History
124 lines (98 loc) · 3.6 KB

how_run_test.md

File metadata and controls

124 lines (98 loc) · 3.6 KB

How Run Test

Expanded external-S3 demo for the Python path.

Scenario:

  • loadtest/run-configs/public-s3-demo-5m-scaleout.json
  • 3 brokers
  • 3 read-only consumers
  • 3 compactors
  • 1024 producer HTTP clients per broker
  • 256 consume HTTP clients per consumer
  • 32 MiB/s produce target per broker
  • 32 MiB/s consume target per consumer
  • 6 topics x 16 partitions = 96 total partitions

Use this from the repo root:

cd /home/ops/diskless-python-kafka-2/diskless-python-kafka

python3 -m venv .venv-loadtest
.venv-loadtest/bin/python -m pip install --upgrade pip setuptools wheel
.venv-loadtest/bin/python -m pip install boto3 oxia

set -a
source .env
set +a

export LLOG_LOADTEST_EXTERNAL_S3=1
export LLOG_S3_BUCKET="$S3_BUCKET"
export AWS_REGION="$S3_REGION"
unset LLOG_S3_ENDPOINT_URL
export LLOG_ROOT_PREFIX="public-demo-scaleout-$(date -u +%Y%m%dT%H%M%SZ)"

export LLOG_OXIA_CLIENT_PYTHON_REPO="${LLOG_OXIA_CLIENT_PYTHON_REPO:-/home/ops/diskless-python-kafka/oxia-client-python-upstream}"
export LLOG_LOADTEST_EXTRA_PYTHONPATH="${LLOG_LOADTEST_EXTRA_PYTHONPATH:-$LLOG_OXIA_CLIENT_PYTHON_REPO/src}"

# This host already uses 9090 and 3000 for the permanent monitoring stack.
export LLOG_OBSERVABILITY_HOST_PROMETHEUS_PORT="${LLOG_OBSERVABILITY_HOST_PROMETHEUS_PORT:-19090}"
export LLOG_OBSERVABILITY_HOST_GRAFANA_PORT="${LLOG_OBSERVABILITY_HOST_GRAFANA_PORT:-13000}"

export LLOG_LOADTEST_PYTHON="$PWD/.venv-loadtest/bin/python"

"$LLOG_LOADTEST_PYTHON" - <<'PY'
import boto3
import os
import oxia

s3 = boto3.client(
    "s3",
    region_name=os.environ["AWS_REGION"],
    aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
    aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"],
)
s3.head_bucket(Bucket=os.environ["LLOG_S3_BUCKET"])
print("runtime ok")
PY

"$LLOG_LOADTEST_PYTHON" -m loadtest.quick_observability run \
  --scenario loadtest/run-configs/public-s3-demo-5m-scaleout.json \
  --warmup-seconds 30 \
  --duration-seconds 300 \
  --cooldown-seconds 30

Grab the run directory and headline numbers:

RUN_DIR="$("$LLOG_LOADTEST_PYTHON" - <<'PY'
import json
from pathlib import Path
state = json.loads(Path("loadtest/observability/runtime/state.json").read_text())
print(state["run_dir"])
PY
)"

jq '{
  producer_successful_mib_per_sec: .rates.successful_mib_per_sec,
  producer_latency_p50_ms: .latency_ms.p50,
  producer_latency_p95_ms: .latency_ms.p95,
  producer_latency_p99_ms: .latency_ms.p99,
  producer_peak_inflight: .inflight_requests.peak
}' "$RUN_DIR/client-summary.json"

jq '{
  consumer_successful_mib_per_sec: .rates.successful_mib_per_sec,
  consumer_latency_p50_ms: .latency_ms.p50,
  consumer_latency_p95_ms: .latency_ms.p95,
  consumer_latency_p99_ms: .latency_ms.p99,
  consumer_empty_response_count: .totals.empty_response_count
}' "$RUN_DIR/consume-client-summary.json"

jq '{
  s3_request_cost_usd: .latest_totals.broker_s3_estimated_request_cost_usd,
  bucket_stored_gb: .latest_totals.bucket_stored_gb,
  estimated_monthly_storage_cost_usd: .latest_totals.estimated_monthly_storage_cost_usd,
  warnings: .warnings
}' "$RUN_DIR/summary.json"

Inspect compactors while the stack is still up:

jq -r '.compactors[] | [.compactor_id, .host, (.port | tostring)] | @tsv' "$RUN_DIR/stack.json" |
while IFS=$'\t' read -r compactor_id host port; do
  printf '== %s ==\n' "$compactor_id"
  curl -sf "http://$host:$port/metrics" | jq '.compactor'
done

Inspect:

  • Grafana: http://127.0.0.1:13000
  • Prometheus: http://127.0.0.1:19090

Cleanup:

./quick_run.sh down
"$LLOG_LOADTEST_PYTHON" -m loadtest.cleanup --run-dir "$RUN_DIR"

Full notes are in loadtest/PUBLIC_S3_DEMO.md.