chore: mv load tests (#2060)

chore: mv load tests

Author: taddes

tools/syncstorage-loadtest/Dockerfile

@@ -0,0 +1,38 @@
+# Mozilla AutoPush Load-Tester
+
+# SyncStorage-LoadTest
+#
+FROM python:3.10-slim
+
+RUN mkdir -p /app
+ADD . /app
+WORKDIR /app
+
+# Building:
+# you can build a local docker image using
+# `docker build . --tag syncstorage-loadtest:local`
+
+# system setup
+RUN \
+    BUILD_DEPS="git build-essential" && \
+    # wget not required but nice to have
+    RUN_DEPS="wget libssl-dev" && \
+    apt-get update && \
+    apt-get install -yq --no-install-recommends ${BUILD_DEPS} ${RUN_DEPS} && \
+    pip install --no-cache-dir poetry && \
+    apt-get purge -yq --auto-remove ${BUILD_DEPS} && \
+    apt-get autoremove -yqq && \
+    apt-get clean -y
+
+# app install
+RUN poetry config virtualenvs.create false && \
+    poetry install --no-dev --no-interaction --no-ansi
+
+# Using:
+# Start an interactive terminal using
+# `docker run --net=host -it syncstorage-loadtest:local`
+# This will start a bash shell as root.
+# You can fire off a load test by calling:
+# `SERVER_URL=http://${HOST}:${PORT}#${SECRET} molotov -v`
+
+ENTRYPOINT ["/bin/bash"]

tools/syncstorage-loadtest/README.md

@@ -0,0 +1,104 @@
+# Storage load test
+
+**Note:** This requires Python version 3.10+
+
+The tests can execute in three modes:
+
+1. **Direct access**: use a known master secret
+2. **Firefox accounts OAuth**: create test accounts and use OAuth JWTs
+3. **Self-signed JWT**: generate and sign JWTs locally with a given private key
+
+## Installation
+
+### Environment Setup
+
+To run the syncstorage load tests, you'll need a Python >=3.10 development environment with `Poetry` installed. You can also directly use `poetry run` to execute commands as described in the usage examples below.
+
+The easiest solution is to use `pyenv` and the `pyenv-virtualenv` plugin for your virtual environments as a way to isolate the dependencies from other projects.
+
+1. Install `pyenv` using the [latest documentation](https://github.com/pyenv/pyenv#installation) for your platform.
+
+2. Follow the instructions to install the `pyenv-virtualenv` plugin.
+   See the [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv) documentation.
+
+3. Ensure you've added `pyenv` and `pyenv-virtualenv` to your PATH.
+
+   Example:
+   ```shell
+   export PATH="$HOME/.pyenv/bin:$PATH"
+   eval "$(pyenv init -)"
+   eval "$(pyenv virtualenv-init -)"
+   ```
+
+4. Install Python version, create virtualenv, activate and install dependencies from inside the project directory.
+
+   **Note:** You can skip creating a virtual environment and invoke commands directly using `poetry run`.
+
+   ```shell
+   $ cd syncstorage-loadtest
+
+   # Install Python 3.10+
+   $ pyenv install 3.10
+
+   # Create named, associated virtualenv
+   $ pyenv virtualenv 3.10 syncstorage-loadtest  # or whatever name you prefer
+   $ pyenv local syncstorage-loadtest  # activates virtual env whenever you enter this directory
+
+   # Install Poetry and dependencies
+   $ pip install poetry
+   $ poetry install
+   ```
+
+5. Once you're in your virtual environment, run the load tests using:
+   ```bash
+   poetry run molotov [options] loadtest.py
+   ```
+
+### Quick Install
+
+If you already have Poetry installed:
+
+```bash
+poetry install
+```
+
+## Mode 1: Direct Access
+
+With a known syncstorage master secret:
+
+```bash
+SERVER_URL="http://localhost:8000#secretValue" poetry run molotov --max-runs 5 -cxv loadtest.py
+```
+
+## Mode 2: Firefox Accounts OAuth
+
+With FxA stage accounts:
+
+```bash
+SERVER_URL="https://token.stage.mozaws.net" \
+  FXA_API_HOST="https://api-accounts.stage.mozaws.net" \
+  FXA_OAUTH_HOST="https://oauth.stage.mozaws.net" \
+  poetry run molotov --workers 3 --duration 60 -v loadtest.py
+```
+
+## Mode 3: Self-Signed JWTs
+
+Assuming an RSA keypair was generated, the private key pem was saved and accessible, and the token server was started with the public key JWK configured. The presence of `OAUTH_PRIVATE_KEY_FILE` triggers this mode when using OAuth.
+
+Run with self-signed JWTs:
+
+```bash
+SERVER_URL="http://localhost:8000" \
+  OAUTH_PRIVATE_KEY_FILE="/path/to/load_test.pem" \
+  poetry run molotov --workers 100 --duration 300 -v loadtest.py
+```
+
+## Docker
+
+To run it inside docker:
+
+```bash
+docker run -e TEST_REPO=https://github.com/mozilla-services/syncstorage-loadtest -e TEST_NAME=test tarekziade/molotov:latest
+```
+
+Happy Breaking!

tools/syncstorage-loadtest/cleanup_orphaned_accounts.py

@@ -0,0 +1,96 @@
+#!/usr/bin/env python3
+"""
+Cleanup orphaned accounts from interrupted tests.
+
+Usage:
+    python cleanup_orphaned_accounts.py
+"""
+
+import os
+import json
+from fxa.core import Client
+from fxa.errors import ClientError, ServerError
+
+ACCT_TRACKING_FILE = os.path.join(os.path.dirname(__file__), ".accounts_tracking.json")
+FXA_API_HOST = os.environ.get("FXA_API_HOST", "https://api-accounts.stage.mozaws.net")
+
+
+def load_tracked_accounts():
+    if not os.path.exists(ACCT_TRACKING_FILE):
+        return []
+
+    try:
+        with open(ACCT_TRACKING_FILE, "r") as f:
+            return json.load(f)
+    except (json.JSONDecodeError, IOError) as e:
+        print(f"Warning: Could not load tracking file: {e}")
+        return []
+
+
+def save_tracked_accounts(accounts):
+    try:
+        if not accounts:
+            if os.path.exists(ACCT_TRACKING_FILE):
+                os.remove(ACCT_TRACKING_FILE)
+        else:
+            with open(ACCT_TRACKING_FILE, "w") as f:
+                json.dump(accounts, f, indent=2)
+    except IOError as e:
+        print(f"Warning: Could not save tracking file: {e}")
+        raise
+
+
+def remove_account_from_tracking(email):
+    accounts = load_tracked_accounts()
+    accounts = [acc for acc in accounts if acc["email"] != email]
+    save_tracked_accounts(accounts)
+
+
+def cleanup_account(client, account):
+    email = account["email"]
+    password = account["password"]
+
+    try:
+        client.destroy_account(email, password)
+        print(f"  ✓ Deleted: {email}")
+        return True
+    except (ServerError, ClientError) as ex:
+        print(f"  ✗ Delete failed: {email} - {ex}")
+        return False
+    except Exception as ex:
+        print(f"  ✗ Delete error: {email} - {ex}")
+        return False
+
+
+def cleanup_all_accounts():
+    accounts = load_tracked_accounts()
+
+    if not accounts:
+        print("No accounts to clean up.")
+        return 0, 0
+
+    print(f"\nFound {len(accounts)} accounts")
+    print("\nAttempting to delete accounts...\n")
+
+    client = Client(FXA_API_HOST)
+    successful = 0
+    failed = 0
+
+    for account in accounts:
+        if cleanup_account(client, account):
+            successful += 1
+            remove_account_from_tracking(account["email"])
+        else:
+            failed += 1
+
+    print(f"\nResults: {successful} deleted, {failed} failed")
+
+    return successful, failed
+
+
+def main():
+    cleanup_all_accounts()
+
+
+if __name__ == "__main__":
+    main()