Create load testing framework and stress-test IDMS service

[MartinPaulEve]

Stress Test Plan: Knowledge Commons Profiles (IDMS)

Context

knowledge-commons-profiles is a Django 5.1+ application using PostgreSQL and Redis, deployed via Docker Compose. It is the Identity Management System for the Knowledge Commons platform and uses CILogon (OAuth 2.0 / OIDC) as the upstream identity provider.

Because this is the IDMS, the OAuth 2.0 Authorization Code login flow is the primary code path under test. Bypassing authentication would defeat the purpose of the exercise. The plan is to stand up a mock OIDC provider that imitates CILogon, point the test deployment at it, and drive the full login flow from Locust at scale.

Goals

• Measure sustainable login throughput (logins/sec before callback latency degrades)
• Test spike behaviour (e.g. 5x traffic surge)
• Mixed traffic simulations (95% browse/5% login)
• Concurrent same-user logins for upsert/IntegrityError/duplicates
• Long-running soak for stability and leaks
• Identify first resource ceiling (Postgres, Redis, Gunicorn, HTTPS sockets, etc.)

Non-goals

• Testing CILogon itself
• UI/JS performance (HTTP-only)
• Benchmarking production DB (test DB only)

Deliverables

• Mock OIDC provider with the same claims as CILogon
• Automated Locust scripts for different traffic types (login, mixed, contention, soak)
• Django management command to seed and clean up loadtest identities
• Docker Compose for the test stack
• Prometheus + Grafana dashboard covering Locust, Django, Postgres, Redis, mock OIDC

Implementation Phases

1. Build a mock OIDC provider based on FastAPI, auto-approve flows, and claim shape matching CILogon exactly
2. Point a test deployment at the mock provider (update .envs)
3. Make a Django management command to seed identities + clean up after
4. Write Locust files for login flooding, mixed traffic, same-user collision, and long soaks
5. Compose file for the stack
6. Integrated observability

Acceptance

• End-to-end login flows are exercised via Locust
• Grafana shows live test + app + DB + Redis metrics
• Soak/throughput runs surface bottlenecks cleanly
• Seeded users and mock deployment never touch production

---

[MartinPaulEve]

Locust testing underway

We have been using a sophisticated Locust load testing system to evaluate P50, P95 and P99 statistics for access across the login cycle in the IDMS. This has involved building a mock OIDC service that simulates CILogon as we go through the login process. We also needed to stub out all of the external API calls so that we don't, for example, hammer UP on login with load testing.

Full details of the framework are available in the LOADTEST README file. This comes with optional Prometheus and Grafana stat collecting. However, the Locus interface, while basic in many ways, does the job just fine and will give you the statistics that you need.

The basic steps to get this up and running (assuming no Prometheus/Grafana) are:

1. Change your environment variables to have:

DJANGO_SETTINGS_MODULE=config.settings.loadtest
LOADTEST=1
CILOGON_DISCOVERY_URL=http://<tester-public-host>:8080/.well-known/openid-configuration
LOADTEST_PROMETHEUS=0
LOADTEST_STUB_EXTERNAL_SYNC=1       # default; explicit for clarity
LOADTEST_STUB_IDMS_API=1            # default; explicit for clarity

2. Start a public mock oidc service:

MOCK_ISSUER=http://<tester-public-host>:8080 docker compose -f docker-compose.loadtest-driver.yml up mock-oidc -d

3. Seed some identities on the app:

LOADTEST=1 ./manage.py seed_loadtest_identities --count 2000

(You may need to add LOADTEST_HOSTNAME_ALLOWLIST= if running from somewhere different)

4. Take the file from that and put it in loadtest/locust as "subjects.txt" (assuming you are in loadtest directory):

cp /tmp/loadtest_subjects.txt ./locust/

5. Run Locust:

MOCK_ISSUER=http://<tester-public-host>:8080 LOADTEST_SUBJECTS_HOST=$PWD/subjects.txt LOCUSTFILE=locustfile_login_only.py LOCUST_HOST=https://address-to-test.org docker compose -f docker-compose.loadtest-driver.yml up --scale locust-worker=32 -d

You should then have Locust on port 8089 of the machine where you ran that.

Let's get distributed

We wanted to really stress test this system, so I got Locust running in distributed mode (it's surprisingly easy!)

The instructions for running Locust distributed are in a separate README.

We basically, though, now have the firepower of 3 machines pointing at the LOADTEST Test site.

Thanks to @tzouris for helping me with infrastructure!

[MartinPaulEve]

Ongoing deployment settings optimisations after load testing

Gunicorn workers scaled for m8g.xlarge

Changing to --workers 3 --threads 8 = to handle 24 concurrent requests per instance. Reasoning on a 3.5 vCPU budget shared with Traefik:

• Traefik is event-loop-based and idles at ~5–15% of one core under reasonable load. Reserve ~0.5 vCPU.
• That leaves ~3 vCPU for Django. With gthread, processes are mostly waiting on I/O, so 3 workers (one per available core) is plenty — adding more processes just increases context-switch overhead.
• 8 threads/worker gives 24 in-flight requests.
• These will now also be specifiable in secrets (GUNICORN_WORKERS, GUNICORN_THREADS) for fast future changes.

Gunicorn writing to slow disk
Gunicorn writes worker-heartbeat timestamps to a tmpfile every loop iteration. On ECS the container's writable layer is slow overlay storage; pointing this at the tmpfs at /dev/shm avoids inflating p99 by tens of ms on busy workers