Skip to content

Commit bcbb896

Browse files
NoahStapptimgraham
NoahStapp
and
timgraham
committed
INTPYTHON-847 Add performance tests
Co-authored-by: Tim Graham <timograham@gmail.com>
1 parent 1498efc commit bcbb896

17 files changed

Lines changed: 848 additions & 0 deletions

‎.evergreen/config.yml‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,65 @@ functions:
9595
- ./.evergreen/run-tests.sh
9696
- encryption
9797

98+
# Performance test functions
99+
"run performance tests":
100+
- command: subprocess.exec
101+
type: test
102+
params:
103+
binary: bash
104+
working_dir: "src"
105+
include_expansions_in_env: [ "DRIVERS_TOOLS", "MONGODB_URI" ]
106+
args:
107+
- ./.evergreen/perf/run-tests.sh
108+
109+
"attach performance test results":
110+
- command: attach.results
111+
params:
112+
file_location: src/report.json
113+
114+
"send dashboard data":
115+
- command: subprocess.exec
116+
params:
117+
binary: bash
118+
args:
119+
- .evergreen/perf/submission-setup.sh
120+
working_dir: src
121+
include_expansions_in_env: [
122+
"requester",
123+
"revision_order_id",
124+
"project_id",
125+
"version_id",
126+
"build_variant",
127+
"parsed_order_id",
128+
"task_name",
129+
"task_id",
130+
"execution",
131+
"is_mainline",
132+
]
133+
type: test
134+
- command: expansions.update
135+
params:
136+
file: src/expansion.yml
137+
- command: subprocess.exec
138+
params:
139+
binary: bash
140+
args:
141+
- .evergreen/perf/submission.sh
142+
working_dir: src
143+
include_expansions_in_env: [
144+
"requester",
145+
"revision_order_id",
146+
"project_id",
147+
"version_id",
148+
"build_variant",
149+
"parsed_order_id",
150+
"task_name",
151+
"task_id",
152+
"execution",
153+
"is_mainline",
154+
]
155+
type: test
156+
98157
pre:
99158
- func: setup
100159
- func: bootstrap mongo-orchestration
@@ -113,6 +172,12 @@ tasks:
113172
- func: "run encryption tests"
114173
- func: "teardown csfle"
115174

175+
- name: run-performance-tests
176+
commands:
177+
- func: "run performance tests"
178+
- func: "attach performance test results"
179+
- func: "send dashboard data"
180+
116181
buildvariants:
117182
- name: tests-7-noauth-nossl
118183
display_name: Run Tests 7.0 NoAuth NoSSL
@@ -177,3 +242,11 @@ buildvariants:
177242
DJANGO_SETTINGS_MODULE: "encrypted_aws_settings"
178243
tasks:
179244
- name: run-encryption-tests
245+
246+
- name: performance-tests
247+
display_name: Performance Tests
248+
run_on:
249+
- rhel90-dbx-perf-large
250+
batchtime: 1440 # Daily
251+
tasks:
252+
- name: run-performance-tests

‎.evergreen/perf/run-tests.sh‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
#!/usr/bin/bash
2+
3+
set -eux
4+
5+
export OUTPUT_FILE="results.json"
6+
7+
# Install django-mongodb-backend
8+
/opt/python/3.12/bin/python3 -m venv venv
9+
. venv/bin/activate
10+
python -m pip install -U pip
11+
pip install -e .
12+
13+
python .evergreen/perf/runtests.py
14+
mv performance_tests/$OUTPUT_FILE $OUTPUT_FILE
15+
mv performance_tests/report.json report.json

‎.evergreen/perf/runtests.py‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
import json
2+
import logging
3+
import os
4+
import shlex
5+
import subprocess
6+
import sys
7+
from datetime import datetime
8+
from pathlib import Path
9+
10+
LOGGER = logging.getLogger("test")
11+
logging.basicConfig(level=logging.INFO, format="%(levelname)-8s %(message)s")
12+
OUTPUT_FILE = os.environ.get("OUTPUT_FILE", "results.json")
13+
14+
15+
def create_report(start_time: datetime):
16+
"""Format the output from the performance tests into a report.json file."""
17+
end_time = datetime.now()
18+
elapsed_secs = (end_time - start_time).total_seconds()
19+
with open(OUTPUT_FILE) as fid: # noqa: PTH123
20+
results = json.load(fid)
21+
LOGGER.info("%s:\n%s", OUTPUT_FILE, json.dumps(results, indent=2))
22+
results = {
23+
"status": "PASS",
24+
"exit_code": 0,
25+
"test_file": "BenchmarkTests",
26+
"start": int(start_time.timestamp()),
27+
"end": int(end_time.timestamp()),
28+
"elapsed": elapsed_secs,
29+
}
30+
report = {"results": [results]}
31+
LOGGER.info("report.json\n%s", json.dumps(report, indent=2))
32+
with open("report.json", "w", newline="\n") as fid: # noqa: PTH123
33+
json.dump(report, fid)
34+
35+
36+
def run_command(cmd: str | list[str], **kwargs) -> None:
37+
"""Run a shell command. Exit on failure."""
38+
if isinstance(cmd, list):
39+
cmd = " ".join(cmd)
40+
LOGGER.info("Running command '%s'...", cmd)
41+
kwargs.setdefault("check", True)
42+
try:
43+
subprocess.run(shlex.split(cmd), **kwargs) # noqa: PLW1510, S603
44+
except subprocess.CalledProcessError as e:
45+
LOGGER.error(e.output)
46+
LOGGER.error(str(e))
47+
sys.exit(e.returncode)
48+
LOGGER.info("Running command '%s'... done.", cmd)
49+
50+
51+
start_time = datetime.now()
52+
ROOT = Path(__file__).absolute().parent.parent.parent
53+
data_dir = ROOT / "specifications/source/benchmarking/odm-data"
54+
if not data_dir.exists():
55+
run_command("git clone --depth 1 https://github.com/mongodb/specifications.git")
56+
run_command("tar xf flat_models.tgz", cwd=data_dir)
57+
run_command("tar xf nested_models.tgz", cwd=data_dir)
58+
59+
os.chdir("performance_tests")
60+
start_time = datetime.now()
61+
run_command(
62+
"python runtests.py",
63+
env=os.environ
64+
| {
65+
"DJANGO_MONGODB_PERFORMANCE_TEST_DATA_PATH": str(data_dir),
66+
"OUTPUT_FILE": OUTPUT_FILE,
67+
},
68+
)
69+
create_report(start_time)

‎.evergreen/perf/submission-setup.sh‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
#!/bin/bash
2+
set -eu
3+
4+
# Determine whether or not the data is from a mainline evergreen run.
5+
# shellcheck disable=SC2154
6+
if [ "${requester}" == "commit" ]; then
7+
echo "is_mainline: true" >> expansion.yml
8+
else
9+
echo "is_mainline: false" >> expansion.yml
10+
fi
11+
12+
# Parse the username out of the order_id as patches append that in and SPS does
13+
# not need that information.
14+
# shellcheck disable=SC2154
15+
echo "parsed_order_id: $(echo "${revision_order_id}" | awk -F'_' '{print $NF}')" >> expansion.yml

‎.evergreen/perf/submission.sh‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
#!/bin/bash
2+
set -eu
3+
4+
# Submit the performance data to the SPS endpoint.
5+
# shellcheck disable=SC2154
6+
response=$(curl -s -w "\nHTTP_STATUS:%{http_code}" -X 'POST' \
7+
"https://performance-monitoring-api.corp.mongodb.com/raw_perf_results/cedar_report?project=${project_id}&version=${version_id}&variant=${build_variant}&order=${parsed_order_id}&task_name=${task_name}&task_id=${task_id}&execution=${execution}&mainline=${is_mainline}" \
8+
-H 'accept: application/json' \
9+
-H 'Content-Type: application/json' \
10+
-d @results.json)
11+
12+
http_status=$(echo "$response" | grep "HTTP_STATUS" | awk -F':' '{print $2}')
13+
response_body=$(echo "$response" | sed '/HTTP_STATUS/d')
14+
15+
# Display an error if the data was not successfully submitted.
16+
if [ "$http_status" -ne 200 ]; then
17+
echo "Error: Received HTTP status $http_status"
18+
echo "Response Body: $response_body"
19+
exit 1
20+
fi
21+
22+
echo "Response Body: $response_body"
23+
echo "HTTP Status: $http_status"

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@ __pycache__
4444
.coverage
4545
.pytest_cache/
4646
.benchmarks
47+
performance_tests/odm-data
4748

4849
# documentation build artifacts
4950

‎docs/internals/contributing/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,4 +9,5 @@ Backend? Here are some resources:
99

1010
coding-style
1111
unit-tests
12+
performance-tests
1213
writing-documentation

‎docs/internals/contributing/performance-tests.rst‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
=================
2+
Performance tests
3+
=================
4+
5+
Django MongoDB Backend uses a performance test suite to catch regressions.
6+
7+
See the `ODM Performance Benchmarking specification
8+
<https://github.com/mongodb/specifications/blob/master/source/benchmarking/odm-benchmarking.md>`__
9+
for an overview of design and implementation decisions.
10+
11+
.. _running-perf-tests:
12+
13+
Running the performance tests
14+
=============================
15+
16+
First, :ref:`setup your environment to run the unit tests<running-unit-tests>`.
17+
18+
Then, set up the benchmark data, running the following commands from the
19+
``django-mongodb-backend`` repository's root directory::
20+
21+
.. code-block:: bash
22+
23+
$ git clone --depth 1 https://github.com/mongodb/specifications.git
24+
$ mkdir performance_tests/odm-data
25+
$ cp specifications/source/benchmarking/odm-data/flat_models.tgz performance_tests/odm-data/flat_models.tgz
26+
$ cp specifications/source/benchmarking/odm-data/nested_models.tgz performance_tests/odm-data/nested_models.tgz
27+
$ pushd performance_tests/odm-data
28+
$ tar xf flat_models.tgz
29+
$ tar xf nested_models.tgz
30+
$ popd
31+
32+
To run the tests:
33+
34+
.. code-block:: bash
35+
36+
$ cd performance_tests
37+
$ FASTBENCH=1 ./runtests.py
38+
39+
.. warning::
40+
41+
Running the full test suite without ``FASTBENCH=1`` will take a significant
42+
amount of time.
43+
44+
To run an individual test (again, you may want to add ``FASTBENCH=1``):
45+
46+
.. code-block:: bash
47+
48+
$ ./runtests.py perf.tests.TestLargeNestedDocFilterArray

‎performance_tests/perf/__init__.py‎

Whitespace-only changes.

0 commit comments

Comments
 (0)