INTPYTHON-847 Add performance tests by NoahStapp · Pull Request #366 · mongodb/django-mongodb-backend

NoahStapp · 2025-08-14T19:43:28Z

Purpose

This is the first draft of the Django implementation of the MongoDB ODM Benchmarking suite. It contains a small suite of benchmark tests designed to measure django-mongodb-backend performance across data sizes and structures for what we expect to be common user operations.

This is NOT intended to be a comprehensive test suite for every operation, only the most common and widely applicable. This is also not intended to be Django specific: each of these tests must be implementable across all of our ODMs, up to the features supported by each library.

Structure

The benchmarks are contained within a separate Django application within tests/performance/perftest. This application exists only to execute the benchmarking suite.

Running Locally

After starting a local MongoDB server, the tests can be run by running python manage.py test from the tests/performance directory. The full suite run locally is expected to take approximately 10 minutes. For faster benchmarks, pass FASTBENCH=1 as an environment variable totests/performance/perftest/tests.py.

Review Items

Do the Django operations being performed conform to our expectations of how django-mongodb-backend should be used?
Are any operations missing that we expect to be frequently performed by users?
Is the scope and breadth of the benchmarks sufficient for coverage without being cumbersome to implement across potentially dozens of libraries, first and third-party?
Are the data sizes and model structures being used representative of real-world situations?

aclark4life · 2025-08-18T20:20:27Z

In general, awesome!

Re:

Structure

The benchmarks are contained within a separate Django application within tests/performance/perftest. This application exists only to execute the benchmarking suite.

Can this be integrated with existing tests such that some established testing conventions are followed? For example,

tests/<testsuite>_ - contains test suites for various features
performance - Django project for perf testing that contains perf testing app?

IOW If the performance directory "contained a test suite for testing performance" then we could follow the convention of adding the _ at the end so the Django tests could run the performance tests. Since performance appears to be a standalone Django project, I'm not sure what the best thing to do is (especially given the need to be able to run without the Django test runner.) I wonder if a top level perf_testing project that contains the perftest app may be better than including the perf tests in tests/. Or maybe a top level perftest/perftest, project and app respectively. Also not sure if it helps, but apps don't have to be in projects …

NoahStapp · 2025-08-18T21:24:59Z

In general, awesome!

Re:

Structure

The benchmarks are contained within a separate Django application within tests/performance/perftest. This application exists only to execute the benchmarking suite.

Can this be integrated with existing tests such that some established testing conventions are followed? For example,

tests/<testsuite>_ - contains test suites for various features

performance - Django project for perf testing that contains perf testing app?

IOW If the performance directory "contained a test suite for testing performance" then we could follow the convention of adding the _ at the end so the Django tests could run the performance tests. Since performance appears to be a standalone Django project, I'm not sure what the best thing to do is (especially given the need to be able to run without the Django test runner.) I wonder if a top level perf_testing project that contains the perftest app may be better than including the perf tests in tests/. Or maybe a top level perftest/perftest, project and app respectively. Also not sure if it helps, but apps don't have to be in projects …

Running the performance tests alongside the Django tests would add too much runtime to be practical. Putting the perftest project somewhere not in the tests directory is fine with me, but having every test suite be within tests makes more sense from a consistency standpoint.

Is there a way to independently run tests on the perftest without a project? If we can do away with needing to have a full (if minimal) Django project at all here that would be ideal, but I couldn't find a way to do so with how the other tests here are set up.

timgraham · 2025-08-18T23:37:22Z

I'm inclined to agree that the performance tests should be in their own directory performance_tests or test_performance? There not meant to be run like the other tests, so to intermingle them seems confusing (and requires workarounds in the test runner, as you added).

Probably integrating with Django's runtests.py isn't the way to go, but I'll think about it. At least the settings file you added could be minimized. I could do that without much effort. I have a few comments from a cursory glance and will dig in more later.

timgraham · 2025-08-18T23:21:33Z

+class TestSmallFlatDocCreation(SmallFlatDocTest, TestCase):
+    def do_task(self):
+        for doc in self.documents:
+            model = SmallFlatModel(**doc)
+            model.save()
+
+    def after(self):
+        SmallFlatModel.objects.all().delete()
+


I'm a little uncertain how this whole thing fits together. On the hand, you inherit django.test.TestCase which has its own setUp/tearDown strategy, on the other hand, you seem to implement your scheme with an after() method. Some classes use after(), others tearDown()... is it intentional? (edit: perhaps answered by the benchmark spec.)

The ODM benchmark spec (draft PR here: mongodb/specifications#1828) is helpful here, but the short answer is yes, we need the ability to do setup or teardown both before and after every iteration of a test as well as the entire set of iterations.

Jibola · 2025-08-19T14:11:32Z

+# Install django-mongodb-backend
+/opt/python/3.10/bin/python3 -m venv venv
+. venv/bin/activate


NIT:
Whilst I'd rather not have a new dependency on drivers-evergreen-tools, to future-proof the binary usage, you could add in the find_python function

This I copied over from the existing run-tests.sh script here. Let's update them both in a separate ticket.

We're going to bring in drivers-evergreen-tools anyway for QE

Jibola · 2025-12-16T17:22:11Z

+
+
+def handle_perf(start_time: datetime):
+    end_time = datetime.now()


For duration calculations, it's better practice to use time.monotonic(); any particular reason to use datetime?

This is copied from how PyMongo handles it for consistency. I believe we use it there for ease of the different ways we use the timestamps.

Jibola · 2025-12-16T17:56:58Z

I will give this file another look over -- only part remaining to be reviewed for me.

timgraham · 2026-01-04T02:06:55Z

+result_data: list = []
+
+
+def tearDownModule():
+    output = json.dumps(result_data, indent=4)
+    if OUTPUT_FILE:
+        with open(OUTPUT_FILE, "w") as opf:  # noqa: PTH123
+            opf.write(output)
+    else:
+        print(output)  # noqa: T201


Uff... I wonder if this blocks splitting up the tests into multiple files? At least it may need to become "append aware."

Yes, splitting into multiple files is complicated by our infrastructure expecting a single JSON output file for the entire suite.

After I suggested a default for OUTPUT_FILE, I now see that would break this if/else. Maybe we just remove OUTPUT_FILE from the documentation on how to run the tests? And actually, perhaps it's appropriate to remove this print statement also as I imagine the test by test output would be more useful than the JSON version meant for machine consumption?

The JSON version is a nice simple summary that's useful for local use as well. Setting a default in the run-perf-tests.sh script that CI/CD uses makes sense, but we shouldn't set a default in the actual test suite iteself.

I'm not going to argue, but my feeling is that:
TestSmallFlatDocUpdate 0.007 MB/s, MEDIAN=4.282s, total time=8.648s, iterations=2
is much easier to read than:

{ "info": { "test_name": "SmallFlatDocUpdate" }, "metrics": [ { "name": "megabytes_per_sec", "type": "MEDIAN", "value": 0.007472334166114841, "metadata": { "improvement_direction": "up", "measurement_unit": "megabytes_per_second" } } ] }

and to get to the former, I have to scroll up past the huge JSON dump. (Of course, nothing is stopping someone from adding or removing that print statement while testing their changes.)

Incidentally, the JSON results don't include the elapsed time. Is it correct that we won't track that metric? The spec says, "In addition to latency data, ..."

Where can details about the metadata format (improvement_direction, etc.) be found?

We don't report elapsed time in the JSON metrics, correct. The JSON metrics are tracked by Evergreen's charting, and we want a single metric easily comparable across ODMs there. We record elapsed time individually as part of each suite run as a secondary metric if needed for investigation.

The metadata format is for Evergreen chart tracking.

timgraham · 2026-01-04T02:57:18Z

+    tasks:
+      - name: perf-tests


In the PR description you wrote, "The full suite run locally is expected to take approximately 10 minutes." On the latest Evergreen build for this PR, it took 35 minutes... is it expected? That's a bit slower than the rest of our builds (~20 minutes). Is this going to run for every push to every PR? It's probably unnecessary. Perhaps it could be a very fast version with very low iterations just to ensure these tests don't break.

That was for an earlier iteration that didn't fully implement the spec.

No, these tests are going to run once for each commit merged into master, not once per PR push.

So they won't run at all on PRs? What do you think about my point about running with one/low iterations on PRs to ensure a PR doesn't break the suite?

Since the suite runs public APIs only, I don't think having an extra suite just to test that the suite itself doesn't break is worth it. Anything that breaks this suite should be caught by other tests failing.

But the performance tests are running on this PR... Is there some change we need to make before merging this so that they don't run on PRs? What happens when we need to make a change to the performance tests? Do we temporarily modify some trigger to have them run on that PR, then revert the trigger before merging?

We'll have to update the Evergreen project settings to exclude the benchmark tests on PRs. Changes to the perf tests would require scheduling Evergreen patches to confirm changes work as expected.

timgraham · 2026-01-27T02:52:20Z

+Django MongoDB Backend uses a benchmarking suite to catch performance regressions.
+This suite is located in the top-level ``performance_tests`` directory of the Django MongoDB Backend repository.


I'd also like to see a discussion (linking to the benchmark spec as appropriate) about what this is benchmarking (both Python code and MongoDB queries), how to interpret the results (Mb/sec??), where do we view the ongoing results, etc.

I'm not sure what you mean here. We could add a link to the benchmark spec, which details the purpose and scope of the suite? There's already a link in the actual test file. Each run will have an evergreen chart attached to its run, there isn't a single place with all the results added dynamically.

Basically, how can I use this tool to catch performance regressions? Am I supposed to run it locally and compare before and after? Is there a histogram of results? Will regressions be automatically detected? How will we know if the results are statistically significant?

Django has https://github.com/django/django-asv/tree/main and results displayed at https://django.github.io/django-asv/ but it's not obvious how to interpret and use the results, so I have similar concerns here.

Is this supposed to catch performance regressions in Django MongoDB Backend, MongoDB, or both? How can we differentiate?

The spec answers some questions but it's a long document with a lot of unnecessary details. Maybe high-level "cheat sheet" would be useful or other ODMs as well.

I suppose not all questions need to be answered before merging this (perhaps some answers are unknown), but this is what I've been thinking about.

Short answer: https://jira.mongodb.org/browse/INTPYTHON-530 will set up automated alerting to be identical to our current PyMongo automated regression alerts. Here's a link to what the charting looks like for that, it'll look similar for Django: https://spruce.mongodb.com/task/mongo_python_driver_performance_benchmarks_perf_8.0_standalone_a89c5e3a89fa7e621f03fc9173067256a5fdc300_26_01_26_19_36_51/trend-charts?execution=0.

The expectation is that the benchmark suite will run up to once per day on commit merge and automatically alert us in Slack if any regressions are detected. No local runs are expected generally.

timgraham · 2026-02-03T01:10:01Z

+
+OUTPUT_FILE = os.environ.get("OUTPUT_FILE")
+
+if os.environ.get("FASTBENCH"):


Nit: It seems it would be more convenient for developers if "FASTBENCH" were the default and FULLBENCH were set by CI. Any objection to changing the default? Probably I should have put FASTBENCH=1 in my .bashrc by now because I keep forgetting to prefix test invokations with it, but why force devs to do that for the common case?

We don't expect devs to regularly run these locally, so the default case is the CI run of the full benchmarks.

timgraham · 2026-02-03T01:12:19Z

+            self.document = json_util.loads(data.read())
+
+        self.data_size = len(encode(self.document)) * self.num_docs
+        self.documents = [self.document.copy() for _ in range(self.num_docs)]


I notice this pattern of initializing self.documents but it seems wasteful of CPU/RAM to create all those identical documents.

Any reason not to refactor patterns like this:

for doc in self.documents: LargeFlatModel.objects.create(**doc)

to:

for _ in self.num_docs: LargeFlatModel.objects.create(**self.document)

?

Good catch! Since we're creating Model instances with the documents, we don't actually need more than the single document.

Here's some interesting data from making this change to the TestLargeFlatDocCreation benchmark:
before: 83.173 MB/s, MEDIAN=29.102s, total time=307.418s, iterations=10
after: 111.719 MB/s, MEDIAN=21.666s, total time=222.919s, iterations=10

It seems that iterating through a list is much costlier than iterating through a range.

I'm not a benchmarking guru, but to me, this perhaps highlights the importance of making benchmarks minimal so that we're benchmarking the backend and the database and not the efficiency of the benchmark task itself. In other words, it may be better to have many iterations of the same task rather than few iterations of a long task that has a loop within it. This is the approach taken for the benchmarks in Django's djangobench tool.

I offer this as food for thought, and I don't insist on revising the benchmarks prior to merging this.

timgraham · 2026-02-03T01:16:27Z

+class TestLargeFlatDocFilterPkByIn(LargeFlatDocTest, TestCase):
+    """Benchmark for filtering large flat documents using the __in operator for primary keys."""
+
+    def setUp(self):
+        super().setUp()
+        models = []
+        for doc in self.documents:
+            models.append(LargeFlatModel(**doc))
+        LargeFlatModel.objects.bulk_create(models)
+        self.ids = [model.id for model in models]
+
+    def do_task(self):
+        list(LargeFlatModel.objects.filter(id__in=self.ids))


I see 10 benchmaks in the spec and 12 benchmarks implemented. I think this is the extra one. Did I miss it? I ask especially because this seems like a very inefficient operation that perhaps should be avoided. At least in SQL, huge "in" clauses are often naive solutions by inexperienced devs. The better solution is to to instead include the filter criteria that generated the huge list of ids in the subsequent query.

If this is a likely operation for devs new to Django or MongoDB + Django, it's valuable to benchmark despite the inefficiency. Do we explicitly call out that this is a poor pattern in our docs?

The spec explicitly allows for extra benchmarks specific to each ODM, such as this one.

Django MongoDB Backend doesen't have any query optimization docs like Django's. I thought maybe that Django docs mentions it, but I don't see anything.

Unless my intuition is incorrect, it seems that a lookup with a list of 10,000 ids (num docs) would be slow.

I'm just giving feedback based on the review item you wrote, "Do the Django operations being performed conform to our expectations of how django-mongodb-backend should be used?" I would say, "No, this is stupid code."

If this operation is objectively inefficient for both SQL and MongoDB and we expect users to be aware of that, I'm fine removing it.

I modified the __in benchmarks to query with a subset of ids which is more realistic.

timgraham · 2026-02-03T01:21:06Z

+    class Meta:
+        indexes = [
+            models.Index(fields=["field1"], name="field1_idx"),
+        ]


Does the spec say anything about indexing (searching "index" didn't give any results)? I couldn't spot a reason why this field (and only this field) is indexed.

This is the field used for the update operation tests. We expect ODMs to index fields if doing so is the typical use-case for their library.

But the filter criteria for the update is the document's id. Naively, I wouldn't think an index on the value being updated would affect its performance (other than the the fact that the server needs to be incorporate the new value in the index, so the performance might be a bit worse, right?).

Is it important for the spec to dictate which fields should be indexed so that results can be fairly compared across ODMs?

A future revision of the spec will likely specify which fields must be indexed. This first version is largely about catching regressions within an ODM, not comparing across ODMs.

timgraham · 2026-02-03T01:22:26Z

+
+
+class StringEmbeddedModel(EmbeddedModel):
+    unique_field = models.CharField(max_length=100)


Should it have a unique constraint?

I don't think it's relevant in this case, as this field always contains a stringified ObjectId.

I ask because unique constraints may affect the query planner.

I'd rather leave it as without a unique constraint to catch a wider spread of user scenarios.

timgraham · 2026-02-23T19:04:38Z

+  - name: performance-benchmarks
+    display_name: Performance Benchmarks


I noticed "benchmark" and "performance test" used interchangeably. I doubt it will cause much confusion, but it creates some ambiguity and a need to search for both terms to find related things.

I'd say prefer "performance test" over "benchmark" …

Co-authored-by: Tim Graham <timograham@gmail.com>

changes addressed

NoahStapp requested a review from Jibola August 18, 2025 11:59

NoahStapp assigned aclark4life and unassigned aclark4life Aug 18, 2025

NoahStapp requested a review from aclark4life August 18, 2025 11:59

NoahStapp mentioned this pull request Aug 18, 2025

DRIVERS-2917 - Standardized Performance Testing of ODMs and Integrations mongodb/specifications#1828

Merged

3 tasks

timgraham reviewed Aug 19, 2025

View reviewed changes

aclark4life mentioned this pull request Aug 19, 2025

Trying solve Queryable Encryption CI #360

Closed

Jibola requested changes Aug 19, 2025

View reviewed changes

NoahStapp requested review from Jibola and timgraham August 20, 2025 13:51

NoahStapp marked this pull request as ready for review October 7, 2025 12:34

NoahStapp changed the title ~~DRIVERS-2917 - Standardized Performance Testing of ODMs and Integrations~~ INTPYTHON-847 - Standardize performance testing infrastructure Dec 10, 2025

Jibola previously requested changes Dec 16, 2025

View reviewed changes

NoahStapp requested a review from Jibola December 17, 2025 20:10

timgraham reviewed Jan 4, 2026

View reviewed changes

timgraham reviewed Jan 7, 2026

View reviewed changes

Comment thread tests/performance/perftest/tests.py Outdated

Comment thread tests/performance/perftest/tests.py Outdated

NoahStapp requested a review from timgraham January 7, 2026 17:01

timgraham reviewed Jan 8, 2026

View reviewed changes

Comment thread performance_tests/perftest/tests.py Outdated

NoahStapp requested a review from timgraham January 21, 2026 17:40

timgraham changed the title ~~INTPYTHON-847 - Standardize performance testing infrastructure~~ INTPYTHON-847 Add performance tests Jan 27, 2026

timgraham force-pushed the DRIVERS-2917 branch from bbf4547 to 92a680e Compare January 27, 2026 02:16

timgraham reviewed Jan 27, 2026

View reviewed changes

Comment thread .evergreen/run_perf_test.py Outdated

Comment thread docs/internals/contributing/performance-tests.rst Outdated

Comment thread performance_tests/perftest/tests.py Outdated

Comment thread performance_tests/perftest/tests.py Outdated

NoahStapp requested a review from timgraham January 28, 2026 15:14

timgraham force-pushed the DRIVERS-2917 branch 3 times, most recently from 1fffd97 to d26a8f2 Compare February 3, 2026 01:39

timgraham force-pushed the DRIVERS-2917 branch from d26a8f2 to d25c35d Compare February 3, 2026 14:44

timgraham reviewed Feb 3, 2026

View reviewed changes

timgraham force-pushed the DRIVERS-2917 branch from d25c35d to 2bb8e0d Compare February 4, 2026 23:31

timgraham reviewed Feb 6, 2026

View reviewed changes

Comment thread performance_tests/perf/base.py Outdated

timgraham force-pushed the DRIVERS-2917 branch from 612977c to 5184bf7 Compare February 23, 2026 15:47

timgraham reviewed Feb 23, 2026

View reviewed changes

NoahStapp commented Feb 24, 2026

View reviewed changes

Comment thread .evergreen/perf/runtests.py Outdated

INTPYTHON-847 Add performance tests

c122f5b

Co-authored-by: Tim Graham <timograham@gmail.com>

timgraham force-pushed the DRIVERS-2917 branch from 9bac8a5 to c122f5b Compare March 2, 2026 14:49

timgraham approved these changes Mar 2, 2026

View reviewed changes

timgraham merged commit bcbb896 into mongodb:main Mar 2, 2026
21 checks passed



		def handle_perf(start_time: datetime):
		end_time = datetime.now()

		Django MongoDB Backend uses a benchmarking suite to catch performance regressions.
		This suite is located in the top-level ``performance_tests`` directory of the Django MongoDB Backend repository.


		OUTPUT_FILE = os.environ.get("OUTPUT_FILE")

		if os.environ.get("FASTBENCH"):



		class StringEmbeddedModel(EmbeddedModel):
		unique_field = models.CharField(max_length=100)

		- name: performance-benchmarks
		display_name: Performance Benchmarks

Uh oh!

Conversation

NoahStapp commented Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Structure

Running Locally

Review Items

Uh oh!

aclark4life commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Structure

Uh oh!

NoahStapp commented Aug 18, 2025

Structure

Uh oh!

timgraham commented Aug 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NoahStapp commented Aug 14, 2025 •

edited

Loading

aclark4life commented Aug 18, 2025 •

edited

Loading

NoahStapp Jan 27, 2026 •

edited

Loading