Optimize some management endpoints via the querysets by MyPyDavid · Pull Request #1600 · rdmorganiser/rdmo

MyPyDavid · 2026-05-07T07:18:02Z

Description

Related issue: "I think we had an issue from even before the migration to the React interface"

Some managment pages were still loading slowly, one prefetch field default_option for the questions.Models was missing. This may improve some queries.

…and exports Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

…or catalog index Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

coveralls · 2026-05-07T08:02:14Z

Coverage is 94.984% — 2.4.5/fix/optimize-management-querysets into 2.4.5/release. No base build found for 2.4.5/release.

…sets Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

MyPyDavid · 2026-05-07T13:13:45Z

+        "queries": {
+            "index": 3,
+            "list": 9,
+            "detail": 9,


this mapper defines the max query counts per api endpoint action during a test.
Very simple test at the moment. They could be expanded by creating 10000 Options, Attributes, etc dynamically during test as well..

MyPyDavid · 2026-05-07T13:14:29Z

        'editors'
    )

+    def get_queryset(self):


nearly forgot the Tasks and Views ;)

jochenklar

I am a bit puzzled, why I cannot follow part of your optimizations, maybe we should talks next week.

jochenklar · 2026-05-10T08:59:16Z

+        queryset = Attribute.objects.all().order_by('path')
+        if self.action in ['index']:
+            return queryset
+        elif self.action in ('nested', 'export', 'detail_export'):


I don't think this is needed since we use the mptt tree here.

For the nested view, editors causes one query per Attribute to Site and this does not work with prefetch_related due to how the MPTT-Tree works. However we could just remove editors from the nested serializer.

jochenklar · 2026-05-10T09:18:07Z

+        elif self.action in ('nested', 'export', 'detail_export'):
+            return queryset.select_related('parent')
+        else:
+            return queryset.annotate(


This is actually not needed since the list serializer does not use the prefetched fields and for one object there is no gain. But it was like this before, so i would keep it. Just remove the parent.

jochenklar · 2026-05-10T09:19:29Z

+}
+
+
+def get_params(action):


I think the dict above could be in a shape that we dont need this function.

jochenklar · 2026-05-10T09:20:41Z

Should be named test_api_performance.py or test_max_num_queries.py. This is very interesting, I was not aware of django_assert_max_num_queries.

I think this should be distributed to the different django apps, which would reduce the complexity a bit.

jochenklar · 2026-05-10T09:35:04Z

        'catalog_sections__section__section_pages__page__page_questions__question__attribute',
        'catalog_sections__section__section_pages__page__page_questions__question__conditions',
        'catalog_sections__section__section_pages__page__page_questions__question__optionsets',
+        'catalog_sections__section__section_pages__page__page_questions__question__default_option',


I am not convinced prefetching the default_option is needed, since we only use its id in the serializer and do not resolve the instance.

And what about the QuestionExportSerializer? It uses default_option = serializers.CharField(source='default_option.uri', default=None, read_only=True) so default_option.uri needs to be resolved for exports right?! It would be used by all Export serializers up to the CatalogExport. Would it be worth to add the prefetch field only for export or apply it only for the export actions? Otherwise we can leave it out like you said.

rdmo/rdmo/questions/serializers/export.py

Line 26 in e77e2ca

default_option = serializers.CharField(source='default_option.uri', default=None, read_only=True)

So actually, the export of a Catalog via the management interface can make almost 3000(!) queries and debug toolbar crashed my machine a bit.
Is it because of a missing prefetch_elements call in the detail_export?

rdmo/rdmo/questions/viewsets.py

Line 97 in e77e2ca

serializer = CatalogExportSerializer(self.get_object())

jochenklar · 2026-05-10T09:38:36Z

                'questionsets',
                'editors'
-            ).select_related('attribute')
+            ).select_related('attribute', 'default_option')


Same as above, I don't think we use this.

…elated Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

…unctinos and select_related calls Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

MyPyDavid · 2026-05-13T09:09:18Z

so, these were the query counts from before the optimizations of the export and detail_export actions:

 Error: test_page_endpoints_query_counts[detail_export-13-url_kwargs4]

Failed: Expected to perform 13 queries or less but 20 were done (add -v option to show queries)
Error: test_views_endpoints_query_counts[detail_export-10-url_kwargs4]

Failed: Expected to perform 10 queries or less but 12 were done (add -v option to show queries)
Error: test_page_endpoints_query_counts[export-25-url_kwargs2]

Failed: Expected to perform 25 queries or less but 409 were done (add -v option to show queries)
Error: test_question_endpoints_query_counts[export-12-url_kwargs2]

Failed: Expected to perform 12 queries or less but 398 were done (add -v option to show queries)
Error: test_optionset_endpoints_query_counts[detail_export-10-url_kwargs4]

Failed: Expected to perform 10 queries or less but 14 were done (add -v option to show queries)
Error: test_actions_max_query_counts[export-11-url_kwargs2]

Failed: Expected to perform 11 queries or less but 26 were done (add -v option to show queries)
Error: test_actions_max_query_counts[detail_export-9-url_kwargs4]

Failed: Expected to perform 9 queries or less but 17 were done (add -v option to show queries)
Error: test_catalog_endpoints_query_counts[detail_export-29-url_kwargs4]

Failed: Expected to perform 29 queries or less but 425 were done (add -v option to show queries)
Error: test_catalog_endpoints_query_counts[export-30-url_kwargs2]

Failed: Expected to perform 30 queries or less but 408 were done (add -v option to show queries)
.F.........F..F..............FF........................F.FF.........F... [ 99%]
Error: test_questionset_endpoints_query_counts[export-17-url_kwargs2]

Failed: Expected to perform 17 queries or less but 70 were done (add -v option to show queries)
Error: test_questionset_endpoints_query_counts[detail_export-13-url_kwargs4]

Failed: Expected to perform 13 queries or less but 26 were done (add -v option to show queries)
Error: test_section_endpoints_query_counts[export-27-url_kwargs2]

Failed: Expected to perform 27 queries or less but 406 were done (add -v option to show queries)
Error: test_section_endpoints_query_counts[detail_export-12-url_kwargs4]

Failed: Expected to perform 12 queries or less but 59 were done (add -v option to show queries)
Error: test_domain_endpoints_query_counts[detail_export-12-url_kwargs4]

Failed: Expected to perform 12 queries or less but 30 were done (add -v option to show queries)
Error: test_domain_endpoints_query_counts[export-12-url_kwargs2]

Failed: Expected to perform 12 queries or less but 160 were done (add -v option to show queries)
Error: test_domain_endpoints_query_counts[index-3-None]

KeyError: 'index'
Error: test_domain_endpoints_query_counts[list-10-None]

Failed: Expected to perform 10 queries or less but 114 were done (add -v option to show queries)
Error: test_tasks_endpoints_query_counts[detail_export-12-url_kwargs4]

Failed: Expected to perform 12 queries or less but 16 were done (add -v option to show queries)
...F..F.FF.F.F.FF...........F........................................... [ 99%]

For a catalog 😅 :
Expected to perform 29 queries or less but 425 were done

MyPyDavid added 2 commits May 7, 2026 09:13

Add select_related for Attribute queryset and split for index,nested …

aa71f3a

…and exports Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Add default_option to questions models prefetch and do not annotate f…

f5582c3

…or catalog index Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

MyPyDavid added this to the RDMO 2.4.5 milestone May 7, 2026

MyPyDavid self-assigned this May 7, 2026

MyPyDavid changed the title ~~Optimize management querysets~~ Optimize some management querysets May 7, 2026

MyPyDavid changed the base branch from main to 2.4.5/release May 7, 2026 07:47

MyPyDavid added 3 commits May 7, 2026 14:23

Do not prefetch for the index action on conditions,options and option…

ad813f2

…sets Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Also do not prefetch for the index action on views and tasks

f1ec6c9

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Add simple performance tests for query counts on api actions

1a38bf6

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

MyPyDavid commented May 7, 2026

View reviewed changes

jochenklar requested changes May 10, 2026

View reviewed changes

Remove editors from AttributeViewSet prefetch and do not use select_r…

757e139

…elated Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

MyPyDavid changed the title ~~Optimize some management querysets~~ Optimize some management endpoints via the querysets May 12, 2026

MyPyDavid added 5 commits May 13, 2026 10:30

Add tests for api performance of all elements

248e5bb

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Clean up tests

4bf0ba3

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Optimize queries for export and detail_export actions on elements

0ef1b74

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Add test for index attributes action

5f4c0a8

Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Refactor and optimize prefetch lookups for questions with recursive f…

1140b53

…unctinos and select_related calls Signed-off-by: David Wallace <david.wallace@tu-darmstadt.de>

Conversation

MyPyDavid commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

coveralls commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MyPyDavid May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jochenklar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MyPyDavid May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MyPyDavid commented May 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

MyPyDavid commented May 7, 2026 •

edited

Loading

coveralls commented May 7, 2026 •

edited

Loading

MyPyDavid May 7, 2026 •

edited

Loading

MyPyDavid May 11, 2026 •

edited

Loading