perf: DataEntry metadata endpoint with fewer n+1s [DHIS2-21757](2.42)

[david-mackessy]

/api/dataEntry/metadata — Multiple N+1 query reduction (2.42)

Endpoint: DataSetMetadataController.getMetadata →
DefaultDataSetMetadataExportService.getDataSetMetadata().

Problem

The endpoint builds the full data-entry metadata payload (all writable data sets + their data
elements, category combos, category option combos, categories, options, option sets). Several
associations were initialised lazily one primary key at a time, producing large N+1 query
storms. On an implementation production instance the top three query shapes alone were ~86% of all
statements.

Fixes

Fix A — batch the data-element graph (new service method + query)

getDataElementsByDataSet(Collection<DataSet>) loads the data elements of the exported data sets in
a single query, eagerly fetching each element's dataSetElements collection and its categoryCombo:

select distinct de from DataSetElement dse
join dse.dataElement de
left join fetch de.dataSetElements
left join fetch de.categoryCombo
where dse.dataSet in :dataSets

Replaces flatMapToSet(dataSets, DataSet::getDataElements) at the call site. Kills the per-element
dataelement and datasetelement loads and most of the per-combo categorycombo loads.

Files: DataSetStore, HibernateDataSetStore, DataSetService, DefaultDataSetService,
DefaultDataSetMetadataExportService.

Fix B — preload category-option-combo options (new service method + query)

Fixes the categoryoptioncombos_categoryoptions N+1 — CategoryOptionCombo.getCategoryOptions()
loaded once per category option combo.
getCategoryOptionCombosWithCategoryOptions(Collection<CategoryCombo>) primes the session so
field-filter serialization of categoryOptionCombos[...,categoryOptions~pluck[id]] does not fire a
per-combo N+1:

select distinct coc from CategoryCombo cc
join cc.optionCombos coc
left join fetch coc.categoryOptions
where cc in :categoryCombos

Called for the data-element category combos before the payload is serialized.

Files: CategoryOptionComboStore, HibernateCategoryOptionComboStore, CategoryService,
DefaultCategoryService, DefaultDataSetMetadataExportService.

Fixes don't impede on existing ACL checks at the start of the service call for entire scope of feature:

• idObjectManager.getDataWriteAll(DataSet.class)

Implementation in production — top-10 query shapes (measured pre-fix)

Pre-fix counts from the implementation's production instance. Not re-measured post-fix — the projected
effect below is inferred from the local dev before/after run, which eliminates the same query shapes.

#	Query (lazily loaded)	Execs	% of all	Fix
1	datasetelement WHERE dataelementid=? (DataElement.dataSetElements inverse)	3,566	35.0%	✅ Fix A
2	dataelement WHERE dataelementid=? (DataElement proxy per DSE)	3,062	30.0%	✅ Fix A
3	categoryoptioncombos_categoryoptions WHERE categoryoptioncomboid=? (COC → its category options — the "139 COCs")	2,187	21.4%	✅ Fix B
4	datasetelement WHERE datasetelementid=? (DSE by PK)	591	5.8%	✅ Fix A (indirect)
5	categoryoption_organisationunits WHERE categoryoptionid=?	258	2.5%	❌ not addressed
6	categorycombos_categories WHERE categorycomboid=?	111	1.1%	❌ not addressed
7	categorycombo WHERE categorycomboid=? (by PK)	111	1.1%	⚠️ partial (Fix A covers DE combos)
8	categories_categoryoptions WHERE categoryid=?	97	1.0%	❌ not addressed
9	categorycombos_optioncombos WHERE categorycomboid=?	88	0.9%	❌ not addressed
10	userinfo WHERE userinfoid=? (createdBy/lastUpdatedBy)	57	0.6%	❌ not addressed

Fixes target #1 + #2 + #3 + #4 (+ part of #7) ≈ 92% of all queries — the implementation's "top three"
plus the DSE-by-PK loads. The local dev run below directly confirms #1, #3 and the category-combo
loads collapse to single queries; #2 and #4 resolve by the same batched-fetch mechanism, so the
implementation's equivalents are expected to resolve equivalently in production.

Not addressed (small individually; would need broader fetch-joins / @BatchSize, out of scope).
Note these are different associations from the COC→category-options join fixed by Fix B (#3),
despite the similar names:

• #5 categoryoption_organisationunits — CategoryOption → org units
• Adjusted timestamps cache in mem value #6 categorycombos_categories — CategoryCombo → its categories
• revert expiry days property range for now #8 categories_categoryoptions — Category → its category options
• Remove pre-auth check for 'ALL' on api/dataSets/<uid>/form.  #9 categorycombos_optioncombos — CategoryCombo → its option combos
• Switch to new user profile app #10 userinfo — createdBy / lastUpdatedBy per entity

Local dev result — Sierra Leone DB (measured before → after)

Profiled on this branch (/api/dataEntry/metadata, cold first call after restart), fixes off vs on.
These are the numbers we directly measured and can stand behind.

N+1 query (loaded one PK at a time)	Before	After
datasetelement WHERE dataelementid=? (COC / getCategoryCombos() inverse)	601	0
categoryoptioncombos_categoryoptions WHERE categoryoptioncomboid=? (the "139 COCs")	139	0
categorycombo WHERE categorycomboid=?	22	5
new — batched data-element load (Fix A)	—	1 (634 rows)
new — COC category-options preload (Fix B)	—	1 (247 rows)
Total statements (approx.)	~1,080	~324

The two N+1 storms collapse into two single fetch-join queries and categorycombo drops to the 5
data-set combos — ~70% fewer statements, with nothing left that scales per data element / data
set element / category option combo.

After the fixes the endpoint is dominated by two single, non-N+1 queries, out of scope for
batching (data-volume, not query-count):

• agg_ou_uid org-unit associations — 1 execution, ~66 ms.
• categoryoption → org units — 1 execution, 4,664 rows, ~60 ms.

The residual counts are the expected per-data-set (×26) and per-section (×21) collection loads.

Note on dev vs prod: the standalone dataelement WHERE dataelementid=? N+1 (implementation prod #2)
does not appear on this dev DB — on 2.42 the data elements load inside section join-fetches (see
root-cause note below), so dev cannot demonstrate #2 directly. Fix A still loads them explicitly, and
the implementation's #2 resolves by the same batched-fetch mechanism as #1.

Note on the 2.42-vs-2.44 "one N+1 vs two" difference (root-cause record)

Early profiling showed 2.42 exposing only the datasetelement loop while 2.44 also exposed the
dataelement loop. This was not caching and not a difference in the export code — the
export path is effectively identical. It is a fetch-strategy difference from the hbm→annotation
entity mapping migration:

• 2.42 (hbm): Section.dataElements (and the operand collections) join-fetch the full
  DataElement entity — SQL-confirmed: DataElement columns appear only inside
  from sectiondataelements … inner join dataelement …, never as standalone where dataelementid=?.
  So the entities load inside collection queries and no standalone dataelement N+1 shows.
• 2.44 (annotations): the equivalent associations are lazy with no join-fetch, so the same
  access loads each DataElement standalone → the visible dataelement N+1.

Verified with a Hibernate L2 probe (DataElement region cold on the first call after restart:
miss=601, put=601, hit=0) and startup SQL logs. The N+1 is latent in both versions; whether it
manifests as a standalone loop depends on the mapping's fetch strategy. Either way, the fixes load
everything explicitly and deterministically, independent of mapping/cache behaviour.

Testing

Both fixes have an endpoint-level regression guard (they call getDataSetMetadata() and count
queries), each verified to fail if its fix is removed:

Test	Type	Guards	Verified fails without fix
dataElementCountDoesNotScaleQueryCount	query count (aggregate, scaling)	Fix A	✅ (30 → 40 selects)
categoryOptionComboOptionsAreNotLoadedPerCombo	query count (per-pattern ≤ 1)	Fix B	✅ ("join table queried 5 times")
dataEntryMetadataReturnsDistinctDataElements	content	output unchanged (dedup, combo, option set)	n/a — output guard
dataEntryMetadataReturnsCategoryComboOverrides	content	output unchanged (override, all combos)	n/a — output guard

Per-pattern counting is added to QueryCountDataSourceProxy aggregate counting.

Test files: DataSetMetadataExportServiceQueryCountTest, (dhis-test-integration); DataSetMetadataControllerTest (+2 methods) and
dataset/dataentry_metadata_query_coverage.json fixture (dhis-test-web-api);
net.ttddyy:datasource-proxy test dependency added to dhis-test-integration/pom.xml.

** Manual testing of the endpoint response against SL DB produced the same json response data (before v after)

I tried to replicate the DB/query shape from the production implementation. It's almost impossible to get like for like but this is as close as I got. The main culprits were reproducible and are gone. 42-dev v fix branch

Query shape	2.42 branch	Fix branch
1 datasetelement WHERE dataelementid	3,566	0
2 categoryoptioncombos_categoryoptions	2,187	0
3 categorycombos_categories	112	112
4 categorycombos_optioncombos	111	111
5 categorycombo (by PK)	111	0

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[jason-p-pickering]

💘