perf: Parallelize bundled app install at startup

[david-mackessy]

Summary

1. Bundled App installs on 1st startup (no apps exist) are now installed in parallel (apps & files), starting with the largest apps first
2. App updates use parallel file uploads as a result (small bonus, most noticeable for the larger apps)

This change was prompted after AWS S3 (filestore provider) testing during JClouds replacement.

Installation on AWS S3 goes from ~30mins to ~1min 15seconds.

• Up to 16 apps install in parallel
• Up to 32 file uploads concurrently (IO bound, not CPU bound)
  • tested 64 but no real difference in results (32 seems to be the sweet spot)

---
title: New Bundled App Installation Process
---
flowchart TD
    M[main thread<br/>startup] --> A0[app-install-0<br/>Maintenance]
    M --> A1[app-install-1<br/>Metadata Mgmt]
    M --> A2[app-install-...<br/>up to 16 apps in parallel,<br/>largest first]
    A0 --> P
    A1 --> P
    A2 --> P
    P[shared upload pool<br/>32 concurrent file uploads<br/>bounded across all apps] --> S[(S3 / file store)]

Loading

Performance

Iterations

Several iterations resulted in reduced upload times as follows:

Improvement	From	To
+ Upload app files in parallel	~30 mins	~7 mins
+ skip old app removal on empty store	~7 mins	~4 mins
+ Upload apps in parallel	~4 mins	~2.5 mins
+ Upload largest apps first	~2.5 mins	~1 min

App startup

The app will now use more CPU on startup and slightly more RAM but takes far less time to startup on first install when using file storage over the network.

Before

• ~30 mins to install bundled apps
• hardly any CPU utilisation during install
• ~150 to ~300 MB RAM during install
• ~20 threads during install

After

• ~1 min to install bundled apps
• more utilisation of CPU
• ~300 to ~500 MB RAM during install
• ~68 threads during install

App file count

Total: 8,252 files across 30 bundled apps.

Files	App
2,882	maintenance
1,829	metadata-management
1,133	maps
443	event-visualizer
443	event-reports
167	aggregate-data-entry
97	capture
94	data-visualizer
87	user
61	line-listing
58	dashboard
55	user-profile, reports
54	translations, settings, data-quality
52	import-export
50	scheduler, messaging, menu-management, global-shell, data-administration, app-management
49	usage-analytics, sms-configuration, login, datastore, cache-cleaner, approval
39	interpretation

Testing

Mac M2 Pro 16GB
Network: 80 Mbps download, 25 Mbps upload

Tested filesystem, MinIO, transient & AWS S3

• first install
• subsequent startup (apps already installed)
• updated app
• downgraded app

[codecov]

Codecov Report

❌ Patch coverage is 37.03704% with 51 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.95%. Comparing base (17f8133) to head (e600736).
⚠️ Report is 22 commits behind head on master.

Files with missing lines	Patch %	Lines
...va/org/hisp/dhis/appmanager/DefaultAppManager.java	10.86%	39 Missing and 2 partials ⚠️
...sp/dhis/appmanager/BlobStoreAppStorageService.java	71.42%	8 Missing and 2 partials ⚠️

❗ There is a different number of reports uploaded between BASE (17f8133) and HEAD (e600736). Click for more details.

HEAD has 3 uploads less than BASE

Flag	BASE (17f8133)	HEAD (e600736)
integration	4	2
integration-h2	1	0

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #24109       +/-   ##
=============================================
- Coverage     69.04%   56.95%   -12.09%     
+ Complexity      709      618       -91     
=============================================
  Files          3684     3686        +2     
  Lines        141646   142012      +366     
  Branches      16453    16517       +64     
=============================================
- Hits          97797    80887    -16910     
- Misses        36243    54285    +18042     
+ Partials       7606     6840      -766     

Flag	Coverage Δ
integration	39.94% <37.03%> (-9.71%)	⬇️
integration-h2	?
unit	34.98% <3.70%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...sp/dhis/appmanager/BlobStoreAppStorageService.java	54.86% <71.42%> (ø)
...va/org/hisp/dhis/appmanager/DefaultAppManager.java	26.20% <10.86%> (-21.67%)	⬇️

... and 1058 files with indirect coverage changes

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5a1b89b...e600736. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
1 Accepted issue

Measures
1 Security Hotspot
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[jbee]

👍