Skip to content

chore: Extract FileResource assignment update [DHIS2-21378]#23879

Merged
enricocolasante merged 4 commits into
masterfrom
DHIS2-21378-file-resource
May 21, 2026
Merged

chore: Extract FileResource assignment update [DHIS2-21378]#23879
enricocolasante merged 4 commits into
masterfrom
DHIS2-21378-file-resource

Conversation

@enricocolasante

@enricocolasante enricocolasante commented May 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Extracts the FileResource assignment update during tracker import out of the Hibernate persistence context and into a direct JDBC UPDATE. This avoids dirty-marking the entity and prevents PostUpdateAuditListener (and other Hibernate audit listeners) from firing for every file-resource assignment during a tracker import.

  • Adds FileResourceStore.updateAssignment(uid, assigned, fileResourceOwner) backed by JdbcTemplate, setting isassigned, fileresourceowner, and lastupdated = now() in a single statement.
  • Replaces entityManager.merge(fileResource) in AbstractTrackerPersister#assignFileResource with the new store method. The in-memory copy in the preheat is intentionally left as-is — nothing reads it back later in the import.
  • Threads FileResourceStore through all concrete persisters: EnrollmentPersister, RelationshipPersister, SingleEventPersister, TrackedEntityPersister, TrackerEventPersister.

Performance

25726592602
25726590062
25726586827
25726584415

compare mode: 4 baseline run(s), 4 candidate run(s) on load profile

Median Response Time (p50) (ms)

Requests baseline req/s candidate req/s Diff (ms) Change
Login 119 0.37 108 0.36 -11 ⬇️ -9.2%
MNCH import 1,046 1.84 1,050 1.80 +4 ⬆️ +0.4%
Child Programme import 465 1.84 466 1.80 +0 ⬆️ +0.1%
ANC import 362 1.84 364 1.80 +2 ⬆️ +0.7%

95th Percentile Response Time (p95) (ms)

Requests baseline req/s candidate req/s Diff (ms) Change
Login 154 0.37 134 0.36 -20 ⬇️ -13.0%
MNCH import 1,896 1.84 2,009 1.80 +114 ⬆️ +6.0%
Child Programme import 508 1.84 510 1.80 +2 ⬆️ +0.4%
ANC import 649 1.84 520 1.80 -129 ⬇️ -19.9%

⬇️ = faster, ⬆️ = slower

Next steps

  • Quick fixformatDate: SimpleDateFormatstatic final DateTimeFormatter
  • Phase 0 ✅ Remove EntityManager from TrackerPersister interface and all method parameters - feat: decouple ChangeLogAccumulator from Hibernate Session [DHIS2-21378] #23836
  • Phase 1 ✅ Decouple ChangeLogAccumulator from Hibernate Session; obtain the
    transaction-bound Connection via DataSourceUtils instead of session.doWork()
  • Phase 2 ✅ Extract FileResource assignment update out of AbstractTrackerPersister
    into FileResourceStore
  • Phase 3 — Introduce EntityWriteBatch; accumulate and batch-flush TEAV writes
    (multi-row INSERT / unnest UPDATE / tuple DELETE)
  • Phase 4 — Pre-allocate IDs from PostgreSQL sequences; stage top-level entity
    inserts and updates in EntityWriteBatch; remove all entityManager.flush() calls
  • Phase 5 — Replace updateTrackedEntitiesLastUpdated named query with
    NamedParameterJdbcTemplate; remove last Hibernate dependency from
    DefaultTrackerBundleService
  • Phase 6 — Implement JDBC flush methods one entity type at a time
    (TrackedEntityEnrollmentTrackerEvent/SingleEventRelationship)

@enricocolasante
enricocolasante force-pushed the DHIS2-21378-file-resource branch from cf6e2a6 to 68da439 Compare May 12, 2026 09:43
@enricocolasante
enricocolasante force-pushed the DHIS2-21378-file-resource branch from 68da439 to 6e51039 Compare May 12, 2026 12:52
@enricocolasante
enricocolasante marked this pull request as ready for review May 15, 2026 07:22
@enricocolasante
enricocolasante requested a review from a team as a code owner May 15, 2026 07:22
List<TrackedEntityAttributeValue> attributeValues =
trackedEntityAttributeValueService.getTrackedEntityAttributeValues(trackedEntity);
assertEquals(5, attributeValues.size());
manager.clear();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are these needed now?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and if you clear before flush I think there is nothing to flush

@teleivo
teleivo requested a review from a team May 15, 2026 11:58
@enricocolasante
enricocolasante enabled auto-merge (squash) May 21, 2026 12:54
@sonarqubecloud

Copy link
Copy Markdown

@enricocolasante
enricocolasante merged commit 83542f4 into master May 21, 2026
18 checks passed
@enricocolasante
enricocolasante deleted the DHIS2-21378-file-resource branch May 21, 2026 13:10
enricocolasante added a commit that referenced this pull request Jul 7, 2026
… (2.43) (#24286)

* feat: remove EntityManager from TrackerPersister [DHIS2-21378] (#23797)

* feat: remove EntityManager from TrackerPersister interface and method parameters [DHIS2-21378]

* Use default timezone

(cherry picked from commit 052d3ac)

* feat: decouple ChangeLogAccumulator from Hibernate Session [DHIS2-21378] (#23836)

(cherry picked from commit 0c1779f)

* chore: Extract FileResource assignment update [DHIS2-21378] (#23879)

* chore: Extract FileResource assignment update [DHIS2-21378]

* Code review fixes

* Code review fixes

(cherry picked from commit 83542f4)

* feat: stage TEAV writes through EntityWriteBatch [DHIS2-21378] (#23970)

(cherry picked from commit b30c1ce)

* feat: Batch entity when persisting [DHIS2-21378] (#24005)

(cherry picked from commit 100dff1)

* feat: remove EntityManager from DefaultTrackerBundleService [DHIS2-21378] (#24024)

(cherry picked from commit 3759453)

* feat: TrackedEntity JDBC insert [DHIS2-21378] (#24027)

(cherry picked from commit 4eaf6ba)

* feat: TrackedEntity JDBC update [DHIS2-21378] (#24079)

(cherry picked from commit d4af287)

* feat: Enrollment JDBC insert/update [DHIS2-21378] (#24081)

* feat: TrackedEntity JDBC update [DHIS2-21378]

* feat: Enrollment JDBC insert/update [DHIS2-21378]

* Remove L2 caching from enrollment

* Code review fixes

* Code review fixes

(cherry picked from commit 34d629e)

* feat: Tracker event JDBC insert/update [DHIS2-21378] (#24082)

* feat: TrackedEntity JDBC insert [DHIS2-21378]

* feat: TrackedEntity JDBC update [DHIS2-21378]

* feat: Enrollment JDBC insert/update [DHIS2-21378]

* feat: Tracker event JDBC insert/update [DHIS2-21378]

(cherry picked from commit 435872b)

* feat: Single event JDBC insert/update [DHIS2-21378] (#24083)

* feat: TrackedEntity JDBC insert [DHIS2-21378]

* feat: TrackedEntity JDBC update [DHIS2-21378]

* feat: Enrollment JDBC insert/update [DHIS2-21378]

* feat: Tracker event JDBC insert/update [DHIS2-21378]

* feat: Single event JDBC insert/update [DHIS2-21378]

* Format code

(cherry picked from commit 9703e4f)

* feat: Relationship JDBC insert/update [DHIS2-21378] (#24093)

* feat: TrackedEntity JDBC insert [DHIS2-21378]

* feat: TrackedEntity JDBC update [DHIS2-21378]

* feat: Enrollment JDBC insert/update [DHIS2-21378]

* feat: Tracker event JDBC insert/update [DHIS2-21378]

* feat: Single event JDBC insert/update [DHIS2-21378]

* feat: Relationship JDBC insert/update [DHIS2-21378]

(cherry picked from commit 658f0a8)

* feat: TEAV JDBC insert/update [DHIS2-21378] (#24098)

* feat: TrackedEntity JDBC update [DHIS2-21378]

* feat: Enrollment JDBC insert/update [DHIS2-21378]

* feat: Tracker event JDBC insert/update [DHIS2-21378]

* feat: Single event JDBC insert/update [DHIS2-21378]

* feat: Relationship JDBC insert/update [DHIS2-21378]

* feat: TEAV JDBC insert/update [DHIS2-21378]

* Code review fixes

* Fix tests

* Fix idScheme issue for TEAV

(cherry picked from commit 4e3367c)

* fix: Refactor persistence layer in tracker importer [DHIS2-21378] (#24197)

* feat: TrackedEntity JDBC update [DHIS2-21378]

* feat: Enrollment JDBC insert/update [DHIS2-21378]

* feat: Tracker event JDBC insert/update [DHIS2-21378]

* feat: Single event JDBC insert/update [DHIS2-21378]

* feat: Relationship JDBC insert/update [DHIS2-21378]

* feat: TEAV JDBC insert/update [DHIS2-21378]

* Code review fixes

* Fix tests

* Fix idScheme issue for TEAV

* fix: Refactor persistence layer in tracker importer [DHIS2-21378]

* Fix merge

(cherry picked from commit 194dcd1)

* fix: Harmonize db sequences for tracker [DHIS2-21378] (#24208)

* fix: Harmonize db sequences for tracker [DHIS2-21378]

* Fix relationshipitemid

(cherry picked from commit f52eefa)

* fix: Rename conflicting migration [DHIS2-21378] (#24218)

(cherry picked from commit 7aefc1c)

No-op on 2.43: this PR renamed V2_44_12 -> V2_44_13 to avoid a master-only
Flyway version clash. On 2.43 the migration is numbered V2_43_63 (its own
series), so no rename is needed. Empty commit kept to preserve the full
DHIS2-21378 backport history.

* refactor: Split EntityWriteBatch into per-entity writers [DHIS2-21378] (#24227)

* refactor: Split EntityWriteBatch into per-entity writers [DHIS2-21378]

* chore: Migrate ownership from Hibernate to JDBC [DHIS2-21378]

* Code review fixes

(cherry picked from commit 331503a)

* chore: Migrate ownership from Hibernate to JDBC [DHIS2-21378] (#24239)

* refactor: Split EntityWriteBatch into per-entity writers [DHIS2-21378]

* chore: Migrate ownership from Hibernate to JDBC [DHIS2-21378]

(cherry picked from commit 7ba2431)

* perf: Batch existing-TEAV lookup in tracker import [DHIS2-21378] (#24244)

* refactor: Split EntityWriteBatch into per-entity writers [DHIS2-21378]

* chore: Migrate ownership from Hibernate to JDBC [DHIS2-21378]

* Code review fixes

* perf: Batch existing-TEAV lookup in tracker import [DHIS2-21378

* Code review fixes

* Code review fixes

(cherry picked from commit 2972a82)

* perf: Constant-text unnest INSERTs in tracker import [DHIS2-21378] (#24250)

* refactor: Split EntityWriteBatch into per-entity writers [DHIS2-21378]

* chore: Migrate ownership from Hibernate to JDBC [DHIS2-21378]

* Code review fixes

* perf: Batch existing-TEAV lookup in tracker import [DHIS2-21378

* perf: Constant-text unnest INSERTs in tracker import [DHIS2-21378]

* perf: Cache UserInfoSnapshot JSON per batch in tracker import [DHIS2-21378]

* Revert "perf: Cache UserInfoSnapshot JSON per batch in tracker import [DHIS2-21378]"

This reverts commit 7084379.

* Add tests to cover fields import

(cherry picked from commit 7305f9f)

* perf: Cache UserInfoSnapshot JSON per batch in tracker import [DHIS2-21378] (#24252)

* refactor: Split EntityWriteBatch into per-entity writers [DHIS2-21378]

* chore: Migrate ownership from Hibernate to JDBC [DHIS2-21378]

* Code review fixes

* perf: Batch existing-TEAV lookup in tracker import [DHIS2-21378

* perf: Constant-text unnest INSERTs in tracker import [DHIS2-21378]

* perf: Cache UserInfoSnapshot JSON per batch in tracker import [DHIS2-21378]

* Fix duration based Tracker perf tests

* Fix formatting

(cherry picked from commit a03204c)

* fix: Preserve storedby on tracker JDBC writers for 2.43 [DHIS2-21378]

Backport adaptation. On master, DHIS2-21537 removed storedBy from the
TrackedEntity/Enrollment/Event write path (and renamed TEAV storedBy to
updatedBy). 2.43 deliberately keeps storedBy, so the JDBC writers backported
from master must persist the storedby column that 2.43's tables still have.

TrackerObjectsMapper still sets storedBy on the new-entity branch for TE,
enrollment and events, so storedby is insert-only here (mirrors createdby):
added the storedby column + bind to the INSERT of TrackedEntityWriter,
EnrollmentWriter, SingleEventWriter and TrackerEventWriter. TeavWriter was
already adapted (storedby column + getStoredBy) when the earlier commits were
cherry-picked. Keeps the 2.43-only shouldSetStoredByToAuthenticatedUser* tests
green.

* Fix note persister

* Fix formatting
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants