Skip to content

feat: Batch entity when persisting [DHIS2-21378]#24005

Merged
enricocolasante merged 1 commit into
masterfrom
DHIS2-21378-te-stage
May 28, 2026
Merged

feat: Batch entity when persisting [DHIS2-21378]#24005
enricocolasante merged 1 commit into
masterfrom
DHIS2-21378-te-stage

Conversation

@enricocolasante

@enricocolasante enricocolasante commented May 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Phase 3 of the tracker import Hibernate → JDBC migration (DHIS2-21378). Extends EntityWriteBatch (introduced in
#23970) to also stage top-level entity writes — TrackedEntity, Enrollment, TrackerEvent, SingleEvent, and Relationship — alongside
the TEAV writes already routed through it.

  • Scope: inserts and updates for TrackedEntity, Enrollment, TrackerEvent, SingleEvent; inserts only for Relationship (updates are rejected upstream by
    AbstractTrackerPersister, so no stageUpdate overload exists for it). AbstractTrackerPersister now calls stageInsert(...) / stageUpdate(...) instead of entityManager.persist(...)
    / entityManager.merge(...).
  • No wire-path behaviour change yet: EntityWriteBatch.flush() still delegates back to EntityManager. Phase 6 swaps this for JDBC multi-row INSERT / unnest UPDATE / tuple DELETE
    statements.
  • Flush ordering: top-level entities are flushed before TEAVs so Hibernate assigns ids before any TEAV references them. The top-level order (TE → Enrollment → TrackerEvent → SingleEvent
    → Relationship) matches the persister call order enforced by DefaultTrackerBundleService.commit() and is FK-safe.
  • Mark/rollback: Mark now tracks per-type counts via nested EntityCounts / TeavCounts records, preserving the per-entity error-isolation contract used in non-atomic mode.

Notable fix: transient TE in handleTrackedEntityAttributeValues

The cross-persister TEAV dedup lookup (added in Phase 3) issues a JPQL query bound to the owning TrackedEntity. Once TE inserts are also deferred through the batch, a freshly-staged TE has
id == 0 and no DB rows yet — binding it as a query parameter throws TransientObjectException. The lookup now short-circuits to an empty map when te.getId() == 0 and relies on the
batch overlay alone, which is sufficient because a transient TE cannot have any prior TEAVs in the DB or in another persister's session.

Next steps

  • Quick fixformatDate: SimpleDateFormatstatic final DateTimeFormatter
  • Phase 0 ✅ Remove EntityManager from TrackerPersister interface and all method parameters -feat: decouple ChangeLogAccumulator from Hibernate Session [DHIS2-21378] #23836
  • Phase 1 ✅ Decouple ChangeLogAccumulator from Hibernate Session; obtain the transaction-bound Connection via DataSourceUtils instead of session.doWork()
  • Phase 2 ✅ Extract FileResource assignment update out of AbstractTrackerPersister into FileResourceStore
  • Phase 3 ✅ Introduce EntityWriteBatch; accumulate and batch-flush TEAV writes (multi-row INSERT / unnest UPDATE)
  • Phase 4 — Pre-allocate IDs from PostgreSQL sequences; stage top-level entity inserts and updates in EntityWriteBatch; remove all entityManager.flush() calls
  • Phase 5 — Replace updateTrackedEntitiesLastUpdated named query with NamedParameterJdbcTemplate; remove last Hibernate dependency from DefaultTrackerBundleService
  • Phase 6 — Implement JDBC flush methods one entity type at a time (TrackedEntityEnrollmentTrackerEvent/SingleEventRelationship)

@enricocolasante
enricocolasante force-pushed the DHIS2-21378-te-stage branch 2 times, most recently from 02e705c to 2c40b61 Compare May 27, 2026 09:11
@sonarqubecloud

Copy link
Copy Markdown

@enricocolasante
enricocolasante marked this pull request as ready for review May 27, 2026 13:48
@enricocolasante
enricocolasante requested a review from a team as a code owner May 27, 2026 13:48
@muilpp
muilpp requested a review from ameenhere May 27, 2026 15:21
@enricocolasante
enricocolasante merged commit 100dff1 into master May 28, 2026
24 checks passed
@enricocolasante
enricocolasante deleted the DHIS2-21378-te-stage branch May 28, 2026 07:06
enricocolasante added a commit that referenced this pull request Jul 7, 2026
… (2.43) (#24286)

* feat: remove EntityManager from TrackerPersister [DHIS2-21378] (#23797)

* feat: remove EntityManager from TrackerPersister interface and method parameters [DHIS2-21378]

* Use default timezone

(cherry picked from commit 052d3ac)

* feat: decouple ChangeLogAccumulator from Hibernate Session [DHIS2-21378] (#23836)

(cherry picked from commit 0c1779f)

* chore: Extract FileResource assignment update [DHIS2-21378] (#23879)

* chore: Extract FileResource assignment update [DHIS2-21378]

* Code review fixes

* Code review fixes

(cherry picked from commit 83542f4)

* feat: stage TEAV writes through EntityWriteBatch [DHIS2-21378] (#23970)

(cherry picked from commit b30c1ce)

* feat: Batch entity when persisting [DHIS2-21378] (#24005)

(cherry picked from commit 100dff1)

* feat: remove EntityManager from DefaultTrackerBundleService [DHIS2-21378] (#24024)

(cherry picked from commit 3759453)

* feat: TrackedEntity JDBC insert [DHIS2-21378] (#24027)

(cherry picked from commit 4eaf6ba)

* feat: TrackedEntity JDBC update [DHIS2-21378] (#24079)

(cherry picked from commit d4af287)

* feat: Enrollment JDBC insert/update [DHIS2-21378] (#24081)

* feat: TrackedEntity JDBC update [DHIS2-21378]

* feat: Enrollment JDBC insert/update [DHIS2-21378]

* Remove L2 caching from enrollment

* Code review fixes

* Code review fixes

(cherry picked from commit 34d629e)

* feat: Tracker event JDBC insert/update [DHIS2-21378] (#24082)

* feat: TrackedEntity JDBC insert [DHIS2-21378]

* feat: TrackedEntity JDBC update [DHIS2-21378]

* feat: Enrollment JDBC insert/update [DHIS2-21378]

* feat: Tracker event JDBC insert/update [DHIS2-21378]

(cherry picked from commit 435872b)

* feat: Single event JDBC insert/update [DHIS2-21378] (#24083)

* feat: TrackedEntity JDBC insert [DHIS2-21378]

* feat: TrackedEntity JDBC update [DHIS2-21378]

* feat: Enrollment JDBC insert/update [DHIS2-21378]

* feat: Tracker event JDBC insert/update [DHIS2-21378]

* feat: Single event JDBC insert/update [DHIS2-21378]

* Format code

(cherry picked from commit 9703e4f)

* feat: Relationship JDBC insert/update [DHIS2-21378] (#24093)

* feat: TrackedEntity JDBC insert [DHIS2-21378]

* feat: TrackedEntity JDBC update [DHIS2-21378]

* feat: Enrollment JDBC insert/update [DHIS2-21378]

* feat: Tracker event JDBC insert/update [DHIS2-21378]

* feat: Single event JDBC insert/update [DHIS2-21378]

* feat: Relationship JDBC insert/update [DHIS2-21378]

(cherry picked from commit 658f0a8)

* feat: TEAV JDBC insert/update [DHIS2-21378] (#24098)

* feat: TrackedEntity JDBC update [DHIS2-21378]

* feat: Enrollment JDBC insert/update [DHIS2-21378]

* feat: Tracker event JDBC insert/update [DHIS2-21378]

* feat: Single event JDBC insert/update [DHIS2-21378]

* feat: Relationship JDBC insert/update [DHIS2-21378]

* feat: TEAV JDBC insert/update [DHIS2-21378]

* Code review fixes

* Fix tests

* Fix idScheme issue for TEAV

(cherry picked from commit 4e3367c)

* fix: Refactor persistence layer in tracker importer [DHIS2-21378] (#24197)

* feat: TrackedEntity JDBC update [DHIS2-21378]

* feat: Enrollment JDBC insert/update [DHIS2-21378]

* feat: Tracker event JDBC insert/update [DHIS2-21378]

* feat: Single event JDBC insert/update [DHIS2-21378]

* feat: Relationship JDBC insert/update [DHIS2-21378]

* feat: TEAV JDBC insert/update [DHIS2-21378]

* Code review fixes

* Fix tests

* Fix idScheme issue for TEAV

* fix: Refactor persistence layer in tracker importer [DHIS2-21378]

* Fix merge

(cherry picked from commit 194dcd1)

* fix: Harmonize db sequences for tracker [DHIS2-21378] (#24208)

* fix: Harmonize db sequences for tracker [DHIS2-21378]

* Fix relationshipitemid

(cherry picked from commit f52eefa)

* fix: Rename conflicting migration [DHIS2-21378] (#24218)

(cherry picked from commit 7aefc1c)

No-op on 2.43: this PR renamed V2_44_12 -> V2_44_13 to avoid a master-only
Flyway version clash. On 2.43 the migration is numbered V2_43_63 (its own
series), so no rename is needed. Empty commit kept to preserve the full
DHIS2-21378 backport history.

* refactor: Split EntityWriteBatch into per-entity writers [DHIS2-21378] (#24227)

* refactor: Split EntityWriteBatch into per-entity writers [DHIS2-21378]

* chore: Migrate ownership from Hibernate to JDBC [DHIS2-21378]

* Code review fixes

(cherry picked from commit 331503a)

* chore: Migrate ownership from Hibernate to JDBC [DHIS2-21378] (#24239)

* refactor: Split EntityWriteBatch into per-entity writers [DHIS2-21378]

* chore: Migrate ownership from Hibernate to JDBC [DHIS2-21378]

(cherry picked from commit 7ba2431)

* perf: Batch existing-TEAV lookup in tracker import [DHIS2-21378] (#24244)

* refactor: Split EntityWriteBatch into per-entity writers [DHIS2-21378]

* chore: Migrate ownership from Hibernate to JDBC [DHIS2-21378]

* Code review fixes

* perf: Batch existing-TEAV lookup in tracker import [DHIS2-21378

* Code review fixes

* Code review fixes

(cherry picked from commit 2972a82)

* perf: Constant-text unnest INSERTs in tracker import [DHIS2-21378] (#24250)

* refactor: Split EntityWriteBatch into per-entity writers [DHIS2-21378]

* chore: Migrate ownership from Hibernate to JDBC [DHIS2-21378]

* Code review fixes

* perf: Batch existing-TEAV lookup in tracker import [DHIS2-21378

* perf: Constant-text unnest INSERTs in tracker import [DHIS2-21378]

* perf: Cache UserInfoSnapshot JSON per batch in tracker import [DHIS2-21378]

* Revert "perf: Cache UserInfoSnapshot JSON per batch in tracker import [DHIS2-21378]"

This reverts commit 7084379.

* Add tests to cover fields import

(cherry picked from commit 7305f9f)

* perf: Cache UserInfoSnapshot JSON per batch in tracker import [DHIS2-21378] (#24252)

* refactor: Split EntityWriteBatch into per-entity writers [DHIS2-21378]

* chore: Migrate ownership from Hibernate to JDBC [DHIS2-21378]

* Code review fixes

* perf: Batch existing-TEAV lookup in tracker import [DHIS2-21378

* perf: Constant-text unnest INSERTs in tracker import [DHIS2-21378]

* perf: Cache UserInfoSnapshot JSON per batch in tracker import [DHIS2-21378]

* Fix duration based Tracker perf tests

* Fix formatting

(cherry picked from commit a03204c)

* fix: Preserve storedby on tracker JDBC writers for 2.43 [DHIS2-21378]

Backport adaptation. On master, DHIS2-21537 removed storedBy from the
TrackedEntity/Enrollment/Event write path (and renamed TEAV storedBy to
updatedBy). 2.43 deliberately keeps storedBy, so the JDBC writers backported
from master must persist the storedby column that 2.43's tables still have.

TrackerObjectsMapper still sets storedBy on the new-entity branch for TE,
enrollment and events, so storedby is insert-only here (mirrors createdby):
added the storedby column + bind to the INSERT of TrackedEntityWriter,
EnrollmentWriter, SingleEventWriter and TrackerEventWriter. TeavWriter was
already adapted (storedby column + getStoredBy) when the earlier commits were
cherry-picked. Keeps the 2.43-only shouldSetStoredByToAuthenticatedUser* tests
green.

* Fix note persister

* Fix formatting
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants