Skip to content

Direct insert of nrt_observation_* data#755

Open
mpeels wants to merge 17 commits intomainfrom
mp/APP-519/write-to-nrt-tables
Open

Direct insert of nrt_observation_* data#755
mpeels wants to merge 17 commits intomainfrom
mp/APP-519/write-to-nrt-tables

Conversation

@mpeels
Copy link
Copy Markdown
Contributor

@mpeels mpeels commented Apr 20, 2026

Description

In order to resolve the race condition between the PostProcessingService and the Kafka Sink Connector for observation data, the ObservationService code was updated to directly persist to the following tables:

  • nrt_observation_material
  • nrt_observation_coded
  • nrt_observation_date
  • nrt_observation_edx
  • nrt_observation_numeric
  • nrt_observation_reason
  • nrt_observation_txt

Related Issue

APP-519

Additional Notes

  1. The ObservationService.java class was previously responsible for listening for Kafka messages as well as some of the business logic for processing the messages. It has been renamed to ObservationConsumer.java and updated to only validate messages and extract Organization_uid values. The new ObservationProcessor.java class now handles the business logic of processing the data.
  2. The NrtObservationWriter is using raw sql upsert statements, with a few more hours work, JPA Repositories could be created with @Entity's defined for each table to handle the insert/update if desired.
  3. The previous code had a single class, ProcessObservationDataUtil.java, that was responsible for both parsing the Observation data and posting messages to the relevant Kafka Topics. This was split into an ObservationParser.java and the NrtObservationWriter.java.
  4. Resolves a possible thread safety issue where the ObservationKey was declared at the class level but multiple threads were setting its value simultaneously.

Checklist

  • I have ensured that the pull request is of a manageable size, allowing it to be reviewed within a single session.
  • I have reviewed my changes to ensure they are clear, concise, and well-documented.
  • I have updated the documentation, if applicable.
  • I have added or updated test cases to cover my changes, if applicable.

@mpeels mpeels marked this pull request as ready for review April 20, 2026 15:16
@mpeels mpeels requested a review from a team as a code owner April 20, 2026 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants