Support checksums and conditional loading for a given CSV row

[mseaton]

Currently, Iniz is able to determine whether a domain resource has changes based on computing checksums at the file level. This works well to reduce the startup time and ensure metadata isn't being re-saved unnecessarily each time Initializer runs.

However, once a file is determined to have changed, there is no further granular checking done to determine what entities defined in the file have actually changed. Given most of these files are CSV files, where one row within the CSV file represents one entity, it is likely quite possible to introduce a finer-level of granularity, and to maintain information to determine which specific rows in a domain CSV file should be processed. By introducing such capabilities at the level of the Base CSV Processor, we may be able to introduce such a feature for the majority of domains without much effort.

The idea would be to support having a checksum generated for each row in a given csv file that could be used to determine if that row has been changed. An additional file under configuration_checksums/domain that contains a properties file of = would suffice. This would only generate and verify a checksum if a Uuid has been specified on a given row. During processing of a given csv line, the uuid would be retrieved, and if it is not null, a checksum of the entire row would be generated and compared against the last saved checksum for that row, and if it is different then the row would be processed and the new checksum would be saved. Any changes to the header row would also need to be accounted for.

In particular, this would likely speed up processing of domains when there is a large amount of metadata (eg. it could better enable maintaining a single concepts.csv for a dictionary), and would allow better logging of exactly what metadata has been updated in a given update.

Interested in others thoughts about the usefulness and viability of this proposal @mks-d / @Ruhanga / @rbuisson / @brandones / @mogoodrich

[brandones]

I think that the usual pattern for addressing this problem is to split into multiple CSVs. This seems like it would be more maintainable, anyway.

[mogoodrich]

My gut would be to agree with @brandones ... not sure how valuable it would be to individually verify each line vs the overhead. Do we have a case where a single file is very slow to load (and wouldn't be better to be broken up into multiple files like Brandon suggests)?

[mseaton]

As far as performance is concerned, Concepts is the obvious area, and this is specifically looking towards improving our concept import loading times. But it is also targeting something that I have always wanted to improve - namely, that we should not execute an API save/update on metadata if it has not changed. The reasons for this are many, but in particular there can be lots of downstream implications to saving metadata, particularly in systems like ours that use a lot of AOP and Hibernate Interceptors to kick off events, sync data, etc if certain methods are executed, and "false updates" could happen with things like dateChanged, changedBy, etc.

I guess what I'd throw back to you @mogoodrich and @brandones is - why specifically are you opposed to this? Do you have specific concerns that this would be adversely impact Iniz or your experience with it in some way? What is the basis for your opposition?

As far as what is "maintainable", people can still choose to have one CSV or multiple CSVs, this doesn't force either. But in my long experience with MDS packages, splitting concepts across them is done for performance reasons only, and has the adverse effect that one never knows what package a given concept is in, or if a concept is in multiple packages, etc. So, I'd like to hear some more convincing arguments against this feature.

[mogoodrich]

I'm not strongly opposed to it... risks would be a little more complexity and potential bugs introduced when adding.

[brandones]

Yeah, I'm not really opposed, the question is whether the value add merits the complexity. The biggest difference with MDS is that you can grep CSVs, so it's trivial to find out where concepts are. The point about there being downstream implications to saving sounds like it would be compelling, but I'm not familiar with that stuff, so I don't know of any examples. Interested to hear what the Mekom team says.

[mks-d]

@mseaton is this still current or we can close this issue?

[mseaton]

It is still current, in that it has not been implemented. I leave it to you @mks-d to decide whether to close this as Won't Fix. I made whatever case I need to make in the comments above, and others have made reasonable arguments that this adds unnecessary complexity. So we could leave it open as a "maybe someday" or close as won't fix. I don't plan on tackling this at PIH, probably ever.

[mks-d]

@mseaton I will give it a label, which means that it's been considered to be a valid feature and I will also close it as "unplanned".

[mseaton]

I have re-opened this ticket and have a PR out for it (see above), as I have recently used the concepts domain as a stepping stone towards getting concepts out of the database and (ideally) on their way to OCL, but found that the performance of the domains is slower than the OCL counterpart due to the lack of this feature.

I have tested this PR out on this implementation and this seems to do the trick. Here is my initial loading metrics:

Concepts:

DEBUG - CsvParser.process(195) |2026-04-29T14:14:19,899| class org.openmrs.module.initializer.api.c.ConceptsCsvParser processing 8419 lines
INFO - BaseCsvLoader.load(155) |2026-04-29T14:21:30,695| 8419 entities were saved

Concept Answers:

DEBUG - CsvParser.process(195) |2026-04-29T14:21:30,731| class org.openmrs.module.initializer.api.c.ConceptSetsCsvParser processing 10033 lines
INFO - BaseCsvLoader.load(155) |2026-04-29T14:42:12,859| 10033 entities were saved

Concept Sets:

DEBUG - CsvParser.process(195) |2026-04-29T14:42:12,867| class org.openmrs.module.initializer.api.c.ConceptSetsCsvParser processing 2372 lines
INFO - BaseCsvLoader.load(155) |2026-04-29T14:46:50,569| 2372 entities were saved.

As can be seen here, it took about 7 minutes to load my Concepts in the concepts domain, and then another 25 minutes to load the answer and set member associations within the conceptsets domain.

This is painful, but as a one-off it isn't a total deal breaker. What is a deal breaker is having to incur this cost every time I make even the most minor change to any of the concepts, or answer or set association files, which is what this PR aims to address.

Running this branch, after the initial load above, I made a change one one of the concepts in my concepts.csv, and this was the resulting load time:

INFO - BaseCsvLoader.load(114) |2026-04-29T14:54:16,221| 8418 of 8419 CSV rows in concepts.csv are unchanged since last load and will be skipped.
INFO - BaseCsvLoader.load(124) |2026-04-29T14:54:16,225| Attempting to process 1 CSV lines that have not been processed yet...
DEBUG - CsvParser.process(195) |2026-04-29T14:54:16,225| class org.openmrs.module.initializer.api.c.ConceptsCsvParser processing 1 lines
DEBUG - CsvParser.clearAndFlushSession(230) |2026-04-29T14:54:16,232| Number processed: 0
INFO - BaseCsvLoader.load(154) |2026-04-29T14:54:17,079| concepts.csv ('concepts' domain) was entirely successfully processed.
INFO - BaseCsvLoader.load(155) |2026-04-29T14:54:17,080| 8419 entities were saved.

This correctly identified that only one of the rows in the CSV had changed, processed it, and then marked it as successful, and was completed in < 1 second.

@ibacher flagging you on this PR. Not sure who else is active on this project who is able to weigh in? @mks-d / @rbuisson ?

[ibacher]

I like this idea. How will this work with #319, which is also at least close to mergeable, but a kind of radical departure from the file-system based checksums?