data migration (android room)

[matejbart]

The main task is to migrate the existing json data into an android room database, some main reasons of performing this task:

• The storage with the json data in the public/private storage is out-dated (at the time when this approach was implemented, it was totally ok because the databases weren't supported with Flows/Coroutines, however, with android room support nowadays, this brings up several advantages and makes it more maintainable/sustainable.)
• The json storage is error-prone, there are bugs like:
  • Delete an active document, navigate back to the camera activity and the recently added document still exists.
  • If the active document is deleted, a bottom navigation to the images simply does nothing.
  • Sometimes, page detection does not find anything but marks the entire image, those images cannot be correctly cropped.
• With the recent major android versions, it has become more complicated with the storage handling, e.g. the old backup json in the public storage is not accessible anymore.
• Some image processing data is stored in the file itself, the issue is that there is no way to somehow determine that something has changed for that file, i.e. every time a task has finished, the view has to be manually refreshed. This could be very nicely avoided with the help of Flows if manipulated exif data is saved in the DB.
• DocumentStorage, SyncFile and SyncStorage are somehow very hard to maintain, they need to be updated manually and there is clearly a missing single-source of truth.
• If some meta data, like the orientation is saved into the database, one does not need to worry about reading the exif data and getting a -1.

The migration steps:

• The DB will reflect the current data which consists of a Document (which then consists of one MetaData and multiple Pages).
• The new persistence layer would be a repository layer with Flow support, the main advantage is that whenever something changes in the DB, the currently instantiated flows would emit the new data and all associated UI would update. There is no need to do this manually anymore. If for example, background jobs are used for uploading, then the uploaded state can be saved in the DB and when the file is uploaded the view would be updated, there is no need for a broadcastReceiver with intents to inform the UI.
• The migration will be performed on the json which is in the internal storage (since the backup json cannot be accessed anymore).
• The images will be copied into the internal storage to keep a better control over them, they will be deleted from the public storage. Keeping those images in the public storage may always bring some disadvantages like loosing references, imho a user is nowadays not really aware of poking around in the folders and I therefore think that keeping the images within the scope of the app is a better approach. Furthermore, now that the backup json is unavailable, there is actually not really a way of exporting the data onto another device.
• The images will be added in a structure like /documents/documentId/imageId, so by dropping the whole documentId folder all associated images can be also removed recursively.
• The quite old DocumentMigrator would be entirely dropped.
• (optional) An exporting function could be provided if someone wishes to export the images/json somewhere, just like the PDF export, or even where the "old" structure can be still supported.

Some general open questions that are unanswered:

1. What happens with the document, if an upload has succeeded? Can it be uploaded twice or should this be restricted?
2. After a successful upload, should it be possible to add new images and re-upload the same document again?
3. Is it ok, to "lock" the document until the images are uploaded? This would probably prevent some weird states, where a big upload is ongoing and the user starts to delete/modify or add new images.
4. If the user would logout and then re-login (let's assume with another account) should the document/data be still marked as uploaded?
5. Is it ok to drop the currently unused isFocused flag for pages/document?
6. Is there anything else to maybe also consider at the migration? (Like adding new fields for e.g. using a new API)
7. Is there any specific reason why some image related data has been saved in the image file itself and not in the json? Like the cropping points etc. Would it be ok to move this into the DB? And if necessary, incorporate it into the files only for the export?
8. As already mentioned, the bottom navigation is maybe not the best approach for the document viewer, because those fragments should be independent (data) from each other. However, the images fragment always depends on the selection of the document which is shown before - there will be always the problem when the active document is deleted and then the user navigates to the images fragment in the bottom nav. Is there maybe a possibility to re-work this?
   • I could imagine to use only two bottom nav fragments, documents and exports (currently PDFs), when the user selects a document, the images fragment will be hierarchly shown in a new activity.
   • The navigation drawer could be also dropped, and maybe replaced by a third fragment called "More" where everything else is listed. The drawer from the camera activity is not really necessary, because the document viewer can be still accessed and the settings could be extended so that every previous use-case is covered.

[hollaus]

Thanks! The migration plans sound plausible and I think we should discuss them in a meeting. I have answered your general questions inline:

Some general open questions that are unanswered:

1. What happens with the document, if an upload has succeeded? Can it be uploaded twice or should this be restricted?

No, I think we should prevent a duplicate upload.

2. After a successful upload, should it be possible to add new images and re-upload the same document again?

Yes, it should be possible to re-upload the document again.

3. Is it ok, to "lock" the document until the images are uploaded? This would probably prevent some weird states, where a big upload is ongoing and the user starts to delete/modify or add new images.

I guess it is OK, but then it should be possible to cancel the current upload, because otherwise the user cannot take new pictures until a big upload is finished.

4. If the user would logout and then re-login (let's assume with another account) should the document/data be still marked as uploaded?

I think it would be better to clear then the upload information and it should be marked as not uploaded.

5. Is it ok to drop the currently unused `isFocused` flag for pages/document?

Definitely, this is legacy code.

6. Is there anything else to maybe also consider at the migration? (Like adding new fields for e.g. using a new API)

I do not think so.

7. Is there any specific reason why some image related data has been saved in the image file itself and not in the json? Like the cropping points etc. Would it be ok to move this into the DB? And if necessary, incorporate it into the files only for the export?

The reason is that we wanted to make this data also usable in Transkribus, but it has not been used there. So basically it is OK. However, I would always save Exif orientation in the images itself and not in the DB, because otherwise the images will not be rotated correctly in other applications (image viewers, Transkribus).

8. As already mentioned, the bottom navigation is maybe not the best approach for the document viewer, because those fragments should be independent (data) from each other. However, the images fragment always depends on the selection of the document which is shown before - there will be always the problem when the active document is deleted and then the user navigates to the images fragment in the bottom nav. Is there maybe a possibility to re-work this?
   
   * I could imagine to use only two bottom nav fragments, documents and exports (currently PDFs), when the user selects a document, the images fragment will be hierarchly shown in a new activity.
   * The navigation drawer could be also dropped, and maybe replaced by a third fragment called "More" where everything else is listed. The drawer from the camera activity is not really necessary, because the document viewer can be still accessed and the settings could be extended so that every previous use-case is covered.

It is possible yes, but I think this should be done at a future point.