perf(backup): batch database operations during restore#3267
perf(backup): batch database operations during restore#3267Lolle2000la wants to merge 4 commits into
Conversation
Reduce overhead by chunking manga and repo restoration into transactions of 100 entries. Categories now restored in a single transaction. Optimized DatabaseHandler to reuse existing transaction contexts instead of creating redundant nested ones. Changed TransactionElement to internal to allow AndroidDatabaseHandler to check transaction state from the same package.
|
I'm actually working on redoing database 🫠 |
|
Wow, very quick response! Thank you! I mean this changes nothing on the database, just batches the write requests. Its a fairly standard thing. Just wraps the existing logic with a context so that it makes bigger updates instead of many small ones. The degree devices benefit from that is basically how high their storage latency is as opposed to bandwidth. That's why my Emulator on a fast SSD only has a ~2x speedup and the smartphone has a ~4x one. The lower end the devices, the higher the expected speedup (until bandwidth limit is hit of course). I compared it to the previous PR because this is very very simple and gives a good performance bump with much less changes. |
Use AtomicInteger for progress and synchronizedList for errors to prevent race conditions during concurrent restoration coroutines.
Reduce UI/IPC overhead by moving notification updates out of the inner loop, triggering them only after each 100-item transaction chunk. Reduced on my emulator from 46 seconds to 27
|
Regarding efa4d20, I got another good speedup by not updating the notification on literally every element. It turns out that this is actually itself quite an expensive operation. It now happens on every chunk, leading to another speedup from 46 seconds on my emulator to 27 seconds. |
Resolve conflicts in CategoriesRestorer and BackupRestorer by migrating from legacy DatabaseHandler to the new database.transaction system. Retain performance improvements by keeping 100-item chunked transactions.
|
Merged with main with the changes from #3270 and adjusted to the different API. Got this speedup against my emulator: Main: This branch: |
2 participants
Reduce overhead by chunking manga and repo restoration into transactions of 100 entries. Categories now restored in a single transaction. Optimized DatabaseHandler to reuse existing transaction contexts instead of creating redundant nested ones.
Changed TransactionElement to internal to allow AndroidDatabaseHandler to check transaction state from the same package.
Much simpler and more efficient than #2475. A higher blocking factor yields higher performance on my emulator and lower on my phone, a lower one (i.e. 50) yields less on both. High blocking factor helps with libraries with many small entries (with few chapters), but I don't have statistics on how the users libraries are distributed.
¹ PC Specs: Ryzen 9 9950X, Samsung 990 Pro 4TB
² Snapdragon variant